pax_global_header00006660000000000000000000000064147140013400014504gustar00rootroot0000000000000052 comment=be873af557e1f3df110d72a96a04bb0656bb968c pengcheng.mind-test/000077500000000000000000000000001471400134000147115ustar00rootroot00000000000000pengcheng.mind-test/README.md000077500000000000000000000157421471400134000162040ustar00rootroot00000000000000# 基于MindFormers在云脑二上训练脑海模型 ## 一、介绍 MindSpore Transformers套件的目标是构建一个大模型训练、微调、评估、推理、部署的全流程开发套件： 我们在此版本的MindFormers套件上做修改，形成了脑海模型的下一迭代版本，目前2B版本模型已完成全流程的训练，具体训练配置如下： 其他相关配置的需求请查看MindFormers套件的使用说明[MindFormers](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/src/branch/master/README_mindformers.md) ## 二、版本匹配关系 | 版本对应关系 | MindSpore | CANN | 芯片 | | :----------: |:---------:|:-----:| :---------: | | 版本号 | 2.2 | 7.1.0 | Ascend 910A | ## 三、模型训练 ### 基于modelarts启动训练任务的配置 | modelarts配置参数 | 配置举例 | 参数说明 | |:-----------------:|:-----------------------------------------------------:|:--------------------------------------------------| | 启动文件 | /PengCheng.Mind-/mindformer/run_mindformer.py | 训练任务启动文件 | | train_dataset_dir | obs://path/to/dataset/train/ | train数据集在obs存储的位置 | | eval_dataset_dir | obs://path/to/dataset/eval/ | eval数据集在obs存储的位置 | | config | configs/official_training/run_minicpm_ckpt-2B-v4.yaml | 与训练相关的配置文件，SFT一般数据集循环训练3遍 | | remote_save_url | obs://path/to/save/ckpt/ | 训练相关checkpoinr、日志、strategy文件保存目录 | | do_eval | True | 是否打开eval流程 | | consumed_samples | 0 | 训练启动跳过数据集起始的条目数 | | data_parallel | 256 | 数据并行 | | pipeline_stage | 4 | 流水线并行 | | model_parallel | 1 | 模型并行 | | micro_batch_num | 8 | 批大小 | | optimizer_weight_shard_size | 64 | 优化器并行 | | resume_training | True | 是否加载已有模型继续训练, 默认为加载分片模型方式 | | resume_merged_ckpt | False | 加载整合模型方式启动， 此时restore_ckpt_name_prefix文件名为完整的模型名称 | | load_checkpoint | obs://path/to/pre_train_ckpt/ | 加载的预训练模型路径 | | restore_ckpt_name_prefix | naohai_2b6__rank_*-113809_2.ckpt | 模型名：*代替rank分片 | | total_steps | 1000000 | 总训练步数 | | decay_ratio | 0.001 | 进入WSD的学习率衰减占总步数的比例，SFT时可配置为数据集步数占总训练步数的30%~60% | | learning_rate | 0.0001 | 最高学习率 | | lr_middle | 0.00005 | 中段学习率，这里保持和learning_rate一致就是标准CosineLR | | lr_end | 0.00005 | 最低学习率 | 最新版的鹏城·脑海模型还在实验探索阶段，敬请期待 ## 四、模型转换为HuggingFace GPU模型 多芯片适用，NPU模型转换为HF格式GPU模型： * 转换脚本说明[ms_to_torch](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/src/branch/master/scripts/ms_to_torch.py) * 一键将NPU模型转换到GPU，转换后模型文件叠加项目“数据集”当中其余的HF模型文件依赖，整合全部文件作为最终HF格式模型转换结果。按照HF模型的使用规范来体验转换后模型 ## 五、说明 在这个项目中，我们对以下问题进行了探索： * 在模型规模确定的情况下（比如200B或7B或2B），确保模型Loss最优或近似最优收敛需要构建多大的预训练数据集？ * 如何构建确定的超大规模高质量预训练数据集？如何定义和定性定量评估数据的质量？如何合成增广富有知识而数量稀缺的数据？ * 在预训练数据集规模确定的情况下（比如2T或4T Token），如何找到确保Loss最优或近似最优收敛需要的最优数据混合策略？ * 在预训练数据集无法一次性准备好的情况下，如何找到既能避免灾难性遗忘又能持续确保Loss最优或近似最优收敛的持续学习最优数据混合策略？ * ... ## 六、评测表现 我们评测的数据集包括MMLU、C-Eval、 GSM8K、 MATH、HumanEval、MBPP、BBH等数据集，考察的能力包括自然语言理解、知识、数学计算和推理、代码生成、逻辑推理等。
| Model | MMLU | C-Eval | GSM8K | MATH | HumanEval | MBPP | BBH | CMMLU | AGIEval | |:-------------------|:--------:|:--------:|:--------:|:--------:|:---------:|:--------:|:--------:|:--------:|:--------:| | 脑海-v2-2B (660B tokens) | 39.73 | 42.41 | 5.69 | 1.90 | 5.49 | 8.80 | 29.08 | 40.91 | 18.08 | | 脑海-v3-2B (660B tokens) | 34.48 | 36.07 | 17.89 | 3.22 | 9.15 | 13.00 | 30.51 | 37.00 | 15.42 | | 脑海-v4-2B (660B tokens) | 36.63 | 39.13 | 20.92 | 4.94 | 3.05 | 16.60 | 29.67 | 37.18 |18.54 | | 脑海-v4-2B decay-v1 | 48.71 | 57.93 | 59.29 | 17.30 | 31.71 | 25.60 | 34.08 | 55.30 |27.02 | | 脑海-v4-2B decay-v2 | 48.70 | 54.93 | 52.46 | 15.14 | 25.61 | 25.00 | 34.97 | 53.09 |31.25 | | 脑海-v4-2B decay-v3 | 49.51 | 57.42 | 59.59 | 19.04 | 37.20 | 31.40 | 35.84 | 55.48 |26.53 | | 脑海-v4-2B decay-v3_SFT | 50.15 | 62.02 | 64.14 | 23.80 | 50.61 | 35.60 | 37.14 | 61.24 |31.91 |

## 七、许可证 [Apache 2.0许可证](LICENSE)pengcheng.mind-test/README_mindformers.md000077500000000000000000000477321471400134000206150ustar00rootroot00000000000000# 欢迎来到MindSpore Transformers（MindFormers） ## 一、介绍 MindSpore Transformers套件的目标是构建一个大模型训练、微调、评估、推理、部署的全流程开发套件： 提供业内主流的Transformer类预训练模型和SOTA下游任务应用，涵盖丰富的并行特性。期望帮助用户轻松的实现大模型训练和创新研发。 MindSpore Transformers套件基于MindSpore内置的并行技术和组件化设计，具备如下特点： - 一行代码实现从单卡到大规模集群训练的无缝切换； - 提供灵活易用的个性化并行配置； - 能够自动进行拓扑感知，高效地融合数据并行和模型并行策略； - 一键启动任意任务的单卡/多卡训练、微调、评估、推理流程； - 支持用户进行组件化配置任意模块，如优化器、学习策略、网络组装等； - 提供Trainer、pipeline、AutoClass等高阶易用性接口； - 提供预置SOTA权重自动下载及加载功能； - 支持人工智能计算中心无缝迁移部署； 如果您对MindSpore Transformers有任何建议，请通过issue与我们联系，我们将及时处理。 - 模型README：[docs/model_cards](https://gitee.com/mindspore/mindformers/tree/dev/docs/model_cards) - 任务README：[docs/task_cards](https://gitee.com/mindspore/mindformers/tree/dev/docs/task_cards) - MindPet指导教程：[docs/pet_tuners](https://gitee.com/mindspore/mindformers/blob/dev/docs/feature_cards/Pet_Tuners.md) - AICC指导教程：[docs/aicc_cards](https://gitee.com/mindspore/mindformers/blob/dev/docs/aicc_cards/README.md) - 高阶API指导文档: [docs/mindformers_tutorial.md](https://gitee.com/mindspore/mindformers/blob/dev/docs/README.md) - 在线文档：[mindformers](https://mindformers.readthedocs.io/zh_CN/latest) 目前支持的模型列表如下： | 模型 | 任务（task name） | 模型（model name） | | :------------------------------------------: |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [BERT](docs/model_cards/bert.md) | masked_language_modeling
[text_classification](docs/task_cards/text_classification.md)
[token_classification](docs/task_cards/token_classification.md)
[question_answering](docs/task_cards/question_answering.md) | bert_base_uncased
txtcls_bert_base_uncased
txtcls_bert_base_uncased_mnli
tokcls_bert_base_chinese
tokcls_bert_base_chinese_cluener
qa_bert_base_uncased
qa_bert_base_chinese_uncased | | [T5](docs/model_cards/t5.md) | translation | t5_small | | [GPT2](docs/model_cards/gpt2.md) | [text_generation](docs/task_cards/text_generation.md) | gpt2_small
gpt2_13b
gpt2_52b | | [PanGuAlpha](docs/model_cards/pangualpha.md) | [text_generation](docs/task_cards/text_generation.md) | pangualpha_2_6_b
pangualpha_13b | | [GLM](docs/model_cards/glm.md) | [text_generation](docs/task_cards/text_generation.md) | glm_6b
glm_6b_lora | | [GLM2](docs/model_cards/glm2.md) | [text_generation](docs/task_cards/text_generation.md) | glm2_6b
glm2_6b_lora | | [LLama](docs/model_cards/llama.md) | [text_generation](docs/task_cards/text_generation.md) | llama_7b
llama_13b
llama_65b
llama_7b_lora | | [LLama2](docs/model_cards/llama.md) | [text_generation](docs/task_cards/text_generation.md) | llama2_7b
llama2_13b | | [Bloom](docs/model_cards/bloom.md) | [text_generation](docs/task_cards/text_generation.md) | bloom_560m
bloom_7.1b
bloom_65b
bloom_176b | | [MAE](docs/model_cards/mae.md) | masked_image_modeling | mae_vit_base_p16 | | [VIT](docs/model_cards/vit.md) | [image_classification](docs/task_cards/image_classification.md) | vit_base_p16 | | [Swin](docs/model_cards/swin.md) | [image_classification](docs/task_cards/image_classification.md) | swin_base_p4w7 | | [CLIP](docs/model_cards/clip.md) | [contrastive_language_image_pretrain](docs/task_cards/contrastive_language_image_pretrain.md)
[zero_shot_image_classification](docs/task_cards/zero_shot_image_classification.md) | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | | [BLIP2](docs/model_cards/blip2.md) | [contrastive_language_image_pretrain](docs/task_cards/contrastive_language_image_pretrain.md)
[zero_shot_image_classification](docs/task_cards/zero_shot_image_classification.md) | blip2_stage1_vit_g | 目前在research中支持的模型列表如下： | 模型 | 任务（task name） | 模型（model name） | | :------------------------------------------: | :---------------------------------------------------: | :------------------------------ | | [Baichuan](research/baichuan/baichuan.md) | [text_generation](docs/task_cards/text_generation.md) | baichuan_7b
baichuan_13b | | [Baichuan2](research/baichuan2/baichuan2.md) | [text_generation](docs/task_cards/text_generation.md) | baichuan2_7b
baichuan2_13b | | [Internlm](research/internlm/internlm.md) | [text_generation](docs/task_cards/text_generation.md) | InternLM-7B | | [ziya](research/ziya/ziya.md) | [text_generation](docs/task_cards/text_generation.md) | ziya-13B | ## 二、mindformers安装 - 方式1：源码编译安装 支持源码编译安装，用户可以执行下述的命令进行包的安装 ```bash git clone -b dev https://gitee.com/mindspore/mindformers.git cd mindformers bash build.sh ``` - 方式2：镜像 docker下载命令 ```shell docker pull swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_2_0:mindformers_0.6.0dev_20230616_py39_37 ``` 创建容器 ```shell # --device用于控制指定容器的运行NPU卡号和范围 # -v 用于映射容器外的目录 # --name 用于自定义容器名称 docker run -it -u root \ --ipc=host \ --network host \ --device=/dev/davinci0 \ --device=/dev/davinci1 \ --device=/dev/davinci2 \ --device=/dev/davinci3 \ --device=/dev/davinci4 \ --device=/dev/davinci5 \ --device=/dev/davinci6 \ --device=/dev/davinci7 \ --device=/dev/davinci_manager \ --device=/dev/devmm_svm \ --device=/dev/hisi_hdc \ -v /etc/localtime:/etc/localtime \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /var/log/npu/:/usr/slog \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ --name {请手动输入容器名称} \ swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_2_0:mindformers_0.6.0dev_20230616_py39_37 \ /bin/bash ``` ## 三、版本匹配关系 | 版本对应关系 | MindFormers | MindPet | MindSpore | Python | 芯片 | | :----------: | :---------: | :-----: | :-------: | :-------: | :---------: | | 版本号 | dev | 1.0.0 | 2.0/1.10 | 3.7.5/3.9 | Ascend 910A | ## 四、快速使用 MindFormers套件对外提供两种使用和开发形式，为开发者提供灵活且简洁的使用方式和高阶开发接口。 ### 方式一：使用已有脚本启动 用户可以直接clone整个仓库，按照以下步骤即可运行套件中已支持的任意`configs`模型任务配置文件，方便用户快速进行使用和开发： - 准备工作 - step1：git clone mindformers ```shell git clone -b dev https://gitee.com/mindspore/mindformers.git cd mindformers ``` - step2: 准备相应任务的数据集，请参考`docs`目录下各模型的README.md文档准备相应数据集 - step3：修改配置文件`configs/{model_name}/run_{model_name}_***.yaml`中数据集路径 - step4：如果要使用分布式训练，则需提前生成RANK_TABLE_FILE ```shell # 不包含8本身，生成0~7卡的hccl json文件 python mindformers/tools/hccl_tools.py --device_num [0,8] ``` - 单卡启动：统一接口启动，根据模型 CONFIG 完成任意模型的单卡训练、微调、评估、推理流程 ```shell # 训练启动，run_status支持train、finetuen、eval、predict四个关键字，以分别完成模型训练、评估、推理功能，默认使用配置文件中的run_mode python run_mindformer.py --config {CONFIG_PATH} --run_mode {train/finetune/eval/predict} ``` - 多卡启动： scripts 脚本启动，根据模型 CONFIG 完成任意模型的单卡/多卡训练、微调、评估、推理流程 ```shell # 8卡分布式运行， DEVICE_RANGE = [0, 8], 不包含8本身 cd scripts bash run_distribute.sh RANK_TABLE_FILE CONFIG_PATH DEVICE_RANGE RUN_MODE ``` - 常用参数说明 ```text RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的{model_name}/run_*.yaml配置文件 DEVICE_ID: 为设备卡，范围为0~7 DEVICE_RANGE: 为单机分布式卡的范围, 如[0,8]为8卡分布式，不包含8本身 RUN_MODE: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` ### 方式二：调用API启动 **详细高阶API使用教程请参考：**[MindFormers大模型使用教程](docs/README.md) - 准备工作 - step 1：安装mindformers 具体安装请参考[第二章](https://gitee.com/mindspore/mindformers/blob/dev/README.md#%E4%BA%8Cmindformers%E5%AE%89%E8%A3%85) - step2: 准备数据 准备相应任务的数据集，请参考`docs`目录下各模型的README.md文档准备相应数据集。 - Trainer 快速入门 用户可以通过以上方式安装mindformers库，然后利用Trainer高阶接口执行模型任务的训练、微调、评估、推理功能。 - Trainer 训练\微调启动 用户可使用`Trainer.train`接口完成模型的训练\微调\断点续训。 ```python from mindformers import Trainer cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16', # 已支持的模型名 train_dataset="/data/imageNet-1k/train", # 传入标准的训练数据集路径，默认支持ImageNet数据集格式 eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式 # Example 1： 开启训练复现流程 cls_trainer.train() # Example 2： 加载集成的mae权重，开启微调流程 cls_trainer.finetune(finetune_checkpoint='mae_vit_base_p16') # Example 3： 开启断点续训功能 cls_trainer.train(train_checkpoint=True, resume_training=True) ``` - Trainer 评估启动 用户可使用`Trainer.evaluate`接口完成模型的评估流程。 ```python from mindformers import Trainer cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16', # 已支持的模型名 eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式 # Example 1： 开启评估已集成模型权重的复现流程 cls_trainer.evaluate() # Example 2： 开启评估训练得到的最后一个权重 cls_trainer.evaluate(eval_checkpoint=True) # Example 3： 开启评估指定的模型权重 cls_trainer.evaluate(eval_checkpoint='./output/checkpoint/rank_0/mindformers.ckpt') ``` ```text 结果打印示例(已集成的vit_base_p16模型权重评估分数)： Top1 Accuracy=0.8317 ``` - Trainer 推理启动 用户可使用`Trainer.predict`接口完成模型的推理流程。 ```python from mindformers import Trainer cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16') # 已支持的模型名 input_data = './cat.png' # 一张猫的图片 # Example 1： 指定输入的数据完成模型推理 predict_result_d = cls_trainer.predict(input_data=input_data) # Example 2： 开启推理（自动加载训练得到的最后一个权重） predict_result_b = cls_trainer.predict(input_data=input_data, predict_checkpoint=True) # Example 3： 加载指定的权重以完成推理 predict_result_c = cls_trainer.predict(input_data=input_data, predict_checkpoint='./output/checkpoint/rank_0/mindformers.ckpt') print(predict_result_d) ``` ```text 结果打印示例(已集成的vit_base_p16模型权重推理结果)： {‘label’: 'cat', score: 0.99} ``` - pipeline 快速入门 MindFormers套件为用户提供了已集成模型的pipeline推理接口，方便用户体验大模型推理服务。 - pipeline 使用 ```python from mindformers import pipeline from mindformers.tools.image_tools import load_image test_img = load_image("./sunflower.png") # 一朵太阳花图片 classifier = pipeline("zero_shot_image_classification", model='clip_vit_b_32', candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) predict_result = classifier(test_img) print(predict_result) ``` ```text 结果打印示例(已集成的clip_vit_b_32模型权重推理结果)： [[{'score': 0.9999547, 'label': 'sunflower'}, {'score': 1.8684346e-05, 'label': 'toy'}, {'score': 1.3045716e-05, 'label': 'dog'}, {'score': 1.129241e-05, 'label': 'tree'}, {'score': 2.1734568e-06, 'label': 'cat'}]] ``` - AutoClass 快速入门 MindFormers套件为用户提供了高阶AutoClass类，包含AutoConfig、AutoModel、AutoProcessor、AutoTokenizer四类，方便开发者进行调用。 - AutoConfig 获取已支持的任意模型配置 ```python from mindformers import AutoConfig # 获取clip_vit_b_32的模型配置 clip_vit_b_32_config = AutoConfig.from_pretrained('clip_vit_b_32') # 获取vit_base_p16的模型配置 vit_base_p16_config = AutoConfig.from_pretrained('vit_base_p16') ``` - AutoModel 获取已支持的网络模型 ```python from mindformers import AutoModel # 利用from_pretrained功能实现模型的实例化（默认加载对应权重） clip_vit_b_32_a = AutoModel.from_pretrained('clip_vit_b_32') # 利用from_config功能实现模型的实例化（默认加载对应权重） clip_vit_b_32_config = AutoConfig.from_pretrained('clip_vit_b_32') clip_vit_b_32_b = AutoModel.from_config(clip_vit_b_32_config) # 利用save_pretrained功能保存模型对应配置 clip_vit_b_32_b.save_pretrained('./clip', save_name='clip_vit_b_32') ``` - AutoProcessor 获取已支持的预处理方法 ```python from mindformers import AutoProcessor # 通过模型名关键字获取对应模型预处理过程（实例化clip的预处理过程，通常用于Trainer/pipeline推理入参） clip_processor_a = AutoProcessor.from_pretrained('clip_vit_b_32') # 通过yaml文件获取相应的预处理过程 clip_processor_b = AutoProcessor.from_pretrained('configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml') ``` - AutoTokenizer 获取已支持的tokenizer方法 ```python from mindformers import AutoTokenizer # 通过模型名关键字获取对应模型预处理过程（实例化clip的tokenizer，通常用于Trainer/pipeline推理入参） clip_tokenizer = AutoTokenizer.from_pretrained('clip_vit_b_32') ``` ## 五、贡献 欢迎参与社区贡献，可参考MindSpore贡献要求[Contributor Wiki](https://gitee.com/mindspore/mindspore/blob/dev/CONTRIBUTING_CN.md)。 ## 六、许可证 [Apache 2.0许可证](LICENSE)pengcheng.mind-test/build.sh000077500000000000000000000024541471400134000163540ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "---------------- MindFormers: build start ----------------" BASEPATH=$(cd "$(dirname $0)"; pwd) export BUILD_PATH="${BASEPATH}/build/" python setup.py bdist_wheel -d ${BASEPATH}/output if [ ! -d "${BASEPATH}/output" ]; then echo "The directory ${BASEPATH}/output dose not exist." exit 1 fi cd ${BASEPATH}/output || exit for package in mindformers*whl do [[ -e "${package}" ]] || break sha256sum ${package} > ${package}.sha256 done pip install mindformers*whl -i https://pypi.tuna.tsinghua.edu.cn/simple cd ${BASEPATH} || exit rm -rf *-info echo "---------------- MindFormers: build and install end ----------------" pengcheng.mind-test/configs/000077500000000000000000000000001471400134000163415ustar00rootroot00000000000000pengcheng.mind-test/configs/README.md000066400000000000000000000317041471400134000176250ustar00rootroot00000000000000# Config配置说明 ## 配置项顺序 configs统一在run_xxx.yaml中，排序按照修改频率的顺序和一般的模型训练流程顺序（数据集->模型->训练、评估、推理），具体顺序如下 - 非模块参数：seed、run_mode、output_dir、load_checkpoint、resume_training、auto_trans_ckpt - 环境参数：context - AICC：remote_save_url - 运行参数：runner_config、runner_wrapper - 并行设置：use_parallel、parallel、parallel_config - MOE：moe_config - 重计算：recompute_config - 算子调优：auto_tune - 性能工具：profile - Trainer：trainer - 数据集：train_dataset、train_dataset_task、eval_dataset、eval_dataset_task - 模型：model - 学习率：lr_schedule、layer_scale、layer_decay、lr_scale、lr_scale_factor - 优化器：optimizer - 回调函数：callbacks - 评估函数：metric - Processor：processor ## 详细配置说明 - seed: 随机种子，可以参考[mindspore.set_seed](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_seed.html) - run_mode: 运行模式，可选"train"、"finetune"、"eval"或"predict" - output_dir: './output' 输出路径，当前不支持修改该参数 - load_checkpoint: 加载权重的模型名或权重路径 - auto_trans_ckpt: 是否开启自动在线权重切分或转换（目前尚不支持） - resume_training: 加载方式，为True时会加载训练过程信息，如优化器、epochs数等 - context: 环境配置，可以参考: [mindspore.set_context](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_context.html) - mode: 0代表Graph Mode， 1代表Pynative Mode - device_target: 设备类型，Ascend、CPU或GPU，默认为Ascend - enable_graph_kernel: 是否开启图算融合 - graph_kernel_flags: 图算融合等级 - max_call_depth: 函数调用的最大深度 - save_graphs: 是否保存图 - device_id: 默认设备id - remote_save_url: 使用AICC训练作业时的，目标桶的回传文件夹路径 - runner_config: 运行配置 - epochs: 迭代次数 - batch_size: 数据批次大小，当前在使用yaml初始化训练时，会覆盖数据集配置中的batch_size，后面会删除改配置 - sink_mode: 是否开启数据下沉模式 - sink_size: 每次下沉数据大小，-1代表全量下沉 - runner_wrapper: wrapper配置 - type: wrapper类 - scale_sense: 梯度缩放配置 - type: 梯度缩放类 - use_clip_grad: 是否开启梯度裁剪 - loss_scale_value: 缩放系数 - use_parallel: 是否开启并行 - parallel: 自动并行配置，可以参考：[mindspore.set_auto_parallel_context](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_auto_parallel_context.html) - parallel_mode: 并行模式，0-dataset数据并行, 1-semi半自动并行, 2-auto自动并行, 3-hybrid手工实现并行 - gradients_mean: 是否在梯度AllReduce后执行平均算子。通常半自动并行模式下为False，数据并行模式下为True - enable_alltoall: 允许在通信期间生成AllToAll通信算子的开关。通常仅在MOE场景下打开，默认False - full_batch: 在auto_parallel模式下加载整个batch数据集时为True。半自动并行模式通常设置为True，数据并行模式必须设置为False，否则会报错 - search_mode: 策略搜索模式，有三种，分别是recursive_programming，dynamic_programming和sharding_propagation。仅在全自动并行模式下生效，其他模式不生效，实验性接口，谨慎使用 - enable_parallel_optimizer: 数据并行训练时对权重更新计算进行分片。优化器并行开关，在数据并行训练时默认会将模型权重参数切分成device_num份，与parallel_config中optimizer_shard保持一致；半自动并行时默认将模型权重参数切份data_parallel份 - strategy_ckpt_save_file: 保存并行切分策略的路径。 - parallel_config: 并行策略配置，可以参考mindformers.modules.transformer.TransformerOpParallelConfig - data_parallel: 数据并行 - model_parallel: 模型并行 - pipeline_stage: 流水线并行 > 需要满足实际运行的卡数 device_num = data_parallel × model_parallel × pipeline_stage - use_seq_parallel: 是否开启序列并行 - optimizer_shard: 是否开启优化器切分。优化器并行开关，通常在半自动并行模式下生效，与parallel中的enable_parallel_optimizer保持一致，默认将模型权重参数切份data_parallel份 - micro_batch_num: 流水线并行的微批次大小。pipeline_satge大于1时，开启流水并行时使用，此处需满足micro_batch_num >= pipeline_satge - gradient_aggregation_group: 梯度通信算子融合组的大小 - micro_batch_interleave_num: batch_size的拆分份数，多副本并行开关，通常在模型并行时使用，用于优化model_parallel时产生的通信损耗，纯流水并行时不建议使用。可以参考[mindspore.nn.MicroBatchInterleaved](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/nn/mindspore.nn.MicroBatchInterleaved.html) - moe_config: 混合专家配置，当前大部分仓上模型不支持，实验性接口，谨慎使用。可以参考mindformers.modules.transformer.moe.MoEConfig - expert_num: 专家数量 - capacity_factor: 专家能力因子 - aux_loss_factor: loss贡献因子 - num_experts_chosen: 每个token选择专家数目 - recompute_config：重计算配置，可以参考mindformers.modules.transformer.TransformerRecomputeConfig - recompute: 是否开启重计算 - select_recompute: 是否开启选择重计算，只针对attention层的算子进行重计算 - parallel_optimizer_comm_recompute: 由优化器并行引入的AllGather通信是否重计算 - mp_comm_recompute: 由模型并行引入的通信操作是否重计算 - recompute_slice_activation: 是否把保留在内存中的Cell输出切片 - auto_tune: 是否开启自动数据加速，可以参考[mindspore.dataset.config.set_enable_autotune](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset/mindspore.dataset.config.set_enable_autotune.html) - filepath_prefix: 优化后的全局配置的保存路径+文件前缀 - autotune_per_step: 设置自动数据加速的配置调整step间隔，可以参考[mindspore.dataset.config.set_autotune_interval](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset/mindspore.dataset.config.set_autotune_interval.html) - profile: 是否开启性能分析工具，可以参考[mindspore.Profiler](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.Profiler.html) - profile_start_step: 性能分析开始的step - profile_stop_step: 性能分析结束的step - profile_communication: 是否在多设备训练中收集通信性能数据 - profile_memory: 是否收集Tensor内存数据 - init_start_profile: 是否在Profiler初始化的时候开启数据采集 - trainer: 训练流程配置 - type: 训练流程类 - model_name: 训练模型名 - do_eval: 是否开启边训练边评估 - eval_step_interval: 评估step间隔, 默认为100，表示每100个step间隔执行一次评估；配置为大于0的数表示每隔所配置的step数后执行一次评估，配置为小于0的数则表示禁用step评估 - eval_epoch_interval: 评估epoch间隔, 默认为-1，表示禁用epoch结束时的评估；配置为大于0的数表示每隔所配置的epoch数后执行一次评估，配置为小于0的数则表示禁用epoch评估；注意：数据下沉模式下，epoch所包含的step数将从数据集大小变为sink size的大小，不建议在数据下沉模式下使用本项配置 - train_dataset: 训练数据集配置，可以参考[mindspore.dataset.GeneratorDataset](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset/mindspore.dataset.GeneratorDataset.html) - seed: 随机种子 - batch_size: 批次大小，当前在使用yaml初始化训练时，该参数会被runner_config中的batch_size覆盖 - data_loader: 数据加载配置，可以参考[mindspore.dataset.ImageFolderDataset](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset/mindspore.dataset.ImageFolderDataset.html) - type: 数据加载类 - dataset_dir: 数据集的根目录或数据集文件的路径 - num_parallel_workers: 读取数据的工作线程数 - shuffle: 是否混洗数据集 - transforms: 数据增强操作 - tokenizer: 分词器配置 - type: 分词器类 - max_length: 分词器输出的最大长度 - mixup_op: 图像随机混合，可以参考[mindspore.dataset.vision.MixUp](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset_vision/mindspore.dataset.vision.MixUp.html) - input_columns: 输入数据列 - output_columns: 输出数据列 - column_order: 输出数据顺序 - num_parallel_workers: 读取数据的工作进程数/线程数 - python_multiprocessing: 启用Python多进程模式加速运算 - drop_remainder: 当最后一个批处理数据包含的数据条目小于batch_size时，是否将该批处理丢弃 - repeat: 重复此数据集count次 - numa_enable: 设置NUMA的默认状态为启动状态 - prefetch_size: 设置管道中线程的队列容量 - eval_dataset: 评估数据集配置，具体配置说明可参考train_dataset - model: 模型配置 - arch: 模型类配置 - type: 模型类 - model_config: 模型参数配置 - type: 模型参数配置类 - checkpoint_name_or_path: 评估时不指定权重，模型默认加载的权重名 *\# 以下配置针对大规模语言模型推理* - top_k: 从概率最大的top_k个tokens中采样 - top_p: 从概率最大且概率累计不超过top_p的tokens中采样 - do_sample: 使能top_k或top_p采样，为False时top_k和top_p均重置为1 - use_past: 使能增量推理，为True时为增量推理，否则为自回归推理，使用时请参考[模型支持列表](https://gitee.com/mindspore/mindformers/tree/dev/docs#text-generator) - max_decode_length: 文本生成最大长度（输入长度统计在内） - repetition_penalty: 重复文本惩罚系数，该值不小于1，等于1时不惩罚 - lr_schedule: 学习率配置 - type: 学习率类 - layer_scale: 是否开启层衰减 - layer_decay: 层衰减系数 - optimizer: 优化器配置 - type: 优化器类 - weight_decay: 权重衰减值 - lr_scale: 是否开启学习率缩放 - lr_scale_factor: 学习率缩放系数 - callbacks: 回调函数配置 - type: 回调函数类 - type: MFLossMonitor: loss打印 - type: SummaryMonitor: 收集summary数据，可以参考[mindspore.SummaryCollector](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.SummaryCollector.html) - type: CheckpointMointor: checkpoint保存，可以参考[mindspore.save_checkpoint](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.save_checkpoint.html) - prefix: 权重文件前缀 - directory: 保存权重的目录 - save_checkpoint_seconds: 设定多少s保存一次ckpt - save_checkpoint_steps: 每多少个step保存一次checkpoint - keep_checkpoint_max: 设定保存ckpt的最大数量，超过则会删除最旧的权重，以保证数量不变 - keep_checkpoint_per_n_minutes: 设定多少minutes保存一次ckpt - integrated_save: 是否聚合保存。True时表示聚合所有卡权重，这时每张卡权重均一致；False时表示每张卡各自保存自己的权重；当半自动并行模式训练大模型时，通常需要设置为False，以保证权重保存时不会因为内存问题而失败 - save_network_params（新增）: 是否额外保存瘦身后的权重。默认为True。 - save_trainable_params（新增）: 是否额外保存可训练的参数权重，即微调部分参数的权重。默认为False。 - async_save: 是否异步执行保存checkpoint文件 - type: ObsMonitor: obs数据上传 - step_upload_frequence: 每隔多少个step上传一次，默认为100，表示每100个step执行一次数据上传；配置为大于0的数时，每隔配置数step后执行一次回传，小于0的数则表示禁用step回传 - epoch_upload_frequence: 每隔多少个epoch上传一次，默认为-1，表示禁用epoch回传；设置大于0的值表示每隔所配置的epoch数后回传；注意：数据下沉模式下，epoch所包含的step数将从数据集大小变为sink size的大小，不建议在数据下沉模式下使用本项配置 - keep_last: 检查obs的文件与AI计算中心平台是否一致，默认True，表示仅保留最后一次回传的内容，前面几次回传内容将会被移除；设为False则会保留每次回传的内容 - metric: 评估指标配置 - type: 评估指标类 - processor: 推理时的数据处理 - return_tensors: 返回张量类型 - type: 数据处理类 - image_processor: 图像处理配置 - type: 图像处理类 - tokenizer: 分词器配置 - type: 分词器类 pengcheng.mind-test/configs/bert/000077500000000000000000000000001471400134000172755ustar00rootroot00000000000000pengcheng.mind-test/configs/bert/run_bert_base_uncased.yaml000066400000000000000000000103041471400134000244730ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 128 sink_size: 2 image_size: 224 sink_mode: True runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: MaskedLanguageModelingTrainer model_name: 'bert_base_uncased' # train dataset train_dataset: &train_dataset data_loader: type: TFRecordDataset dataset_dir: "./wiki_data" shuffle: True shard_equal_rows: True input_columns: ["input_ids", "input_mask", "segment_ids", "next_sentence_labels", "masked_lm_positions", "masked_lm_ids", "masked_lm_weights"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: MaskLanguageModelDataset dataset_config: *train_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 1 dropout_prob: 0.1 batch_size: 128 seq_length: 128 #length of input sentence vocab_size: 30522 #size of vocab hidden_size: 768 #size of text feature num_hidden_layers: 12 #model depth num_attention_heads: 12 #number of attention heads intermediate_size: 3072 #hidden_size*4 hidden_act: "gelu" #activation post_layernorm_residual: True #select postlayernorm or prelayernorm hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False checkpoint_name_or_path: "" arch: type: BertForPreTraining # lr sechdule lr_schedule: type: cosine learning_rate: 0.0001 lr_end: 0.000001 warmup_steps: 10000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: BertProcessor tokenizer: cls_token: '[CLS]' do_basic_tokenize: True do_lower_case: True mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' type: BertTokenizer unk_token: '[UNK]' return_tensors: ms pengcheng.mind-test/configs/bert/run_bert_tiny_uncased.yaml000066400000000000000000000103241471400134000245460ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 128 sink_size: 2 image_size: 224 sink_mode: True runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: MaskedLanguageModelingTrainer model_name: 'bert_tiny_uncased' # train dataset train_dataset: &train_dataset data_loader: type: TFRecordDataset dataset_dir: "./wiki_data" shuffle: True shard_equal_rows: True input_columns: ["input_ids", "input_mask", "segment_ids", "next_sentence_labels", "masked_lm_positions", "masked_lm_ids", "masked_lm_weights"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: MaskLanguageModelDataset dataset_config: *train_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 1 dropout_prob: 0.1 batch_size: 128 seq_length: 128 #length of input sentence vocab_size: 30522 #size of vocab hidden_size: 128 #size of text feature num_hidden_layers: 2 #model depth num_attention_heads: 2 #number of attention heads intermediate_size: 512 #hidden_size*4 hidden_act: "gelu" #activation post_layernorm_residual: True #select postlayernorm or prelayernorm hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False use_moe: False checkpoint_name_or_path: "" arch: type: BertForPreTraining # lr sechdule lr_schedule: type: cosine learning_rate: 0.0001 lr_end: 0.000001 warmup_steps: 10000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: BertProcessor tokenizer: cls_token: '[CLS]' do_basic_tokenize: True do_lower_case: True mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' type: BertTokenizer unk_token: '[UNK]' return_tensors: ms pengcheng.mind-test/configs/blip2/000077500000000000000000000000001471400134000173515ustar00rootroot00000000000000pengcheng.mind-test/configs/blip2/run_blip2_stage1_vit_g_qformer_pretrain.yaml000066400000000000000000000136011471400134000302250ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: 80 sink_size: 2 image_size: 224 sink_mode: True initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: True parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 2 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'blip2_stage1_vit_g' # train dataset train_dataset: &train_dataset data_loader: type: MultiImgCapDataLoader dataset_dir: "/data" annotation_files: [ "vg/annotations/vg_caption.json", "coco2014/coco/annotations/coco_karpathy_train.json" ] image_dirs: [ "vg/images", "coco2014/coco/images" ] stage: "train" column_names: ["image", "text"] transforms: - type: RandomResizedCrop size: 224 scale: [0.5, 1.0] interpolation: "bicubic" - type: RandomHorizontalFlip - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False text_transforms: type: CaptionTransform prompt: "" max_words: 50 max_length: 32 padding: 'max_length' random_seed: 2022 truncation: True tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 32 num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 return_attention_mask: True train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: model_config: type: Blip2Config freeze_vision: True max_txt_len: 32 checkpoint_name_or_path: "" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: true hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: num_hidden_layers: 12 num_attention_heads: 12 query_length: 32 vocab_size: 30522 hidden_size: 768 encoder_width: 1408 bos_token_id: 30522 sep_token_id: 102 pad_token_id: 0 max_position_embeddings: 512 layer_norm_eps: 1.e-12 hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 chunk_size_feed_forward: 0 cross_attention_freq: 2 intermediate_size: 3072 initializer_range: 0.02 hidden_act: "gelu" dtype: "float32" layernorm_dtype: "float32" softmax_dtype: "float32" compute_dtype: "float16" add_cross_attention: True use_relative_positions: False tie_word_embeddings: True output_attentions: False output_hidden_states: False use_return_dict: False convert_param_from_bert: True checkpoint_name_or_path: "bert_base_uncased" arch: type: Blip2Qformer # lr sechdule lr_schedule: type: cosine learning_rate: 1.e-4 lr_end: 1.e-5 warmup_lr_init: 1.e-6 warmup_steps: 5000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False lr_scale: False # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.98 eps: 1.e-8 weight_decay: 0.05 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_qformer" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor step_upload_frequence: 1000 eval_callbacks: - type: ObsMonitor # image processor, tokenizer for prediction processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 32 pengcheng.mind-test/configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yaml000066400000000000000000000135761471400134000305260ustar00rootroot00000000000000seed: 42 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: 100 sink_size: 2 image_size: 224 sink_mode: False initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: True profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_to_text_retrieval model_name: 'blip2_stage1_vit_g' # eval dataset eval_dataset: &eval_dataset data_loader: type: MultiImgCapDataLoader dataset_dir: "/data" annotation_files: [ "flickr30k/annotations/test.json" ] image_dirs: [ "flickr30k/images" ] stage: "eval" column_names: ["image", "text"] shuffle: False transforms: - type: ToPIL # align with torch impl -> check interaction with 'Resize' - type: Resize size: [224, 224] interpolation: "bicubic" - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False text_transforms: type: CaptionTransform prompt: "" max_words: 50 max_length: 35 padding: 'max_length' random_seed: 2022 truncation: True tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 35 k_test: 128 # only top_k takes part in computation add_extra_itm_score: True num_parallel_workers: 1 python_multiprocessing: False drop_remainder: True batch_size: 50 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 return_attention_mask: True eval_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *eval_dataset # model model: model_config: type: Blip2Config freeze_vision: True max_txt_len: 32 checkpoint_name_or_path: "" # pretrained model dict here dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: true hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: num_hidden_layers: 12 num_attention_heads: 12 query_length: 32 vocab_size: 30522 hidden_size: 768 encoder_width: 1408 bos_token_id: 30522 sep_token_id: 102 pad_token_id: 0 max_position_embeddings: 512 layer_norm_eps: 1.e-12 hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 chunk_size_feed_forward: 0 cross_attention_freq: 2 intermediate_size: 3072 initializer_range: 0.02 hidden_act: "gelu" dtype: "float32" layernorm_dtype: "float32" softmax_dtype: "float32" compute_dtype: "float16" add_cross_attention: True use_relative_positions: False tie_word_embeddings: True output_attentions: False output_hidden_states: False use_return_dict: False arch: type: Blip2Qformer # lr sechdule lr_schedule: type: cosine learning_rate: 1.e-4 lr_end: 1.e-5 warmup_lr_init: 1.e-6 warmup_steps: 5000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False lr_scale: False # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.98 eps: 1.e-8 weight_decay: 0.05 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_qformer" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # image processor, tokenizer for prediction processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 32run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml000066400000000000000000000116121471400134000344050ustar00rootroot00000000000000pengcheng.mind-test/configs/blip2seed: 0 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 40 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'blip2_stage1_classification' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" num_parallel_workers: 1 python_multiprocessing: False drop_remainder: True batch_size: 40 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: Blip2Config freeze_vision: True max_txt_len: 32 checkpoint_name_or_path: "blip2_stage1_classification" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: true hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: num_hidden_layers: 12 num_attention_heads: 12 query_length: 32 vocab_size: 30522 hidden_size: 768 encoder_width: 1408 bos_token_id: 30522 sep_token_id: 102 pad_token_id: 0 max_position_embeddings: 512 layer_norm_eps: 1.e-12 hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 chunk_size_feed_forward: 0 cross_attention_freq: 2 intermediate_size: 3072 initializer_range: 0.02 hidden_act: "gelu" dtype: "float32" layernorm_dtype: "float32" softmax_dtype: "float32" compute_dtype: "float16" add_cross_attention: True use_relative_positions: False tie_word_embeddings: True output_attentions: False output_hidden_states: False use_return_dict: False arch: type: Blip2Classifier # metric metric: type: 'Accuracy' # lr sechdule lr_schedule: type: cosine learning_rate: 1.e-4 lr_end: 1.e-5 warmup_lr_init: 1.e-6 warmup_steps: 5000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False lr_scale: False # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.98 eps: 1.e-8 weight_decay: 0.05 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_stage1_classification" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # image processor, tokenizer for prediction processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: [224, 224] interpolation: "bicubic" mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 32pengcheng.mind-test/configs/blip2/run_blip2_stage2_vit_g_baichuan_7b.yaml000066400000000000000000000141061471400134000270120ustar00rootroot00000000000000seed: 42 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: &batch_size 16 sink_size: 2 image_size: 224 sink_mode: True initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'blip2_stage2_vit_g_baichuan_7b' # train dataset train_dataset: &train_dataset data_loader: type: MultiImgCapDataLoader dataset_dir: "/data" annotation_files: [ "vg/annotations/vg_caption.json", "coco2014/coco/annotations/coco_karpathy_train.json" ] image_dirs: [ "vg/images", "coco2014/coco/images" ] stage: "train" column_names: ["image", "text"] shuffle: True transforms: - type: RandomResizedCrop size: 224 scale: [0.5, 1.0] interpolation: "bicubic" - type: RandomHorizontalFlip - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False text_transforms: type: CaptionTransform prompt: "" max_words: 50 max_length: 33 # it equals to Blip2Config.max_txt_len + 1 for constructing labels padding: 'max_length' random_seed: 2022 truncation: True add_special_tokens: True tokenizer: type: LlamaTokenizer unk_token: '' bos_token: '' eos_token: '' pad_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 33 # it equals to Blip2Config.max_txt_len + 1 for constructing labels add_bos_token: False add_eos_token: True vocab_file: "" num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: model_config: type: Blip2Config batch_size: *batch_size freeze_vision: True freeze_text: True max_txt_len: 32 checkpoint_name_or_path: "" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" prompt: False prompt_length: 0 vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: True hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: vocab_size: 44728 text_config: type: LlamaConfig seq_length: 64 # sum of max_txt_len and num_query_token hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 64001 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float16" param_init_type: "float16" use_past: False offset: 0 repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False checkpoint_name_or_path: "" arch: type: Blip2Llama # lr schedule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 lr_end: 1.e-5 warmup_lr_init: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.98 eps: 1.e-8 # 1e-8 weight_decay: 0.05 learning_rate: 1.e-4 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_stage2_vig_g_baichuan_7b" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # input image size tokenizer: type: LlamaTokenizer pad_token: '' bos_token: '' unk_token: '' eos_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 32 pengcheng.mind-test/configs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yaml000066400000000000000000000112251471400134000341340ustar00rootroot00000000000000seed: 42 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: &batch_size 1 sink_size: 2 image_size: 224 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ImageToTextGenerationTrainer model_name: 'itt_blip2_stage2_vit_g_baichuan_7b' # train dataset eval_dataset: &eval_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: [ "image", "text"] hypothesis_template: "" transforms: - type: ToPIL - type: Resize size: 224 interpolation: 'linear' - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: Blip2Config batch_size: *batch_size freeze_vision: True freeze_text: True max_txt_len: 128 checkpoint_name_or_path: "" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: true hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" text_config: type: LlamaConfig seq_length: 160 # sum of max_txt_len and num_query_token hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 64001 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float16" param_init_type: "float16" use_past: True offset: 0 repetition_penalty: 1 max_decode_length: 512 top_k: 1 top_p: 1 do_sample: False checkpoint_name_or_path: "" arch: type: Blip2ImageToTextGeneration # processor processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # input image size tokenizer: type: LlamaTokenizer pad_token: '' bos_token: '' unk_token: '' eos_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 128 vocab_file: "" # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor pengcheng.mind-test/configs/blip2/run_blip2_stage2_vit_g_llama_7b.yaml000066400000000000000000000142001471400134000263210ustar00rootroot00000000000000seed: 42 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: 'blip2_stage1_pretrained' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: &batch_size 24 sink_size: 2 image_size: 224 sink_mode: True initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'blip2_stage2_vit_g_llama_7b' # train dataset train_dataset: &train_dataset data_loader: type: MultiImgCapDataLoader dataset_dir: "/data" annotation_files: [ "vg/annotations/vg_caption.json", "coco2014/coco/annotations/coco_karpathy_train.json" ] image_dirs: [ "vg/images", "coco2014/coco/images" ] stage: "train" column_names: [ "image", "text" ] shuffle: True transforms: - type: RandomResizedCrop size: 224 scale: [ 0.5, 1.0 ] interpolation: "bicubic" - type: RandomHorizontalFlip - type: ToTensor - type: Normalize mean: [ 0.48145466, 0.4578275, 0.40821073 ] std: [ 0.26862954, 0.26130258, 0.27577711 ] is_hwc: False text_transforms: type: CaptionTransform prompt: "" max_words: 50 max_length: 33 # it equals to Blip2Config.max_txt_len + 1 for constructing labels padding: 'max_length' random_seed: 2022 truncation: True add_special_tokens: True tokenizer: type: LlamaTokenizer unk_token: '' bos_token: '' eos_token: '' pad_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 33 # it equals to Blip2Config.max_txt_len + 1 for constructing labels add_bos_token: False add_eos_token: True checkpoint_name_or_path: 'llama_7b' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: model_config: type: Blip2Config batch_size: *batch_size freeze_vision: True freeze_text: True max_txt_len: 32 checkpoint_name_or_path: "blip2_stage1_pretrained" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" prompt: False prompt_length: 0 vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: True hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: vocab_size: 44728 text_config: type: LlamaConfig seq_length: 64 # sum of max_txt_len and num_query_token hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float16" param_init_type: "float16" use_past: False offset: 0 repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False checkpoint_name_or_path: "llama_7b" arch: type: Blip2Llama # lr schedule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 lr_end: 5.e-5 warmup_lr_init: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.98 eps: 1.e-8 # 1e-8 weight_decay: 0.05 learning_rate: 1.e-4 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_stage2_vig_g_llama_7b" save_checkpoint_steps: 5000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # input image size tokenizer: type: LlamaTokenizer pad_token: '' bos_token: '' unk_token: '' eos_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 32 pengcheng.mind-test/configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml000066400000000000000000000113241471400134000334500ustar00rootroot00000000000000seed: 42 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: &batch_size 1 sink_size: 2 image_size: 224 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ImageToTextGenerationTrainer model_name: "itt_blip2_stage2_vit_g_llama_7b" # eval dataset eval_dataset: &eval_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: [ "image", "text"] hypothesis_template: "a picture of " transforms: - type: ToPIL - type: Resize size: 224 interpolation: "linear" - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: Blip2Config batch_size: *batch_size freeze_vision: True freeze_text: True max_txt_len: 128 checkpoint_name_or_path: "itt_blip2_vit_g_llama_7b" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: True hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" text_config: type: LlamaConfig seq_length: 2048 # sum of max_txt_len and num_query_token hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float16" param_init_type: "float16" use_past: True offset: 0 repetition_penalty: 1 max_decode_length: 512 top_k: 1 top_p: 1 do_sample: False checkpoint_name_or_path: "llama_7b" arch: type: Blip2ImageToTextGeneration # processor processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # input image size tokenizer: type: LlamaTokenizer pad_token: '' bos_token: '' unk_token: '' eos_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 128 checkpoint_name_or_path: "llama_7b" # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor pengcheng.mind-test/configs/bloom/000077500000000000000000000000001471400134000174515ustar00rootroot00000000000000pengcheng.mind-test/configs/bloom/run_bloom_176b.yaml000066400000000000000000000100051471400134000230640ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False micro_batch_interleave_num: 1 auto_tune: True filepath_prefix: './autotune' autotune_per_step: 8 profile: False profile_start_step: 10 profile_stop_step: 20 profile_memory: True model: model_config: type: BloomConfig seq_length: 2048 vocab_size: 250880 hidden_size: 14336 num_layers: 70 num_heads: 112 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 initializer_range: 0.02 param_init_type: "float16" embedding_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" use_seq_parallel: False use_select_recompute: False checkpoint_name_or_path: "" eos_token_id: 2 bos_token_id: 1 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 arch: type: BloomLMHeadModel train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset processor: return_tensors: ms tokenizer: type: BloomTokenizer type: BloomProcessor context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_id: 0 device_target: "Ascend" graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # parallel use_parallel: True parallel: parallel_optimizer_config: {"gradient_accumulation_shard": False} parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 16 optimizer_shard: True micro_batch_num: 48 vocab_emb_dp: True gradient_aggregation_group: 4 recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # aicc remote_save_url: "Please input obs url on AICC platform." # runner config runner_config: epochs: 10 batch_size: 1 sink_mode: True sink_size: 8 per_epoch_size: 8 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 weight_decay: 0.1 lr_schedule: type: cosine learning_rate: 0.000001 lr_end: 0.0000001 warmup_steps: 1000 total_steps: 4000 # -1 means it will load the total steps of the dataset callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor trainer: type: CausalLanguageModelingTrainer model_name: 'bloom_176b' pengcheng.mind-test/configs/bloom/run_bloom_560m.yaml000066400000000000000000000100171471400134000230770ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False micro_batch_interleave_num: 1 auto_tune: True filepath_prefix: './autotune' autotune_per_step: 8 profile: False profile_start_step: 10 profile_stop_step: 20 profile_memory: True model: model_config: type: BloomConfig seq_length: 2048 vocab_size: 250880 hidden_size: 1024 num_layers: 24 num_heads: 16 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 initializer_range: 0.02 param_init_type: "float16" embedding_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" use_seq_parallel: False use_select_recompute: False checkpoint_name_or_path: "bloom_560m" eos_token_id: 2 bos_token_id: 1 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 arch: type: BloomLMHeadModel train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset processor: return_tensors: ms tokenizer: type: BloomTokenizer type: BloomProcessor context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_id: 0 device_target: "Ascend" graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # parallel use_parallel: False parallel: parallel_optimizer_config: {"gradient_accumulation_shard": False} parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # aicc remote_save_url: "Please input obs url on AICC platform." # runner config runner_config: epochs: 10 batch_size: 1 sink_mode: True sink_size: 8 per_epoch_size: 8 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 weight_decay: 0.1 lr_schedule: type: cosine learning_rate: 0.000001 lr_end: 0.0000001 warmup_steps: 1000 total_steps: 4000 # -1 means it will load the total steps of the dataset callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor trainer: type: CausalLanguageModelingTrainer model_name: 'bloom_560m' pengcheng.mind-test/configs/bloom/run_bloom_65b.yaml000066400000000000000000000100021471400134000227760ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False micro_batch_interleave_num: 1 auto_tune: True filepath_prefix: './autotune' autotune_per_step: 8 profile: False profile_start_step: 10 profile_stop_step: 20 profile_memory: True model: model_config: type: BloomConfig seq_length: 2048 vocab_size: 250880 hidden_size: 8192 num_layers: 80 num_heads: 64 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 initializer_range: 0.02 param_init_type: "float16" embedding_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" use_seq_parallel: False use_select_recompute: False checkpoint_name_or_path: "" eos_token_id: 2 bos_token_id: 1 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 arch: type: BloomLMHeadModel train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset processor: return_tensors: ms tokenizer: type: BloomTokenizer type: BloomProcessor context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_id: 0 device_target: "Ascend" graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # parallel use_parallel: True parallel: parallel_optimizer_config: {"gradient_accumulation_shard": False} parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 12 optimizer_shard: True micro_batch_num: 48 vocab_emb_dp: True gradient_aggregation_group: 4 recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # aicc remote_save_url: "Please input obs url on AICC platform." # runner config runner_config: epochs: 10 batch_size: 1 sink_mode: True sink_size: 8 per_epoch_size: 8 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 weight_decay: 0.1 lr_schedule: type: cosine learning_rate: 0.000001 lr_end: 0.0000001 warmup_steps: 1000 total_steps: 4000 # -1 means it will load the total steps of the dataset callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor trainer: type: CausalLanguageModelingTrainer model_name: 'bloom_65b' pengcheng.mind-test/configs/bloom/run_bloom_7.1b.yaml000066400000000000000000000100151471400134000230550ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False micro_batch_interleave_num: 1 auto_tune: True filepath_prefix: './autotune' autotune_per_step: 8 profile: False profile_start_step: 10 profile_stop_step: 20 profile_memory: True model: model_config: type: BloomConfig seq_length: 2048 vocab_size: 250880 hidden_size: 4096 num_layers: 30 num_heads: 32 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 initializer_range: 0.02 param_init_type: "float16" embedding_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" use_seq_parallel: False use_select_recompute: False checkpoint_name_or_path: "bloom_7.1b" eos_token_id: 2 bos_token_id: 1 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 arch: type: BloomLMHeadModel train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset processor: return_tensors: ms tokenizer: type: BloomTokenizer type: BloomProcessor context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_id: 0 device_target: "Ascend" graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # parallel use_parallel: True parallel: parallel_optimizer_config: {"gradient_accumulation_shard": False} parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 4 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # aicc remote_save_url: "Please input obs url on AICC platform." # runner config runner_config: epochs: 10 batch_size: 4 sink_mode: True sink_size: 8 per_epoch_size: 8 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 weight_decay: 0.1 lr_schedule: type: cosine learning_rate: 0.000001 lr_end: 0.0000001 warmup_steps: 1000 total_steps: 4000 # -1 means it will load the total steps of the dataset callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor trainer: type: CausalLanguageModelingTrainer model_name: 'bloom_7.1b' pengcheng.mind-test/configs/clip/000077500000000000000000000000001471400134000172705ustar00rootroot00000000000000pengcheng.mind-test/configs/clip/run_clip_vit_b_16_pretrain_flickr8k.yaml000066400000000000000000000101341471400134000271600ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'clip_vit_b_16' # train dataset train_dataset: &train_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: ["image", "text"] text_transforms: type: RandomChoiceTokenizerForward max_length: 77 padding: "max_length" random_seed: 2022 transforms: - type: Resize size: 224 - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: CLIPTokenizer pad_token: '!' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 512 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 8 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 768 intermediate_size: 3072 num_hidden_layers: 12 num_attention_heads: 12 image_size: 224 patch_size: 16 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_b_16 # the loaded model type projection_dim: 512 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 pengcheng.mind-test/configs/clip/run_clip_vit_b_16_zero_shot_image_classification_cifar100.yaml000066400000000000000000000103601471400134000333560ustar00rootroot00000000000000seed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'clip_vit_b_16' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" tokenizer: type: CLIPTokenizer pad_token: '!' text_transforms: type: TokenizerForward max_length: 77 padding: "max_length" transforms: - type: ToPIL - type: Resize size: 224 interpolation: 'linear' - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 512 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 8 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 768 intermediate_size: 3072 num_hidden_layers: 12 num_attention_heads: 12 image_size: 224 patch_size: 16 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_b_16 # the loaded model type projection_dim: 512 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: 'Accuracy' # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 top_k: 3 save_file: 'results.txt' pengcheng.mind-test/configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml000066400000000000000000000101351471400134000271570ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'clip_vit_b_32' # train dataset train_dataset: &train_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: ["image", "text"] text_transforms: type: RandomChoiceTokenizerForward max_length: 77 padding: "max_length" random_seed: 2022 transforms: - type: Resize size: 224 - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: CLIPTokenizer pad_token: '!' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 512 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 8 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 768 intermediate_size: 3072 num_hidden_layers: 12 num_attention_heads: 12 image_size: 224 patch_size: 32 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_b_32 # the loaded model type projection_dim: 512 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 pengcheng.mind-test/configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml000066400000000000000000000103601471400134000333540ustar00rootroot00000000000000seed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'clip_vit_b_32' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" tokenizer: type: CLIPTokenizer pad_token: '!' text_transforms: type: TokenizerForward max_length: 77 padding: "max_length" transforms: - type: ToPIL - type: Resize size: 224 interpolation: 'linear' - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 512 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 8 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 768 intermediate_size: 3072 num_hidden_layers: 12 num_attention_heads: 12 image_size: 224 patch_size: 32 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_b_32 # the loaded model type projection_dim: 512 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: 'Accuracy' # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 top_k: 3 save_file: 'results.txt' pengcheng.mind-test/configs/clip/run_clip_vit_l_14@336_pretrain_flickr8k.yaml000066400000000000000000000101471471400134000275300ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 16 image_size: 336 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'clip_vit_l_14@336' # train dataset train_dataset: &train_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: ["image", "text"] text_transforms: type: RandomChoiceTokenizerForward max_length: 77 padding: "max_length" random_seed: 2022 transforms: - type: Resize size: 336 - type: CenterCrop size: 336 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: CLIPTokenizer pad_token: '!' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 16 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 12 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 1024 intermediate_size: 3072 num_hidden_layers: 24 num_attention_heads: 16 image_size: 336 patch_size: 14 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_l_14@336 # the loaded model type projection_dim: 768 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 336 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 pengcheng.mind-test/configs/clip/run_clip_vit_l_14@336_zero_shot_image_classification_cifar100.yaml000066400000000000000000000103721471400134000337250ustar00rootroot00000000000000seed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 16 image_size: 336 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'clip_vit_l_14@336' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" tokenizer: type: CLIPTokenizer pad_token: '!' text_transforms: type: TokenizerForward max_length: 77 padding: "max_length" transforms: - type: ToPIL - type: Resize size: 336 interpolation: 'linear' - type: CenterCrop size: 336 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 12 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 1024 intermediate_size: 3072 num_hidden_layers: 24 num_attention_heads: 16 image_size: 336 patch_size: 14 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_l_14@336 # the loaded model type projection_dim: 768 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: 'Accuracy' # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 336 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 top_k: 3 save_file: 'results.txt' pengcheng.mind-test/configs/clip/run_clip_vit_l_14_pretrain_flickr8k.yaml000066400000000000000000000101361471400134000271720ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'clip_vit_l_14' # train dataset train_dataset: &train_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: ["image", "text"] text_transforms: type: RandomChoiceTokenizerForward max_length: 77 padding: "max_length" random_seed: 2022 transforms: - type: Resize size: 224 - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: CLIPTokenizer pad_token: '!' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 12 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 1024 intermediate_size: 3072 num_hidden_layers: 24 num_attention_heads: 16 image_size: 224 patch_size: 14 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_l_14 # the loaded model type projection_dim: 768 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 pengcheng.mind-test/configs/clip/run_clip_vit_l_14_zero_shot_image_classification_cifar100.yaml000066400000000000000000000103611471400134000333670ustar00rootroot00000000000000seed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'clip_vit_l_14' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" tokenizer: type: CLIPTokenizer pad_token: '!' text_transforms: type: TokenizerForward max_length: 77 padding: "max_length" transforms: - type: ToPIL - type: Resize size: 224 interpolation: 'linear' - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 12 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 1024 intermediate_size: 3072 num_hidden_layers: 24 num_attention_heads: 16 image_size: 224 patch_size: 14 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_l_14 # the loaded model type projection_dim: 768 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: 'Accuracy' # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 top_k: 3 save_file: 'results.txt' pengcheng.mind-test/configs/filip/000077500000000000000000000000001471400134000174445ustar00rootroot00000000000000pengcheng.mind-test/configs/filip/run_filip_vit_l_14.yaml000066400000000000000000000116011471400134000240170ustar00rootroot00000000000000seed: 0 run_status: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 15 batch_size: 64 image_size: 224 sink_mode: False sink_size: 2 runner_wrapper: type: FilipTrainOneStepWithLossScaleWrapper # parallel use_parallel: True parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: itr model_name: 'filip' # train dataset train_dataset: &train_dataset data_loader: type: ImageToTextRetrievalDataset dataset_dir: "" shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.2, 1.0] interpolation: bicubic - type: RandomHorizontalFlip prob: 0.5 - type: RandomColorAdjust brightness: 0.4 contrast: 0.4 saturation: 0.4 hue: 0.1 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False project_columns: ["image", "token"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: ImageToTextRetrievalDataset dataset_config: *train_dataset # eval dataset eval: &eval_dataset data_loader: type: ImageToTextRetrievalDataset dataset_dir: "" shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.2, 1.0] interpolation: bicubic - type: RandomHorizontalFlip prob: 0.5 - type: RandomColorAdjust brightness: 0.4 contrast: 0.4 saturation: 0.4 hue: 0.1 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False project_columns: ["image", "token"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: ImageToTextRetrievalDataset dataset_config: *eval_dataset # model model: arch: type: FilipModel model_config: type: FilipConfig text_config: type: FilipTextConfig hidden_size: 768 vocab_size: 21128 max_position_embeddins: 32 num_hidden_layers: 12 vision_config: type: FilipVisionConfig hidden_size: 1024 image_size: 224 patch_size: 14 num_hidden_layers: 24 token_learner: True num_tokens: 24 num_token_groups: 8 token_learner_dropout: 1.0 dtype: float16 checkpoint_name_or_path: "" projection_dim: 256 ratio: 64 batch_size: 64 recompute: False # lr sechdule lr_schedule: type: CosineDecayLR max_lr: 0.00005 min_lr: 0. decay_steps: 780 layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: Lamb eps: 0.000001 weight_decay: 0.0003 lr_scale: False lr_scale_factor: 256 no_decay_params: ['positional_embedding', 'class_embedding', 'scale', 'bias', 'ln', 'bn'] # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: FilipProcessor feature_extractor: type: FilipFeatureExtractor image_feature_extractor: type: FilipImageFeatureExtrator image_resolution: 224 tokenizer: type: FilipTokenizer pengcheng.mind-test/configs/general/000077500000000000000000000000001471400134000177565ustar00rootroot00000000000000pengcheng.mind-test/configs/general/run_general_task.yaml000066400000000000000000000052001471400134000241620ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: general model_name: 'general' # please edit by hand. train_dataset: train_dataset_task: eval_dataset: eval_dataset_task: # please edit by hand. model: model_config: checkpoint_name_or_path: arch: # lr sechdule lr_schedule: type: cosine learning_rate: 0.00015 lr_end: 0. warmup_lr_init: 0. warmup_epochs: 40 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # please edit by hand. processor: tokenizer: image_processor: audio_processor: pengcheng.mind-test/configs/glm/000077500000000000000000000000001471400134000171205ustar00rootroot00000000000000pengcheng.mind-test/configs/glm/run_glm_6b_finetune.yaml000066400000000000000000000133021471400134000237320ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: GLMConfig vocab_size: 130528 hidden_size: 4096 num_layers: 28 num_heads: 32 inner_hidden_size: 16384 seq_length: 512 # 推理时, 输入pad到的长度, model里的最大句长 embedding_dropout_prob: 0.0 attention_dropout_rate: 0.0 hidden_dropout_rate: 0.0 hidden_size_per_attention_head: # default "None" means hidden-size/num-attention-heads. layernorm_order: "post" layernorm_epsilon: 1.0e-5 use_final_layernorm: True use_past: False activation_func: 'GELU' position_encoding_2d: True param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" bos_token_id: 130004 eos_token_id: 130005 mask_token_id: 130000 gmask_token_id: 130001 pad_token_id: 3 max_decode_length: 2048 # The maximum length of the generated words. is_enhanced_encoder: True is_sample_acceleration: False checkpoint_name_or_path: "glm_6b" top_k: 1 top_p: 1 repetition_penalty: 1 do_sample: True arch: type: GLMForPreTraining trainer: type: CausalLanguageModelingTrainer model_name: 'glm_6b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLMTokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/train.json" shuffle: True phase: "train" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] max_source_length: 64 max_target_length: 64 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/dev.json" shuffle: False phase: "eval" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 8 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # parallel config use_parallel: False parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm-6b" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-test/configs/glm/run_glm_6b_infer.yaml000066400000000000000000000131631471400134000232250ustar00rootroot00000000000000seed: 0 run_mode: 'predict' load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: GLMConfig vocab_size: 130528 hidden_size: 4096 num_layers: 28 num_heads: 32 inner_hidden_size: 16384 seq_length: 512 # 推理时, 输入pad到的长度, model里的最大句长 embedding_dropout_prob: 0.0 attention_dropout_rate: 0.0 hidden_dropout_rate: 0.0 hidden_size_per_attention_head: # default "None" means hidden-size/num-attention-heads. layernorm_order: "post" layernorm_epsilon: 1.0e-5 use_final_layernorm: True use_past: True # True for incremental infer. activation_func: 'GELU' position_encoding_2d: True param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" bos_token_id: 130004 eos_token_id: 130005 mask_token_id: 130000 gmask_token_id: 130001 pad_token_id: 3 max_decode_length: 2048 # The maximum length of the generated words. is_enhanced_encoder: True is_sample_acceleration: True # sample in npu to acceleration checkpoint_name_or_path: "glm_6b_chat" top_k: 1 top_p: 1 # top_p not support for npu_acceleration yet repetition_penalty: 1 do_sample: True arch: type: GLMChatModel trainer: type: CausalLanguageModelingTrainer model_name: 'glm_6b_chat' metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLMTokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/train.json" shuffle: True phase: "train" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] max_source_length: 64 max_target_length: 64 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/dev.json" shuffle: False phase: "eval" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 # parallel config use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm-6b" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-test/configs/glm/run_glm_6b_lora.yaml000066400000000000000000000134321471400134000230560ustar00rootroot00000000000000seed: 0 run_mode: 'finetune' load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: GLMConfig vocab_size: 130528 hidden_size: 4096 num_layers: 28 num_heads: 32 inner_hidden_size: 16384 seq_length: 512 # 推理时, 输入pad到的长度, model里的最大句长 embedding_dropout_prob: 0.0 attention_dropout_rate: 0.0 hidden_dropout_rate: 0.0 hidden_size_per_attention_head: # default "None" means hidden-size/num-attention-heads. layernorm_order: "post" layernorm_epsilon: 1.0e-5 use_final_layernorm: True use_past: False activation_func: 'GELU' position_encoding_2d: True param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" bos_token_id: 130004 eos_token_id: 130005 mask_token_id: 130000 gmask_token_id: 130001 pad_token_id: 3 max_decode_length: 2048 # The maximum length of the generated words. is_enhanced_encoder: True is_sample_acceleration: False checkpoint_name_or_path: "glm_6b_lora" top_k: 1 top_p: 1 repetition_penalty: 1 do_sample: True pet_config: pet_type: lora lora_rank: 8 lora_alpha: 32 lora_dropout: 0.1 arch: type: GLMForPreTrainingWithLora trainer: type: CausalLanguageModelingTrainer model_name: 'glm_6b_lora' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLMTokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/train.json" shuffle: True phase: "train" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] max_source_length: 64 max_target_length: 64 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/dev.json" shuffle: False phase: "eval" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 8 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 layer_scale: False lr_scale: False # parallel config use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm-6b-lora" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-test/configs/glm/run_glm_6b_lora_infer.yaml000066400000000000000000000133521471400134000242420ustar00rootroot00000000000000seed: 0 run_mode: 'predict' load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: GLMConfig vocab_size: 130528 hidden_size: 4096 num_layers: 28 num_heads: 32 inner_hidden_size: 16384 seq_length: 512 # 推理时, 输入pad到的长度, model里的最大句长 embedding_dropout_prob: 0.0 attention_dropout_rate: 0.0 hidden_dropout_rate: 0.0 hidden_size_per_attention_head: # default "None" means hidden-size/num-attention-heads. layernorm_order: "post" layernorm_epsilon: 1.0e-5 use_final_layernorm: True use_past: True # True for incremental infer. activation_func: 'GELU' position_encoding_2d: True param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" bos_token_id: 130004 eos_token_id: 130005 mask_token_id: 130000 gmask_token_id: 130001 pad_token_id: 3 max_decode_length: 2048 # The maximum length of the generated words. is_enhanced_encoder: True is_sample_acceleration: True # sample in npu to acceleration checkpoint_name_or_path: "glm_6b_lora_chat" top_k: 1 top_p: 1 # top_p not support for npu_acceleration yet repetition_penalty: 1 do_sample: True pet_config: pet_type: lora lora_rank: 8 lora_alpha: 32 lora_dropout: 0.1 arch: type: GLMChatModelWithLora trainer: type: CausalLanguageModelingTrainer model_name: 'glm_6b_lora' metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLMTokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/train.json" shuffle: True phase: "train" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] max_source_length: 64 max_target_length: 64 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/dev.json" shuffle: False phase: "eval" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 # parallel config use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm-6b-lora" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-test/configs/glm2/000077500000000000000000000000001471400134000172025ustar00rootroot00000000000000pengcheng.mind-test/configs/glm2/run_glm2_6b.yaml000066400000000000000000000132321471400134000222030ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" # 59GB for 910B save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: ChatGLM2Config batch_size: 1 # only for incremental infer num_layers: 28 padded_vocab_size: 65024 hidden_size: 4096 ffn_hidden_size: 13696 kv_channels: 128 num_attention_heads: 32 seq_length: 193 hidden_dropout: 0.0 attention_dropout: 0.0 layernorm_epsilon: 1e-5 rmsnorm: True apply_residual_connection_post_layernorm: False post_layer_norm: True add_bias_linear: False add_qkv_bias: True bias_dropout_fusion: True multi_query_attention: True multi_query_group_num: 2 apply_query_key_layer_scaling: True attention_softmax_in_fp32: True fp32_residual_connection: False quantization_bit: 0 pre_seq_len: None prefix_projection: False param_init_type: "float16" compute_dtype: "float16" layernorm_compute_type: "float32" use_past: False eos_token_id: 2 pad_token_id: 0 repetition_penalty: 1.0 max_decode_length: 256 checkpoint_name_or_path: "glm2_6b" top_k: 1 top_p: 1 do_sample: True arch: type: ChatGLM2ForConditionalGeneration trainer: type: CausalLanguageModelingTrainer model_name: 'glm2_6b' # if True do, evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLM2Tokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/train.json" shuffle: True phase: "train" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" input_columns: ["input_ids", "labels"] max_source_length: 64 max_target_length: 128 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/dev.json" shuffle: False phase: "eval" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 8 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 65536 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # parallel config use_parallel: False parallel: parallel_mode: 1 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: True # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm2-6b" save_checkpoint_steps: 1000 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-test/configs/glm2/run_glm2_6b_lora.yaml000066400000000000000000000134511471400134000232230ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: 'glm2_6b' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" # 59GB for 910B save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: ChatGLM2Config batch_size: 1 # only for incremental infer num_layers: 28 padded_vocab_size: 65024 hidden_size: 4096 ffn_hidden_size: 13696 kv_channels: 128 num_attention_heads: 32 seq_length: 193 hidden_dropout: 0.0 attention_dropout: 0.0 layernorm_epsilon: 1e-5 rmsnorm: True apply_residual_connection_post_layernorm: False post_layer_norm: True add_bias_linear: False add_qkv_bias: True bias_dropout_fusion: True multi_query_attention: True multi_query_group_num: 2 apply_query_key_layer_scaling: True attention_softmax_in_fp32: True fp32_residual_connection: False quantization_bit: 0 pre_seq_len: None prefix_projection: False param_init_type: "float16" compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" use_past: False eos_token_id: 2 pad_token_id: 0 max_decode_length: 256 checkpoint_name_or_path: "glm2_6b" top_k: 1 top_p: 1 repetition_penalty: 1 do_sample: True pet_config: pet_type: lora lora_rank: 8 lora_alpha: 32 lora_dropout: 0.1 arch: type: ChatGLM2WithLora trainer: type: CausalLanguageModelingTrainer model_name: 'glm2_6b_lora' # if True do, evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLM2Tokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/train.json" shuffle: True phase: "train" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" input_columns: ["input_ids", "labels"] max_source_length: 64 max_target_length: 128 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/dev.json" shuffle: False phase: "eval" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 8 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # parallel config use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: True # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm2-6b-lora" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-test/configs/gpt2/000077500000000000000000000000001471400134000172155ustar00rootroot00000000000000pengcheng.mind-test/configs/gpt2/run_gpt2.yaml000066400000000000000000000105251471400134000216440ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 # 0--Graph Mode; 1--Pynative Mode device_target: "Ascend" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 4 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" checkpoint_name_or_path: "gpt2" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False arch: type: GPT2LMHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.0001 lr_end: 0.00001 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt" save_checkpoint_steps: 1 integrated_save: True save_network_params: True save_trainable_params: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-test/configs/gpt2/run_gpt2_13b.yaml000066400000000000000000000112311471400134000223040ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 4 model_parallel: 8 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 5120 num_layers: 40 num_heads: 40 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" checkpoint_name_or_path: "" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False arch: type: GPT2LMHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2-13b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-test/configs/gpt2/run_gpt2_52b.yaml000066400000000000000000000112331471400134000223110ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 16 use_seq_parallel: False optimizer_shard: True micro_batch_num: 64 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2_52b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 8192 num_layers: 64 num_heads: 32 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" checkpoint_name_or_path: "" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False arch: type: GPT2LMHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2-52b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-test/configs/gpt2/run_gpt2_lora.yaml000066400000000000000000000105231471400134000226570ustar00rootroot00000000000000seed: 0 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "gpt2" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 # 0--Graph Mode; 1--Pynative Mode device_target: "Ascend" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 8 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2' # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" checkpoint_name_or_path: "gpt2_lora" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False pet_config: pet_type: lora in_channels: 768 out_channels: 768 lora_rank: 64 lora_alpha: 128 lora_dropout: 0.1 arch: type: GPT2WithLora # lr sechdule lr_schedule: type: polynomial learning_rate: 0.001 lr_end: 0.0001 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt" save_checkpoint_steps: 1 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processorpengcheng.mind-test/configs/gpt2/run_gpt2_txtcls.yaml000066400000000000000000000104201471400134000232370ustar00rootroot00000000000000seed: 0 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model resume_training: False # context context: mode: 0 # 0--Graph Mode; 1--Pynative Mode device_target: "Ascend" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 16 sink_mode: False sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: True parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TextClassificationTrainer model_name: 'gpt2_txtcls' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: TextClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: TextClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 768 num_labels: 2 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" checkpoint_name_or_path: "gpt2" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True arch: type: GPT2ForSequenceClassification # lr sechdule lr_schedule: type: polynomial learning_rate: 0.0001 lr_end: 0.00001 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2_txtcls" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-test/configs/gpt2/run_gpt2_xl.yaml000066400000000000000000000112361471400134000223470ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 8 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2_xl' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 1600 num_layers: 48 num_heads: 25 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" checkpoint_name_or_path: "gpt2_xl" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False arch: type: GPT2LMHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2-xl" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-test/configs/gpt2/run_gpt2_xl_lora.yaml000066400000000000000000000114771471400134000233730ustar00rootroot00000000000000seed: 0 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "gpt2_xl" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 8 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2_xl' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 1600 num_layers: 48 num_heads: 25 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" checkpoint_name_or_path: "gpt2_xl_lora" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False pet_config: pet_type: lora in_channels: 768 out_channels: 768 lora_rank: 64 lora_alpha: 128 lora_dropout: 0.1 arch: type: GPT2WithLora # lr sechdule lr_schedule: type: polynomial learning_rate: 0.0005 lr_end: 0.00001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2-xl" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-test/configs/llama/000077500000000000000000000000001471400134000174275ustar00rootroot00000000000000pengcheng.mind-test/configs/llama/run_llama_13b.yaml000066400000000000000000000120751471400134000227370ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 16 910A parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_13b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 5120 num_layers: 40 num_heads: 40 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_13b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama/run_llama_13b_910b.yaml000066400000000000000000000120741471400134000234710ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 4 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 16 910B parallel_config: data_parallel: 16 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 4 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_13b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "57GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 5120 num_layers: 40 num_heads: 40 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_13b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama/run_llama_65b.yaml000066400000000000000000000121001471400134000227330ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_65b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.5e-4 lr_end: 1.5e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 128 910A parallel_config: data_parallel: 4 model_parallel: 2 pipeline_stage: 16 use_seq_parallel: False optimizer_shard: True micro_batch_num: 64 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_65b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_65b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama/run_llama_65b_910b.yaml000066400000000000000000000120731471400134000234770ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_65b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 32 910B parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 4 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_65b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_65b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama/run_llama_7b.yaml000066400000000000000000000120711471400134000226560ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_7b' # runner config runner_config: epochs: 2 batch_size: 4 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 2 model_parallel: 1 pipeline_stage: 4 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama/run_llama_7b_910b.yaml000066400000000000000000000120671471400134000234160ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_7b' # runner config runner_config: epochs: 2 batch_size: 6 sink_mode: True sink_size: 4 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910B parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 4 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama/run_llama_7b_lora.yaml000066400000000000000000000124071471400134000236760ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: 'llama_7b' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'finetune' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_7b_lora' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 learning_rate: 1.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 2 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b_lora" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_dtype: "float32" softmax_compute_dtype: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_7b_lora" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False pet_config: pet_type: lora # configuration of lora in_channels: 4096 out_channels: 4096 lora_rank: 16 lora_alpha: 16 lora_dropout: 0.05 arch: type: LlamaForCausalLMWithLora processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # cfts init config remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/000077500000000000000000000000001471400134000175115ustar00rootroot00000000000000pengcheng.mind-test/configs/llama2/.ipynb_checkpoints/000077500000000000000000000000001471400134000233025ustar00rootroot00000000000000pengcheng.mind-test/configs/llama2/.ipynb_checkpoints/run_llama2_1b-checkpoint.yaml000066400000000000000000000124701471400134000307350ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 100 minimum_loss_scale: 1048576 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/.ipynb_checkpoints/run_llama2_1b_32-checkpoint.yaml000066400000000000000000000124151471400134000312400ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 64 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 20 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/.ipynb_checkpoints/run_llama2_1b_loss_scale-checkpoint.yaml000066400000000000000000000124151471400134000331430ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 100 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/.ipynb_checkpoints/run_llama2_7b-checkpoint.yaml000066400000000000000000000124161471400134000307430ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 20 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 4 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/predict_llama2_70b_910b.yaml000066400000000000000000000075541471400134000244750ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'predict' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_70b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 2 use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 32 910B parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_70b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" ascend_config: precision_mode: "must_keep_origin_dtype" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 n_kv_heads: 8 ffn_dim_multiplier: 1.3 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: True pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: True checkpoint_name_or_path: "llama2_70b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaProcessor vocab_file: "" type: LlamaProcessor # metric metric: type: PerplexityMetric eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_13b.yaml000066400000000000000000000124171471400134000231030ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 16 910A parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_13b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 5120 num_layers: 40 num_heads: 40 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_13b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_13b_910b.yaml000066400000000000000000000120771471400134000236400ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 16 910B parallel_config: data_parallel: 16 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_13b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 5120 num_layers: 40 num_heads: 40 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_13b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_1b.yaml000066400000000000000000000124701471400134000230170ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 100 minimum_loss_scale: 1048576 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_1b_32.yaml000066400000000000000000000124151471400134000233220ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 64 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 20 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_1b_loss_scale.yaml000066400000000000000000000124151471400134000252250ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 100 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_200M_cosD_tokens2b.yaml000066400000000000000000000124201471400134000257050ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 2.e-4 lr_end: 2.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: True eval_step_interval: 800 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 20 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: False gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama2_200M" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 1280 num_layers: 10 num_heads: 32 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_70b.yaml000066400000000000000000000124471471400134000231110ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_70b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 1.5e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 128 910A parallel_config: data_parallel: 4 model_parallel: 2 pipeline_stage: 16 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_70b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 n_kv_heads: 8 ffn_dim_multiplier: 1.3 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 32000 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_70b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LLaMATokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_70b_910b.yaml000066400000000000000000000124371471400134000236430ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_70b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 32 910B parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 4 use_seq_parallel: False optimizer_shard: True micro_batch_num: 32 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_70b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 n_kv_heads: 8 ffn_dim_multiplier: 1.3 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_70b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LLaMATokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_7b.yaml000066400000000000000000000124161471400134000230250ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 20 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 4 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_7b_910b.yaml000066400000000000000000000120621471400134000235550ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 2 batch_size: 4 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910B parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 4 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_width_256_dataset.yaml000066400000000000000000000125661471400134000257430ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 lr_end: 1.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False # important input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 consumed_samples: 0 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 64 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 512 # 2304 num_layers: 10 num_heads: 8 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False mup_width_base: 256 arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 20 minimum_loss_scale: 524288 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/llama2/run_llama2_width_256_dataset_resume.yaml000066400000000000000000000126621471400134000273200ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' restore_ckpt_name_prefix: 'llama_7b_rank_*-500_2.ckpt' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: True run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 lr_end: 1.e-5 warmup_ratio: 0.01 total_steps: 8280 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False # important input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 consumed_samples: 64000 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 64 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 512 # 2304 num_layers: 10 num_heads: 8 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False mup_width_base: 256 arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 20 minimum_loss_scale: 524288 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/mae/000077500000000000000000000000001471400134000171035ustar00rootroot00000000000000pengcheng.mind-test/configs/mae/run_mae_vit_base_p16_224_800ep.yaml000066400000000000000000000113441471400134000252650ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: 64 image_size: 224 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: masked_image_modeling model_name: 'mae_vit_base_p16' # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 64 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: RandomCropDecodeResize size: 224 scale: [0.2, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: Normalize mean: [123.675, 118.575, 103.53] std: [58.395, 62.22, 57.375] - type: HWC2CHW mask_policy: type: MaeMask input_size: 224 patch_size: 16 mask_ratio: 0.75 input_columns: ["image"] output_columns: ["image", "mask", "ids_restore", "unmask_index"] column_order: ["image", "mask", "ids_restore", "unmask_index"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: MIMDataset dataset_config: *train_dataset # model model: arch: type: ViTMAEForPreTraining model_config: type: ViTMAEConfig mask_ratio: 0.75 # mask ratio image_size: 224 # input image size patch_size: 16 # patch size num_channels: 3 # channels of input images initializer_range: 0.02 # initial std of cls_tokens and mask_tokens hidden_size: 768 # embedding dimension num_hidden_layers: 12 # number of transformer blocks num_attention_heads: 12 # number of attention heads intermediate_size: 3072 # ffn_hidden_size of encoder qkv_bias: True # if has bias in qkv dense hidden_act: gelu # activation of MLP post_layernorm_residual: False # if using post layernorm residual layer_norm_eps: 0.000001 # eps of layer_norm attention_probs_dropout_prob: 0. # drop rate of Attention hidden_dropout_prob: 0. # drop rate of MLP decoder_hidden_size: 512 # decoder embedding dim decoder_num_hidden_layers: 8 # number of decoder transformer blocks decoder_num_attention_heads: 16 # number of decoder attention heads decoder_intermediate_size: 2048 # ffn_hidden_size of decoder norm_pix_loss: True # if using norm pixel loss checkpoint_name_or_path: 'mae_vit_base_p16' # lr sechdule lr_schedule: type: cosine learning_rate: 0.00015 lr_end: 0. warmup_lr_init: 0. warmup_epochs: 40 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: ViTMAEProcessor image_processor: type: ViTMAEImageProcessor size: 224 # input image size pengcheng.mind-test/configs/official_training/000077500000000000000000000000001471400134000220105ustar00rootroot00000000000000pengcheng.mind-test/configs/official_training/run_minicpm_ckpt-2B-v4.yaml000066400000000000000000000136001471400134000270250ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False reuse_ckpt: False # If true, data parallel reuse ckpt for optimizer_weight_shard_size 64 resume_merged_ckpt: False reset_steps: False unload_adam: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'naohai_2b6' # runner config runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 1.e-3 weight_decay: 0.1 # lr sechdule lr_schedule: type: WarmUpCosineWithDecayLR # WarmUpCosineWithDecayLR learning_rate: 0.001 lr_middle: 0.001 # lr_middle==learning_rate is same to WarmUpStableDecayLR lr_end: 0.0 warmup_ratio: 0.0004 decay_ratio: 0.15 total_steps: 1000000 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False drop_skip: True prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 drop_skip: False eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True optimizer_weight_shard_size: 32 search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 1 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 8 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: False select_recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "naohai_2b6_" save_checkpoint_steps: 1000 keep_checkpoint_max: 1 integrated_save: False async_save: False - type: ObsMonitor keep_last: False step_upload_frequence: 1000 # keep same to save_checkpoint_steps # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31.5GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2304 num_layers: 40 num_heads: 36 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: True offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_2b6" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False mup_width_base: 256 depth_scale: 0.8 init_std: 0.025 embed_scale: 2.0 arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 40 minimum_loss_scale: 32768 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/official_training/run_minicpm_ckpt-2B-v4_sft.yaml000066400000000000000000000135761471400134000277150ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False reuse_ckpt: False # If true, data parallel reuse ckpt for optimizer_weight_shard_size 64 resume_merged_ckpt: False reset_steps: False unload_adam: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'naohai_2b6' # runner config runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 1.e-3 weight_decay: 0.1 # lr sechdule lr_schedule: type: WarmUpCosineWithDecayLR # WarmUpCosineWithDecayLR learning_rate: 0.001 lr_middle: 0.001 # lr_middle==learning_rate is same to WarmUpStableDecayLR lr_end: 0.0 warmup_ratio: 0.0004 decay_ratio: 0.15 total_steps: 1000000 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 3 numa_enable: False drop_skip: True prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 drop_skip: False eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True optimizer_weight_shard_size: 32 search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 1 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 8 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: False select_recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "naohai_2b6_" save_checkpoint_steps: 200 keep_checkpoint_max: 1 integrated_save: False async_save: False - type: ObsMonitor keep_last: False step_upload_frequence: 200 # keep same to save_checkpoint_steps # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31.5GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2304 num_layers: 40 num_heads: 36 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: True offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_2b6" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False mup_width_base: 256 depth_scale: 0.8 init_std: 0.025 embed_scale: 2.0 arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 40 minimum_loss_scale: 32768 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-test/configs/pangualpha/000077500000000000000000000000001471400134000204615ustar00rootroot00000000000000pengcheng.mind-test/configs/pangualpha/run_pangualpha_13b.yaml000066400000000000000000000115541471400134000250240ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 8 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" # dp 4 mp 4 pp 1 for 2 nodes 16 devices micro_batch_num 1 # dp 4 mp 2 pp 4 for 4 nodes 32 devices micro_batch_num 32 # dp 16 mp 1 pp 8 for 16 nodes 128 devices micro_batch_num 80 parallel_config: data_parallel: 4 model_parallel: 4 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'pangualpha_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] output_columns: ["input_ids", "position_id", "attention_mask"] eos_reset: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] output_columns: ["input_ids", "position_id", "attention_mask"] eos_reset: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: PanguAlphaConfig batch_size: 1 seq_length: 1024 vocab_size: 40000 hidden_size: 5120 ffn_hidden_size: 20480 num_layers: 40 num_heads: 40 pad_token_id: 6 eos_token_id: 6 post_layernorm_residual: False param_init_type: 'float32' compute_dtype: 'float16' softmax_compute_type: 'float16' embedding_dropout_prob: 0.1 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 hidden_act: 'fast_gelu' use_past: False use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_13b" repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True arch: type: PanguAlphaHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "PanguAlpha-13b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: type: PanguAlphaTokenizer type: PanguAlphaProcessorpengcheng.mind-test/configs/pangualpha/run_pangualpha_2_6b.yaml000066400000000000000000000113251471400134000251630ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 16 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" # 1 node 8 device num parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'pangualpha_2_6b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] output_columns: ["input_ids", "position_id", "attention_mask"] eos_reset: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 16 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] output_columns: ["input_ids", "position_id", "attention_mask"] eos_reset: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 16 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: PanguAlphaConfig batch_size: 1 seq_length: 1024 vocab_size: 40000 hidden_size: 2560 ffn_hidden_size: 10240 num_layers: 32 num_heads: 32 pad_token_id: 6 eos_token_id: 6 post_layernorm_residual: False param_init_type: 'float32' compute_dtype: 'float16' softmax_compute_type: 'float16' embedding_dropout_prob: 0.1 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 hidden_act: 'fast_gelu' use_past: False use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_2_6b" repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True arch: type: PanguAlphaHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "PanguAlpha-2_6b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: type: PanguAlphaTokenizer type: PanguAlphaProcessorpengcheng.mind-test/configs/pangualpha/run_pangualpha_2_6b_em_f1.yaml000066400000000000000000000102431471400134000262300ustar00rootroot00000000000000seed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" # 1 node 8 device num parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'pangualpha_2_6b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels"] eos_reset: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 16 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: PanguAlphaConfig batch_size: 16 seq_length: 1024 vocab_size: 40000 hidden_size: 2560 ffn_hidden_size: 10240 num_layers: 32 num_heads: 32 pad_token_id: 6 eos_token_id: 6 post_layernorm_residual: False param_init_type: 'float32' compute_dtype: 'float16' softmax_compute_type: 'float16' embedding_dropout_prob: 0.1 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 hidden_act: 'fast_gelu' use_past: False use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_2_6b" repetition_penalty: 1.5 max_decode_length: 1024 max_new_tokens: 32 top_k: 3 top_p: 0.0 do_sample: True arch: type: PanguAlphaHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "PanguAlpha-2_6b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: EmF1Metric # processor processor: return_tensors: ms tokenizer: type: PanguAlphaTokenizer type: PanguAlphaProcessor pengcheng.mind-test/configs/pangualpha/run_pangualpha_2_6b_prompt_txtcls.yaml000066400000000000000000000103601471400134000301630ustar00rootroot00000000000000seed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "pangualpha_2_6b" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" # 1 node 8 device num parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'pangualpha_2_6b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels", "attention_mask"] eos_reset: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: PanguAlphaConfig batch_size: 1 seq_length: 1024 vocab_size: 40000 hidden_size: 2560 ffn_hidden_size: 10240 num_layers: 32 num_heads: 32 pad_token_id: 6 eos_token_id: 6 post_layernorm_residual: False param_init_type: 'float32' compute_dtype: 'float16' softmax_compute_type: 'float16' embedding_dropout_prob: 0.1 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 hidden_act: 'fast_gelu' use_past: False use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_2_6b" repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True num_labels: 4 arch: type: PanguAlphaPromptTextClassificationModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "PanguAlpha-2_6b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PromptAccMetric # processor processor: return_tensors: ms tokenizer: type: PanguAlphaTokenizer type: PanguAlphaProcessor pengcheng.mind-test/configs/qa/000077500000000000000000000000001471400134000167425ustar00rootroot00000000000000pengcheng.mind-test/configs/qa/run_qa_bert_base_uncased.yaml000066400000000000000000000127161471400134000246320ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 12 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: QuestionAnsweringTrainer model_name: qa_bert_base_uncased # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: SQuADDataLoader dataset_dir: "./squad/" stage: "train" column_names: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' do_basic_tokenize: True do_lower_case: True input_columns: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] output_columns: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] column_order: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] num_parallel_workers: 8 python_multiprocessing: True drop_remainder: True batch_size: 12 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: QuestionAnsweringDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: SQuADDataLoader dataset_dir: "./squad/" stage: "dev" column_names: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' do_basic_tokenize: True do_lower_case: True input_columns: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] output_columns: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] column_order: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] num_parallel_workers: 8 python_multiprocessing: True drop_remainder: True batch_size: 12 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: QuestionAnsweringDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 3 dropout_prob: 0.1 batch_size: 12 seq_length: 384 vocab_size: 30522 embedding_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" post_layernorm_residual: True hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "qa_bert_base_uncased_squad" arch: type: BertForQuestionAnswering # lr sechdule lr_schedule: type: linear learning_rate: 0.00003 # 3e-5 warmup_ratio: 0.1 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 3000 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: SQuADMetric dataset_dir: "./squad/" # processor processor: return_tensors: ms type: BertProcessor tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' do_basic_tokenize: True do_lower_case: True pengcheng.mind-test/configs/sam/000077500000000000000000000000001471400134000171215ustar00rootroot00000000000000pengcheng.mind-test/configs/sam/run_sam_vit-b.yaml000066400000000000000000000152701471400134000225570ustar00rootroot00000000000000seed: 0 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 32 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_segmentation model_name: 'vit_base_p16' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: Sam model_config: type: SAMConfig image_encoder: arch: type: SAMImageEncoder model_config: type: ImageEncoderConfig img_size: 1024 patch_size: 16 in_chans: 3 embed_dim: 768 depth: 12 num_heads: 12 mlp_ratio: 4 out_chans: 256 qkv_bias: True use_abs_pos: True use_rel_pos: True # 仅支持默认False rel_pos_zero_init: True window_size: 14 global_attn_indexes: [2, 5, 8, 11] layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" prompt_config: arch: type: SAMPromptEncoder model_config: type: PromptEncoderConfig prompt_embed_dim: 256 image_embedding_size: [64, 64] input_image_size: [1024, 1024] mask_in_chans: 16 compute_dtype: "float16" decoder_config: arch: type: SAMMaskDecoder model_config: type: MaskDecoderConfig num_multimask_outputs: 3 # transformer: decoder_depth: 2 decoder_embed_dim: 256 decoder_mlp_dim: 2048 decoder_num_heads: 8 transformer_dim: 256 iou_head_depth: 3 iou_head_hidden_dim: 256 layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" # std checkpoint_name_or_path: "sam_vit_b" # lr sechdule lr_schedule: type: cosine learning_rate: 0.0005 lr_end: 0.000001 warmup_epochs: 5 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: SAMProcessor image_processor: type: SAMImageProcessor img_size: 1024 # input image size: int or [h, w] or (h, w) mean: [123.675, 116.28, 103.53] std: [58.395, 57.12, 57.375] pengcheng.mind-test/configs/sam/run_sam_vit-h.yaml000066400000000000000000000152731471400134000225700ustar00rootroot00000000000000seed: 0 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 32 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_segmentation model_name: 'vit_base_p16' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: Sam model_config: type: SAMConfig image_encoder: arch: type: SAMImageEncoder model_config: type: ImageEncoderConfig img_size: 1024 patch_size: 16 in_chans: 3 embed_dim: 1280 depth: 32 num_heads: 16 mlp_ratio: 4 out_chans: 256 qkv_bias: True use_abs_pos: True use_rel_pos: True # 仅支持默认False rel_pos_zero_init: True window_size: 14 global_attn_indexes: [7, 15, 23, 31] layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" prompt_config: arch: type: SAMPromptEncoder model_config: type: PromptEncoderConfig prompt_embed_dim: 256 image_embedding_size: [64, 64] input_image_size: [1024, 1024] mask_in_chans: 16 compute_dtype: "float16" decoder_config: arch: type: SAMMaskDecoder model_config: type: MaskDecoderConfig num_multimask_outputs: 3 # transformer: decoder_depth: 2 decoder_embed_dim: 256 decoder_mlp_dim: 2048 decoder_num_heads: 8 transformer_dim: 256 iou_head_depth: 3 iou_head_hidden_dim: 256 layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" # std checkpoint_name_or_path: "sam_vit_h" # lr sechdule lr_schedule: type: cosine learning_rate: 0.0005 lr_end: 0.000001 warmup_epochs: 5 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: SAMProcessor image_processor: type: SAMImageProcessor img_size: 1024 # input image size: int or [h, w] or (h, w) mean: [123.675, 116.28, 103.53] std: [58.395, 57.12, 57.375] pengcheng.mind-test/configs/sam/run_sam_vit-l.yaml000066400000000000000000000152731471400134000225740ustar00rootroot00000000000000seed: 0 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 32 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_segmentation model_name: 'vit_base_p16' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: Sam model_config: type: SAMConfig image_encoder: arch: type: SAMImageEncoder model_config: type: ImageEncoderConfig img_size: 1024 patch_size: 16 in_chans: 3 embed_dim: 1024 depth: 24 num_heads: 16 mlp_ratio: 4 out_chans: 256 qkv_bias: True use_abs_pos: True use_rel_pos: True # 仅支持默认False rel_pos_zero_init: True window_size: 14 global_attn_indexes: [5, 11, 17, 23] layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" prompt_config: arch: type: SAMPromptEncoder model_config: type: PromptEncoderConfig prompt_embed_dim: 256 image_embedding_size: [64, 64] input_image_size: [1024, 1024] mask_in_chans: 16 compute_dtype: "float16" decoder_config: arch: type: SAMMaskDecoder model_config: type: MaskDecoderConfig num_multimask_outputs: 3 # transformer: decoder_depth: 2 decoder_embed_dim: 256 decoder_mlp_dim: 2048 decoder_num_heads: 8 transformer_dim: 256 iou_head_depth: 3 iou_head_hidden_dim: 256 layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" # std checkpoint_name_or_path: "sam_vit_l" # lr sechdule lr_schedule: type: cosine learning_rate: 0.0005 lr_end: 0.000001 warmup_epochs: 5 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: SAMProcessor image_processor: type: SAMImageProcessor img_size: 1024 # input image size: int or [h, w] or (h, w) mean: [123.675, 116.28, 103.53] std: [58.395, 57.12, 57.375] pengcheng.mind-test/configs/swin/000077500000000000000000000000001471400134000173215ustar00rootroot00000000000000pengcheng.mind-test/configs/swin/run_swin_base_p4w7_224_100ep.yaml000066400000000000000000000140701471400134000252220ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 128 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 use_clip_grad: True max_grad_norm: 5.0 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_classification model_name: 'swin_base_p4w7' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset seed: 0 batch_size: 128 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 0 batch_size: 128 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: SwinForImageClassification model_config: type: SwinConfig num_labels: 1000 # num classes image_size: 224 # input image size patch_size: 4 # patch size num_channels: 3 # channels of input images embed_dim: 128 # embedding dimension depths: [2, 2, 18, 2] # number of transformer blocks for each swin layer num_heads: [4, 8, 16, 32] # number of attention heads for each swin layer window_size: 7 # window size for swin mlp_ratio: 4 # ffn_hidden_size = mlp_ratio * embed_dim qkv_bias: True # has transformer qkv bias or not layer_norm_eps: 0.00001 # eps of layer_norm hidden_dropout_prob: 0. # drop rate of MLP attention_probs_dropout_prob: 0. # drop rate of Attention drop_path_rate: 0.1 # drop path rate of transformer blocks use_absolute_embeddings: False # if using absolute position embedding patch_norm: True # use norm in SwinPatchEmbeddings hidden_act: gelu # activation of MLP weight_init: normal # weight initialize type loss_type: SoftTargetCrossEntropy # loss type checkpoint_name_or_path: swin_base_p4w7 # lr sechdule lr_schedule: type: cosine learning_rate: 0.00125 lr_end: 0.00000025 warmup_lr_init: 0.00000025 warmup_epochs: 20 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.9 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 512 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: SwinProcessor image_processor: type: SwinImageProcessor size: 224 # input image size pengcheng.mind-test/configs/t5/000077500000000000000000000000001471400134000166715ustar00rootroot00000000000000pengcheng.mind-test/configs/t5/run_t5_small_on_wmt16.yaml000066400000000000000000000107011471400134000237120ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell scale_factor: 2 scale_window: 1000 loss_scale_value: 65536 use_clip_grad: True max_grad_norm: 1.0 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: translation model_name: 't5' # train dataset train_dataset: &train_dataset data_loader: type: WMT16DataLoader dataset_dir: "/your/wmt_en_ro" tokenizer: type: t5_small src_max_length: 1024 # this will truncate the source input sequence tgt_max_length: 128 prefix: "translate the English to Romanian:" stage: 'train' input_columns: ["input_ids", "attention_mask", "labels"] # determinied by the model inputs output_columns: ["input_ids", "attention_mask", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: TranslationDataset dataset_config: *train_dataset # model model: arch: type: T5ForConditionalGeneration model_config: batch_size: 1 d_ff: 2048 hidden_size: 512 do_sample: false eos_token_id: 1 has_relative_bias: true hidden_act: relu hidden_dropout_prob: 0.1 initializer_factor: 1.0 initializer_range: 0.02 is_encoder_decoder: true d_kv: 64 layer_norm_epsilon: 1.0e-06 length_penalty_weight: 1.0 max_decode_length: 128 max_length: 32 max_position_embeddings: 1024 num_heads: 8 num_hidden_layers: 6 pad_token_id: 0 relative_attention_num_buckets: 32 repetition_penalty: 1 scale_output: true seq_length: 1024 top_k: 1 top_p: 0.95 type: T5Config use_cache: true vocab_size: 32128 checkpoint_name_or_path: t5_small hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 embedding_dropout_prob: 0.1 param_init_type: float32 layernorm_compute_type: float32 softmax_compute_type: float32 post_layernorm_residual: False offset: 0 use_past: False # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.0 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.0 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: LossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: max_length: 77 padding: max_length return_tensors: ms tokenizer: eos_token: pad_token: type: T5Tokenizer unk_token: type: T5Processor pengcheng.mind-test/configs/t5/run_t5_tiny_on_wmt16.yaml000066400000000000000000000106321471400134000235700ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell scale_factor: 2 scale_window: 1000 loss_scale_value: 65536 use_clip_grad: True max_grad_norm: 1.0 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: translation model_name: 't5' # train dataset train_dataset: &train_dataset data_loader: type: WMT16DataLoader dataset_dir: "/your/wmt_en_ro" tokenizer: type: t5_small src_max_length: 1024 # this will truncate the source input sequence tgt_max_length: 128 prefix: "translate the English to Romanian:" stage: 'train' input_columns: ["input_ids", "attention_mask", "labels"] # determinied by the model inputs output_columns: ["input_ids", "attention_mask", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: TranslationDataset dataset_config: *train_dataset # model model: arch: type: T5ForConditionalGeneration model_config: batch_size: 1 d_ff: 512 hidden_size: 128 do_sample: false eos_token_id: 1 has_relative_bias: true hidden_act: relu hidden_dropout_prob: 0.1 initializer_factor: 1.0 initializer_range: 0.02 is_encoder_decoder: true d_kv: 64 layer_norm_epsilon: 1.0e-06 length_penalty_weight: 1.0 max_decode_length: 128 max_length: 32 max_position_embeddings: 1024 num_heads: 8 num_hidden_layers: 1 pad_token_id: 0 relative_attention_num_buckets: 32 repetition_penalty: 1 scale_output: true seq_length: 1024 top_k: 1 top_p: 0.95 type: T5Config use_cache: true vocab_size: 32128 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 embedding_dropout_prob: 0.1 param_init_type: float32 layernorm_compute_type: float32 softmax_compute_type: float32 post_layernorm_residual: False offset: 0 use_past: False # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.0 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.0 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: LossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: max_length: 77 padding: max_length return_tensors: ms tokenizer: eos_token: pad_token: type: T5Tokenizer unk_token: type: T5Processor pengcheng.mind-test/configs/tokcls/000077500000000000000000000000001471400134000176405ustar00rootroot00000000000000pengcheng.mind-test/configs/tokcls/run_tokcls_bert_base_chinese.yaml000066400000000000000000000132771471400134000264250ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 24 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TokenClassificationTrainer model_name: tokcls_bert_base_chinese # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: CLUENERDataLoader dataset_dir: "./cluener/" stage: "train" column_names: ["text", "label_id"] text_transforms: type: TokenizeWithLabel max_length: 128 padding: "max_length" label_transforms: type: LabelPadding max_length: 128 padding_value: 0 tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese input_columns: ["text", "label_id"] output_columns: ["input_ids", "token_type_ids", "attention_mask", "label_id"] column_order: ["input_ids", "token_type_ids", "attention_mask", "label_id"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 24 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: TokenClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: CLUENERDataLoader dataset_dir: "./cluener/" stage: "dev" column_names: ["text", "label_id"] text_transforms: type: TokenizeWithLabel max_length: 128 padding: "max_length" label_transforms: type: LabelPadding max_length: 128 padding_value: 0 tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese input_columns: ["text", "label_id"] output_columns: ["input_ids", "token_type_ids", "attention_mask", "label_id"] column_order: ["input_ids", "token_type_ids", "attention_mask", "label_id"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 24 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: TokenClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 31 dropout_prob: 0.1 batch_size: 24 seq_length: 128 # length of input sentence vocab_size: 21128 # size of vocab embedding_size: 768 # size of text feature num_layers: 12 # model depth num_heads: 12 # number of attention heads expand_ratio: 4 hidden_act: "gelu" # activation post_layernorm_residual: True # select postlayernorm or prelayernorm hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "tokcls_bert_base_chinese" arch: type: BertForTokenClassification # lr sechdule lr_schedule: type: linear learning_rate: 0.00003 # 3e-5 warmup_ratio: 0.1 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: EntityScore # processor processor: type: BertProcessor return_tensors: ms tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese pengcheng.mind-test/configs/tokcls/run_tokcls_bert_base_chinese_cluener.yaml000066400000000000000000000133171471400134000301350ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 24 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TokenClassificationTrainer model_name: tokcls_bert_base_chinese_cluener # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: CLUENERDataLoader dataset_dir: "./cluener/" stage: "train" column_names: ["text", "label_id"] text_transforms: type: TokenizeWithLabel max_length: 128 padding: "max_length" label_transforms: type: LabelPadding max_length: 128 padding_value: 0 tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese input_columns: ["text", "label_id"] output_columns: ["input_ids", "token_type_ids", "attention_mask", "label_id"] column_order: ["input_ids", "token_type_ids", "attention_mask", "label_id"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 24 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: TokenClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: CLUENERDataLoader dataset_dir: "./cluener/" stage: "dev" column_names: ["text", "label_id"] text_transforms: type: TokenizeWithLabel max_length: 128 padding: "max_length" label_transforms: type: LabelPadding max_length: 128 padding_value: 0 tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese input_columns: ["text", "label_id"] output_columns: ["input_ids", "token_type_ids", "attention_mask", "label_id"] column_order: ["input_ids", "token_type_ids", "attention_mask", "label_id"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 24 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: TokenClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 31 dropout_prob: 0.1 batch_size: 24 seq_length: 128 # length of input sentence vocab_size: 21128 # size of vocab embedding_size: 768 # size of text feature num_layers: 12 # model depth num_heads: 12 # number of attention heads expand_ratio: 4 hidden_act: "gelu" # activation post_layernorm_residual: True # select postlayernorm or prelayernorm hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "tokcls_bert_base_chinese_cluener" arch: type: BertForTokenClassification # lr sechdule lr_schedule: type: linear learning_rate: 0.00003 # 3e-5 warmup_ratio: 0.1 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: EntityScore # processor processor: type: BertProcessor return_tensors: ms tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese pengcheng.mind-test/configs/txtcls/000077500000000000000000000000001471400134000176625ustar00rootroot00000000000000pengcheng.mind-test/configs/txtcls/run_txtcls_bert_base_uncased.yaml000066400000000000000000000105771471400134000264750ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 64 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-standalone, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TextClassificationTrainer model_name: txtcls_bert_base_uncased # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: TFRecordDataset dataset_dir: "./mnli/train" shuffle: True input_columns: ["input_ids", "input_mask", "segment_ids", "label_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 42 train_dataset_task: type: TextClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: TFRecordDataset dataset_dir: "./mnli/eval" shuffle: True input_columns: ["input_ids", "input_mask", "segment_ids", "label_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 42 eval_dataset_task: type: TextClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 3 dropout_prob: 0.1 batch_size: 64 seq_length: 128 vocab_size: 30522 embedding_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" post_layernorm_residual: True hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "txtcls_bert_base_uncased" arch: type: BertForMultipleChoice # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 # 5e-5 lr_end: 0.000001 # 1e-6 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: return_tensors: ms tokenizer: cls_token: '[CLS]' do_basic_tokenize: True do_lower_case: True mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' type: BertTokenizer unk_token: '[UNK]' type: BertProcessor top_k: 1 pengcheng.mind-test/configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml000066400000000000000000000106111471400134000275010ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 64 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-standalone, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TextClassificationTrainer model_name: txtcls_bert_base_uncased_mnli # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: TFRecordDataset dataset_dir: "./mnli/train" shuffle: True input_columns: ["input_ids", "input_mask", "segment_ids", "label_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 42 train_dataset_task: type: TextClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: TFRecordDataset dataset_dir: "./mnli/eval" shuffle: True input_columns: ["input_ids", "input_mask", "segment_ids", "label_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 42 eval_dataset_task: type: TextClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 3 dropout_prob: 0.1 batch_size: 64 seq_length: 128 vocab_size: 30522 embedding_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" post_layernorm_residual: True hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "txtcls_bert_base_uncased_mnli" arch: type: BertForMultipleChoice # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 # 5e-5 lr_end: 0.000001 # 1e-6 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: return_tensors: ms tokenizer: cls_token: '[CLS]' do_basic_tokenize: True do_lower_case: True mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' type: BertTokenizer unk_token: '[UNK]' type: BertProcessor top_k: 1 pengcheng.mind-test/configs/vit/000077500000000000000000000000001471400134000171435ustar00rootroot00000000000000pengcheng.mind-test/configs/vit/run_vit_base_p16_224_100ep.yaml000066400000000000000000000136671471400134000245060ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 32 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_classification model_name: 'vit_base_p16' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: ViTForImageClassification model_config: type: ViTConfig image_size: 224 # input image size patch_size: 16 # patch size num_channels: 3 # channels of input images initializer_range: 0.02 # initial std of cls_tokens, pos_embed and linear weight hidden_size: 768 # embedding dimension num_hidden_layers: 12 # number of transformer blocks num_attention_heads: 12 # number of attention heads intermediate_size: 3072 # ffn_hidden_size of encoder qkv_bias: True # if has bias in qkv dense hidden_act: gelu # activation of MLP post_layernorm_residual: False # if using post layernorm residual layer_norm_eps: 0.000001 # eps of layer_norm attention_probs_dropout_prob: 0. # drop rate of Attention hidden_dropout_prob: 0. # drop rate of MLP drop_path_rate: 0.1 # drop path rate of Block use_mean_pooling: True # if using global mean pooling num_labels: 1000 # num classes loss_type: SoftTargetCrossEntropy # loss type checkpoint_name_or_path: 'vit_base_p16' # lr sechdule lr_schedule: type: cosine learning_rate: 0.0005 lr_end: 0.000001 warmup_epochs: 5 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: ViTProcessor image_processor: type: ViTImageProcessor size: 224 # input image size pengcheng.mind-test/docs/000077500000000000000000000000001471400134000156415ustar00rootroot00000000000000pengcheng.mind-test/docs/FAQ.md000066400000000000000000000117151471400134000165770ustar00rootroot00000000000000# FAQ 目录 [1.1 如何给MindSpore贡献代码](#11-如何给MindSpore贡献代码) [1.2 多次提交记录，如何进行合并？](#12-多次提交记录如何进行合并) [1.3 如何提交只有一个commit？](#13-如何提交只有一个commit) [1.4 多人协作，如何解决代码上传冲突？](#14-多人协作如何解决代码上传冲突) [1.5 开发过程中遇到问题，如果在开源社区中进行提问？](#15-开发过程中遇到问题如果在开源社区中进行提问) [1.6 PR提交者是否需要签署CLA？](#16-pr提交者是否需要签署cla) [1.7 PR提交时显示存在冲突，无法自动合并？](#17-pr提交时显示存在冲突无法自动合并) [1.8 CI门禁检查提示的错误不在PR提交的内容中？](#18-ci门禁检查提示的错误不在pr提交的内容中) ## 1. FAQ 在查看FAQ之前，请仔细阅读[代码提交注意指南](https://gitee.com/mindspore/vision/wikis/%E4%BB%A3%E7%A0%81%E6%8F%90%E4%BA%A4%E6%B3%A8%E6%84%8F%E6%8C%87%E5%8D%97) ### 1.1 如何给MindSpore贡献代码 请参考bilibili视频连接：[如何给MindSpore贡献代码](https://www.bilibili.com/video/BV1kg411F7Pc/) ![输入图片说明](https://images.gitee.com/uploads/images/2022/0414/171942_e6202765_7386274.png "屏幕截图.png") 视频会手把手带着大家如何贡献代码到gitee、如何fork MindSpore仓，并进行代码提交，解决git中username冲突等的问题。请一定要先观看该视频哦，会给您节省后续很多麻烦！ ### 1.2 多次提交记录，如何进行合并 - 问题描述： MindSpore transformer代码仓提交规范要求，提交记录仅保留一条，所以多次提交时，需要进行合并。 ![输入图片说明](https://images.gitee.com/uploads/images/2022/0414/171522_cbcd9488_7386274.png "屏幕截图.png") - 解决方案： 1. 通过 git log 查看代码提交的log记录，使用 git rebase -i 选择要合并的 commit 2. 编辑要合并的版本信息，即修改需要合并的 commit id 前面的pick修改为 squash，并保存提交 3. 修改注释信息后，保存提交，多条合并会出现多次 4. git push 推送远程仓库或合并到主干分支 ### 1.3 如何提交只有一个commit - 问题描述： 为了开源规范和方便审核，注意提交的时候只能有一个 commit： ![输入图片说明](https://images.gitee.com/uploads/images/2022/0414/170606_2dcf4944_7386274.png "屏幕截图.png") - 解决方案： 1. 修改代码之前，请先git fetch --all; git pull; 拉取最新的代码再修改 2. 如果已经提交PR，此时可以基于上一个PR进行修改，使用 git add *; git commit --amend; git push -f; 3. 通过commit --amend可以把当前的修改叠加到上一次代码中 ### 1.4 多人协作，如何解决代码上传冲突 可能是已经有同学先合入代码了，再合入代码时，会报冲突。需要先把个人仓先更新一下。更新方法是： `>>> git remote add upstream https://gitee.com/mindspore/vision.git` 这就把远端主仓添加到 upstream 里面了，然后使用命令 git merge： `>>> git rebase upstream/master` ### 1.5 开发过程中遇到问题，如果在开源社区中进行提问？ 在[transformer仓的issues区](https://gitee.com/mindspore/transformer/issues)提交issues，按照模板来描述问题，我们的工程师会在下面的comment留言哦。 ### 1.6 PR提交者是否需要签署CLA？ PR的提交者，每一个都要根据提醒来签署CLA的，CLA是保障大家贡献的代码都能够得到开源保障，声明开发者是你本人哦。 ### 1.7 PR提交时显示存在冲突，无法自动合并？ 由于存在多人协作开发，因此在开发期间如果没有进行及时同步代码仓，则有可能出现冲突，在PR提交时无法自动合并。例如A团队的commit4与B团队的commit5存在冲突时，A团队的PR则无法提交。因此建议在PR提交前，同步最新的master分支，然后对开发分支进行rebase master操作。 ### 1.8 CI门禁检查提示的错误不在PR提交的内容中？ 无论是使用pycharm还是vscode等IDE进行开发，**请一定要安装autoPEP8等格式代码规范检查的工具**，每一次进行代码修改后，需要规范化代码，然后再进行提交。避免CI要等和阻塞等问题。 因部分静态检查工具引入或升级时，代码中已经存在一些拼写问题，而CI门禁中检查工具并不会对仓库中所有代码进行检查，所以可能会出现PR提交内容之外的错误。该问题涉及的是静态检查工具引入或升级后，该文件首次被修改，一般为一些简单的拼写错误，请按提示进行修正后重新触发门禁。 如果静态检查提示了一些错误，但是按照提示内容检查PR文件却未找到相关代码，请rebase最新代码后再按照提示进行修正。此问题出现的原因是三方开源检查工具小概率失效，导致错误放行并合入代码后，其它PR再次检查该处时报错。pengcheng.mind-test/docs/aicc_cards/000077500000000000000000000000001471400134000177145ustar00rootroot00000000000000pengcheng.mind-test/docs/aicc_cards/README.md000066400000000000000000000330571471400134000212030ustar00rootroot00000000000000# AICC上使用Mindformers教程 AICC，人工智能计算中心，提供[ModelArts](https://support.huaweicloud.com/bestpractice-modelarts/modelarts_10_0080.html)服务和Ascend 910算力。 MindFormers套件当前具备AICC适配的特性，用户在ModelArts平台启动集群训练时，无需额外增加上云训练适配代码，套件将自动进行： 1. 从OBS拉取训练所需数据至训练镜像 2. 统一训练输出路径 3. 训练后将训练输出整体回传指定文件夹 4. 训练过程中，通过回调函数将训练输出回传至指定文件夹 下面将提供在AICC上完成GPT2模型的训练样例，用户可以按照此样例学习MindFormers套件在ModelArts训练作业上的使用方式 使用前需对AICC的[AI开发平台ModelArts](https://support.huaweicloud.com/modelarts/index.html)，[对象存储服务 OBS](https://support.huaweicloud.com/obs/index.html)和[容器镜像服务 SWR](https://support.huaweicloud.com/swr/index.html)有初步的了解 ## 准备工作 ### 模型准备 本案例使用MindFormers套件内的GPT2模型作为教程案例，请参照[GPT2](../model_cards/gpt2.md)进行模型代码和训练数据集的准备 - 模型代码 ```bash git clone https://gitee.com/mindspore/mindformers.git ``` - 数据集预处理 以Wikitext2数据集为例 - 数据集下载：[WikiText2数据集](https://gitee.com/link?target=https%3A%2F%2Fs3.amazonaws.com%2Fresearch.metamind.io%2Fwikitext%2Fwikitext-2-v1.zip) - 词表下载：[vocab.json](https://gitee.com/link?target=https%3A%2F%2Fhuggingface.co%2Fgpt2%2Fblob%2Fmain%2Fvocab.json)，[merges.txt](https://gitee.com/link?target=https%3A%2F%2Fhuggingface.co%2Fgpt2%2Fresolve%2Fmain%2Fmerges.txt) - 参考[ModelZoo](https://gitee.com/mindspore/models/tree/master/research/nlp/gpt2#language-modeling-%E8%AF%AD%E8%A8%80%E5%BB%BA%E6%A8%A1%E4%BB%BB%E5%8A%A1)，将数据处理成Mindrecord格式。注：训练数据处理时，长度应等于模型接收长度加一 ### 云上对象存储准备 准备好代码与数据集后，需将其上传至AICC的对象存储服务上 - 进入`存储`标题下的`对象存储服务`，将训练代码和数据等输入内容上传，并准备好用于回传训练输出的路径； ![obs_prepare](assets/obs_prepare.png) - 进入`对象存储服务`的任一文件夹，点击界面上方的复制按钮，即可获得当前文件夹的OBS路径映射 如：obs://huawei/xxx/mindformers ![obs_url_bt](assets/obs_url_bt.png) ### 镜像准备 ModelArts上的所提供的预置训练镜像，通常MindSpore的版本较为老旧，不满足MindFormers套件的运行需求，所以通常需要自定义镜像，安装合适版本的MindSpore包以运行套件 我们在[镜像仓库网 (hqcases.com)](http://mirrors.cn-central-221.ovaijisuan.com/)上发布了一些经过验证的**标准镜像版本**，可以通过几行简单的docker命令的形式，直接使用验证过的标准镜像拉起MindFormers套件的训练任务，而无需进行较为繁琐的自定义镜像并上传的步骤 - 在镜像仓库网上找到当前版本的[MindFormers镜像](http://mirrors.cn-central-221.ovaijisuan.com/detail/78.html)，打开显示如下（截图部分仅供参考，具体镜像版本以链接为主） ![aihub](assets/aihub.png) > 可以复制其docker pull命令拉取该镜像，在镜像仓库网上的镜像权限均为公开，无需登录可以直接拉取 > > 注：当前版本镜像为每日开发版，稳定版将在后续推送至镜像仓库网 - 镜像列表 ```text 1. swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39 ``` - 在一台准备好docker引擎的计算机上，root用户执行docker pull命令拉取该镜像 ```bash docker pull swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39 ``` - 进入`容器镜像服务`的控制台界面，找到`客户端上传`镜像按钮，将会提示如何上传上一步拉取的镜像 ![mirrors_upload](assets/mirrors_upload.png) 按照操作执行完后将会在该镜像列表中看到上传的镜像，并能够在ModelArts中选取 ## 拉起训练流程 ### 训练作业配置 - 进入`ModelArts控制台`的训练作业功能区，点击页面右上角的`创建作业`按钮，开始创建训练作业 ![train_entry](assets/train_entry.png) ![train_task1](assets/train_task1.png) - 名称/描述/新建实验名称：按实际填写即可 - 创建方式/启动方式： 我们需要使用之前准备的镜像环境，但拉取方式仍可复用预置命令 ![choose_mirror](assets/choose_mirror.png) 这里选择自定义，即可在镜像一栏选择之前准备并上传好的镜像 - 代码目录选择上传的代码路径 - 启动文件选择run_mindformers.py脚本 > 注意，在AICC上拉起训练任务时，即使需要多卡启动，也只需指定python脚本作为启动文件，作业启动引擎会自动完成ranktable与环境变量等内容的配置，并多进程拉起python脚本，不需要额外的分布式启动shell脚本 - 本地代码目录与工作目录保持默认即可 ![train_nodes](assets/train_nodes.png) - 按需求选择资源池，规格和计算节点数，配置作业日志回传路径 卡数=单节点规格数 \* 节点数 **下面将较为详细地解释不同情况下拉起训练时所需的每个入参及其含义** ### 从头训练 ![train_args](assets/train_args.png) - 训练输入： - `train_dataset_dir`：选择数据集在云上的数据存储文件夹位置，作为训练数据集的路径；该参数名应当能够被启动脚本识别 > 选择后，ModelArts将会在拉起训练作业时，将**云上的数据集拉取至指定的目录下**，然后将该入参的值从云上路径修改为镜像上被拉取的路径 - 训练输出： - 启用时，需要将文件输出在超参传入的路径下，训练结束后会进行回传； - 由于输出通常较大，默认的路径下磁盘空间不足，我们会将输出统一放在/cahce目录下，因此MindFormers**不启用**该项参数 - 必需超参： - `config`：填入对应config文件在训练服务器上的绝对路径，在本案例下为 `/home/ma-user/modelarts/user-job-dir/mindformers/configs/gpt2/run_gpt2.yaml`，亦或是通过环境变量`/${MA_MOUNT_PATH}/mindformers/configs/gpt2/run_gpt2.yaml` - `run_mode`：train，训练模式，将不加载权重进行训练 - `use_parallel`：True，在所选规格参数大于1卡时需要设置为True，单卡运行为False - `remote_save_url`：obs://xxx/xxx/outputs/gpt2_test，填入对象存储服务中实际准备的OBS路径，**训练输出将会回传至该路径下，需要显式写出obs://** > 其余可选超参可参照 `run_mindformers.py` 脚本入参进行选择 ### 全参微调 ![finetune_args](assets/finetune_resume.png) 以下列出需额外增加部分参数： - 训练输入： - `load_checkpoint`: 在finetune模式下必须指定，需要finetune的权重的obs路径(通过数据存储位置直接选择obs中位置) > 选择后，ModelArts将会在拉起训练作业时，将**云上的数据集拉取至指定的目录下**，然后将该入参的值从云上路径修改为镜像上被拉取的路径 - 必需超参： - `run_mode`：finetune，微调模式，必须加载权重（在训练输入中设置） - `resume_training`: 断点续训所需参数，正常微调需设为false，或者不设置 > 其余输入/输出/超参与全参微调一致 ### 低参微调 使用lora等算法进行低参微调时，与全参微调没有区别，需要把config指定为使用lora微调的config - 训练输入： - `load_checkpoint`: 低参微调时，该路径与全参微调一致，需指定finetune使用的原始权重 > 选择后，ModelArts将会在拉起训练作业时，将**云上的数据集拉取至指定的目录下**，然后将该入参的值从云上路径修改为镜像上被拉取的路径 - 必需超参： - `config`：填入对应config文件在镜像上的绝对路径，在本案例下为 `/home/ma-user/modelarts/user-job-dir/mindformers/configs/gpt2/run_gpt2_lora.yaml` > 其余输入/输出/超参与全参微调一致 ### [断点续训](../feature_cards/Resume_Training.md) 训练或微调，遇到训练中断的情况，如果保存了ckpt，则可以保存处继续训练。 如果`run_mode`为`train`的情况下训练中断，恢复训练步骤如下： 1. 重建训练 2. 其余设置不改变，增加超参`resume_training`设置为true 3. 增加训练输入`load_checkpoint`，通过数据存储位置直接选择obs中需要恢复的ckpt 4. 拉起训练即可加载权重，优化器状态，loss scale，step等继续训练 如果`run_mode`为`finetune`的情况下训练中断，恢复训练步骤如下： 1. 重建训练 2. 其余设置不改变，增加超参`resume_training`设置为true 3. 需要变更`load_checkpoint`，通过数据存储位置选择obs中需要恢复的ckpt 4. 拉起训练即可加载权重，优化器状态，loss scale，step等继续训练 finetune恢复训练示例如下 ![finetune_args](assets/finetune_resume.png) ### 权重保存相关修改 保存ckpt相关的配置需要修改`config`配置文件中`CheckpointMointor` `save_checkpoint_steps`:可修改保存ckpt频率; `prefix`:可修改保存ckpt名字前缀 ```yaml # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "gpt2" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor ``` ### 训练结果查看 完成后拉起训练作业，耐心等待输出作业日志 如果一切正常，将会输出如下日志： ![train_logs](assets/train_logs.png) **并且在 `remote_save_url` 参数所对应的OBS路径下有输出回传**： ![train_outputs](assets/train_outputs.png) ### 常见问题 - 启动作业后报**module not found**： 标准镜像可能缺少几个pip安装包，将相应的包名加入代码主目录下的requirements.txt文件下，拉起时会自动读取并安装相应的包 - 回传过于频繁导致影响训练 可以通过所训练模型的配置文件`callback`中`ObsMonitor` 新增参数`upload_sequence`：表示多少step回传一次output目录下的内容到obs中。 ```yaml # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor upload_sequence: 10000 eval_callbacks: - type: ObsMonitor ``` ### [权重离线切分转换](../feature_cards/Transform_Ckpt.md) 注意：`mindformers/tools/transform_ckpt.py`转换接口仅在`MindSpore2.0`环境下使用，可以在`NoteBook`中安装`MindSpore2.0 CPU`版本，进行离线转换； - step1 在`config`或者`训练作业`中配置超参`only_save_strategy=True`，拉起训练作业，仅生成分布式策略文件。 生成的分布式策略文件保存在`remote_save_url/strategy`目录下。(`remote_save_url`表示obs存储路径) 训练作业界面中增加`only_save_strategy`超参 ![](https://foruda.gitee.com/images/1687916968422798842/7975fa97_11500692.png) - step2 启动`NoteBook`任务，运行如下脚本完成权重切分转换（注意在`MindSpore 2.0`版本下） ```shell python mindformers/tools/transform_ckpt.py --src_ckpt_strategy SRC_CKPT_STRATEGY --dst_ckpt_strategy DST_CKPT_STRATEGY --src_ckpt_dir SRC_CKPT_DIR --dst_ckpt_dir DST_CKPT_DIR ``` 参数说明 `src_ckpt_strategy`：待转权重的分布式策略文件路径。 若为None,表示待转权重为完整权重; 若为切分策略文件,表示原始的权重对应的策略文件; 若为文件夹,表示需要合并文件夹内策略文件(仅在流水并行生成的策略文件时需要),合并后的策略文件保存在`SRC_CKPT_STRATEGY/merged_ckpt_strategy.ckpt`路径下; `dst_ckpt_strategy`：目标权重的分布式策略文件路径。即step1中生成的分布式策略文件路径。 若为None,表示将待转权重合并为完整权重; 若为切分策略文件,表示目标卡数对应的策略文件 若为文件夹,表示需要合并文件夹内策略文件(仅在流水并行生成的策略文件时需要),合并后的策略文件保存在`DST_CKPT_STRATEGY/merged_ckpt_strategy.ckpt`路径下; `src_ckpt_dir`: 待转权重路径，须按照`SRC_CKPT_DIR/rank_{i}/checkpoint_{i}.ckpt`存放，比如单一权重存放格式为`SRC_CKPT_DIR/rank_0/checkpoint_0.ckpt`。 `dst_ckpt_dir`：目标权重保存路径，为自定义空文件夹路径，转换后模型以`DST_CKPT_DIR/rank_{i}/xxx.ckpt`存放。 - step3 将切分转换好的权重传到`OBS` - step4 训练作业界面中增加`load_checkpoint`超参，输入转换好的checkpoint的文件夹obs路径。 ### [profiler](../feature_cards/Performance_Tuning.md#profiler数据采集) 训练作业界面中增加`profile`超参，设置为True，即可开启profile功能。此时profiler参数均为默认值。 采集完毕后，profiler文件数据会回传至`remote_save_url`路径下 ### [dump](../feature_cards/Precision_Optimization.md#dump使用方法) 参考高阶特性精度调优中dump使用方法，创建json文件，并上传至obs中。 json中dump数据输出路径指定为`mindformers/output/dump` 训练作业界面中增加`MINDSPORE_DUMP_CONFIG`环境变量，并设置为modelarts中json文件绝对路径。 训练作业结束后，dump数据会自动回传至obs中。 pengcheng.mind-test/docs/aicc_cards/assets/000077500000000000000000000000001471400134000212165ustar00rootroot00000000000000pengcheng.mind-test/docs/aicc_cards/assets/aihub.png000066400000000000000000002161031471400134000230170ustar00rootroot00000000000000PNG  IHDRP3 pHYs+tEXtSoftwareSnipaste] IDATx\Te?k+ iXuh$\((" bB*f+jD]fi?f; e QE$3g`a9gsϙ9ǹ:}r9tuzA, `A, `A, `A, ` ?O~F}ƍza=O.wu_s3N酸|YvaW,}$]LpwS""+圿yŭʵ̖Sn%WnXֆ\3սھtehG*WtبX!&l̛?DTrfiɅҊő~}H>Q,}OP?_< ixު (Z2tǢߴOe Zg;sFŚQ*ܗwN*hDdl`5Jjqf|>=lx_Tp0Neg/o;~@*jFU]g f"\Nr0'd3QHص$'E3rҙecK%"_ZU;Z/nl!Dlgv rn ZQq_ZQ9}%\6N@Zm(b˾ӒK7-bu9DA{<.t7Ϟ~'GDD<bHY,P>U%GĮerZ1e;)XjFUO`jYR~8e D](wk{](^tS]K'H~\Fwt'OC~OڅF%bUVW4[lH8UHhgu+)&tb"ufT+d9&rn֚Zvh?]/}?WQeUgq脥wrSVN>7: tT ]kٿrkUpufr5꥕Ѫ&'8e׭V9(:RҿUwwU^5finÛyF}_qg>Lyo˟<3vsuq7 Dܥ_{(S- v e6fY|9 9p&"JOw>tkUbX~;ugǯ(ܱw".nWle't:B"Ouu9;C0Z@GsvIltݭ2yO?FDTS= SQ.yKD;v_,Y;7twDFzW[ݪO\}Rk<],zb&B ײ-%R?y*6 , k=]֪/() uͻ\1%4مgMl&=(Y`KeMHDDyw!Itl٪ qRf*eivr9q5HTjk.r3mk[!"xo"6. }Ek3ʥDW*{*|T_qYW#Շil[Z<(-+Uǚ4rP푓^ǻ޶m ;7]9YI^ɀn]ּ?inz:zAU e%:]cg9ݩmxl)+x<׊Ji2iΛO1rv;KOs ?QGY__}߼)<S7'={pG'^!-^ej<ϫ[leyfz;T{|okbݽ}zɍѹ-9:H0[cL@%sdBY(/(ۉSݽvT)޳[مx~WLXQ^"9L1[EBUB(\%9[*UߦD'_&d焇~#`}ⷪlg{l|-L]^z?V/-V^)_rCeYZ;=1@4qxv% Νp0;DWd؉$8 W\s壯hE8DW{wzl#uLQ.J{Ǣu9s^av@yw[%v=WG4`K{m*ظ`ˣGCK`T #ގ`,Lђ+oމǦby$4OZi_b" pPIK;K|Kv^x:YRV{Q{ K)fiC1=}o1wl%7o%w1ܶk+.49+Φm;zdLpf""rpką!41w|AÓv") k}~iD"|UAr:u$S6m>PN ƛќF9yOSsb'uѦ1qCrghŜvTԏopV㻣0^Un3vD3W_BK"*;,vX`THFǓkQ`FIL-NB= L-5Nax3|;?ײoh4S//ot)!^5nr˺v6ܼMd.x:3hH{"Wh6iؚ1L7qxdWHӯ=1q _mu{+f|GhV0!@y8'Vgbi":Yq%evuO&h/ ɳ?UCg?C {G_R՚K뚍Ysj]bZZqjZk"-{-qrug_:?&tՔθ[/<޶Dlg̪+ '`tV'MXnf>@/4ֲKyێbl9g_ə349yL5/⯊ݫ2a~i:5&9o22cٝu2f\vx\\N_+w xmH`'viZUc76yVN Vmg:̄몗V۹04N>YE[s[g>lx]|cB_Ohٵ9+/֎jrg9a[9s4#uV>KCt ~ z<|&obяw}jٹI&V 9癊pYv4Vs/_znKQ[ܗyֽkȹR媋T.G{]VoGmVZ3ZsRMf_Zs_zi=]* ܚB@ ~ܘq]-^K[pYṶfFI^UCsҽa~iEE,=Es53WvP[m.&FV95qFfFؖx$+͎$~9o66)8ME+)Ig7tnql>[;{:Kݐq_fE Y^-n^ꥉm&uK+KĭVv\Վr7}k!1Wccq]݋n[Czr?UUQQ`ƩVRV.RK+ˇ9ΚluYN3=5VHw7wůX6$M/kݻH/k?{MuXl~x Ӧ6kؙFx4b<;5M.ƈߜl3wKn8+3Ϫ]*R˚g+&Кf>{Zr+$W[Xaܯ(G[4YTy):=lxyitRK7œ $ f;Ӓ  O6ɸ_߱CM;ؘwu_Kͻu= y< ?&"# ϵekS ʫ7>};w <c񒉟n0oBGv4U>|,/IЊ8 @=i|zN]ayͻuv]Gߨ S'i?tXt={&ox3ym<2Ǹ_>}0^1t,.x X0 X0 chٮ׻ cWw!/ `\.>$K4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK[\ IDAT4aK4aK4aK4aKƨQrfiɅҊ'O;aw=qc.,v1^xrCwPr]i\]^0ס6R"'{(ދb':IFZ IrjL (DT$QuvjBh@ `&3/pΆ⺁w{jl)?~-L˓wEy/-,_#G4'?zQFjS#=EFKB$&,׈T Pnvo~#uɌ ݕ}DNE&g ЙLG/ K[u(gDtAW5p橤Z'n !LEXWOu[\tUrhê3״w9$`-;ѻpzͬ~Z.(dg䙮UϢu 6Vf7.k(=jlf?؅e9Y9w'r\.r7kl+g3 )#=g3+gYM2r6&Yͬ&f/ q}~T{g3I+O8fGrG$({/c-y4W*tMP\ѓ6@ʃ^HyL"jnf eyTl3+g3#2#lzז4.tR4[m;Yqv.z_nONO?;@^ tu 9D>+sWG_4{+DuH,"ᴍO5"""# c&&O8HDu1=s?p!*O;H$#]atjMjSwY6֒ҏ ˬݗΧͮ!"Rp:"/zb|U"埮vw6+>>UAZk.fHCJ)bbuDՇ37F_uݾMs${{ ޳lݹo$K"fx̗R dir9$մ .=#SA+3Dү2۱\B, =S"迧^VΏP@E1LpKO5KvBʔSUtWr*FMFdOX9IV,}`{imzsS-Ԟ/>y凔(/j-Yʝ⪱Y}硓u*m- V¥jż̥[^'Zqn\?}>OkʼOB'Y9ۨN<&"IãvOA"lvGJgeR=WKW:vP4qx8٠dsxΨ\YمPfK-eǏꀗ -F6{NqHv*T+2E#&lJ_KaZ՞>-S{ cɮݳš>]!$bW""2}wĸ?gEk^o5d;gծl}k|mkduBS&D+dD"woe=XYK53-=}~;ȑ 3{wiZr:C#*=҉ߝ$ˉhukBtʝD9aNګwp0~QR#2Lfbh{o$"5*gHEd)vmN"ȏ3~u,О8\Z޸Yˆ]P|oJO`|)3͉rNp \|^j$ekL<$~ɟAvQFZFb1't5QEdO"7B\HRZZlS ?S^,,""Eb.11AJ"Z( 1UkuݾuATzbp^ tS=Y:MûS^{qFr]AjǪx,b63o{.†p\)lLu⿂ s[9sor(,69e4GҖlF]!]X~dW?\qo_f`Z!V=주 UA8t2bT!sF`+"}Myxhgn6jYDtlޞR?ϟ. "s4v @bWkvsjdej1'*8vVbpfD3:UZtY~s j1Կ8Ƚ$_. %MAmQϥkF0M-Xb"N\9/hg^Qk\7~~|cg 5廗D&e&c !fOiNTL2~Iff[S?xwDN5QeT9=ӏ&?꽭)$hw&8<έfīe&*g>r.?7RF.\3CO{;dDmpzCu"[iA7g+2Զ^&),4=!pUCM8xDDC֌Ү5F%y%n [صD}_eǏm&ϸ閭?]Y e+^(""w獩ho}X;*^JLzZ֖]/& Y=%"^Lzz̰Q_>{/7d(=!}{ym"/=#o؍HH;I]ގesJ*"~mB&?:|YTcI+눘ٖI޵l1'&ƦDuDK7[<¹諾6''YPh+3ɩD3-HGR0b{{ԾvQNDDD>޶Μ$'-'8 >ɛGDDwwGFMΉXe:> bKMzoT_YX]i%޸f͙7ؚ!|ED4EӪ\ZCD"f6JDwˀQ4Z+:rHW!#eZÏs2"j"r?̭o/CfsNʧlFiI̳ "2`?2 ].|UkjPQmcC3i[.o$D8y caК}Ak$;##!TvFueaNȁ?l7`l=a}x)]~FE=Y;o\"+K5,0hAD;2"p\Jد$!}[ Kq$./VPITi_lD4jTg$r-5ojeeD4jf\ܢ0Uk7jO8}L3%NS׉|Ttγc?eh .ɑj o&NZc*Ɔz""S~-^qQQvB1].ҨIˉFBD套5y?DDNY[mK/yW/kIb_FmF大֑dUB;{sqco5R#5Y,+ BB"j%VxCJDvB6_ӥ >Y:UۏsfƳ MS*[uz} C'. rxT%K+ ǫ_A.ge6}icetD~]M҉ԖkBD9k~;ND5B)%&i~.zE|[(`>k3[FCE\OxnК̈Wg>M܉mK$9~"]-XK!"wwe65?,MJ}IYe3hcoX}F}GD>9 HV/j3ӉgxBTvFɉD+Y(1Y}(F,ify|"rs>iLiik'%4%8?71/\s#J>z/(1"" GEoVh6w椐%P}[Y9?$5I'H9FH W/>݇'K8 !"2jsg5W5ra.#ĵSEXO"ѶDmѾ+o(jro$/`_rq왓Nkdʩ-7Rv r>vlfFFRy!2MD6^N~Ԣ(=FwerǒEKvY-qi sM LtWd}o6==˧cFܬadb"sWkEch`M}$j޺-W0OJP:{•DB߹AµQӤDL|O5bxn+(=fUT{L/=dOu?\u⽨9g}N-f7c?lM)>l ]@2]E^6($lQ%[+7s Tzhv\3i%CK D]$d:Nj?oKGDW"E%ߓ}6c&-*2 +svYy((|GO(xbN? j*>uh;o1;a[[/\mt#,M,""Jrg{eOf1ً5R6ڞCC=&˽z_9SE;+TV\]@w>TLrstcTqlZx_m18&. dQM|Z9fV@Dvwdo{4>_A;琍UVWRm\{1{4aw\5f/ثcy, _hkrVaf\tlTz8.*Q`y?򞐈lmsIgXZP5DdU!ZẋK#3(7TI>&"6 rtla Z7n[}4hIBE#[, IDATpް@+F%"m{ .aDN/cuZM $_&49Í7[,7=q[NnS13)_ l{E<̌ NJ}1Ay#c&OK NޥzĎjV"פ3|G_,${Ή[նO7b>{$:+qa.9buBnx}׵<#}]}},LR(4Xx͌;2~^kz14ϊӪj^E1gO7)Yx͌K>[ZTg_l^ӱ,m1Aދ>s>FϾJhzX&J"w׈}RֻhSG۠0 |KgT{NI'rS53# jɛ¼M$WIMzjWsSs;.H8;cI coVmz$L챻m>)OhX-);JW?a I ix}G_mpӝpt}낄":IDz4|˦wpqcj$] ;Zz{R=(G1"ܝN/ N kKQk&7-l>[Z:3XgQ>ViwfY  mz2QZz{G <;_НNO{S9A, `Xڸ_߇ O:z/=Æ'vu/@/K_{R^^:<^4YTy):=lxyitR &5NK.V4wA, `A, `Ћ]Kc&q߱ʑ^m&L r$"-gkU&n6tu7:7|c,6꾴^EOgFFh>3M0ds/gEd^}f"fع׼g e/#쏠vZޝh D'$)؎FEJM䥙Uӷ$VH#åĢ+|?'Y%Id;N)ЁjgLګ7{D4?򛬽bis_!g}ZSY RcnaFʄ6}E/_*Kz 2l,e++ڿ 4<ێ˼=JfMP\s+Ϧcc˛F%/zFWfrs܊$6'T_UI4D=`&"z ٿ۾ݡ-twzf+qJ/Gopy~dzϷ\﬏qeG-z*{RMְKwL\SBTTW@mh\%aX h@+?P+y15gU28&+|ʞ#pl:-{gZ'=;݃@DS(pޯO<~RD?ZY>?7{Z]3:wl}"TY/u4m |Zֿ=>{:CKDD?)_Ȱ_g"Nw/S|Ҳ(mfu.1/3?ߺUPa}?j o/EqN2hnX[yx6Up?,}Ͽ/G=xс/bލxhdi|, ާ=[/|w=3GկW5)=xÏ=/tcg~g2~O>AŞ?/9;oR|{'.W_|FP?q?y8O{տpt}>C.Rxf|g?'F3{]~}?*"zD䩧裏~7[W}4uo}di|<|OgC>w+WogvOܞ_6>wk1ȗiYM3P{[?2'4[kOPvn_ e?f~K~ %ʳ_bnпڷ?O|G SS}/wnolO}"2~ԑO|_秞[] <{~&򫍊_WsGW_6=?hB>15%7j+2=:~}"y#r_ko~sziݡÿKIs`onŅ?낄q̧?̿w䏵s//y7KsC]DCǼ$|BTß '϶tiޥz}=#?џտ sx|8KeGW/4jK?p8ągD&-g~)v'jO{Ȑ+g~ !V":㵥o|?_>zcG><؏|bP~nkoGDɟ|=Aho7{]'Χ@?vÿo~|mNkٝ"vlW~^_7?~?pW;3Y,݉8 Oǐ?~gzTcدC 4ᐥ,S{]' ;? g` Kc Onx)RO"wfO_p 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KcP/G^*EFzh8eWdždžv]ʫcC}<;<6}M/4vFE/<ўȾIGο}"b׻g~ݨ^5;&)/N/>{sCrfFrcdi7Gο1YQϺXYS\zѱEDnNWZf==vc] EIgm:יE;oůfoӇ#Zk槻>k_z`qrli.\:|s'ە `ϐ1n_;".1{$%-9U"cAS)-r}ɷE4ߐ1} G' yU2:vRr5/>83Q8{Y췼r5pPͼGyƝrr7<^H:F[F|pukvD^r9[Dk{[}; ${37VZ|5M;{sj1Rw6K Y_OQCNWWЧEc-N9vM_9«? O,"jMi`7{ܿjxn(rlTnjLۈb٣[.v~♷M_;ډcAnqz82;w40XTߩs#)R(o2XBEOب1͊vs3)U@^)`tm;Hzg嘈߿*C}Gi!Ȅ8#MQ_V˸v+ա;Y@YIQθUЮ7+gޙ:-i}ldWVM\Dku=M540ᐥ, @8di!KYpC 4ᐥ, @8di!K;Ѣ%[̌+m=iZdh#BƀKQM[/(Ĵ`ʹlbt˲帲˫ىGMӺ_0,A^9cN6-ޥd9 <-U阷M3rVVdEekmN^#bbhsʹZ6j- (ꥱoYn۲הڅ-~!YjWAj6U Uw7*o7J1-S*vZŪg-rrTDDF mۯũjnTݬUd*Q'jimd")YΤnHtc]3FA~Q,֍q]DקR3ED6juC _*ֆZ1n܈_OEoP15wm/6떗ȨaNMr%Mivل ėh=: G w㶋e:(;KSk^MZ1,VUˬSqí7Sz-[F}5#OnEtczꏲv)$έ[ ;|I]QTg߶Ȥz=yVܥϗUTEYQV 7bnWkb*k*loE Y,"vXiDٍRLTr]t>o$c(۶7*[(e+[v{-=0OjF9--[`z*=1"JED6JyeJF&[xFbK^۔)mEI("eR4R QDiKcǍbw#gDrr[\ۥu_q ۮJ3P)M/YVv"ڲ!Fzy[Dd{T\mj~ ۮF}DeR7nd ا&uC*k䬌O)mY.QeF4MDDrM}LsU""vj$,." 5is""ƂL|E&٘7ڬQ_hsSZ/' M{/(O=x`< fF3MwF)dإhfS[P_ rug](]Ẻ.Yo;G>j6W9KcKQM1L~x| , @8di!KYpC 4ᐥ, bfXiC̴+Aj+m2e}!ẺrfYkAR݌Ζ.Jokf,XYV]o^{;7ײz۠#l52QGFg8U(zJ`"|cw{xe3k>mm[Qn57ՉCkTWjq㮻.JBMs5hs*խ щ(dlDc PJ9̌ru۵j}tbܟ4M]VYw]3+lNTkU!g!sSb,;ZYVfF.:"SqQo?f.ĂxNIZQ[nfmG≩zd66GL$G+E6O4H|)^;n-eϖ"fq2}ܐDܶ]WYlJ#W6J1M+*園l{_gy x-Ӽmdj7Jm7viG, 9Q|Fmzi¡^pC 4ᐥ, @8di!KYpX1ED.Ĵv҆n3Әfui_hVdZfMB,hv)iY X] vrx,d*g3[]3+K.Lkՠ'FeRZ5Wu7EO#ƗsXnY#~B,bN{RD.""lyi&`1v!YtgI^_?1B@j$ҔVQgYYMi-1XMUTZ1.2Uږ8,/y˚/vXJX#pG[Lk+;ٹ8"bnumb'ʂ4)"fFdtlZ&(V|Dd-x1yˊb'6n[Rk-ZURzJF+ uRt[Et{6TJ+3ΪR4 6,6W.1Cg}(E#uq(T'ja3󪁙RoB,#bQ0V&)zbVd2-[F'ƝIݐֆU@di >/v`F#j#geuK*"v!Wr֤HPE"4^RNٳ수i-e'q;IKьzV4rlbY(9[T"j[-% nQ$VpՉvml]WkqeDdͬ#H`01Z%J[Ζ-MS˲Q'?ꨈ5S5z>k2k5 q{J[?[#$g-ǥKIkz_3E"bLb-zR=R=QLuJE-0Tdֲ^+%ڄ9c1:ZPL弅+דXtШXaT\U>OTOhiex``Q/6rVV_~ƸHȉض$g嵌_ 9hkV1 Z/[RɎ'>uڒ6ېi4׍UQDZUlkmN^_GKK1-~.{˖J{{?/ύ~׺Qlv!YTrZFb ?E@jqjhݐl5s?1$b[l`TK[씶uvũʚNkE4iN;`j[TgL\WO9kv$t]b' dGմ7jwsf/HӮuu֫榴TɌx1|Cd9V} @m=k.EXjr@M}EDu9زb",(fZzmעޘ{*gYR4ݷr@M<EQ#%+2Gi :97m5z,[eYzT3>ݘִN2d9E5-Z[׵Ҧll٢h4cިUו#"#]Y5̖WD/E*ϪN?poCo2ZG9M&T$cE՚-mn20{e.Ĵhi7Jm2Fkgp> Kc? 1M"J|IiZx]}-'%Prjf׍RL4-#[x-+[8wy,e6Ѵ(کmCQ9c56'^]]0Ӵ$1vP^Oʢ;:[Ĝַ,"9xnNnxefdMӴ@gi nqh_Ϸ.:Ԯ @[[^[>|Koz}ymEL:8?HۢV^;+X0}mY%v8^B=NPd!ux - ?L7 IfOu1Ϋɪ;m1Vn_?p5V7F]ꋽ_y׮d#Kc M,"+LJ+ͧ[~{E)k[>Z}+cWNwn#C~uۖ3oܹߜz_vcuR^y+"r㙶H^~]= Kc\r~dBҎ9vcur]7 >n<٨>wr#=t:Kkn^tW_gxW.ԏL+}-u#m9s5Ai`o4o-RqlNNKp]%Wd.S޹LD̳'v{}/-銜A໑:]3W噕Wǚyxelm5{sCɷD})?9$Nru3cg]9|_My.l왧 m{lmW\BKRs-h?l diO.,*9˲,ZP'/n}?R)3Kr\##%k)>檡O3l_skfXبUebCQ9˲ KH~͌'lbsvz.]+$VQg˖e-Ճo;Wq/gt9fZKɲeYUSnδKH-DR4eϖ^1]ʳحѼ56G̴Z5of84WkhNoۻL.7i-SpFhҚwz-LkU(;e|W[*dB,VeJiKo]iR=!⿎޺RcE6W|D˘n9K]߮Ʈuu6&Drb.<S}doإh05%fQ/P d2ǴUpčRL˘ ICվfVt]-'VhEhR!ifH~]*s4h}u9YP>;V8|hkD.^߻-?T (||(y'X?}+n,#Ѷ؞;-W c5MKSXקY8Z>ĺ|)4MEZb[NWDZ{W uEmν4ٵj}4p/YR3A4U15 e&aWk55wt}JZ1Ƚm y2?s6Jź*^"[ެU[2>:lfc-OVQz jzl,ze]rqmKK7DWUxy.FǢo?xӘQSE^Njo|6(|jKgn7ѧ&1Ywýۼmh62ﵖ0h3pxeUcW~r:m{'կ5ֻy%tu=FV#wǴsկ;nk,r_F[jc63O+j.H9qNFy|7O~WSb] whZs?}|X;אַ˻mwٱ[anJ8Z>Ă|N_n9jOn8-/r c9]V?Ϩ>x Q>u @om%TsJT0`^iͬ4-ijU~P&eQ_,bJ_5f.RFmfJՑxvV%n%z<]}HJ|iDSj{V 1kMU'Zpu5Iݐfgt˥qkż4v9c]n^Gė{q] &:L{6r/YT{uض>o5g[jj;u#7_Ohd|bTCUx[zQVol8%F2bo㜟YF%w{yւ[s@56J,67P7ו[oy/=P8-_I `M[zt|˱.ﶢIݐF˦ZuϮ."jt T#x)'XRzKQ*f`K^+m-^Ojuow Y5MTq~liiE^Sk^>{]`'-[F'퍯ۮh*7E OLySui-ٶOP2r@q~w(2,{}S3eu}iH*zǏ㮛}ۜ^hy45|=G'*U!YM6r]Z#Yz9F)v"vdPfb*E{wR1O9m(lԪ2 ,QSQT飃/ʶ벚V:U#z*g{o52g[vڶH98*mll٢PЎ 1__SaNn"}ܐ9ӜZu.eEmK?}~Os5hw^S//P"]TNvZoi;re+e+*/ǻ?a,+W,l?]G|*խ 'M[DزG'Lf-+v+XϗшorJYsFM-o~3}|K=*WF)v"De|B]4y}s]GS)s-2%+ݖ &t ٲ5؅X֧%Xz yQu5o\U'+"=c,XKbѳu9uqX#g-bfYaosi֚m\Yf)mUҌb*x`}|L8ChR7$ee#xXr{oo5#7 ϗKЗymH2F?^o=߀Xmۗ-EM^o6HLS53;\]3|Oxo5ugե( 6HE֛͓Emi4~)GS'fʜ3J9{ز;3_,kijkO IDAT|[wh(EQi|8̆8sm]\ĝX3+r1`fYUZdQu mzo]F3۹?ѠJ|ٲ\`Ơ֡N_hخԖr=;ɬ3mݬ!- [dovrl4Z ImfTV+ryiж۷-UD=M-[TEݱ֜{-[.N۹r)Xc<"wkD^Y8v) oϿO)]hPogFB>Ywq]JoA:mԪ+di DF)d*3 c=i6<Vq[3&)9D*%oDFeT\z XYx]J=Xq.*b{InɽmcɄDO4~+esۛ#mv.g򋕖/kKq[ںfV'{ ŗmcd^k׫nûӧIۯcA-]s[\3>llᛯ(g؅ֵTV?݊tXGzw!q_ObU5Z>z8=˥ۗcZOIe1_3=UnLGQu-sJǍ d^gN31\EH+7Gfi]g0hėsZD[#9Ow=k2t_ly64g&c]-d;[EN}l4h(Gew삡yrjjdN1lkÓ! 웶Mlrb4mܦ蘺]ԍTD&/L榴!q4Y^0"sFfq.RO:0nG.E#Uݙ"6^oֳ)3٤V-FMs^qBD%]rrɻ݊nǨ,Q{Z }۷H|ia^Yaf^uef)gkc&j,; Xζ_.]׊qCV+6,"uz3' WNNƂloly.k-d4߷ykۙ8Fw!EdOſOZ~W4?dyg0z^h ."vb >3dǹ(NT'fڥhF^ߕI}6CQ-6΀j:_km.7ڞ]Fo*E!7[4sטi-p ]h;Ic |Rjph @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di .Kcӗ'V^;{kͳyꗣCKw{Fzh8+otdi 滷EUgWN7W8zYN셺H=stlh.X#Kc`mnjan ç+mٸxfENMLgn~ޝߵS"vѻ.>GƠZYtKEyygDlgD\sޝͷQ]ľ9:?xF[joJv<ў;rWg^jwHr:j W';&+=vuѱ{$mJb{[n>>tADy;SUۋ `1VoH7C+ރ 'Gο"b[s3 ο;[kt뗣/T.5'+"Ƶ{3"r낓 娓cWDwWxD" >>rzBs"/Hݲr5p͎c/;q]ݛ>.p~z3.ͨ"J̳7qZ]iD}9Ә ,AΝWDV^;)￡UMi/W/}u"'vl~~Ƶ{; `/Eӻnt湋מ;wol>w'~ ;=ɞew(sjձW͐ޗ#_ 5,A}["oN|wvn&x/~mmivʄmuM?~Jw2 62YhA_m(/i7C+g޹/KnX-7%B-N̓! IE4ڽa|-߿wcܪ79|3WnnqzxlS]9=6tt+ 3di 闊cO5߽8ke}(rojtbkmfܿ7?}8=|N'ܮ{W?s^ri7tErWݹJy;+v")HrthܕݹO4[熎eBWEDK_:v!RR,Z,z fF:ʹQi-;?x=ښY(6ju#1ifKދ]hsu44مXĜ(/ź`4jgYWi)j#Z`e'[KfP̴Vc[ll٣JD[l_SN#H}N8LyQz9+-d Kc ;x4MmΉum5pamQy,eZ[[hHkK1eTLXh$֒ʹ6~VdtQ&473*S+Z4#ge;{YKցLl+" /!9if=[+ ؉|ˎqF)vbs'[<H̒5#"^`Ué[VV2ZM2ZQ*oFRDlzD#+ ʊͭku#j\.d°>_Nڑb!(VQґU╹LP=kYY_܊⎪r(V['uEs.4})UxTDdYq-")S9k=obZ̾n E@K9ֶ-VSlbF)v"_5y})>"v!DgVkTB]$U3AuU϶H%lŷVd-Zm-^.2Wmm#1gL7S{SF픦Ljūԍ׊u1D/޵V:X d}qA$,޹q2B&ƺH|:錟hƂ."z<(zGj{s]ͱyTmj I;#"DZ|٥I1Ӛ9ne'Eƥ^'bfZo6{w}ѳ4ۀfiV;V&w`(88kQ _502A/ cl*9^ClPKOϥEnD*W$r.(AAA>ЖF>,{ߞmCv osKܺdf[kI667HAĚK3/~  @[EkMO [fǴђȌv vm6GPi*ot'hN ioqZ{߸s!L=F TF_Dʺ.iIuoECwl wFK]V:ˣaۋAAmiC'cxSxW쿕D{ߘyT5QK;6\{'t 葝o.hxA~-;zJ}%O6U(hzMa֥šϑ )լni@^ );AAAȇ=F =&h$*mfdޱx?P:^i ;]'Mi@iE|~5KWM9S=Nx68vhy .Y/Qq'=}+`2싖>C  ZA)w}{ ٔRJZ%Aȅ#Z-Vc:<_L|ӘQ{9$>+xjFw$ϻ

yȋN<AAA8e@)i .AA[@[AAAAAAA:ЖFAAA@[AAAAmiAAAAAA:ЖFAAA@[AAAAmiAAAAAA:ЖFAAA@['N۟/O")17~J+657}c ee*s;),jdv.iA+{ߨ?ƼJ+Ӭ@VeEuIMjS.Cuv*|^}n43=e>$߃WTA{|9 %hK#3{\Zb|7\{}^ p0ĵ/O^(hi5 ى1ճK4zd]g 7hm$?;&L(Zut%Usa.TItI"16ZFEhv B)tYG!m;w <b>GQ-G4]y*፼y:F1_*yqnWeɑ%$Xȟ/zkS6}ƙF2{r{8('tG]m+qkm;Pq&m]>OeJFz澻A4ZD7qJs/ kk62u07ŽF&nSea6*2FPjDߠ[9姉hy"Aߘ/D )Zt=}Fym@UB5Eds(@oz*2}%p㟖-0'*s?#t,-~~ mi"FJݡ$(Mۓmr*hVȬ1 r,7y7$ϣ<sÆ4yp#Ƌ %|n'9ˈM_E 92t k5#%a)KS^e*35uܘ!|eZleJԛ'KmNMciR174UQ(shyK+dۦ8 HJìT%!jW~p@i+Mer4 {8ٔuC1]i`ޥMLH +uJ]+LzwK4O(oi'Yz\ W^22A*s_C\wU|OG>vw\S '%AÖM7q1NHm~O}Б7éQj@6nGĝ͉-l.T<}4u=pmZI`ÑeI-kzO -O2ID^Rom:/ ֲReWS\9엛W2& \-`—v :ٴj[EB6k>v6#+mqS_ǜJ7@e܈C#$s@R6k$˜P~DDNUY5VZ("Ӹ A>&ou3i&ߋ|n Lfx?o>i}l>&7O^Ef?Aky}|nZMσV%r)3STQIpo7 h&1 -`;)Q՚lR=$sr('gVUQ 5 ||NCoD9 ϼާSҊ>O&)QWH*ϛT<&~vL÷ b}eieVECB˾ɬ uq퐵FTzoЪ0ܕ*FTݐiIm̭&e-l<,ȥ+2TrQW@jͫnO8 nPxKf$n!SK ?^+WS韫R7o>߆iD3Nܤ;geϗ볲V^XQeŞVR,9|{bJ)0Sݡ80^{#!V\;'7AH`_eitU E0cID7,/c=P>:c޷g`}u$]y@p:.I\d5ѵl;J)Mw6!pmZCY#]'ѡ@9ƺδu}6QZX>`%ŽX_vi/Fk#4uX &펖&BrTezxP_tN/;Twb)IWPk4a)qC;WVg~ ]$wY:I3~}SdN4KW(>8l?x'YX IrLMrtEȷh.i7O wf=wg5w.=,'/{+/Y98bwh5=bW- IDAT8t5@7G{A:2HfůzP %N ͤʹl͑=*clb=92 ei!]CX؇κ H]S韮&9?i*u>߆.A(fp"P/}VSH a{[g0ʹ*m(TdȲ`:l",]0'NE[Y󠙌- sXthwNڬX\GBS(4nHR;A[A`mWbRn!}jv< ŝn6S]FJoܜYKǝM-@',S4+.inEQEOq Ղ UUD|.( vKj<^ƒoQ5{Mj[ٲ^&N3MMvF{,}yx+I%TvJ6t~$S[l~0r|NEmSWh%{;jsc{x`5)I̍؊NnU3IKGmJaq˹'b\Xy;,\a[\"e>bmӰFP .? ; %Md"ʆ(! 6ciTg8l u%:umSe$$0زpӏ4ea>N9SFj& V ;rv%ARzYZ_Zȍ};jj6iEmRC1|Wk;ѣP:RݟVU{mӐE m%@\Lүn!e2;=]It(rP_+/Hʹяjy#}Wuɚ"@688#m+{@>Tr>RnB8oE~}iI).Gw}$jY|e5`84;9q[s' |xk8K cˀA)^ el6Q3vc5Iwеhٱ[ǡSl8?q_4r/H<^xzjE;qb]Y;8B)gQ|t늦Z_ z8@VD& @)kt6wѷ̯x8ᮓ Obmn4V= %rnES"@W0B Pxy=? ڭqew&yoeGIcNrLw NnWh3r #pz+G դc?8GxCn37/I%˃m"md+W/ɵa(Cjb&p]'r%>%zNH!rHOꔋޱx?=5j93O0X~]:X~;9-c|yM;Ѧ?t?vœ˸ՃW\Ǯ|<:9K^,Ȝ'>~{a/Ui7בδGS9>PCkmHJ17ًQVs$W@y%L#{Ϝjg,IQi-F ;/R||z|LHSC"Xftu  $ iDnl(rd)mdI'%b>O49i- _/~4'y葃I Wks'ѧOzp-A';zXM -e2T|p!4,+L&?MciSiT(e:"c= 5etд#m| VSXC q6ZL$4]Hk\9 K \VCjkTK#Ժxt1&^[1I0*4ї4ea >@>qsj,QydG`!ix$x_5LI^МrбPލ b8CV,ˎ\)jw@ T| җoϸ5C? Cޟ4Hj,8{A#wF=enIt s ŶQ] y*G1ћ.8T\'tҬoZ-EpI? K#   u>   rhK#   u-    ׁ4   \   rhK#   u-    ׁ4   \   rhK#4 MO3/`ms{8!{]'Og+h'k4ҝr]}Rcʥ 郼KFnTUL&X/I{v{߸sA \Ѹ OXmw@ijL|sØ=!>oMۮu:qVGE+2:2JP6 㫉Kuty.I#qȹuv17ӽԻMuoGP_wկ]up)rwzYuom:oTx핲i*r#*˿+{G'4zWr ЖF>(ܦc1ZFD/<̀N}c? \"aBݹmks,"~l8X)THzϮE}PvkӂH>}17;M 5C$ZE_Q]q>1f&Fj4П"Ƴ$ggSTL6}t=sG]9Qff[9_S~nFџ3SȿWωW$ff*E_n9=/޾v*oϑ郰LPhK#5@*Ƴ/pofUW+(Yr*RjrL2eu~(l늍-ڔߎ;ք ͭfٔ>Ĭ0L% l?9tI-(-4̔pٺty7g)fЧGJۺe4ɫ?R-{炵fgoXZ3gņǦ!< 42̭T]-'s]ѫ\zu_l!-ikǮ":~AE.>Xbns'䝹9[ F嵸x=XؠZ$Z6nGĝ$3ZQG,2sd$|b\܋¸**^=$ak>.ڦicwaB_2\:* :rso:RcP; "PL;|7~1M:_MTnN/PyKP$N7h&ߓV|I5~~7SS+%WVky}}}LZ$edgc S+d7[LxLRi0K3i1>䟇ok7BԺ.*g3iM6ؚ|k5֧SA'q$LkdKO ZM|U>=*3hyTI[6|xRZp>%UCfD&+VQYjW Jg~ƊaA\XQm>++QJf&բܐuH2vQ$I(#p|O1d?+QTzoq@lx=o e|=ȯSo^{jy)/ͤUzF֑lxKAӤ$٩m6.>+m.o^sY(oÊ;B,TקB[U._J{G^T7QeX06|AFN6֥S3^TSeo{!7fnʙm+0nMk(g4ch#':yA0r_>`-pܯδxnLzC{LOr+-I^tRd?*ގ(T;rRy(Xڍ4]=61(#MKGQT7}KQM{?Sښ,G(ZZjUq׉DQb#,+ؔsߐWl-I0\ZMПb2)_qiS1qsڱx Makomhw[N3 KthZKyҚʐKmIz]ڄQyzUr 0ӍVTOX_r/) &bnW4N_~?%m\?4ͮK 6Mv>vG[q ;]큇A6XEZ{N+YLIfcVjBhq/Z3~^Bū|ݟh,rXM93փ %!mq@Dڝ3UV9In m&%zfY4.-!oy(`M]Mxպ}z8-3geoej/xOY,Nx (4BN YLA\P4Xa{;X/_kŮJLk*/PY $MTy{zrJ+qww6aC}{4 7N8lGNm G8`wx\5o"oX;|&;%#G횯놈;`xq&|֢dZ6Шf_:gN@ǖDG R7JQ@z8ԣud#_gPFcj먪ٗ(̲I!ar 0> >?C>+U-f@iZl?E4݃h:Nxpq}V-[ 9;iw&ܛJ(7 NQD M:܊/G Nƍ SC}3U*^omiC|(1v+wVE&EKW>LY]LlWiq{pQVⰅmAex\//10 vb'mȭO#V~/Ӟ w26>!Z3;ƒxRI/Toj.q.4Lgdxk`5;7Ba6kǟ5[*ijrmqmzco>d›Ai .*j0-`xKK$ ktf{O-}>oĊ֟EI{!A|B476PA]z._OհWϻ֘rlYX3Mkdw;m0ӻ%GeǬC9D!,-Lvؒ6a78 k)结r CLoh*?KiPRu>CѼ%vpqj*̢C=jrk-|XqBas!'P3brgd-6jbu"{! 6/~$'@o+tH)[cÃ9M~2vX6dk${5A:%r/ٟ) ֥Qh~1[oO@y%}š]UrgT(zk_:1;P^Kl[Bb >~pa\tOAo:=ookr(OyesOvp ʃ8 j#w"32j ,i-AIdQNes=1b ]i JÖ=F @kD!t=yKE;qp0T'&Oa(@CCAڋ2ÅHR,_QRBR{|%R 8oV}9_ӹD#FN5.hLeu5 X66G?At f6wR\3 }fI͸ODţx\7rCJ]Pdi.}Rf|ӹ |%]?0 ly N2?1Ȁ~==TS~`4AE IDAT '-v.JEVdV`ѵ &O bÑI.DǩAl8҂ܦ^_,>?8bˋ7_ΒmӀ=! U֨?=D2=#IK;.ڡa̅TĆ#%!L=YJ *2; Obhe 1ZREhgTF2pCEQ|O?x@&[YM8H̼!Ϧ0Ѣ}l{yirH4- O"8:BV k5"yxS:ZhSJ靻\5]{Ng]DJ{߸2?!B^O|Sy /P>tTm ;w+ uRɭr]Y@n;#5 ݡjKO+Djst/ uUW82T}oe+!AVuu\PGY?Oņܐ ո!;v<ڐfXν+dge%  7֟&ȿ$ v3PGıpq]@6{+.h;]坡G'>h2'D f.9}:Nuk81^UUS4!s/\$.apy;JiY=Ndx1e$ə̚jjO:s<}LzY_] ii}֠/sG;H@ёrqR.G|Qa{[CMܸ!ú<߳Dl0ئQ7]$TEt#W+#-,jħSh" j'4;FĨY/PA.oؐ}#U9Qv S̉tLO-Yy}J37V$܎Y"$ niܼ u_͝v#v擎x;;11eF TƥXFtDjB姲z1 MkOP%h\f@4CT^=+r_,!-].!(_4Sn\X*>ЛFc4 s<|u7؄_ச:ngH#szXHmPē\uV*/fQ1 0:MdoJk!83\\R4#H@'kmȰs/qV}uU'HjFdj8="mmt[Nr ygF\}ls윤N exRvOZ겊e 81c'' %s;TTMܦ3T*N<_sTUeI(BRCR|E$R!ci ԦrjKro4OZ'+V{[r͕dmS lGdR揆VV?SG!8!/?^__9 1$~J ^%|Xq^D[7% 4sY!4F A dq)o!ސߏڏr/D]uF%y#hK#   u>   rhK#   u-    ׁ4   \   rhK#   u-    ׁ4   \   rhK#pR{@AAЖF>.ܦ ]Fr٘:=w]y?k'ԐQ7b%\۽o ykI=]ƛbsXR@AA~hK#nӱ-(%Ʊy;Ɯ'>$wȧ6siAAW ZCsyZPI'}&Yޏͪ<'d\WSLd՛__x互u6r'(7V+M0Q];-!!u9}6h?A5Y,2XD 2 Nx$EIzIL0YSz|+0xRUՔr %qAA?Ͷi%,Szj$[<.\3 p~|;Iy"z6 poDȮ3UuIi G䤎\?Ymvڍ sߦ{jsg IQhE9Z)6^,|ejj'ۄ!k) %7 '!m`+@׉+^9oWZR4b L]'Lq̗;Y1nitb Q{Enw@UaWAA74Avq1M=H^S6em};sZjq- :Zh_x(-?8Jia:6V=h+ׅ|uBT,/rWpq~2[fQXo~ISI]HinYhGQlauKGAA~@ ^>in43t$]'z)񭉱-Fw&<4}7RSt ;5eߍ7Pm?r ڧ.,/Jz݌{g92wIr\Ajs_я5O#"75zIe~6wΛ9I4ݢwS҇R?ޱx_*A+/[!i-k؉<=PUr^9/qԺ}Xc~t932aܧea>.A|>6 KkqrzaS (@71AAZ6B_6|MLqwVqEbAA~9 E2nkK&  KAo9}wO#4Lj:^iS/D8  /}AAA:AAAAmiAAAAAA:ЖFAAA@[AAAAmiAAAAAA:ЖFAAA@[AAAAmiP:W?i~0YOh|,5Nf'ŧ+Z۵ j$Bm(qjZ"U|[IT17+EM*Qٴq;L0}m:2~U ,KY jkp!3}b&Vk\ū+K oߓȿ țrqOU}{9+MG'=7b kO Cԅ_$:^k4r9tk{>d5џm"+B 1 74el>36J뼐} Qh}MSS7p .-xZjMӠ)r@")V0"p=8ˣai'%,M@ 5=M[y]Ro:/ mPС?O</qMkl4c>&+IvY[-I^0AUDZ17N)d.btEt#ܦchF?j>?nHm:{}nP{z kuե]>% _jyM$/nzYDM-|P?d=3nӾ}v4uea1w >@6r C}TIM 僻.eZ4.4)|wS,dVަ?E0g!so :2-J4aE7l)'{Ǥ2ݖloiyD}j#Pt7_L)u O].y H6 UJ1)Y؟Q}8 m:_5L R񽕔%\Lʓ5 oB I&q<`$8-CE_sVL=E0y$rF>&p,SA*~8ݡZ8϶ꟹo轟8M]y3̜'d%W!_W1Mm_^/{Nxx%sIFT&vÓr\mMBz/ ܠt OQ}wKbDKQjJèv.Қ2z,Hr.S|c4S? zvhW;wCNRvr媉۴0˲>^, [BTwЬ (0e A4TnK T17<DSXZ\)7bU9vuQ%nh_{ˎ$*b wjo2*,4 ҪzMt3f6Τ %y=)lg?n_=٘ScG ūvRQ\?Ħ?3 E1|)/,[IG!;曥<s:|i&C} k>G9b֞ӧU%fTZeiZK[.UfI8ɷ:$wU熿NO Jm[r!&%i2")hEBtXz@M,7R L/ݘ^?oJU.6Z~O"*^óZK[o+>i}ޤ ?_OZo}j۾OZm?|OO4VkIDQ*I5.%QU|nD3KQʲN [ Eܚ|KL^s rObIkyL> 'ϵiN甯$ޠ5u~6*|nTg:nxOKXz4Ico& ѶBW7fu`,m]δG>7mE.Z@ΔIkOJ_dL{ԛ=ޮqkZ\ hÞ;ӽ}.Ջ}H6=@q&Y/Q(.EM)t vF)ӔCqO(poF/DOLc6Z.n۶>#*f,̷4A۟8q,[˥32LV6$(nq4usYdJLs 'ە{A3OF=伥5 9|_͝:?Iy 2=dS7Sc=(+9sDhdgFL6\{Br]K'O4œE̽ k;v}oE\e]:$(4R,Bf-[5k\.٩17MgV EXe@v[rשJo"ii+x duXA"e:xO.aJF}|:u/D1-|hZ :8&E!g!@i.h @v@S)j<F'(` |J0EpP97H$oݜc aЀLٔz_%S3O(oGNẀ-Mu9j5- IR-6 )@P'L8嘇~=Ėj-F+&L~'[% 7\8G^31L6$ vf,/vp=DUS0pB5 DS 5Ցjpa?.k:`c7 *_unةBC6 @oWܘlקW/c~"M8T\Fje{jgwlW#v7oK\Ҩ1էW8m:VLPUzUjfV7nv@CMS撪j9OJVT(QpKŽǪ3T]`_y!s ָs8{e4E/>TU{)ld}>rJŶqyIf<]{;$CJ^V&m 􋽁$%lܤ`7Bp~i⪓k0tCh%H*gىmϸg{ں*,Ar9ΟsNŮ֦v8i0؟sgΫpki2ZG;Bxt_4?28@KaLZ߆+%/+ IDAT;ȊNu iM/|V䌰_hnr/I1qa]^Ź]ƇH|cp05,9=1Gz'mnm7kZZZs{->]].({z?ЛjaumgJ(Ն:3B*i6ӚE2ejZ\Z]Gw8A$iӸRl aWr!{*rK̉ LjQ%)i̎2 IhzZTr!yBM Bδ9ӥiQXte)iӛӹA [YW3w/s pvF,b1*sF*u=5*,ƖᄒL)9 hOsGjv=K)%~ǚZIzGB@R{"+ügИJ-emq[I,ʾ 3#;W;='"7;a=FYC ƨ$V{[ܑ^- Χ62kn)\*2i jn;eE-qM/K%'\ =u},emMK e]q|ChP@G %ΝzIE-H&$}=VnRd^/dѥ!DQ̷-NiVei̾F+OX{|/G K[ E6TVe;[ ګ-usdz>27E])<[]WGSfu.՞-yBd!פ IRq2gYC1p@ ;5]dPa/s_$E(#<{3iQi *F 'f%"?՘8.tOG/5=FQSΘIdlN=ѥLM"ۨ6i|ok(ܒN\X0ccZMbs&уT55]gz) ,$fHBbRZHWWŔ1v S(3KkDB,_RկieuʢH^w|.y66Dԗj9Ym{: ʸ%& e6B7KȰ 呌Dh$,w >AQj>i_tp))_bOL׬Vn%+q}ĺb7H-D:|!ُ؇g[Xh.MЛE" &bZafl94>N^ WVV|%tl-V JzHD7JZKc*;WӞB_μ@c`YQdXaİ-.XjCB\>KZGŻRLҖtͷl7YcXigL# f5}ў553_P$涙i9[J &bT~R+KFM-uǨT [+TkoOkhu_fB/_s/߈S>Ϻ|}oVw$-~H|w2,J]]Ι8=O?e9z.#>d5f?03N1e,?z,vJЪL3]E1kXY`Sd̏XRўLٲ/\V YQZ5|Rw''5thkkkINI Tc^`N+X%>ְZ/]@0R~kҞoxո x _$d)FG\Z+Ggm~g+hOx/  }i`З}i`З}i`З}i`З}i8@Zl }שAGFN9fp. gP g'"Z- ܭ7Eh3ne!si1r5{*_夜=O p`.W UjNTZ=WP Q/aO!'q0{'.F2 /?vfkSLSfzD/$/R[dD/i1YdrqHGgTͥB+4i0_i1IWk^8ߞS&[KbtZmiJQcSL^R^ (&M򻳓r9I񲵌csci<+bGx2ɴ9MWlb4[n׎3h1rJL(),ܓ Q;JלR-G G72>-s}s9vbuf:F`ܳ:.uy@SsXѺ=qi۸vh"]HtmĴENrnt۬ٞH!V&j!=)dki`nas|v21൘BkF2U-ݭ9HG>mWVjiX*g11rm.$r{zSc͍vrAjV^<^)07R/W~BZA9IƵQ.l1vQNcWͣdFtv'ey\HY[ŢEy.#\)ƺ3D2C\>"-sBd}\]֞ѿrz3ڳL6*槆q@"yd#%WQw|Ch%2 /?5,H1JJy96KDXn|6'KjYEݨ>_£ӵud<1A)Y~ښ>zbz:cBEu=re5.)z>*?NY\AD.\~w^C {3h1z8K^.-TA Y.{f"yBejO!u4ja2euͮz.&P&m%@_sfPM'>-`Z7 BY_Y^JaoMJ(J˴[T+l(e="a0x~/$BNV ͒{ڳBkxyM!Jdڈva9S\xkeacD.H٘-DzZcH O)Dڍ%ɥUS4+RXf2>&+(3!sc^qǜyva>tlћ:S"ҚB-!5L4Fc>1gq( !"ɞemG| Z𼕶QV[@l 9w2YmٱϞ3#e] 'huF3JLF Z/ @_Y0N\+e"aNf} -J$R+KZ3v}ixE#ӵk\P #/(qU9%&ĢZIR4(ݤA[5'ÉknMLJ聿3Աlm-ig~e=?H[zeZiWJm;F͌%-lmDP|V̥űX~ཝ59VrJdk$/D.H+k b_xfX%j;&Չun){i2$t RK,Y3Fp7wZ+UMvz$.2/ ؗ %ml>u :9,5{zEhcc:[+H) ܋YD'k;ub]`e8r[eu,z}os .VBU䌰>{2B) ry33q؎ #pFqOLU0Kϥm2dHNi,@m0ЗOӨ\ fsieX =V^R*O  }ix56ը0/Fޗ /HڸRP+Z;T* |*P",˕"tO{FhL'"y "㵉uc`JE^r]TZ#fXPǷ\זbZu7ќ\ LQּv*[֍t-Qc;>'NRxT};Tfo5I4)Vڸ$ļumrZ%ֿ-ź[[:v] J,% w!B++,V L&D ݎVkbyGOd3KbKu"uSҐpo4f]z9-&幌$鉉Dp@AEee`7ZuBRY[[r󭴓@1󸜹8Fv8n'5Wt'R{rAPܽ>uꈈMؕv9#sg𛱕@_ǂШ\r{/ͅ=~'0'lr_Z >G*HΜ>_ڶxfg "`o8"ruVcKŧklTW.mYX_&wR^G R.̆f ܤ yzż: HãbA[U 4<.W D{],OGXё&"(SܲN$E8-ͰqľeknjRRYަ>>#%uN}Q1.C:N6L ,ղZ!~q"v1ʺ֑Rf&+%}R__KtuHR(ٜ[]|U 1& *- smh1i]uY{GB4+",r6eE2[Dv1;W8Awad8ϩHK'K$gc˹v4'\6!PhkquQH[֫w2qKo^wo&+ܖ=o$H|ɩ.k[Nr!I<[xv+M_ꎥ\A[Y]:`+*,h->5Lr=hS\nmuGKsРT_ /$?e["ZVh0{.L`(G3tǨU|r'fc㵴Eu:[E39>P73;_U2iQHmj5HZ˨kI찒z+ό6wȕbX&ҿ~IDl2 %adX.-!CĖQQ۴ϿQ_rfwʺBQ{SIRق,Dh8!+iN69nt!efHDDrIXqzm=927Lr`N.%6T޶lmEq=l4]&ePCKh"\&r/Չbšz ab=xIDAT%ejٴn3D 1+|9Q8FyͲJ* TBKv OOgZ٦̅.0a) hƜUdz w,rJw"i?\h2/ +rOww:HfmN{3ߜPq۴ 7"vR->=Cj lrUU^FiqLZ 1m, KlFSEsޠ\mgHhl2pAؙC[[[z|:澞0$\sX$nˁ bOVW9UJa MڿOrf /˞Jao-^>ޛ|t[i3§=3pF/,N@ǩ.+-5]yl+8}~r^7vreU]\..ְM OʶUjj|+ö/ Ga?ݭ 6aJMO=twy /=G]ӑp༊t[<2rJL‰z3߰}RrcՑAx[/\ͥűbxtA`o`З}i`З}i`З}i`З}i`З}i`З}i`З}i`Зv|do7o-? ;~e4Hh>J}iG OH#= u4@0K4@0K4@0K4@0K4@0K~gl=wa#߷ɟϏ:t}.u4AK=ѡ߾Pn0_`"Ç>އu-uP7: }iݿ6nnlQDg~.~ܾ$6n?+n毷Z꼻AD.$X'S¹ܵ*Y"%7o_\d%^,XbrG3^oxRL,㌦͸̣'^ن.߿[l/ .g\khvWhODD©NG7't/bu19gώKۋ܍Ͽz5_vW<(}roSwio?+xЗז𵩿qYk`_yz2R|4qwJYkRrG¹:7Qv~|sCVn,j?/Yo/l8= |O>8;ۮFOOD[vCF/a6>YfeGUꉸm;͸!l/}囷gEo A ^-{fv6|̱,n47 ]dѵ'Zp}㙎Գw}գj5J ]z>\rGn_SoOm3|B|60hԯYn~9Q:SepN'"9yv4orwe7=qc+Szy%x %-▜3OK䛲ecP~vG:Bŧ_T;n@Yr[TT z"p9ώRpUҽc_ϾѓQbȩk\D{Lez+:~v ".X]3jN?uhͦ StD:]\Xonlo;vRQfSxeY}{n9a}A]l^sdyfvQFe)'| ^OW?RNF&>iV9lKoUK\ZZ8sƵR;XH''ݪ?ZݸizhCe=Z^xtE: C k?m|}^hscх!Svɋ+k _M qwXo\A™=qOk^W͟ -L]̐znGtΦf}{;g&c }nSU-$Ŧ իU;SvqĽ{'^ ZS]ў11vd]Sv%"47vritq8ґGXv^81s_m6;;M-L ]xv΄󿱅ٍ޺M۳DQ?3X=s5ٍtI]͢zdz2EzR"W\%̗xn'2ſ5c%&ӆD>9!#sӋvb/dE(r6Z^UW;{v&n-Y:H+uw.O -;؉{C7 ;o9D3 I;.M=ƷRJ~65t"' zG1J-Uj^>bůq\u*\9D=!Z|߉L/ S-po\׾e["D߿i4$Ψ|󔎾ժ#`8ζ>Z1ұ탣ϾҨ"'Cshf(_֫G?T2_O^KkxB갥/lb_=qI{{ImozDDPbք`xdۋOz~z"YAP4囧._} 21?s'~j~zVZK=f5]|nڎH׸4m#,qfk7{$B{FUϰGܣHq͖V]Ϲ!cqwg[m ُ*iQ2Fwڨop!=!o4'BDOr=9,XHԭ2h;"m bճ1UI#=ɻrUmd{jxφ_l">_,ٵq"R2?my e淴M͋W<mQZt=Y/kuRۧIUG{,^ VK cu|6~E['%_6W.u5M64s]n5 i~Kzh;˜$tns4+k׈Hĭqsn7/0nPQ=<]u%_nm6"؇G03t5A~g?-8d i{nO5ko{ Z-><EcN~}PmX5bGm_PF$ݠULɊnw=QOg>.SGO}P.J֗uo7swl?] #~*1POdVW;޹2X@'>4f)t65tHY{Ƶ'.}:Z O"Tש5"D938:s3C"s.zamGnl 4pzϟoӽ4NOtڡL6][9}~5#G"Uo郟j"b׽c.9`W{ё62u,ynݦ_*6Mjj1ffMkqktT ]U'JG3/7ɓJKRGI免 QLȊO9QJ{d1W RC ~\*9lhg?qX#7D4-eOYݶm-{ܥvfzkةoNÇgow'Ç\${?ݑxA{5ou\Z5 ֟T{ ;GjSR?Qlk~:qX~LcM7WuA.w>9zބ=G Z`ؒ{s!D0ou"h6֡)uu궏B4GzRnɱm -e6_*l]5Y4_`UQ@Ŕxa ޱFqkJV"gsJ+wH}sF}qreɫNѵ8 fq׉0xWKl)oHč[l6WWK`8{hQX9ŦyGO說C46nzoqwn'}5/ ȏW7©wjesCg-TxiE6qERWWsQYS "̠$wkEP/hXy0M.'|UK2'9 WI%uw;|15v57}jPL9g`:QOgWnF[Um­ci\ uPN6iƭYqx8n[$^'tJa`<-7K72TsQgھy*t'!cm)2ctЗ0"Y\abnWpZY.?^Cm?hk_0w`_M TW|sBN6Uա&芺G$zNv?3t/}|klw7v?D=R9Ao4d`qkl~^ڨ9 |i{z\w/=])-O^"uM8 _s2ux4L? 8fSr^xPkxniF@O.ۓ.PS-wq'Lwu7mlG©r׻<"'O/wN}trQLdŀ8QxݍvsUݿHN"Cm;3{a[ĝN}t,<?|T.c{3D ER%Y㘈~vKsnDi{dT+-֦[q\S]S?}m>8B|FKG8z6s7a~| 7[|LXx_.Ů  _"`bƭ_ǻ" >Boyjo{;o;=Z ʡ^1x/ N>Z~I/Kz}/Re2?Eݮ}a{R?q_AH&[lbc[[]`/ #xIA$oI!@::.#Cm/9~Tv?KweK?`hH/ێkXIėu/]D};j_7o_\lG*Ϳ>:CyǾx= o7x\ /  /  /  /  /  /  / w|?aC*{Λow(^Row凡RiΛo0ߡxI0$  /  /  /  /  /  /  /  /  /  / "MjFIENDB`pengcheng.mind-test/docs/aicc_cards/assets/choose_mirror.png000066400000000000000000000573561471400134000246160ustar00rootroot00000000000000PNG  IHDR_L IDATx{X/ EU5EmZJ%jpT&ZEqSM.>ZѨE QĐm"&bQP@cf`Tv9g\3nFB!0=!B! B!hHB!FC­B!0n{BkȹQu0v>/㘲!:(wnߩC b@e<a s$ʭ~=/Ut0שOÑp+D' zUwC!z$ !B!V!B B!hHB!FC­B!0FnywU9= !kj O!vqFZ-[LnBNˆÊ@v8+6|oGCs ðp͉l>Ɂf|.v1Ҳzl*Z.ρ fN7W?ryO8.WMJ䆰6w쟯g.`>B[Y¾-k1d]Y8f˫`s:|Zl]\s|_ VL}w\_J#SePӖed OMe JƵx|C1kᠱ38XYZ_r2ÐESqz sÜI[6i3ex9m ~%W@U-$h[1k#c>OQ|R^#9^u'x nNl'"YeX:M"`'-WuBfz/q9I+v?g|Ӫe83eEOg1e#UCL|> z6"~Bt.(7q>xPT+U?72;9j*ޠ ;e# 4rle_ηS<~$ -Y̒ܶ+Nl+m1g` UT@ 'VrJklo6|=r{29v02űIUW}uv~643sCtQͽH}w|迯& XT U_*2`NUO^)W}YuJu@]"aRKTnKᳯ1yz*Tߤ&BRYT}]ŋ1oO:.puE9bp T oj^ޢrn;-ThGxU HpdYуlUym"VPպWv+ ߪn{A9mC۳ݙR*Su}zN[Bw]2^s">xUUU^uĦ}MS 9AgyWܨg} .y~6p_6AnV-_nW`T==e=tv a8M+y|~CG'pk9eoAd%RG#I(l#NmhPEs>@F 9 pyRFܶ12̉]cGP6 iuAQ?h@UIlce{R6CsZ-髬XӬpwf:lΞ`mYCԕ1,KnT68k>BZlv>\v  *Z@˾^ę|M_U=`y|~"MeB>5WS6UxE73U[1PB]wBi-uiht$Qv~O_PmXS!#ZMẳ._b"\ y"H=m j#h3x}R_ct3.ĵ *b5͘9ׅPۊ>_ĒJl ُU;@UŕM\+ άOzբż+ܜ/RrOuLw/_ U_lՉg|/> ƍLr]JSF]nTXaCQt&йv8j+ & HoI)g/o//\ܦ+IHDGgGhӈL2[nEeh[@Xgp*ߦ\bZ"|^XxrF̔3f&0Ylն[tQR(#St__;s9v]p/+,`;3c StQ7[,1A;r3v3^ȖCѼ]UWcY GωkY ˓qlY Aoe.2"VΫm _ e$7T漚}AB}Mq3g0E}BL{DeVG)ުBnM"@wjƁ;G?ߢB@]~C:;D/ոH`źdACKŎ-sVU[&##$ОVT'7jF4f#֜#(Jf^=m mANpDP'lr[f:fN[ײ \L9r9 prgtM'Mr`܁`׌ּ37'UU<^z{n[I&^joT\kπ|9A16yxּ:~V!Ć D*b=-ܛ*~^>_92EOCS{Y| lױv=Bv=qq"qvW!B!z B!hHB!FC­B!0nB!ѐp+D'*_X79W!:J"|щ(=/׵گ79lbB\ #E!会† aBCr I]ϣ T!z.Q,v9m`[^`GPFf$Z5܏-ۂ yd,)=T9F sv̫; 9c5YBPm>-fSl.>!.a\KBn]n͆aRoXu3<7̊y_O}ۙۢF+D@s(v \MPVϷqu$hߟ<U*}2w&sv1 Gcm@l+p{V&L'j7O_S_KfF3Ss#5+ U(wfWCe7O_c\2$ˏJ+37Sy=E csS68u}>nE9rEp[}Sﳨ'cvT;hKPeM-V$8C¬JؼF rs5U9E`+xk˪\f6W|dlIl ĩUks@l/*fPZd˞(MŶ{*-]4_um>}Z)57TN;,A"2I݉ºF>ަ,c2yLO\O^FLsb;A1̉l3<C;+Doj `wWsfmsF˪6wLYn# BW;3+ܝ)߳%k1y:p-Vn[0p1R ~:C{pk9eoAd%RG#I(ڙ-9楪&8olOڮ<%d(hM?o vGUAVj31ctgh4'Fջiac((O3=ZuoSpש~<纰b4ck>΄,~.X?~b6h>Ya[ W!LܚRvFaAiCj6c5CAz%w\P8_`S[kBl31\_ȱWW-Hf8Ep+z~,0A@h;m:s@PA$'iBr분V74im5B_cp[ZNbbG3l5א̈"v>BRٸ!űb!E,V+u†?BMvC*ҕ{ tv &ot% [gFdv+z|PRh -jiV;Oo0ð!hޠ: O6gYk:tBJuZ>[i{ 6>cHVa$(ܖP_`*ֳ(ۛrЙsVwr:i$óST)puio?ܑ6 Z='^^M]!zUUu (Qbm"yf4MS~iĦ<}v.{i r0^ef /nL{m 9B#EֆsltMlcdL@%$Nd7 V& " زt]p~lYV=/.~l7KB~!8ۭd,HVݞ߯`+Dd?Ep+B![!Ba4$ !B!V!B B!hHB!FC­m|ciHQm{'{܁A&F0BJr!z,&}'_Dzu0_g!/(ٺrj9{a5Vȇ~_=[!zI(+✒©z/|=v.'y/Tg޼MHbwC yXSE|[=}# u%|/iOO$IW>FE|H!W B1 B@>MgoV ųwɬ,kc,PmEXy'8,HY;P]ժ!<'.!lP=Fާ[X,[Zүs X_K_B*X&/USSmGgǑ9d頩f/bWPf~!ov.#l'!oPX>f1^=O%g Uw˯G[HhA&P>8G?/@֛lM)l쭹첄+}t71D?as}|8I] ^[0>ރ([U^[PucB|<"H!|+Jx\jUb /)4?/.nqQ&tZ v%R4 { NM"-x  'c˻ҵCǸs}̦V2X>q¸.`xu CۚZ~&0 Ͱ IDATC̾I.e^χ0b'rB=MJ | ~< ۉ5_j+bw^ @Yg:BK.i /H:~1?gkg9j8 \,_U{yAI'3oX&3e;RddlY,_/p<&[l2 (KL!ӟ`7 ƅ$)ًyc;\>q_+SZ~-4rK%TF3dȠC "|@调\tWƿv` x1'%=wU=ٝ;w~ܹ̾sGԻHV%yJi};:&X˷Y+_ޛѳ,.+k_SnO}~wgZΒ#;3Gu;e'~`5ݬzZmz\!1췞ŧ Rz:sWx |KP@lc9)`'|ıWuc=hёctz0ynuJ%eYHC[艑$|L +߇]G7jUKqT[!z򓜺6/-C('l" ;$5[GRf5Q~Ϊ<@ vlh>L56SV_qq }qj{ҙ/;r {ǔ|x!hnS6FA├|98$.U-UݴgGLaU]C5^с$5Tִ?0_K8הS0!@tR=f}$q<'sQs2e 90 e飼 5TWn㖂2D\G0O=CRy؊wҰf^= gTwYd_Zۖl r{Xnڂ;Ka$ ct23ݕ%/?9č*?i;#5_繀HEN~S}񞷓G`&{}<緜!O1@ڍ7y+oH.df4hOs,S˳`#<~Y/O;#YD~;=yi'} ԋkxLa褱J^v-_gگmX> >dY:VoSOV_.!q3illlAa>I$?ĩǻ=fog# 0W ~- I+6{mv&K?m9sc6Wt[j046s+nyV#pHJ)f|itAvA@$ ϶\Zo1[!x:ƊDPiGvH1P![!x09Bt)9L!B B!hH[BB&tTndܽWܽWy卍X0%'F%ЕW>ǦTnd'lyjjj{(B<4 ?C߯뎡 !X0CӾV͕8!_ !vַ/ʭBY]N_Ph566+V!A _+xt!U;jiim_ Җ ۆV#UlMLL055mp5u5fRB!ASSS,,,Acc#M?ϧL&]m!444Ѐy)DOfbbyϧ!mCF*B!:D7Jg111v_Րp+:y&У)jWmAznBaf=Ю644PQ)WF}[cJϭB'Yϩ[!=!ESB<+!W­B鋿,m$ !谾qE_p+/H!Q_Yp+ժA_p+`}U_p+/J!z*V!B B!H= !D !y2+I-:cJ#F?B>D{B3dUe[NY=cQJN4/q1ȗuk\6\nnl 7ʚа{ǭ'5`6jh׻|nd7ydz멫W^y<Ɓz;l{c-oTȭn(ݷJJn}U?׶߲!7VKeu?~a*O0sWب6O8n?,#XZMݓ.- c'[VC;%$ .e^adapg}}s#ե{͒ {k&bZN&Ll?(,&.=3Vn}iW~=nb@-h\~g+8pK&7V߾f3Kj-?XZ3i:j(+W0` `>G| %ʖ>?q[Ϳ}J[F!n8" ϸʫx'Y`'7NN}ԊT}[]n&_UVFrC>D­VMKWaoV:??+f<<# f*7lnyW!?guǖ3jӼ@.$l2` y׉<&?wu>#wxZ7P-k^gr7FvljuQC.ݷԡ Pkgx|׵xjMv^5Cb`snWsj2 zj`fůUShĞo;,(W1e'}>\85*\Z5?"c6j1!Z sVmPÍ> `uie\΅?Tfsm0@e^~kƔ?IuvQ0g||PBꑽ%B!fb(6n_ګ*Vcu6{yj? jUre~h+VTڿH1Kprxj i {&󕦒h>cW= |LO%=WBNV'vۤ]7`(07Sk_}lS+eέM9z8_Sޮ4h]k`I$yۣ-zA?ORnñK @5bE7xwX?zKjw5{<Sk¥xB !LMp%R7<\'CYIWX p\KM`ЯqJÍDn.nfMCk \H+s,گ͉{ç'k)EnUOVN.10iqO{əsN*0xͨvbK)gu+<_lFQPɅcYDn *D_;GԖSv=hڳz&d#V!Den 41I$rw7xǺ5R`UgJeMֺq Է˶)g6>?i3?0U񻵔 d8f37{+UUO`o7I2a?uTZi:S:G ȑpBwk)5s8vl؍`yS\Ȅ+n`KTI ޻~Znv)Ϻ Xt%!&.) @=Y٠YaF+l嗅U I5\,"T_eF1lyFO$ܳѤz*x[e-.n޷ Nkr.}uJq-#VPEڹ4Ldxʡkgm4`auwZ]-ՆJNܣ); `m@񷷸V<5rnGXpONW>NUYl}%>#ВPJF9Ń,NOBa T<߆Jes^+![!K{-?o;ׯ|53ʦm-@)`ڏwVD`fe%n'0iEĜHs WI 0sޙmTF-/; \W8WO1n$ |W/u"~pnR-#o(),& ,b@#W]Cu;(; @م|?i@q/ _)g3A>ohoQWۈDgVi^'+L0#>YL$Ւqj ޘ7+*9a{τF7(>[|ؑrY6fxIc_](D#!C1d*<ʎo_OQ zU6/Qy[Veڲ&ˋ177 333  L'Xt(;Gܦ@~FfkH`E#;y)=yxř6IlkXi$=Euw,%Éf=WWejQ<)4iUx4)uyܿ*0dԶ7@G0|dqyGԷ&q(oBMa8og]mVvymS 2Ooɼ'qd1Lb7l {{X~GͶ 4 CGB;I([ҳgQ_ d%Ux*02%sH/%&ja!}Q"2I݉ºF>ަ,c2yLOƼdE/fo $8o /HAN(?A2/ e dհJӶp5U[uBG±=\NcgjjB"xCq} Iz1Ok^Ϥw%zQ<^BWjb qɑXGI _QA\RcD9嵅c,C_r h&%+^K㐒O@uN¶0TkI^ȎgmYAܲ5d iR>ϡ9УܵZ<Z@+K=j @wﾼyΙ9dCǪB! Zz[5YTH%!o gD0'FgpՌ9ړefh "D'ilhZ5u!uBg)\ n1%LǓpo)SLcbNK_\qO!xd/E#;>1" x<>E Y/O!sWKY:T2o,F['IJL`w e*ü;qN:x,?{jBB]skZK]9IԦ]~#`~N3g/8'9nͽGN{W>r-U5FnO_b_ZvncuU6A|j $i[ue $]?@q Ky ]'?B. hr{ǨP:cs4/8j!hjKH&ʭ+A|tU[E4ǂX.|>H2[3?`B:21i56L#ӷ 32z88.PTm8crFh&.,,!o$!CP_P^B+yxcˡyx2(?Zm??Aġ /!lEeP0t]/'nU[gYNyNç3,>n)(T'|V_GR ;G,( gؿxd;.kx ϳj44iחߩ^Hɿ}<]wHJ/`Kx$ Pƿh /=w/ʭ b j1[Ձ9JƳqj vlKU m j{Y4q|V,HPNMڮd|xSx!.oG6q9K~7wZ3s!!(Nx~ ٱRS@>PgDȎ}%y fNdF0o"x%VY;QxꭳG8o='IYz`,&.iþdKldRvzpk$;\Ipn4ЪKB_#>=M0'&S :hS}Tv6"rݚ/,lyԃīZdf;q$=*sbAFzC.g !:H.x/ۨ1*c- #l]_X6N?GabbBa_ԓQ'ݔc ~kV#^t:D^b4큗@ae'cﷄ~߾xqK!#p+D`bb!CSRBG; ff^/;%P!DʊÆPO`iiY/B!Dאʭ=%_QajBr+DP(a![!:9wv0B>Ad'lyjjj{(BtFihhP5wZȯyBL­e.Z_,Z8N9 _{Vc&祫 ]f!]$KRr a$`VQuȅ<=qkΦ7kmrgap$]sIDATXҠx?j'vҰUgqdB x.~\?HĖsiѽJRBej{*Kȳveh^ɯN"Z˩6~~I,29|xz(唕y9}'moekRΦxdzӂ TŮ@l:ejTOvW,K8,m;Ϯy$\Z|\Rt|^k}H%em<0f ,yFsͿzu.*ᶵၮ:ʣo K@Q kjdE\/O^-nYcN_gQ3SWrsmlrKY?oz(>SOȏ$9%Žn#(+okc&n|d9/HЖ6_Óp+KA蘱#9q9>T&-Q?k{o#O$far_lLҾ8#;!iJ>^C=,x%i)I, E#p.*U/!v/ˡ8ٶui9u9+?TiWM,+*=OPdymSTcLxv~8#~'1mWRWĎ'KAZY6??ȹl%]Y`+}\u00c w@+ h& :9$tܓ+å4Lͻ[]-٥,E(ڝdc\%Ij û4`+@n 8 3C Cu]yw>[EyKX?MF{n TaU;N^0{dށ6ԷVȗDndž=}2ۺ#V!Dߨ9JFf Hc`PjqMai?Ԁzh|\K=7itG4h# fS-3Yd7k,c c]J\N"l K!X-PO/Oudt|{%5;Xxzۚ%ƟPM6Qz|E箇IYTLJA94r331E:R7}&~hg{FEqZ2poFdis'h]$]d[--Ԁҗ^du.IͭBq IR6Lz)1{اK? aw`{Rqf|XLI10pΏG惾u p⭶zhk}RNB3~87[!} wf#]ͨ xN2.fwbRJUefǺXq_}!4<&xq2S`d^oeIRK;4iNL { vιkRm& GX{0aN2o_bCs#ns{!\hXkGIJ慰 T@p[#-ݗ^cJ\1~vXt%!=rJLߏtbLMcjÇ..>Wabǚ8r܍Y3] Hz"a35̤47"52.3|ٽ8? ^jnYdشhhE=حs>BwW}?DGm+&#y~&!!Dg3]lXZewYՠ 1nF6݉Y<`$c?eQp:Jtdh1p~H bcF)O\U"=Tp+aV1#t)Igڹ)b/xE.;@^HBC6f'+CF6Z-Ճͯ/Δunϵq?xu445?RV-QUrR=}0G^Dyuڋ%1h iL%^WEXT>GbeB>{ k$o\ cf>Iuٻ0+3<`LVnØevtFhrfj6'gvwAxhˬ6JUL!q6oS;&gj=gw UPp%XMG *R lUXPZB!,سBȑ#B\b:Ix0Kzz>v\\\pvvRRDPdT(}˞!5B! s5Clhؿ23]Y#~Dn{s.4/&%Dnn 'SvƢbCB­BqL&jD ]+}݋]+'ebI?)Q 7g&e B!:%e BܚrY\P&B! B!ߐp+B! B!ߐp+B! B!ߐyna/q466uWpXT`@KK`?ǽBXZ|H_wH:%5 }h_wE3^^ !uR Dkll`+B B!ߐ!0',W6Z?6mW(pV*mYo=(fۜmGz6 ./Ztp˓p+f\V\a56۸N٩mtqse!~yxcH 5 ]9mq`| \83|\ io}=k899T:T*qvvIb p{3_hroV4PLs39⥥FȧZCC=_?Gŕ`hif VSS555 899BJB$,Zo ZQ}('4QlߗJחٲ1_ Vn_ilhnrZZZhllp+n/[/Z{[42eyrمw_&AWe?A/8Pl 7I8v{bG3htWIfkW8֦bN]~L` !Ѽ:6 lL=1SܭsہdbH_ \ 5:_4 ]EM_wO!n)n~K=6^{!L}F̈ ~` sp19z `ƥZ&M4;}eBS_x3֘m5]W  'fs43܀qlx?>5Ǣ;:;#oE|gjG7S)4Ɍ#Vd6>hX"bŁZ;p/9^ڵltx[=B܊z=:@e.QDvL;6A|z.ORgn-wR:3)봁?]0+xCݍ7?8,Kl!x'xLB+gB02}Sd͗ॎ*8j{jv,Z͗Bj>>#6hXL2p#x}ӈvm5ހlAfLyҶ^! ib C*aWm>}FkPED.$\/ANv\^Z4?רڜxG^A*vo5h-ԗM>%ֱZo~hSm}Wf1emT10- Rb;H['a,|3=j)ٲ_枥b&)XGk\fX8;Rm\֬*W3bp dw* iߟjA͔$Gb:A;v)$E D/"i|ھ;W@IO&@_Lj:~E#JDֆYO"8ɃԸ  /|o1(AӺ^ac?u1v۵WIz.77z,7 RS T1š̥PI3' ͧJ_>nJ: 7r׃8 ĥPG.NQYFqpe}@)>8M7o ^/+,Ky5D]+Ɛh^k6{6w>7\dۦtκԼW( BtC\L[𙉸@.דYHZLKT䌩%5:$Gv8NNGoYk'-db<jax*M`|%|{}ٟ Ԓdr q4 ϗvu_Sٷl@CZbEŚ8E;qLK$q\_35Ͽ!;+YJR3qƫR,<1ĀQ=Nя0^u+cཤfj9yɃIPX VSC ^ˠ,|UO-v#\uJ1PˢU7))/!exָe:ҿ1]VD|qsn+ufƐ}8?tM"Ts ӛE+w]_`[VQ#&+rvNn2W&sUTo‘ nu ՝RLTB$mjy65TKPz3 g:Bm%?:il pN棬5.X%jO]<"3%/UWn簡Q m8֚ SQWvtd8ڐC/j#4 ̪ j-kaLz@xb <ijݡ pe4Cy}뷭^tV+ils!B nݹmv[&/ve~̈"Vv$)2PFcHZ+{RVnRJ771DOdft'kaJ4}u8+^wR٤μF(prr܇ywS3v *^ =r9!<*+~^3 j=lڿz$xi1B?YEBۉ$3),Z3]Gμϲd 7e!i*AMm tGd7?q%rN%0-ƅnCŚU D=(|(S\|m^U˺*Kh5ҹ4bB~Oa0U=yjeTD@Neez,fBY5o\$"Zs}swu&e:ؠ/кWu|ʆu GwVNcQrҗw +u\t;;₋ U R4HeWAj*V{`u\Fm]={+7d _ϭ%^G;澇x\epV-]7`Eb/MT9E߲h͚۶syƨ}޿#{_>qvvFTT*Q(m b7DMN*y@]DW睧Fhxilo ۴TľEDhjl2g%׮ЪRqQPTb ..8tv'Zbb --4U"R5f\{{Ǚuق8nꇻsw2;E{h 446ctr2=ȭfBI1z}]aT~5!UŽ(vt.!:B![!BoHB![!zJw ĿUL6nÓ ZgK0 ZZZfM7i}I!D ŀ nacx3vL.y%D 5ؚOwB39)KBa@}˲BN%/Ih>|8s"w|}{+ĭFUTT-кNNNm])KBF) q0!3nBŲuϖ&6BٛLjnB8Ecw:ȭBN) /05 899ȵkmu+ !n k~ZZx% !|4HTnnvWqVpkp-Go([!1-2ߦjpeVem y(֜[!]j-M ̿@­=2Z\p;fOp+!$x_[Vg-v6ky{ EB2ZvlFú( V!eVa돤,A!ݬߚd 5mtMz IENDB`pengcheng.mind-test/docs/aicc_cards/assets/finetune_resume.png000066400000000000000000001220051471400134000251210ustar00rootroot00000000000000PNG  IHDR0q@BsRGBgAMA a pHYsodIDATx^pg91q:ޙ9k w×5^3{Yig 渍c̲`wӧՌ{0i $Vja~ɲ%e3¶0B[=|3|2+TU*TW7TϧL͸teDlƅKMlq&@6| M3>lodȼ'F2d< y 0@`#G2d< y"o֛ooyY=W,GV[&I_jHa//7((ƭ~MCV^j%37e\m|[.Z72#˩Xcn94tL|j?f J2((W^h[qhuH ~ZnD|DDAo C@AQEQT)65կNJ_GA_?B 0#ȸonW;  (ƫn?5GA8Ĕ. BAfi*3COiIznEF}5-m^-6.A̮n9wݎIEAQEQx gm0kπǩڕBa׉L^7=aiz{>L?& 56">%>< uy<}& EQEQ*_/*,X\ D 0.ɓNpc4;Vdk^ij w2qi$/] 3r獫i`2`PEQU܋vr݋얞fzZ4nR`75{`8!FVD8U)Z'Ѱ"1J<1lU?#/+Wy<&i ȃ]EQEQk*/SSK4Rzc`7O!)`=(cͶWED<(Y,{!i}| z͋9&yIWpAmSI& EQEQ K>+ЪSjA 05) Za<_ViN)"#lώzaW7r~9w7`3o۳Zo=n&;=-/Njz[]I~b۠JZCU"_O!a,lU)S{cbP;SFZ6:̿ hO N=GÉ^YÔ A_c/?79|..;gS2 6O+ .|hl> %9f (҆~&jy?`Mf̿+cWkjy/&]0Lr&8 ncl*b{_3!̞r?z1OŻ&F1VEhXQR9zdMcvy?/MJGQuKMBvI&j/jT9 rl8idԥh 3$6~۸"-KWJ;XVUVLjj~\ڽߧvZeQ~\Ez̭KWʷ_KZ?':cU M婜eZZi{__<+fw8 8NUs_#+󯃡ӪduyYjYe2GaԒoWoyS F]fyÝcUVWQ˝++>839wȜ6XMXۆ9\,~'zV`U2br* s-elS.j~m4v/y ְq45VWʆh`<*.t 22++벫uQ`}iؗ.J 0tj6K^y|r&K%QeC m2V`84h`_ipZDS׆+`ؔX0`p}v_|ܚC%WLUuVRxO BP_>ڛrI$U_vy}2ay|Aƒr w}Lkbm`N#kK˜{gK-$kk̴Yٶ(x4:C4K'?)kij3f` tx4x[ny&[϶JSefzv+@*lrMq̼w4Ɖ6u֬ݛ\(^;v\,]ۥ/9q]gKLOL~'owrSVmO?l!vc@~`m"G&MM8 kS}ϟҹv<׺6 y.7M~-ir!2~!>*td2Ϗ6^xfz4 <_Xiy IۣCm8i^/6_ciDC|$Y]/waiH/B5{I$U_zzqB|8wf9́`xBFt<&&Z4nZ7xH,ם|h{͟Jo:G!306F=n`.9c_Grfne@XyRҲ,/oY6zBÇEL`}{ۆҿs].0Qkl@3 do=ZW*xMc岢ߦj ;Npu[_.f͇nz:?˙[ݛΠa.iX2[N_y_:B˶m07wd;}H1=7OV4q ޸/sGȜ 4Wp~ԭz\4 =hr}a1Ҷ^t;nʶB@|}{tٙ|˾-2gi VW\וi$hؑ3Ý۝16*g~QaGrϢ{ vrΝg5r؀kA[I56RwT9tpAVRۏkc68M!Rp\n.uy~c^CW/ 4ԑ=5׊ީ,z:Ox\8-{qtZ\5A鴪䕏]/kuH/jW_}NL  OwѦ 0ks#_׮4Wp~cvV_`4l&oe5yͅu`J*wY.i̶4@f97r4pt斝YujQ28 B06*G {`h<؆ ow ~RZ.?ΫftzkdQfl:`4涡ouN!)lKyoS,|G)Ƕin >7n_}ѫFn-uyOՆpX旧w; ;BIuIND{̼G5nD x+ iUgGto>%Z俔H/jW9_g^[.3]c^`ii\,_떦%\yB=i2 K7 dVbYni1ßDl7,5mi7$zَhl{ޞ_oƭnY\?AG~\r$z`˽e]/ۖm4|C)kp,uN)4x~ K^}[{?n7h4:a]j[{K_'MK͸+y}kBͳSD܋yz;,lKyۦ0Y**$-GB1mtۆ_i7|-Ky{lحF |WOo?"{ iUgK䙮q#ReU~`)7U"dNmmM= &әm"aQOMիJ O_Fn~Ua0Ư16mk_{zaF)˳^]dx旧w ey-_4ax=0jS* g*So/dMկJ&%JY-{oy ݹ"YS#0 Ӫ:+_&_?/]U 䀩H/(J؇+ey#D<Ƿ$=bEPΗD__mm@%QC<7m pgBiAnX@m`)7QŗvcPw|,_mjgyfO)SOыw:M#AX@m`)7ŗvcPw|,O=uD;=O ">bEPD__mm@%QC<Wl} i#Z:-1P[@ Ldi7nڹtVqҺ6VC}HT30gyfK\B{_#"jHa޾ 8⤦Wu+ d>$Z. _ē噭*>WNxL@`)UEQEQW!U*ۥ.Go-="jH(ᫍSw;&Go#=  (pw.qjJ ^䡇_Dr{]EEQEQn)Z TmK{Y/ra[W\EEQEQn#8."T[* .4맾4+ 00((r+7zBĔ.w .txsU@`)`PEQVp 8dTeh/ ,\EEQEQ ǩJ/h(*/~)i:U@`)eEEQEQY7^Z{a516m-g?%(4|ڻ1'4Tho ]^g׳(8pOKPc@ e|iQEQEirw47(kvﱃ.]?g[^Bt]i7Id|<P;@ |Yb|Sdh\P7D~[wah=G($.%KD^;)cB08{Vd?S{Pqt p#- \4pB+3ihtRXcB0i>-~^{hhþX 8* Rn?]s: _7>MqL_VڴROfi~PzJ7a =mqL_V`B qL_V- 1!P[@ |YcB0DŽ@m`)eZ~z(jt1!P[@ |Y=q&EQԴ+ Fj!(jj#R @`P5]@`)eZ0(E0P-EM"Pm@ |Y k`6 T EQӵ0T_Vϕ/+~UVsyO?tswj@:%o{۵:\ěP#%=HKQt/01&uM'?5TcdUH/+sDdÒVdϔoENj{e*Ity| +^k*g?-Go:\ѳq`B,N&O@]wpu>abMl&&=VF}3st;j+`XFk,/%>VIvb+M6N*qLu`EPꗕP"0 5!/e4|CȞf4xt9/(F 1e'.K׉&rh(a/(FJ 1۪[aYie!uE?s3Ea[v\\Mz*%04 Uy0kOR1QaW۝rt8jBR+?|FFG'^Ѥq*R?6H1a ~U0"R~ kjcBl54_ Jٸ6ixRghϊ‹w:F6F{t{(A_.=-D{V^t<Il;ˢ')Ip-$[6`\:̾C{@Ooo@!i1otș/ytfn{!19y䰴: KßǖfA7Hic./ơ9~ȼ3_oew{l>`tOqY~3z@:>wƞ~^?o}#Y.{sޅ~y[k|@r(i\]fcL\cN[fr.^Gubwsz>ooeUy݁#k3moA]qru<]})sȾY_af-r_oڬW]W-{r>t-y|xo1yfuu1fc> f9{ݮJJ>k٫`@B_V'_SD഑4r-3,Jh:< HmϽk`x _WhcZAh,^"U@mC+)$(F>/ضA(|Ng=0b PmGGaO9$Oj#/`1#mt{Ny46;%ow; m|b{5xB+Krݟ,i N\45f}=lZ .~r(a\}4Ͷg']5ۜ`=EU9y, 5:vskU|} bP`str[ftYn D{涮y}Mxyu(47exۑkm+g;q<)/%c910TBY_V p0 gE@ƖӀMhynF/аBx!+,:ܰr u+X 1q%}ϋrW1}=+)\ɈJ1co;,o~Gj밍 {wK/ڹ4^s|xL^iD8h{{zq@h0~I/yXN~A.`ixG̿[>/8X-ޯ <{]YKK Up8KP~/eyׇk0>M~t`כiHuKBe|K9uKN'o= w}x}"]{FqI;z?^b='9ek4vNv-VH/+P3&E^UilMnE 5zf:'ۘ%=$` z^Kt|}^)O:*r ;K|۪nKҊmo91X!`zlп<`n{Zi̘~󼁼_KFN7|gS{\ÝiD8G_3zzJޗcI=Ao0<(͏.H,X2zZ/|NCىvO9O/DY/lmޏ-_7ӿ~/vWv\#zCZlx4|Vh hpg)%:ᓝA y=ι+ FP֗U"4mtuO0@$1. קx^() !v!N7A?y2`xzNb~"Fd؊ {uYU"!Pl޲^(6#W=Ta zv)W>xGZ线_#rPAü雺h{: 7[{DXzEi)$=wϤky8SHjly_~{r{-|y Av=AN4m=Iy^wŖYt4s .ơSHF/< oy.]~tLZ:?2/'`))/!oRNA+^%y*^ rUa =5 <_z~%W/]{\+)NW/]01, FPV !~#KKCuhsb \k'5b E_~`togWZBRJO!1#X^AC۬wӅ>u5૷~jv Ipng#o;.zt=oa{ȿaKיX쪼]n9y<@Ôvۢ<$oyG{J>9Y0 }_~F^@  ϐ^L׈7 N@3|;rk?c6^Q/w{]3hP `~+H~9\ NS6DO_ur¾u2zENuE% 1:h$3q +h_4&}!i{!>g s5,XY+9åpe-ԠEQӵ0T_VӋڳBOѠ":\?H#vC{\C4.+ /P k`6 =-Dm԰BO!ѿz_wT-VvTEQӵ0T_VZH/+BAQt- Fj!(jj#R @`P5]@`)eZ(jt1!P[@ |YcB0DŽ@m`)e "R E8&jH/+pL_V- 1!P[@ |YcB0DŽ@m`)e "R E8&jHA((( @`#G2d< y 0@`#RŞ6((ꫂ@ErrEQEQT "R=ˌ((P{@ EQEQ]@`)`PEQEw`G@AQEQUEEQEQT}P{@ EQEQ]@`)`PEQEw`G@AQEQUEd}Ûj[]I~b۠JZC^UN@m`)VYJoܸiӍ[]I~b[%-!Q,/ 0&; sKm;}H>R= j~)0J؇cyN/x~#R_ sKm;}H>RWutto֦8EP/4vcPw|,饒sܩ~9w>CP/4vcPw|,饒9 +J!zd`)T/Mq@46TU}ȥ[dsYJB,珔sEv."=֥+[w̯%r }^ꪏ婜eZZi{__nyղGe#OU{Ml\'Edu7>P0*BUfCZeO+K՘:+Mz) 6VYƫX WUIt5Q 9m51-3_+;4H`\ڽFwPS|G)D)n֖6_bx#XO`(9헧'0`p v%Ϸ-o|''<ִ0Ǝ 䕏d~:&B澜v8IUf5{CY 0DFΝ1{*SXSfx~ܨҲ.+(p\!_my,blT΍8(P=HU{g`h#%|`EMUU>d.?uɤxlggj4]J{,im A&9buׇFh6܍sbosVʶm>U3ׯPfʝwFW#$B*|G)LMہ6vi^!!hmx4OAݘmZ/O7t}V4d0ϏtyΟrsd}ƫ:(r~sSed0jaIƫ/L=@]r~!n"+eE?ǝVܦ\Y} }Cv[_NYfy1-gbYۙ}[ujn{y2O}n=MğX͸['Lj'rϷ"T>U;"]?q ?x2Gzq 9 j~0֥1 FP&3zٰEYtpKl0 &hc-.>G:a'ﲿHT\*8^w3&B|^w7!{0s_:)ן ]3O/7<E6ܿ@Na^o-EBq9"m+HG#>oKZ~o-2?O)37_gT;}HQ- ۠qvϿ ]Ӹ5}63ިi<;?uxO"w* "RD֑y/HjrxUk5.~sFh|,v P_0嗦Jtz iF슆\KzLXδSx 6o~4Eۺe҅@6"b[^ 0yp%\6:'_m$ח@6PG.7I˾-2g?om"6u}V( F4шO_5 tx$p_3g;Rᶔg~Q!"R ,!N/[s?Iq!ϛ]Z6.4.r>sfD1LJه\>#D-t[Ll+^`\W0}$zq{? Q̼ǿk#*y>*QW=t ?&.`)T/MzA(.ip纠LȳvPxאb|gpCr D/l)oq43ԙ]lӟm P B pmM$6V6 )bәm eipQOS8x#ˬg@ ۘ;'h2;B\^#ik_56`8̝*S10+EIll0r Oy1rt*9ϢYfG,3;~& ~4t xs35UwTg#*6h[:L?Cl}M9aFs5MinE F߹:h(>|WjߋdO~%_ryH.[}[e8f'@ KSa^Lm|{zHiE/hq/YaK^}[r!jfY?xtU,Xh҃$0}ҴԌk/]oڍ^2C/hޏ{ARB4 ^yz04m׋ٞ63-zT=hp,4EBu5k|^LK|iG=sifݿ2wSƩ,v`F`D|h-9-\H, 9m\)ˆ]87,Du,E>sf *I jҿ_ƣ]_kt707kZp\ӻBgF^ x8r$FXz*yxBLe*Y}o35KS^XJ7n׫@P/4U)*ވncnoܯ *_ugeۊS_niO,Qg./d;_jH_~i//6ƶ(!XK%3^_0嗦œncl*b9T>jH_~i//6ƶ(!XK%3^_0嗦œncl*b9T>jH_~i//6ƶ(!XK%3^_0嗦tȍ7`u8iݏWl@}d>$9T>Ed}5J׭u?^mPT}W%-!byNt}K 0 +鋌(("jH((=  (. 00((0#R (("jH((=  (. 00((0#Rv9/3((jz "R}OK(( _XA0oG@}x= ǃ@`)P8jH/, P{@ |a7#R q<_XA0P-g?=OQ5mk:x= T߸q(jj#R @`P5]@`)Z0(E0P-EM"Pm@ |a k`6 T EQӵ0T_XZH/ʗo*<ԧWp}|qk;5 o7ZڽZ}c`|-xʬSmr?ꕽ{䳤J*y9vH?KzE}v=Z3Qmq*U?7.xUaY`6 򿰮%[ KZ咽?>SuGb/%t 'Ez6YQ/xvWp}\+GƕS w;8g?ŠvmD~yUr_o`|SyZ昴9XNtfFK_]mHt6~˟KU܏lUeM0JҀ J$f86d7%:^9!Fluet}:D %::^)!Ft[5{s? 0Kz:`,dg|"|zEF˕i\K1ϛcb€ -M?|fP~MLQltgH8TQ=F`GP*օVِkix;qmCO ўnxϽSHXx:xxa|m9lGbBgxE@K:oQIۺ, nϺ| ߂OeCt<$-{W-3~@w> 3k)`|%#{[;@.z cro\I!Am4ly Bs> V}4:6/(ovI F_M#To跍qz`he1>F>{59gnfNQswx@JOzs>g~#RKKy/miܵi]S/WkӠ58 uo~OƵ6:m5ƽƪw2Uԃ-ރt<>|GްE塏]x4pr?r>wAXVC_签pErrIvJ%Ni͠e׼y-GS ;#mmw=`nY%9L2N2, .N*c䀹"Pm@ e}a!HFW^# Dc:B B{/~}() !v!N7A?y2`xzNb~"Fd؊ {uYU"!Pl޲=4d޵_h<{BƒvzAv X}|L0 `u%Ol 3 4]ٓ{U;`RF~i u/RO9%֠ hN!_(,X-mϿ$'j˽g},8$o5&aL&_žny#ހ5po*.m;aLh#9\֧w:Mn[*붡/>?k9F/~ or{; w۽'WuϤM}v9!9Ael>u} /BˡvWrUԙ2|?}+r>?( FPV !~#KKCuhsb \k'5b E_~`togWZBR50?p`=z {T m^l#[OG׉׀)$mgEi`|lm3 AT%KFH~9 ]踦~LOH@|Y!AI]`RC%ynC{/kBh#:_: f": S/a+9s<<׹E~imPnZy(@^(|]&}f~6^McԛV[<%o:bMw˟]goًO&4M#:1PQp}u]Zmmk7>_p"Ὗ4c0~FkCe#>}eXh9YU^O2䓛2a[f]4 0TB_Xa,lĹ-m>[aWpIE {\XKtҰ"艡BtO~v: \@+g&\nU -\~O;m;9,ڸx Gb$w~З7xe0(9m׸Zd~{8zôO#tqnr`6 ҿ`6""kAC0,x8qxH  3DK7x:xF vADt}'?g]xën&ߨz{>$(zyϋpIG\FM^?t넜OLMfQ/LjՓ8!{ `)VUvSGyU=j#R kn>7nxL#|m\炨 .8j|dH:{uÜEͯ:/EJijp)\Y 5EQӷ0T_XӋڳBOѠ":\?H#vC{\C4.+ /P k`6 =-Dm԰BﴑHO k[:5Z}>i# k`6 T EQӵ0T_XZH/,BAQt- Fj!(jj#R @>EQtA0oG@}x= ǃ@`)P8jH/, P{@ |a7#R q<_XA0oG@}x= ǃ@`)P8jH/, P{@ EQEQEw- y 0@`#G2d< y 0@`#G2dP%__@}m2@~`<'dw\xh؎1u˺Kݘa3fȌ>L, ;ȷzI>f} /=mY$1Ww\@0,>;C׺+:8"$:p}ތ }N$ֻnxwEwb4ڻ\xdH\h`A8\>?aXN~ NϖB95w?g|:arp =LQxH/ "SlPѽ4waAÁuF1?q4aDJ'r!׮ɵkO>vYtΌUn ?f}^.(`׉ߏLCZݔ`5r;$"@*NoA㍲KOf,nﳧPg& 0Ean;BM#x;=wÆke~$)@"Rqm{mGz4PA~ȕ34$ ~yoJǍ01/`k`)]!}{QOwZ8F, x׷p{>D8!#xܴ;@ R 4^ E{`am ?8צ8.L Bt .& 19y:IACߖo:پG J̰^Fx? H (0/ȅAaa!^HfrNcw|5ֺF5'ßfrPQ1PNwj$N' @۞z? (-"l8]"|Jk'Bֿ.FgOo(1G2d< y 0@`#G2d< y 0@`#G2d< y 0@`#G2d< y 0@`d77FoFΰ?U=[&nvH-|5-m^-6?gvu˹v$d׆w?k_!jz6y .v>{NJ {+MBnPѷU7v]<:xW~D~PΏW:AyY0|h?X*O Y8W6cLuݲ}7f،2cmkjpB0pӫ ! Z'Ѱ"1JӍ&#'G)n鬒t].|ȵ˟72dܱ@K5; pyo˂%?3]a1ud1~ҽ}^e^'1sfXgfs^'4Δ?h-yh0QȄo?i`q\^Y;G6I`X\xF20f`rQ6 A{Wk| ,eww]G.C{RzP7e?>FSsF<+tZJɼs7GG4s-eߏ7/?'ԻϞZ CN!)`x6, 0f6t^#7-7p"0zchaK Xss)`l O闦ɓ#kƤr@BX@G/  5 0IF䂟:nq#3tkXư7y8/t?e AJ%%{xy^RSK;DnJ/56зNkҳ33졗Q;@8 :rg>Z62N39;kÓaBP`LC~&jy?`Mf̿+cWkjy/&]0dµ%Z/ɾɽ2 C7p@$\"†u->RDN!Y_C_#3#37#c]s-==MuKMBvI&F]fyWGzlsٞ y 0@`#G2d< y 0@`#G2d< y 0@`#G2dǃC~:,~ zD=}BQT~u?HA/7nܴ~ ro=F?z@oY/H`?^`)`    2M\(䛲ŎugdXڟ^VsdZqO)USr} &40`"vm]n'a}[Ɏa{( jQ("g#aEi?D 7Ѱ6h=2aEd|{H@yjp;+-Y ei; "s"2xʿCc۱]5w˼o OBzjY#9>~s['=(f&h/09ԸU=Ui( [[Ϛ#)OlkL͏.0mLoD> j9kiV珯:hgzbKfi,q%U9޴ iSGD^y3||ЌNכt (o6md(=L8hqlcʝcAd\yϺۉn7ގSb6*w'U)* 0 ^]%ibF"[ٿ0WFt"@jpi[#vք5ry0Yrq44~`ם㘷v4Zר6hNGHaN,ܼ.%x8ojZ6󣡊ij`O`8#7LiA[yW{ܬ; >}mhn;+f{ӱsakba77ɣwOͰMwV׵7cϵ,Wt{ݼ0}~L<;~cڛNn^/=O0}[y{uclkYN Yooiq@ `H4=\=:r& 0zI7ȻE녑 /4c=(U  %3z`['Ȭ~h!ȳ dMmܻVNs.^)s{;7ɝilIT큡BИ`*T6")?wIH׆iDiDJNokI;,=M܆}s <'ί%.ŖN=+m@; tntoSit.3Wc^^b=b{Lt5|FOދ`%ΛVzޘOA[|{̕,șmߓxGsv{`]9IX>r/\o>N+~e^~@L77}e\VlXo9:h#FJm\j C@ϖ{dqt|kcѪ.>7 i`3U.Ӑ pily\z4}NH׶oƠ6T͸lh bLҽFJPC݃AvuI)T^cL5ZƸEC]s]N^k:bړ%w ӌO/x/p 0e| <'ۏ) tbn2o7لB4mon:">IfEuwݦ]>fZz uxsMW̺bύ̇yo E˼ 0F'qhR{f:f !S7r[2in#+= 7BKƺi 7"$~c*)G<; RCwx -4w4M;xNde]tӮ"R~ә7i86FгŌ0Y(wLSIkC[>fyM0&3_nT|1ƌinǺNJPûvO~%_ryH.yx?50W@ud ,ƮHˊŲ^Ěr1t:G_HrL1&4t qd,鱁D"NgxRJQC*w֯{>&Bf'ds=~bY3%JR޿4 K~zOdJ$ofsm -ܽ_Mc7U^O=x%>^V͗7/esujVY 0Ɔu N~5lGA&Ŕ끑0(*Uq@ Z`b?^`)`    zA@@VP/0vȍ7=F?ȂEO*Po@ z+~ zk/E__P0@`#G2d< y 0@`) |<({Z;{(ϿEweiHAnܸi7@?[G00-+G@ (   `ȊFd~c6u``)`@ +< ==ǝs7c00h0 ƮHXQZ#@`)LClk{J3%mհ\Ly폮w <"6.1~R}?P*쐖ˬ6Kw9_C7Wd璕.YݸXfݻPVq=gʆ}ݲybsY2 voƆ2szi`G|BB`qom -㶜zqyeCw 0o0I~bՔMP| {˜3^όvJi7[f.!nyϒ6ݷ[(+^?+f{r2ߥ+;_[/ ӹNYȾUrƾU%~izx4aiz<氽GPVqB|E{d:^.9ˣƬ'ddܹ-X)MӆC7b6 63dHn/5MCeY؀r'`fqt4?[+eeHqyΧ9>l{htj0H(r.M&w0(]&Ov`<@ eA~6 g=rQ=g.H .=6z`$'ۃ#W/PF'n@OzJS2y={&”0"FGr8n3@1 t Qva=&b* 0O*Qÿï{6/"N1G:-kkSʑ &.. KG>O6AvYj?r`ܹUpG`DS=r띲 wɜu<;/;S^(MvSVE£e#ȃGmTO)yكrNjH @jFO6ܱEz=S_UGu^i0*\|c@`)|pU-iKn"O-~lh\#-ꃷˏ[,Gu`3e;dibuBYv>& -r7띲zlYrEl[Uұ:e͢s%'탱}P&i:LƎ 䕏d~3ݶL  P Vil;7 1n60rlo_7n|߹¼^3eE%Mi/5qY ^C4SiOF Ƶ6N `RU%~izx4jcoP{@ Qвb\,~O-ȝ qFsVx\lyQ6pq.`hz`[tH 0.ˢf;ee3] Թ4*W? 0.Lq ?gs~sS1(]&Ov`<@ nx5m).U}s=%׊SV43 } v`]as^b2:PƓAJS2y={&B e}rJÂQ9qA^Yr]/w.M>#.H&9rO6ߵI:ICL0GEH<=X.;t|'\y}o0d78m,gi,ˆYzj[G5.~?4`zb#R(@|̼w4,yQ:mhpSV S W=r 68Xx_)M~0Բ{~CzYS#hbxvkeh;$ףvY`eQӸ_dQ`T .+-׺eҠg$ /Q^#0骵?:gsXqUk|JZ&d7\jnj.v?|e!*700j`9K69:p YQ{[eYbU2qk@@~`lX[']Pg]H Y~@ R@`?^`)p@VP/08P@ +@ {Z;ƍzd#eiHaA`A(J?GUߕL4 y 0@`#G2d< g )HaA!FQT~u?G72c:/kI%EQWYD#RЃ7n{~u?G=fgԶ޾8&ee?Pl>~s?T?yq zA` %K  ` K  `ȊFd~c6u``)` U]N˹1D@ E>{Ȯ`lG6wI#Sh슜U=2:EE)`ZgoɆU6eUrjEUfG׻GWGÏz?R嚝 BKWN_00E`X3bߨc\h;e2Zٹd8>ٰU.{SEn9gyFUGu^i0*\|c@`)|`-!-//Y/fˏ˜2gbze2e@N$ d~\~`o$̰~5ny5xZ-BnKe y{<5ȑffުf*~DHX);}i`l?XfhK"d1q̺wn; uz,}w-yg,[nzh, am y2ϿO6ܱEz.ѽwi0=sxRhAvy}2K_-߹k ViXr[x[><`gҹvd>qB6ܷFگQpu*7è韱WJk,!0 "{WÝVAn`2Ӛۆȳ dMϻ3xOwI =u}s^|9ӊL*KˤW?L{s#R(+p"l~O_Ãxp~]5ƀA·uedC:0e+z[ iEÃبyEizoD^$G L}S.(u*q޽u^z^[/lשOFz/h 0rôƨOT`c샆^[nMҗk^Nܴg :PƓd45.޳w``) 0F; 7z[ SC,n7S?<0JioNgy赕|5k6O3=8ԧf*`<{+}aN!)nt=,8wA y~/ݺ>*ip2Z J`!Cxm<=/af?ƻsOY~Nou jm4ܿ vO3jo4>ql>.ݲaO@Gw[, WJS,2~Ss[ske^p1*.45xň~E<z{1Oо^쐦g`jY\3`mƧUa$@ Yi==mEa_Y껲?F 7nܴ~ -K#h@ #  `  0dH2M#q<@ 4`Ȋ.HOq L"  0dEE+r6VVCP{@ 4`Ȋ׻GWGÏz?R嚝  4>yMT85[dNBS ;Yg?&O6"MWW{jHaz}aI\J5r;20UM#d>wl{Wqyq򱽟yo&0y0J>P*쐖ˬ6m9ˆQ3+}3qƋ%eÖey‰8}MҙSt 0| hD~~{l"ԸL^xI@:0 Aԭs}G *6^.p(A</;eE9c} _#SGCO!)%(g`׸x{A^)&00#˝KO!s#[ݲf9ʑ 0ԒYzz @u\^HKػvY%00ETPjg-Sӿs\~@MUktnj"|? 0 04쯮+e5v&i?#ޖ/.%/J-r ch&ir78t>Pf<k-zp>seZq]V;[f5./7j L5 grkݲmi)}?رNW/bVmUh@ YiL e]2dY6u MmF^fi WƭC+HaJ6ɶ-/HuNzX04运_t(@`ȊƆu N~ 00h0   `  0dHzACnܸi7@?[G00>,`DQ]YD#G2d< y 0@`#G2d< y 0@`UadvZ0 (s?~B>pΕ-.ev;cy&dP $\>,xyPc +; Ba1udQj$y3f̈hL1 <ΕCdֻnwEwbP?pl|u C,d - ?!>,A@C_ 0Fw+@`|-Wq,uS1(`]+y6zAnG5_'k"v BXNHGF׮ɵkO>vYtΌUn ?`xfh{VH.B A}?(TdƟR>{jIB #r N!)`z`D·{zJY5DA"rP LßX´sEzyN[I:&ҵI2(W|wC `8a':,Vv܅>[C/Ј40y0e{DGu@`@^z?['Aٿ]Kg"\mH<l{dEÉlp ,"}wNd^zIx!DpMp0|^cao\ O rNm z`"K2d< y 0@`#G2d< y 0@`#G2d< y 0@`#G2d< y 0@`#k@Lyo jwludǰߌ{:>{rHX5ڱPmrK?!K[C@yfmd^H+MOE{/ 09IaCV&>0 I FFAfB=%$"w7lA{Wk| ,eww]G.NB9q!@$,  }}\.:*J3mt*vYtΌUn 7D.@(#H "9W&jLK_OovD{Ę=0vXTH4DǽڻC ~. `q{[ ʾo?^*2Odw{=$WQVc>w IX,>6=3 !q^$dD oy y^oU b~vwSi$9 @+$OhN'NG}ްsF=+ =WfbP%xz4AB$8'B &Ëċcڻ?ݛݓCe*"=b`?}P % yyp‰=!I8=( G\ȠloWUߵ'hƼ׃aw8 DC 6\pC-׏_B!F/ //B iLI0 A 'wrzgx0v.LEQF?|=2xmPzouf=|0js!CR@yAG^ } 殁ԑ ^p 4ĆyyME^x@B3} `];O^.ީ;+#_<J6Tp{=D"㹞 @%虡eK"?5#5d< y 0@`#G2d< y 0@`#G2d< y 0@`#G2d< y 0@`#G2dޔ 0I;}[Ɏa{wco\owiowJ7O+M mrюqM,2E9~}iebO[Ei4uPϦMAML Lya N9 ;!jXڟ*^>n^z.&ӵk#_M !YZPXO 'nWr=AȠ#@@".c#: 7 5 3"mf{ |4(##Wz[c@"Mk@==0" ,[y&聑 b;]%BDB},u" 'B1'H)}Mzb7 蕦0p_I=%О`!>h7Vb* N`T 1L#t 0on,RCl$x"y|IqhyVtN%POdG'5À@Òz8遡i6'<_o`ig^1?h׷(>'F)'nw%]bVv0M} G`oT%x})AzrsaDyz1=N!FdGbį8N#:mOwÅ!DR/_nFf,>'F)S5O}ߨځF=0P  ^s#%NEF`@mMN!"R"P[@ U- _$|7鲬tx`)]6zbc /5`8CO [E:5:ZkqzDTH{CLdk@"D{Foʨt:-&.; y 0@`#G2d< y 0@`#G2d< y 0@`#G2o f|>|& q"LVYIENDB`pengcheng.mind-test/docs/aicc_cards/assets/mirrors_upload.png000066400000000000000000002515631471400134000250010ustar00rootroot00000000000000PNG  IHDR r_E IDATxyxW&{B HHHACm/j9xRd9G--(K QD(k@Bl31L2l$!u"sߟsgIj^~V:@D*JR@SzVW?IV/{b own,`0(((Hr=4W%<'yIRhh5eee ]BT ŪrT%W`` `088t-HPPA;0Z\nHhPrHj*ld\?\-աkڛwM_7uhbbbFѮ];Y,L&ѯhZLnD:Cd5Y$sef}wUJ*srlh? k ôUTO! +88XVUfYeee2L2LM]f>\K Xm.Dr16D+OcBuWd=wMJ˯3n\~~~ T``؅ "w:@741Z~ ~:Uֶck&ߠ)_~VYU2;ъ ʼ fhmndztDYMzkblFKrRu\)mu{D5:T~Ъ5h=2QavLu]5#>BE&}W#EEuWdџ_ӑ6].! 4hhX, >J{S$JGU7')Qǵ' VhhhSڲZ%?NԷo_[mڴ$Y,+++K{n֜5Cv̬:k&ϣU:E-8:pYiNlc^i#Z^*oB@t]z#myX5z2Ũ7UҌ5G.3'[He7 В %$tSD۶:.p0뮻*٬>L'N>nz ѣr`0(&&FcƌѡC{nYV4Ofk1LсPX' U=<Q(?>/E^E>R1AzS.6kUTjج m;5sn:ȍxnI%{:zT] 4swJ\sU<Go9V)s~P;5=r2 CR^m|MZC Vh4*66VfY۷o׹s犋)ժ޽{{k6Ae-+WN5?;+bթ2}zL[.z/RN!e\끒rMT`ү6SfExzn۫iNaGcd U_ﺶf3^yLOܽ"gT1?[L*kMWoIB+>^g;{K+Cv7?"""K.d3gΨGPppJK绚f/KY{.T躆(cMzUef/6+P_ 6_njUd~ÍܑYקs5s>ITMJ-lcdTsjTy Uer ޷WY+rJib9PˑAU;uĺ*sdhpIII۷6nQF)22Rk׮m&OB}g5s999Y7nTQQ$i„ :x;`uqgI6o 5jT\\>cpz0ͲX,2 ׄ+ @&IfefYe}S**ybU\*ӣF&F@iܱ2ߠmT]>'%c&>C| U7" 2[G')d!UA0Wq $.Ct5#['S\VJI7j .7}nuzX-V|>b!^ j"gi5jp7aaa{>LIIIСկ_?Iҷ~[XII֮]/*""B.\pj2p@O֭i=EDDhȐ!d K.]Vsn_kݺuVTTRRRظN^DF'x]y]s)kgG1ɓ O@XRKTZ]zU{UyyKx]vK.^C5Uu,t]HEa[40|رޅgϞ:u6n6fIҀ\:M6\1*..Z ŋ*..V.]С9w]vTRR#G[nQ>}}-zsI\VE m|҈]4[ۜIv ~U-o{T-@Of,jm,w/4#]\>W;W 07§{ϸ6 OUo wsPt=t|TTTB 01>J5++˧=Ծ}{8q1ctqرC/@9V$mVŎc={TYYUPP6@'Iz@mXWKtnt&I/_nappeXo߾Qdqb mdX.5L7$ ~j')?T'Ϋ@2/6OSERoհ߶HI1efhjukNkOxy._:VW 5=ž+%00e<ٳ]رqff&L$;vLcƌх ܆ꪠ@7nTQQc<`egg+1H',,Lڶm |֨X,.y|ڰaCS`M.Y Ƚ_ńSFFi_Y75OdJoI!SÎu|mI=J>W#>UxS:&x.LOU1Q^eOyx6"V9U_c9R1Q$-ozߪ{V/I9ʑg5}t.s~ Vd2)##|KJJR߾}qF瑩vEEEO5yd7N|]C1z]Tk KeXJ 4P[x.bUXё;s&\=^')ј'5izJlsd6_4CoO1 ںI=${ mZ>'UM6۽*j׊6:I$*t*aڛJy^wo kt6]IM+V9P:{Nė`\ddd[TT$ɤkusv7+..N/^tt֭e65j(^m|,ЂԫW/U;g6uQ]v */BCC5zhF޽[999M]R4r]4[d 0(%_iWtK1V;}MGjSwkz]ÈZ˨Qԯ_?9sF-r9l%%%W$O>qu';;[]tqۜ;vڱQFd'|8n2ܹx/ 0@Q@c{&+((_e˪oWY5p e4߷$7ZzC5V4޷kRrLSUzca4`kgCBB^_^Dޒ̇._&:*_j憒_luEGGkڽ{Nt iׯ]›fl($.A)k}Z;j SYY6oa\wK*X4$_CA^o?Y-V*}P=p]xQ_KxNu[re ty߿_E| 94f5UΘʵ2T3H[/t zd^Zst6Jvv u U5tuYbF _4&_UVU[,6ٻ@Kg2_KA+:}YdVQU?ooA_3iOIy`И`x^JxЌtt\%&P &HwZb3Yu}.O BE+蜅MOV+ CIEjp POm?H1SEPl_[J*X͂b_SZ@nOioOA~~$?ŪBU9f7[K?s;XVeG"?Y%ȪsxO [^|w O^|k%2{]3~k65~uMIn5}hk}1P7-4'jl M]MU"@cVkSW4u@sb-/rsV??ӧ  8tj\$h\XuZΞSؘ.h2.'-ȩS'矓lv{bu# r>zoH{L&PdvJIN^{+**KZ%@֮R ~Z;8<ڶPHp {u' PwVEdWd Pl86u)L M]Z,2\ЊI]y.]$Z0$YA:@D*J:@D*JM]ɀӚ`K@3B'4Vh?j{ogWRJv@h؁nZ9 [ R;孟釥Kd))iR !!0A9NhY~{ܷAw Q76KI~XhY~{B7$Q7|>Zx8uN)zv,{jB^n]sfnھ#s$-Xۆ8Zz枫#Zz>&cmXfݧ>{j/(ܧSXXXEEEm8#Ih{P7nvsQ^x;w~[W8$u ֜UK5aki֕+W=_- FsyL}z$7uNG6{`l9 vIQ(kgZQi9iN IDAT= Z߰Ya!c;w~:jс΍޽zTXX.Z,IpBK\c)I:p jl$[HwuIz5S>&Z9"mt1Rf|m;vZ pbuUA~ZAmTj1w$<#@WE=էwOZA]^M^3{QzSBBjj@eԝ?8:k:eKbhꔉ.]eg$8B{uzG':.rm\V;ԁ9zg|G>bbb4$%*(c֮Z yz湗uSl]̞vR9wd K{VVo$gu(=~GN>#ֻWvVIRjJBCBӭmok#5lPG;gFJ2lPo7_^ՎtќӇ^R#CCjZ?\]rvR%Y_zf^|aNtWUsu?J;~bl4 saܵWŠ!m=ISƹ1y0@W^S4u1h^"W.SStU؃oRHR{W967?y/<*9W5tv(oج{PK4ijWWޠ4uDZAyyեs|1H:͹`Ỏvqq%޷侩vR4S;Ȭ³GuWp+m3Zw;@3A czXQu|+u}}z|׀ҩj=+cu}_.U#i:v[?NIlnÌWs.Jf.P~g&ṽ1@W֩kۣS}]5K)ާaǘi^m$)uȟh˯):>7lPu;{NN^xe y)7$[IǨWTw#SC ց9.ᵝ;ʿSLLL)/(t<'{/&:*tpbZASLB9\Ãcu\y;k uNlGVw )[֟}Xiq_:]f/{FΜX{d<<=X8H-y/iɍ!Nns37JLqj'8 [j[,e힩C:Ud{m)s6YZ43KjTG>\F _u[E|(uVB(CKJ|ALT *DWקؖQnq6#Y6&7CV>,Z3N^@IRWƪ (SJ~ʶ39GϦ)EJ7DF:: zxή6!:U}M%*$[ɪک1eΣSޫ9|VM*kj9jN,̙=KsfҪqE).V=o.;2}G$UZxS_:9KMIQi/mSo4eVbb"?^aێ}BCB4')`zݬH.}KU>KW;ԗ{ ϵ ?oܫjc\Ut*9A=ۨ׳_ٺ%2ȱ_strG:F=rκt;WݞŏjQ\o9S{xHG6~]djԦ&MT0Is';3nT jdciiG'HI5qiNaA*("4HW*Vus۩!=aFif+뽊NktizE 9o1FjxAksWq]H]ֺۘzv鐤:6m|\с7uN"w 9wes$,Q;BCBӭZju}]g@gzpb۰YF%%vck5tD(6T_\ϸ"WTTVא.9FҁޗIbUǺ:YiXK>4"ֽNv]ٵcՈgP/EISyVUێX[Aiy/W:źn.Q57uc\mS{؟Ǫ4bxV~Zwd*.bm! mSǷ^$]׼_StQ#G \rgw(\*IiKf%VI%UV pSuDLnͷ2NC-Շ#\wu.ۤ_ ם=wұ.ϧr͞ŏ*kzjtARv.&u^:^r2][.J{`us:Pڪ_+htueK.qPVacktpVT\ێuCx(ru o 5~I2V slNy0uJ3#뜏UpVuoZӻ# vs:j>&ʥQ#~{+9vNPܼ@M̪rS5]&>tY )RfVEι^f?"-z'Kߢb+e.Uϡ>צQufL-FWZq yzONA:O$7O Z sjwѭ򚈂:>bW\?km+:EVeĵl꣫n=XXm)cǑCFzvU9S렓.oeD#\m4I\jaV<-͔KW.oX+Yɒ23]t}2BY Oi\mȔ,rbB'H*"ҙK$u=UKER6ІӔi([]9V쁹3 }@!5}-zhL=TQ/lTGWs+uS'_jvGO% O`7RtVTV[U9+ZcjFEZޛ},R򽓕1|}bMj Ved(l\;5a tlg];)1#C]>S7e(^I:+u{^Ũ{[^tϒTѭm'7$[x߫tI.t_?V_:VڽTidI)|ʯkkjYS^Ɵ]4{x.e^sҞ3ck*% 6_׎AN u)yrMW;ŕ0: I蟨Mڷ=+M=we%:8Ͷ$1Y߮UzW*9* T4sW}45 c\",#zc OGhbzxX<x[6Ԋ6q}A6'CiGRܼ5=I3HM՞lHS ק[YVw:x )RfN͕+XNYZe>Kɿ[?[)Vwf"p@Y=8ʥݖW|Ϲ]VW8&:hu` yS`sFzDoOWQ{[͋ Ђ[ٕ t6W'2+mqjgruX\p'7r<1V:I2%)YCSSr]ܿAKw /@QR¤ ~vܨ jb9Ip보h\nv]*Cmc꼱SFHEST "wl=4jQ ]5cwf-)NqfRjbr{ tۿޯȏ6kgmխOzvjUx:2)h~jGtZ,np%:8. :lҋa5IR i^Q'K+l)~KZΣt)IgVl>X;o*s \]q.uc51ee.VZ$u֔rἍڕ2KgUp}Q1uZ'=T7vګlB*6i&SߘGʲ6uuu\:4u@Y  ~~_S@pN:AssZ:|Ly} A@qsy׳ee @#2Z>-K}oO<%0At !!$Df>X|,}=Z#GQM])>$CS@S @hZ%tVU"@hZ.ɀӚ7(thKpc+U]g5u h:4u H.ԁ쳺@u_b6 ;!~Y=`HR/f>2 NKukumoNv÷v_~Ҷ@5XY JsM+jP/@g?Vl:k5ƾP3 T7Qdg~YYY:]i9fڄ,&t^ JG#kj><'pʗv:8^GOKB j6u hA@+P]HMe&UGӆt[ݠz)~eu; 4r$a:@c[.*A~ج̢4:v(qϕXrIE/ii<`ٯP E"ŗ{EtJc=J ×1SWuNۺپ>nN>Cԭ@ 4抎uʪo?w /ھl>-xd |sySBe 2 2I?ڶ ip!8 x._-i)w/4׼8:kbj'"@@c"4 a:14t#\+ES!j41BsQ1NtZ&P:heN:h%LZtpkiV/>SDDTl:ӘiRIͲ kt_gͮ\-Ws\ /K\H3/P)I^@ρAmyޟ}v<}j@@ZDkJ{@{æ^#H-:h+FPT0:Fh{[j#D-:@@SXa:@@Z @sP}kCB-Sco2 adXƮdh*Em&@G{$I}rZiiCn5 jO tpsk&iæO4sjOwhUWstHrr8~5:V!> Z\GV48M;,]IcC5a?}z+v#8A:BtмDUj͚~^t߽aJK;zs$X U#C}GQP5\S ^S9a<hU"19E_cC-?h,;Cs~{:59X 9"Hְ[]%j5>ƯdhæO? IDATS}Ch*FXe=eC#D:;[ڸMacGO@I҄qceh~ :+g˶O޽zZ鹹u`Emئ~k>X>>=w>нVGOOI;۴Nvrskwf>9~I85Q >)4 Z3kC~ 6{Z3UKLN$=6T,^(@qZwmΝ>m:|Ԧim۶ZwDc5vt9x`tqQ;76Fu$6mzvGh Tp<n5 9z^CAjѡQ <7,f}5kZm>U#p<t4tu68ʵ0]ek:*!8hy@S0YC#$zX]E$HX!:xwkh=ʿm j7p28<@WyJ_|GNW4C5nտ/-Nc_K.  (**Rzz,c#|M#K4R?_ #zQojB} Zڋz4G^ `;{@K⢀ |/ h~V.?vM/7UAQ/ 1 WWj-QVpgRsh\\\VkU[}S$A>WosV\57F*qr|xh ~мի]!NｳQq>th߶VVj#L|ح1+W5}{N9 @͚KsEc^!" A7sM # ͓SON'nӠ; =^6rKBhȉ4UrDЍZxOjh~+RUL^S$lN˛tފjbN_?3]+*ʳ~Iʴ{2zgȸx WD*ܓc0zWW [dhrm^yWg A>@k{e@ZvQ|_<~f {*RUqIҵ|wO75s,U/^n0S// 8P HR*Ǟ^ rJ:I{&@*]u5*I9Z4lB[}Z6$%] RؐE._&++G]jv{r|'R1 7Z&Gb|> @kSUͯp:rQvei]^]_fVjU]\7sRި8m]vWDPW}*e&RZBvեB.W_W )Vt/99I_𜤣11Oj2Th^P x-\C,I%yVoޓW|#4\\B'#8hrZU#KtR* bYA.upaE~Y}t>4z=|/Kj A.9Il+5(:&WCRI!0T0.|l:d@VLl e ҭKz}B=2QIl烞Z_?SʋHrr@C٨>^CUs |$iKM+ЕIEm5 _eI7P2X*0W _R$}٭ITM ˺$dz ZAqC4+b@Sd,KmwuW9ʔ]Y-:jHLQf}6/~@nHT$98(/͋Ь2bHmT 6pc5PG>tZZpi+ewT3$++^&_>1P [}*`^>h>v_Z.Gzg^iekضmIj`Vk*I2j=>R/^Ep9$AQ|Y5C;*EhVetLb_TRxfY?1Ze26)W;vrikc7T`,{\V#p^6OH ߽s>6،eud]*g''999ɩMX栝+pj:BsFUqr~rw?BH.//_NLx|}z}IRA5nWu~U5|hmCWka>K4:vEdizotXlwX7|Դ*6m?[a<5-_t/U~Z&Go__o͘6I'MժS^Ϳs'MՌiV5y/]Zsmi$I02.I*5-_=UO^:i{[Wr$SPdjN>KW -ǏH$yG4y>W]Ѿ$Io IR O͞@E9G}(LJ{F.Iz Zu'${ə96[]9h`:vE.؝讅@m؉$:):y*WN\^֣Smu tT/xjZroh,֞?j?j=7@m@ck$Ig5hxz64[վj*.){$=#$s<9ӦmkCjzsDV@G%ljfnf)P[USmy+p]ب4&CE%zoz~tg'}5MDȖ@U1\v]{nЪwyIuоD@Z/t5p%Pk>^y7/SAS2ԯGwsin,VnY93 uQyƍm1(_Nʶ$Owmi_77ɾZϳx@~z/9u ř!P[ŽC{l,1±Q#%4́@BmNu}9Muq6w$IruuX%r2_tNOOwK>S7[.xl:;;)RSmNSRJ=9]coY3d\l-?V Ym3 VС~ܓU9bp#`{9g/ gX vWvhL;7n)t:vNuD#T2_4jx']׍Ŋږ&ɶj$ 򔏗$)7HOB',W%IZsN2Nk/`Ҩ$ZL dnUq@kw!Bkvtu&Gnt Se+T/Wz^ڻ9},+7}rϲ[6jy0|O_0a9VuYi -S~FwnT@ku &t >@K{^VuhP|s tpp?jtd4J:wQN{\I'Y-'v| 4J! 5˅,=4S^W ,VǞUs5^M;j*Lqׯ_מe4+sw쫒\jfƌWzGiƭ*.)ѿwWWYA$7/_qu7f? zɹwV9B ɓղ߿Z%̺M%%εvhޯ4>}VNNcJ;{NS@e9h@ VUX|d_֙3gw4!IG*S PUTTd e̹~||ueߟ˯{WyzzUw9LԿ__5}.dff!d0d(T9;9G yMII'T\RLIR±d=yz_km&*8GZ ^MSAC(ۥ!.oM-H'IGݥnݺH9WTnکcG s/\}، k7* z?U<^u0ZV+Vg_ZZ_Js4,Ϝ9SjXk}`P!ɎThך|1oo/62G@^>@FPWΕu$D/ѡzH+= sTеT6qhn[ nݿӆM׽"zۜF>u r8.@We[V 0Y[5 sB;:[;Ѻz5WWsĤ"a<=<4եKgwkWIj_ڹyyT{|t=GjMʸ)m}x搆N{luqtH$ݽsTXtC׮]өӒ^ԹouJJJ/WW Хo]c#V93נ픷hq>5Q@>>zfΣZ}e*..$ 'O[Y8XG_+I҄wk/šZE__\Hhĝô*,,Ҧ*7/2?Ywf^x7k$L@kP]IبuW~kk]*>n{c<碾#S.d)7/_Աïս[Wkڶzwno־}ly{uT`|%[yyrvvV]doo/ '@W%+ulSziKzH7/~~7YA ̆Yނøhϩ[.zpDoMի/Ug۫WO5aծx\_~97nЙ36c]vΟPfE]vMݺv>'/ݳTrb}d>>ӫW^~W{B9Iz7\jo ժt_iiڴiSt[*n/lZ$);ev}sQΕP"!oHMeok~fH]Pmp5ŋt4!YNNz+r%&++ݹgΜs~!wxۂ!h)6s?Oz?ݺud*,,Rm=*Zyx$?gXpkZ svvVw몎=)˱Oɔ$Iyyz+ѱ}f|| nu %ݽ4k)<UNE}v]&} } ŕ Q(\!CmݔttT\Rbp֤cF[;z>K?" Wsrvr%I**!@z IDAT-O*m]7-c:t._b ̋J;{~^*,,ۓaG>t]i~ $II'|zMv^?1=Gq T.)2n}8o,o/9sVVժkmmV>>*,,ԌS'u+ʺh {I=*}h6(=ݠ$I\]Ag(A]%I[FޥW_o_оoD,[+gT?nr޽ʋ7n،|mPƅLE߬]qU\Rݯ?Ox+u ЩPދW>8LqAHMԭەmVƍ:]{<=ݠ7n7'}A?T`uwW7nuUjŪC: ?ͱ{W`O|hJ=#G,~ =Lo߾V{cNr~}eϞGShMN4ik^Zk$S WkgҕsU.p K=uЁs czꩿQ\tE}._ m\p#3j)J>w@1_~b}m RfNo" :=Do%GxQ+.*X)n^8@?k/h%\]]ѡ{<;'G_퍫0~߽tYNwןÚU28Xo!թkVݣPHۂӦM U.^-*,(Ԩ2d\΍ d]*g''999ɩMMzA;=5.ݶNc?ڥW?^:"D|}j=Hvr򔛛n*_XXcQwyG۽7|F~p[rn^^N|!\ꫯ,۔MZwUq<hmUWv:@CW @hZ%tVmco)2 qƎTށ$!YK}Qx{Yo424D ;~r~JLNfϚZu"ԧw$ SO[;ϸ;PߟNGQ[d4VXwz$ |rrOZyuPAa6EЦ68qA>:,I5!}uY?WQ53-:;Φ4zԝ:CɫfK+9tge~brG=4kCrskg` }3F={=_#4mޮ{mBgObri34e׮{M%%Iԡ[coT!;;[׮]SiiiߦMuA ~SF# nnJNN})}5zG r~҆7l9K IvtQRn5yٹ"+7:9SgTTtTV}zɯ{:MY}^߼єgwvvJKKս{w9999xw&%%%Qvvv +'99n>I>^ʹknU2W-?Aފ[5 T9/H2U3|"Ew> thKJ93T <@=={+@WspxCєkMY}v_vAs$///eddkf|HWsۇh,묙HBS4a}:o+9QNN^nn=k%`w`}:M>:<*  4Y==Ua7ym) ϙ9995Xئ]9Fc6lDg;>׃W9JvSC;v`֬f,{/'7v6s>$B[W7vČ rNRKٜ;PFcIJLNcC'Ir+9ݐdj:{TXg_їl6Edj9[u&1Wx3 zG 𫲅qc,[}=X$o+jbmUVN_.D^{3Ŧ+9 SI۴W{U@sEΎ ]i͞5U~ݻi˶O+k4胿m$=c:qA\?n yuлf&kM?Ou9qڔ&蜜LJ=:ť_ :]["r{/dˑܼFʪK[g_Q.5!}:MZ-P)\͕&N>u$!Yqzl*&ԬIB [;=7wei(jB=*TP[m޳76-ک32x:zz꼫yJ8vVsW j*`K - B͎ 4We#雗hj4#^RZ0}ko[qmckP@d5Zdd[雗(,{ܲv#>guliʄ`RE!@W=d xa5O@(Irsk_ZﭶPJڵ֔ݶJ=|R %]3i$..I2$ea-;.׈ZZaWIRGOUK߼D 6I1)Z+wðIz`yZݥR[VoW&_Dg< ^ۿ`^0pi̝!Z^)Nt*ؐҶkZ6_鋖(Gk (mעme?D #M_Gg鯪V5K߼DSZBEs%IhV͟e?WF -X]V3ي~JUJ5(t|whn1i?XmN*eK7(=6^ 6hֲ% :bfn1pNE6U*p#*|κ9=EfPz תpzK h2>k4mi֌@icU̐O_ҎsʞGslS^)n\k8+?MJ\>(Y{OъE7GmMe]Nӽo&.)nEjf*B5Kk45&ivgɢ9RƌxM+|ABՖqS7~%S^ikn]]\ =TUl4]VQNAgT)>Rì  RXkiT?O]MIUv?OYW0~z5=Q1БUbm#IWsrѫZqƆyJ%7j>m[I~Mji|v*K2@)]^U>eKU6]}yt%WM[C-3pa:@ +\\2p1U3[n{$Jw^4vOuQhzISUU3d^;ɿY "]ӱ9쪿~|cZ8mܫ TR@JJKUZ\"I.J$T$ZlyHʮ&T8UeLṴZ_,LF[1*B\TF=_+zT- *(PQ-{` M9*M)wYײV˧RhՑA32w]պvRK:yYlR>_ЬrsM_t1s#_wUwt~{_^T%U--ԦkT<%]dnSNNA"ɐl}9\.(q|GtJ=|R׺S}PWe-]5E+6U|E߹Dܘj%,gbmfb]%_b\ڍB4{Ylh{5|*}'E$D)\oq*]/m u˩%]5/D))3J}4=]ZLmOH\N4edWŭ6jjxNua4Ѧ*d \Sѣ5kizzV 3yg(|5>ͿVҕ8wZ-6aO:KRsYµv\/5W)}ʱڥnOE(q`( IDATh\[O}7Wkh W|UƻYyÒ_ו(pM rVQr:Vqi;0R~)뒩\8'EGTp꧊OV3t_aۋTx4\qv%c4R6$_afxNr\R{۵Ay-m~;S p.\N}ֶB{|*:Ff}pM;mUnRpQWovb8Q=˲. [(&kp@q;w]3e>H3`U˝`1*O]E(k54 {,+L%&gY67gjcp y[zcyg>x>\G4ڷ5Di c lT ׼aZL49i@eij#JҍAUϮ9S;TjCu VB .@,xMF:Mg(:  8mBK2,F#op\޶D+ֹ _ZξZZw/sWk!ڗlxWofh; 9]{?}..߫u޿w⴪<W]]4G:%՜^U|EI޵>0w})H i#64g+6Ў9_UrXU2bI,Zz ]ZUZq WIҌpmڊ4Ϯ\>5/nZشĪX^;fa.,k$]KgΗ|upd}𼳐.`˝%\E:+Ct>kBg,+ɺdMAmGWܿJ*[+C(u&QnB'Ht~GW:Uu^}~HwW:]{Pӯgm}&@MMs#@גk_f۞j3U<ڟ*9L^UPOk uE7_ !2/2B;wQM&E"#\7@ ֳJqV}yv<p ^m= hokӻh_( TPͺZfHGtsgknfMu+Pg+՜sF?xv$zQWx{'; ~WoXj#0w||h @'ʳ…e oon@'Eg*=sJ>>U?7ڳ\\hZ .Ȼ\pA^^^:F{hxK0|ҭ.s"#sԜotQȈf%76ڳvBp*** x6@\Xh/ji+m\΂ti?~\/^l1뫠V=#@XPPP ] 0.Tt?FGՄ*+$_ @x ר+gB-;5V1.W6{Z:,-k^zs6`JGtmHtԵTG:yh :@qbi̇UYUen/9Vѷީaٸ1YcTPC I㔐4N;4όYN.j@!@}Y(%ܾ.cQDDoƸ'((PV.׾mWM?Qиz wT>]ӧ+4}Ɯz;`]eU[N70h]zJsSey)Ok^vۊh-%ICc%I;KsdX乳4wLIwߡXo% SRR Iv}ߐr Z:@ﯕY64/iXƎR,j%f\8ͺn=4#.[B{ouNP^=UYYvk1J_q`$ x:@RrTJVqzl+A'Vz6 ts93qO׶RZK/jn1nCJg֌jnbʿU| k5q]5ah־3U$d *>b F+66bgi t)B3JuY͘,$X3㌐cΨF'Ioe.uzKI]dw4 n\RCʬWڍ;JƎ2-]Xs{ɱRM7NLeKh⭣5h-[|o h}RS*564'IOWh9kQ%Yʍݫ/=^Tq*/a (}ɂ=|[֭uj?Smo[5t1n͙s%\K0ΞSh5ݼ/.u:v>6uΞ`.QW孡JtGR=:_=:@'E)tJR3UUW0@}Zcͳ3{&Mm딘0X}G=v~7F]=:-Q~P˯ݘoS|Wii dX >rX55tOG={Оx,@mO!3*W_%ՆWgHڥ5jo@xαOSg?^\I G~ܲRRF{c !##9 j?k7G$嚟K}5$jˇ_tw'6ieϞv 5wb- ch~aYvޯ+f+(s\ F%w*󓞚~Iw?MO{Q8µ)%&=/X+oUDg}_+`;o̕;lMA@]$))1U'I[d'InfݭJv(/-q㖕vvfMn e+2myY%pϮ\fB7;]iuRҰڻ*g*C RḐRk'OZoUY4DIQT:UNױJן?өh{E #h.[gB-UdkLkIG~>ޤHT]O鱵(h=;*B'+$U(te* 5gN7?ӤI'--Z?4"8Zy.{_ߺ70 l*Еg+MQ>bzlh+_B5t-_{wKsH*?#H=4{w+ՖoGOIш@4/@w@95EߺmI; A(kbfi\ f*|B-(IVӖוg^۽Vi٧4HEt!kmhI#T-CR!e>OW-"3%sRȜ~UxI34/cxYqC^ й9Vs/ɮZ]yvzRVQ2W1DOi|k5ExN}@K8Jג+Kd8Oo>p[qQcoKhŇRr!1;EkӕB=hlgokv8E6ƭ t!##$jkc+ZC46)|iؽ:P֊rRyTWέyjY4{ ;݀ sf Tx_Mz2EZ}۵fvW!)P{KG(HҬ}J]ڱ1Gohzn_̓sl㖕wvfMne+2mY+yyI7ܦ{yn*i<[[yNRyQc4k?g/_=4lfUQ:=iNhٯ'Θmy>^W?l 6@5R.t ɵ g=^%|Gڐ\$Ҭv9`$t'+@W(xRjҲI#:}eΗe%^hBFI'-Mzrb@A կ@W-IC:Ơ4u~۷ek.pkZi5~GsX{׾.%=o~7rU>^=8eMEc4Iiͳ!!J/˞]A\cbz|˹FԁRўFd9$elZ#{tފ6[h KjBG%/k9y7`{$%YQk[$*o_7sC$խr)/ɭM6̣.ӷn Mֲ:f/zj҅Jyv=4#QwfJ׳z0dּa >ue\)_ӞݫYH!Fd_!ieZ-a;#5nh+'cYN^_ݯ|<b Wvu-v\eEr):idɗ֦+Vrgk?w@%s.')gTdצ+d~^^?i1!#6_W[JVԴU`}΄SZwNfi{Ng$I^y_=m=*9^&~8uFWx{[^^2hlg]p]^^Ffo_(!!YǶo7o_tD[Vj3.QnE~5۶9.n9O,֬k{^$iڕ+}qcG| w(Uzt7{{﬷/%uFH_eU&*B7mKK$I^У<e,Ҍ_2O?FK?g&KyڭǙ.V;~'Zv%JWhoQs/,mҌY8r6Wg*(뜶1Y:T|DKm{(((P˖.15yZ0wv57k tVMz8MwT{n{uO>wCZU\=ltRxAvn)tGkA:..q_BzI(:Ed,<*>.VO,VATA=ӧ+4/z&Ks0Cc ]bllVoQ J1NKy Hҟ^^^ŒVS%昕UUy)}ƍe7v9'0zWb̻b [GW_wYh̕(gslx`1 Rb%%%W:3h0jFJ?c[;tI3|<^ر㪮>kwܺիWFII ڴŢc:+B4EReO>ڹkjj^.ӻO+ 930Mq30dTrfTQbI>/}/e-s6n2_Ʋee:T|D?1/$Ejn$kżYk͟ÏU7|={(f`:k @/0ymxPq^3gṜjo=oc>qmCB=kMLצUVV=¯wʪ*}UMv֥)_F$Q t.;={*g%OzԄc-jT{zG]vI˂;QY)]F54gKЂ?~[}&:sjVZtgW1ж٣.V/O曫4fHM}+9 ݸ[J'N/F[GWgn= k%JUTsƨ6 Ѻ7B *n3f>:>RݹuV`f4llIwQL S.b5cs!1c)Əѡ%JL;gj(P~7췢⌺[h9z)}z /ޡIzUxXe.kT[B[FH2|}# XR566FXGb.#bMt'cNkC(22gJ'ʛԏld,mCnkq:q w(>.|ܩO-Vll[v#@Tr6n{י@g,)ЌϨplNw8[zQcըnVLL,TllƍFjlaFcIλMևrn\:[~U t6gg;Id K~:>F`qA5#g;v˞. l;7omR0P-d%ǜ]5$q9CG3XEJ](-%uFָ̟#jX9l+_ 7iǎ"Q#7W~ ן^^^w*GЩlb]&a'W$4Jeۇ qFx@a7 \eUΞ@r?% TPC[-1ZPCh3h\/ΖupRJӒg7䰻}YNsֱ\ll^|53xqmlא9UAz} |/Ѣ*TT|'//^fN:T-==>pϏ mi\Rtm=V3r7@לvn;q_cm/G[GۅZXRj tXuUNYcK=U،]&!ʪ*ʪ*%YC=ڿ]@+_n\3,Izg-5td Yr쪚5d\ml h乳4a([ThHOݔ0DUg5y}vϫj禲ZvN׌qvlu.V]yBz7;MXHݿt kvc<tBXFe;v%P=Q)KkxR[ӌeر^Pm{)y,+f`f "&ϝ}EF@*42.mv;PrNe0BcSR/]Spׄc4jDY Ψf;oci m2jj:,Yo UO;gf?@ ~nP/a[:.n{^A.csiRqOJT%l2Ue,Ң4jD^|5`1QN^OӜm]ؑ}}:]Jo#? %ORKgQ,E 3 ɸosfޭW_7+3rİ?@gvFws~9MW+$ɷ[{.|[FWmmݺռM:s#@>k:;SuΣ;FCUжƍhUE@ﶞmS"@:%tNK[O.5ݿh>~ mz^ZO{\8#cρC.\ׯ6w].s_~uom<ە>l,.*:؁g]w% hmTtJ:@ԥ'vx,VM&E"#ګftgu:=Qs:pX P&wJ;w!@IN&:ϥzhhNԜorxN7j: Ѕ5ݺ[o9X|es?]*9?ָl=J 6K^e*]6Q_| m~u?U?egϺlsdλ|\wu6Pb&mw Gtkmזft\tvK: g)W@[+Cr6nRB8$.RT02g;6*fRR完YoeJHcZe'\_t{}[CRh|KN%2%+,厲Meu ]إ}[Q ;g"FhaZ-]XIRWYU_}]OWhxR$ 0~$k0\svWHHOIRd*> G}իW{!=TQqFǎױcPr$=>0dG9~e`mJ/mz6C]qGUuo;3I==OYΪEOUGEh뷷k]@J}z{s&/~lK@ۼsgPʛdwP,4Xgt״Izouv.k9vhǹ6QӯMfw&iPߚy/o꽯}խ^y||Le4̖3g*C0 =+"^}Y=1k$ۡ#P3Cqf6 [oe.U.9 4P|BC͍;J?]_yM)=mh.BoD铏o]eUf|XCG+B ~{qZ|%I!ZOV|\,9.y/$xCz@9qꌮ$/ݿfCym܎ܙ;.q)s{{NE~5q舶njLm9x>Sw?s\ThTtJ:@D)tJRxy[OUЁ\?pxGos [ݸߢ)ޱғ2YoxymooഭMg |tO"@l/@cy]s9}.Ga!=ێt kvc<]:*tAtqCyOәqgrLޢ#R^uܼztNS"@:%tNS"@蔺3/?O\gzb:S^.}w;m~Mw_OL-e\ mzx̅?l3Z.c%r檞.\r8SC{YUw_kxz(cJnN!t*th6xG.B=9 y$@w0E-u =n_ޣ5>3h\O @y/^ӲެEOLРF kܷւJZ:,-Ҳ te0=0^~n9su*'qբct`jj78.P^: {*Ҭcw@ۿ Vr弙H5 5qp6zuJީqh́;>~303C&4@E ZByS2T`+S+iV8kڦE"bjcWh-xx"K4F2Ќ ĵcV$~zֳ54iݪjjo:oEv}SlaNKR8~!͎~H+ϟ sw=+**(o~.)-1qzpdlqcguuKP``:7&QYf7^nUVf]Bnk9?8>wrLp`>-ǔ/l9J1Ϝss!sCz[Ͻe_g}\}X.1nQ]c}kchۇk'Ӏv$Z\xDC1el˔,;Uy)RoO jWT[w%@o_-',wB :Gobct|7]aan&I|?ĥ:t`أ&W?;G4JNY2)R[W#s$Eԇ;B kaE$x{+-u522B9Y>mbcB EZ#G+,,T;Mi=a0!hwV',Իy7v櫪{%v:d"itHsg9}N<}}$S֥~V6F:qn'$Y W*zo^h5m0[|\Uj-VrzU*--Suu渻[n4gTOXP ӖB E;B~LDUTR³+e7wU #{5*R)ɉZ  U켹JY%?_.;yKp>ͮ*zÊ7)> hw2ŗf?3ج~/wӕ u^[f)6.A N`[Ĭwe58WНc;[:iuelq fѨ/R{q[TJQQF g;8(@>>W I+1 IMʸIRچͶ:ePmMl_p:_΁Ӿ;/ʮNc?^Qn5za4Re(E4CFCQƶL"i7))qsrUUU:G/$PUU**NsFԇ;aETQQUHHgaanRh߾nB yn CoO P``]fev 6%v،5¼W;MiaM2QIK7\ԨH|7]A*,<5JJ\С{tqc^wŝNJevm:,_973߸0i[nא5xhY^aazAynYDbcHm[7$I ;_25…[n6uM:u Q?yD߾[ߢ]KTuZUgu`/ʼ4Q?_Ґ<vdZհt[rz3$5 nxXʺTs%ڷ :ըfǀ0IMUF jRQqJUUZDCy6r.g;~ꔉfP(5ۀQͶe%ojՋ/KꗴevIRS0~B RsK\RS%¨S͵VTTJJ\ms;UUQQilݳ;os A41cDG%>oϖit%j|3CsVyE%f;[/$0äҕW|rkÏXq1lIҳ񼤦cEfh=+IMtP -u -\2[k|1}?zUQ~#:/iV2KF5,YYv!,GF LWsUTTNڷd[2sWɒzm9B%] ;cVۛeQ\l]Qv###5%F#<.\&5U|WEEꗔ徣F*mf0ivm7Zvnٺ]69QG*,hT5!AΛv-]3UTT!ASl\ηLuu ZygN_ӷjz7c2eOkڊakkn/$رQKhѵpeꔉv/#P?GNJ*)<GFF(%9g9|;E?_._NJtXWϜ|eewJL͜~0ܹ<׊Jʢ6:IѳtAgtAIh@ 6{hynNCo[{C1t@ #?UUUن֌}dd-wYp߄qW<Jpox{̌^ߤ ͠Zss Ym7v*s|ڙ8'56nJא[.4*<2 Ra*pvmTn6wkl( $57{a*Xv_}o:$tCzhrfw#j=OKcy{{Xiz^n!ګf׮qX= mv ι ҹusz_p+C|Et?bv[ڿ[⚿j_ TtH:@D!tH:@D!u ں c|=n<-Q^嘠X-Ƴܳ[3>mvm>7h%m܅Kc+,\嘁oSqrL`o?1'OsGp1-R1#&j+|l9f{nJ5Q[@m]W|rIRIi&=v̐ac~?aEjYةWƶL~KEmC檤#NS֥ڝOʺT 6F2ݞk*)-wyN%)>aWm]kC|r]s[()-C3c[kpFV󻺮|QcjwVk7d&e]Ӻr5vxm\砹3'>ajJIiSQQ I+4uDKUچZmzں:%(go$)jTRݪJҖEE&+:TxDk%IO%'??nW``)ISL)%5/w$//O-_"Iz>oHv5',׾>;_|ys%IF*mf=V45S (?Rm]+9{ѓӧMl۬zyukÏٽO}RpPO%hmrBBŗUTT1q%)9eWbc4:j]g<=e]޺]s!ImJK]c#.6<+Nۖ۝7N6YIK]\l>@@SZV#GtVU)$8: 7@ᘛ0sRSk? I  #gˈCHYj͎4_EEY=W!!ؖҵWŗ5}dM2Qôo6Y3g< ~%//O j?J|n婨SUUumj  UچZ8^aa|? 9:u&=y?  p '{aWMm@;@EEvmjllTfmvＩ媨TJrr0~]:\N9kg#e]8^ik GOQϸs60ȸ\9{4n1|jAgx@iw&|C ze%| }sU+]͵4iT3a~h6}j2 }wd=ϻ z:tڝ .:<'I>ޭZ[ӡ#*,<:ej q9IZ2I}U/aNկ~NXV*++7nM6Y,xʬ"'ITW.Gaڝfe9oo}k$5Ukڢ>avߚ3ώڅ[U0녤cے֕So}sl{#X]IWYy]8x{G8kTUt cvs=wgFhӱ5dq}VU|?k$UUUY3uIW8JÛ:&tc(ETl9B޻ m^+7q1KTTT?~iM5}k@Eadl˔$ΛkWQ.f+UdeUQQں:klj*m9r eUTRIi޺]BoWiiَ9}C4bPl\*&,eW?}_bJ|P%>uu ůVLҡ#7vtZh:VuK{}jn~kWj +.6*نg1ﯻ{ IDAT`[/,,T+UM<9>?mi,rlooݮ3PiiTUUm8FE}R(7/#@#Z|O꾰X.Yi8rLΖCt9y<Ͻ嘺jc _liA&Y3;+nۖs˨fIɌr0~Su>TϯYM_Ruu&'j㦷z+N _VoNNJ]FpɶNĵ1Lu߄q:zHK'5`@+RUUS3w+뒤>{i;o:mt3ZѦmج?Z ٛ6eTBsg_<$affyܧ~CgfSV,)-׆OY}/,,T3g<`W/К~ϯ$-"_kժ3[N###{ We+Z6of;Vd9G?na=w]`9J1=}RxX',ouU"-ǜ:]c9kпTI^=ѽGw1G >Ss#gY=b9fchZ[} cWL]hY3춏вy߄q)I.+] WRR%ڲu<zfS*+Pm]r>SUU 7O~^~/*+_Vyl*lI{sUWר!JJùqZ0ǚ:eJKT]]#//O$'*8(@ĩܜRQ"5zTS;*** /OJ\5NXQ| !8P_Jjy; YYٛqcG7[J2[ޝ7ֹ#骪`sjy-4Ё=:t`vNL{}rijƪXGW7"ujk4bPmج<B5g󫔳7|%U(^e啭V7*:hS&6[S^mǵmJsq 6mٺ 2ee(-uݾt:zȬd=EP+?m+SL3gs_F͉k2ej3pa>mFFF(˰7BҊfCMHbc7$\i*2r5<,ԩRZvN$itH3+(0@aazfSmϬ']f\#zΛkm9HM-_hw+2.6FcO6٩j#d5*RcF^aa|?KΚ kVrܵZ\=wq1:VTlV8a,sLcA=%~:f<o>C_Q%e7غ/^UQΝչsguId²9-(!~ Ԅ zkmµmEL[µs jvnZ ݺXtݬْ%ڹ oر &cJw(Mp ;V7X=tڝ@e+C"@:$tC"@萺p-YKKi+_Wrohoj|?W)7f/ WL[}F~FFx,:*&:ە]US[Ϩ{zͭ'lPnn/(߷KbIV?ܮ}[v華VK cTZѨܮ;<+R/q1|=_PXTh_ӮJ~93y~A?٠'gk^*TUyjomx{{+-uNc3 Rߐ )RXq1nCzk^L l3$ݻ2紽TyH}:!ؼ?{+OWk{o;v Ҏ*s{qY3Sԣ{͙OOHyi٪U\p{|綾֥$oH?e.׸^i;aw/-t\WէWwYGOzsK8Y}`slї;<]6.c:W?RRSXk雑u9qW6va\݌9VzsYe\MzPt _K;+\Ґac\n߲uzt%|VP޼fJRXX^[ysqyڲuݮV}E56~atd e}IC\lΜmγs5BS'4=Wsf3CgN$TCW(o'x|ϯ9=h<>=\#Iɡ}ޞ ֘GT_7q^]ힳ Ci\9 ])+u{ltzKMJtyN+֓^?y\|zsm7\Ԏ]4޹b+F*WlOuOO1^~Vk̑96: ;ʮԎ]]S߬9hkG~Z毡}UpQWZ4>HGZg|RYi,;o2e[a˳Y-߷hڐ~Bwm%:#Ё=vZv:7sܼ|m$)(0i|ƶL-_t썛޻@g{->^ .hTAт'9ӑm%}ѣzzz8ÆU@s據LƷϲHjxDl0|=}M/3=2~k$ ]iyj#k6vwx9]Z]U ''YOZ@D1tn%en>op i_^]|M{o/;'*[l:+urRj {JU5wkkG4vn9pۃo =}YsiU9Vjݹ+W'g˳DzrvꮕɟjQ Jڂ^?Ul+K$34d@%>D\|~D3H7uDM2QRSnt$'Zl?5خ }v*/Ï ΋`Q)wwUlꯀ.˳V}cr*kMDۯ{=W-oK ]NJ]ٕ;PUK#л2΁>ݯ5uzz8P3vi`UV?:@#G4[ a{C^ݯ}mk NiiudkzѩV.ǺzZ׺?k}XbڏI2/\K-=@{Ռs<ۚc ]M;WR痴NU/}gA ]|z~S1erTpQou©u1VQk΢~5NlIK̿$|Z3 Y%T]V3+UTRIir>SsK\UU$)go&?S} czzs'<Va;þfly{N9rS;P-/ϮɜZ@mdKڕ]i>szEyS%8ǖ935`T3*ܵp<}Non= .V3*͋w2Ku[.:[uA!=Ty.RNrjܟFV۵,UCj/iWvb~kbNHPocp]6wg[:Wl54wloS1v*]|л>RM=ztfӲUv-YeMsضpY5}7v=2$I=qe`O#=2SN?_-[4eh`_Pk+rJ(!APʺT s IMU+25өkIi%-]3t 0kwSg遉wvW>=\ f%9Wc˳Bz5ӫz謀ݝec8*йRpQ_=ݯ|֌݆uOOko_Ӛ>w ֩3'MAy[5똑hyE=zԛ5 U+9EC{)_^]8.\pІDؼ< eqrG˧=f/f1k$]oLSo~fokj Е֝1[~!y:~ T螻Z7\4+jj0vb)땜^RS WI%I-qga-w\jhg\+_PQQQ"]~%;Y֭!I[G茀تCJ/YfV ]UnB=iĝ~zl+9u]:RSE~^gOm!ſ;_ͿSom/ш;4t kk)5P/yyv5O8P6^fAaalYeN6-Xj;v+cg|=jiH_mJ~v zk{JUUs 0vR8~:V\P/sbQW-\#"iՋ/i7$ȩꜫvҕ t^~%M^z#m tk70[~c8[sfSsCPonZB[qYAPogi٢jlBVhn3t2ۊu{ ԣ{gz:zٕ]nV4jQ̠f+I2<3 ;Pe:6gG.f]ceޏϪ>?wIouBFwg~3fLkuueT47ڞqF?^pGs{p-7Qx :sN=] 3߬wF9y~7+1LI/>c`Eկ־ՖUz|>a>=\կփo48ߺ r !CjsE1. <-@g0ڞʮk6̛:eH:t`YƶLmܔDx{kO%?Wةv9-?MN{Duuou 'îhҒڕ]{-]U(sd`?_=va >Z~Vkm#Btť*U՜jx KFzwׂ'ϛ4cV_3xkB>3ZʀfBs6>>ښYj7m+׏ Q{to Tiy`9Ưg|ɱΜnv7h%ҙӧ,r̹ ,tҢ5Yc[9ܢ5(3bP1~ݭ\d}}s?o9 tC+vBG…ãSHpuZJhw-҈wǻUiO t&t^hw.\$xjگ^Ňouf/C"@萺ZNc}Re9rgyt吰~ޖcza},nfv[q3rL^>c.ԟsI17X=v/.s.ue1-QXc9Қj1~>-0嘆s_XW嘳gz[=C"@p IY!(e]1uu;_s競齫 ]'c[25vrURZIC荴uӈC7$躜m()-1q***vB~.ڸ-j@xNՄgX[n7|F n㦷4d@}Cߔ?&+jTOC(.6FlqFFF\\h͐y֖ۯ[f/ETJr|-S;_9{$IiJJ\ܼ|=jI I+4uDel%>m$ivyyyꍴua瘛Y2xƎRNVkwV&'jdd9Nٛ~2n`~'fdHN`xpz)։04s) 4cD3y씏t111 EMZ0؂N9c ݯ^׺uZ\ٳ< FDkY"[ %IˊJM[6~g gJ xsp0N8.rAp85`E+YJJLTn&3*ƭ._S\ǎW^ 1ε54k̀\]]O[njjjV cLU ?_ZZ/[{I?DC"U S-8s?Ok<pŹ96F~':u0nDu㵝7JfhuJ7v\̻ڛf;;tݚ땒luEGE{v䚧1WU!bѠA=x5ۺ7s,EE p9> s51) ӇsgTKOobʜ2]/Zm4h=MxIs ?Zq͊lܖ'=v˞|keݣAz쓝3#tVۦ4n|7f ~q?SfݢHUTV+.~Jʻ}eJN:UTVtLԢOzkm=|kyTy33T˗iРZHUĵTvl Jejhh](nU@w#@5h@rgڌ!)1Aw](DI*mK VsIY&RAN:X;wP퇵u;#ݣ^PB}˞ 59#]!o^[%_KFRbjk*.h .KvSyVl+)]U﹇ qÿ%IZ]R"ݸ沴GΓzkuZZYf?Sqo6+-_VdrD?x|Hh3qBh JJL[ۃXM*zW#_P`+8(H Z]gFں_kxov̘1]:~$cuйߩٝmHw@sU\Ζv} tGӽ=U:Е~rsW;~|NB]}{].W>ro_'u]s4?@z{Эz9?ʾ(\Rph:|𵴵u_n󳽽}Ͽo|y*("(kgm}}p !(t.W.Q[Jy;8^mHw|==z:@@"@H ~3g\ۃXIYBz{@9QpP}((kwUidE];/w8Zӥ t<5ٛy7):jHVs֘Qt=ZU[;n76ٛUy}oHI<[o[ g=#Γu3VMCժ77s ҙgozlkS攇f~}]}U>ۋ歷>G_[6}C:p!}~a]jwq5רd>PW J'N|c:vxgNUۦ/wq Q۪/_}AS~t}=tz\l{C_WG璿lՏ|7|]Jʵ?ߥ[ ܧ~H_|vo/:T߶^w@=ݭ5ٛ5}Ff^*)VRbBo ~TTVkn~bbU%TP--GܳZvjw۴uJ}ݶw-ˊ4qB$^bhg/t_,I8|k_I*mw&XcyV;v\˗iuJ9NY5-+gn{muz,Ϫ ͱy3ZZRǎ7Ǿ~k>cP=1W4f~nkkS^[v@OdCpp{g_x_m7\x|md2\p9FΜ9N8jll Ӯެsku/ڵۦ۵n&{ZZxIz%ߑ+Ͷ:xArVU/-0CS\iYȺHO<> [ :6#a'Ƥ2Cө_ݧ0EI PiYfkUۦ6/07z{ \ QLLKށw#Gy|鉹Nx͞3_ti9W Nc_}?n:;{3+WGLRddǶ6+66FI >bMMRǦhUnU7xt%W3>:*RSzkuϝjUo{ϝ:<`?٧EOn?,KHXeݣO,"/)x~YAcSTjc7BnC"õgW3\myo?a4#tiz-9N.FGΓwUX,z<*t~X Uǟ77t9p8{b43Da$3\0ٸihrF3zX--GUZVnSIU~5FP&bQD`p2B5W\!^kɧui]_?2+:Nվo3+ߋMfOO{Le嚖in_ɾқڽQtTjmmUTm}g݊A0$sg墢ZmZI*%ߙ)w|=LED /a5jz| Hsշ=CUGe.7]-^[S&*.~S&jNS9UQYm+-+W:Zъq3Eݣ_F騯&{&M5g̖;cd%mT5ٛmV{rgc-0|~~܏6iۻ'sRIeejk:-^WسJwEO+:*RknPkkɃ˝ˊ~&gZ]&KHb4qB54*ZjM+zrgQJJ4S٬89O}ٳ̐߻jTQy~cnk챽j~ >} \2_}?:̯?P$תP\hjh4Rs <Êw}׾oS:Ko݋F9m|`^/_zr,]hϮ*t-^߰Yhj"sռm[+""\CDjۚ8!UI ^9)iG0EeOEe54jZ=:[oڽza>\;ɾ ;5i$%>u%_mݛ9ܿ%\劍%N/YO݋55jmm7KrUes:>gRIJ&{j߷z~%<*L#Iݦ-ZW>*m:̷gyRva}Y pI,Z$别WI,%ߑ~ /g$`X$k~g}kHdGiYhI 5ٛvƤY>`]Pp3`PWW6Ѯ""(e3_W~MjV3Umi9㎎:i{muzM\Taw MHM8N"IRkI=yvIG!D{t8^.;w؛3UۦjY\ _#G1&%&葜e]PҒb._wH+chm=hu:/-)$IQQJ<}OU5&kdN4z .'N۴wkj:iࠠN j@m:vSLmdC%Ǐ:ms ~?JlEcg[3cJ7FSk|<=Og{1bQ%ߑ0EMf-)^9X&v-J}~aaF~n5h@;v\N;:*|}ogFCCGH}ߚZED+/w$7ݨm5$WlW":GKWh劥湸%\BKjh&V񨨗|gϒ|] x3"]SN{vU2tCSˊeښ mܴcdCI|O~G+PR[nLij4eUTV+.~^X ݗ \A]m雒?}Jk4-+ӬO$s?#Q3=W|U$#0"I!4YAXa1C fiYaLiYԇdou!6+66Ƭ>}/6ٛM[̥C V{Kթ]Uoۡw*iTEGEj>زEr/s60YYrmzMH?,vY}F,)1Aw$WY09#] *zeQi.(Ҏڥ䔉.xF7DGi>s޳ʬTZun;<ϝ#%N ?;U>@C_tt&{f??t7\˩-+^:xAa""UwBƶ5kכ_ahKrbccf855:|ujjj}֬ݠEGEjTr%[InJ#QyV9NN~|iKppz^MMjU}}q?]1XK ۪SMH״LL3ҵgxÆ}S7m&Lќُ(/wvfR~,ZZꇓra9z[ym"#jp:]IeŋTZVm>fIRhH^|N }NViY˞ݯݢd^iz\ڒ?܌p`ioHvJP]4/-G,-+׎].̢g7x6mkիo,驹R}jr:Oo}ОҲr8|ΙiY}zyrI=s//ޙ?D--GTZRZ_~$|WTV0!!^59iwq)k\򐊗,4n>Ⳕ!RI JJL0q;,sY&{s޳%%&u*|y͛Gykk*~!m;I fBrН$yӕc%%&no-԰sDE QkkVY,ji9(IJ4U#n-$W?]iYU;%>' !B/x6ٛ5}F59#g 1wsmi9[v}boy;#ݼne> SsJsߋ7WiϮ*IRvl¦f6TTHr=[z0'O s"bqq IDATD}~9Ns9VNrm3 5ww=kDA{lt aȗ'/8X  濮rM7 tu9iﵿ~k;f̘.Jo0pN^z'*b^[ @ΨZ^/ZhVAjhNYxaߠ+v*^Rzţ=tW?Mi7iruaaij:iXGn;9uTmv6n C:me;2hN$m~o7k>6Õz/fҲrnӖ @{%ю;F>󷭽vhN81tt+āp;v\|YQ2\׿MgϞ5#@1t::Jp%:uw%\g4p`;rEWӉ.W)((H>wQt*EufwVK﹕'c@BWJK]ѓ˳vEWg.,'+Mpo@K`N)6iʷ+.~<^Se JHտ$wVR3KNV|Eƍh;CrtQuu!MNrm$mܴE3T۶[ ;,F4,fjM,._&{ T@݊TiY$iTrZ[O*%%7 @ pWn2L~X?w,C ==L:Tn7Fe/>5k77ݨ|k$ɿ+;g**}rgHY ttoUhhBCC4dHyTߴE<*I VK Jkt>8g@ @@7:t'HN 4-+SQfH=sV|ES4,fh@ֿy8Ţ+8!U'*pUTV+.~JJW)vXC ~fa/2\iY>͛hO C:zQUm~[Q=:@@"@Ho@qUoՖ~}{槭Ϋv u$gcA6>^{5P@5=&{EǶ6i~}H뗝oH. gk$)8(HUW PPPP/ #@Kb_0@117꺰^x p%r8ʷ|IdoֽY9k}ެK>ﯫ\e;N:??'0:B8Nzߺ|}.Bꔜ2Qiެ|k**/xViY{=sq8Ni)]KX8NeV\hUTVk?c1쀲sf[iYG;Ҳ./]_Fndo֏>(sn g:upE}Tŏ5ٛ6iGs<92BQ55%NI !.r8\U#+)1 By>MfM?Teiks/xV׽on?3Ҳr3;Еo-Tvl7k*)]䔉).~FJM[4n|9kXP9_9Ɛo-M[TRsjÑ4-+Sw*%%Yk֮W29eb9sǡFߩY9~wEo\_%\FSEhnf>/zo%%&pN9&M*-{U>!:s/MgϞ5_˼֐͙@e/G2]va]mUUTTd>PK^/,QtT"۵WJJLR;Ξ=3g3k.fKQtT**5[T'tZ\pmۺ^_Y/xd_'>B?_Z0|=XZVUZH'zkN^}KdIrg(,>$pW'jL}rg(wV54a͊QRbji9%|GV4VTV1>'өoҴ{d]P'i%ZԱ)kcyVI};=p:4f()1A5oUnjw47|?Dk|Uo9+g\gtZq{%`.UNIE$Y,=^.ԉBդ3nig՛}cil6ͷ_T}>}>ngϞթӾUvqoP4IҎ]܇C!'|)ڳקNt^iuօ7iwwt__~wv&{lд{$I5hXPIX-q U=7Shxsf[JUM{vU1sEe֬]ҒbY,r8ͳvMqIRLP@ $ g<6nRRbV3ʷ~}h9r7VJJYy*x>;.Ghr34:yJJWy=N֬ݠiwI\zjc}k.~ ײEѻvx,Xn|ߩښ.-ZZV#ڳJwj"ҋwj_N-^] 473s >""UmIҊh״~m]Wsվh+Qy}Nk[6--G6nڢQi}$׳?+44l?7@qsqu_,9O{Z=QKkTmK6ٛGhn~JS=x77;'s1[W~&g+/wԞ]U2JQQC<+|yMQx ^y,3'/@_CK`'59#]--G3'f(D3pᄄx%wGIJII֐p_P_uuQEhZ=:xAMfݫ1)ڭf] ! #t1n|GkL}ɧji9*I!:JLjwP͙9Vxn>M}'D89#$9' '8_Kufr:O*!į0oSEZӥ-U:ެ5 IS5D?<hU)%W07.~T:3>q9NqcTsZv~ۍfɿ+;g**;}ϱQI\WT}ٳfz%LTǟjrF4Be$ݓ'3\pڞ]U7Qs[Ee;vOH9&-_V#Gje6o/uH#9+;+J t#G)mT5ٛuQU>9N@%T x>OC ~M%O(44Tٖ;+GwPCG{.g_4tGp=^պ-:*RUҢ9e0?!߿E-]έYARpwV[˴~ƙKq3rD~fmܴE!!ך&NH'M#Xu͊g+WV! ѐ!7Q~3rxT߰cZԲEzkueeB~ t=I [U;Uۦ_(Sڤ9*M/x νe5n3˻^[v)oJ)UoѴLsIG\Caݢm[kmhK6z-MH eXݲE:hzEFMf[ JfY~Чo6]VH3uP,EDz{U2/5~f}d h qjjjuYY_F嶚Zs5k7(66FQz3$V?xOJ#QyV9NN~|ilk WyeXPվo44a{kSRb^*)2ב |}]ppK8˟w%6ðRU[SoVvl;\JNҽY9j7ެ{rVgVN3eŋ~RǍҘO+CW7xӲ2=EGEꑜb+q91ݗK^mݴLL3ҵg tn7q`͙rghgNZV /Yإ9')$ZIcysUP3f~}ò(,5Çzϛ~[̰a3kZVbŘa’U*)]JNhV(ͷtHM˺GSرQK/CŢ-Um'%J},Ϫъ#<骶hͳj䨴sm3eWݷ kJ<mEO<H0\UyTe2_ZRQi/wV54j4-_VߠARIGXBCuJK%sݦ-9*M ԏ33$~mͳvM|Ysϵ^!C"_h:տ_?:}Km]DѺRWW/%DÇa|k>f>|mNr󬒤K-)^+ED`=$3O^ IS3CSEWm?qBKv]IeŋTZVm Qًi~Oz8*-+}ٳ45[4l+mT磷;͹HWZoמ@j*<|?)"ͳY~sbhTٸI=s}*~ @7KJL;T7v[ T[SwB ]~!m;I fHDrUru{5.EO=Lk{;}a:~NN>g<߿5$)HAAr6={VgϞՙ3mmm>˂5>sݱګVzxtk2cV,-+װZVH{mu):v6Z鮥n1Zsf?UTVkzI Z`g_ϗ_b4/wpXEejhw.f}9PáF!jmmѪ0E--G}e3Iijm_dn?,IRM ~TӲ23T^}Sw&*,,,do +^nwMfMFMHٿBCCݟQȑ˳J2)f0EVox^}UО={.ƺi:|TC0^s`C@{w<>O[W&8l_YorQfϙ9~+']ΞmөSti˿t/:mӯ_OiLmw3gtY}k OW ];=ֱ':msԩNۤi[xctmiڤg[SN:κ;#jgҲrnD:ji9 \</RoW+ohsV3n~66z˻vrkǽp#@py!@wyH| }ɕFXiQ]Z 7$pymmRPЅȅuzb B>λ`X_܏ [+ $tDR}՗CC:Fg=p+ $tD:@@"@H $tD:@@"@H $tD:@@"@H $tD:@@"@H $tD:@@"@H $tDo^ۚ4̬MsNv Rd`#⁊<IDATx E*AQ fJalO<["$զ{B \Wzu|@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@`(ѿƕo}'n޸1 b8y8cC? :r<qƍ 0J9[o@t$I@@t$I@@t$I@@t=\x㎜z0|J5<칸p""Sq8ϯRSw'Xqw⍿^+[[@tС׿8rg|#O&""Npzu\Dޛ&__":~RZ p&ZٺZ+[rL`Kk@f&l7|QeF sg] :(miƠARZo-#lfI@0skilQד^9(6ۚ`t3h9Hj`,ptWg\L`v`Uq%22!@j2@l:kb F+{#lo`׸8 %k߼׷v_[k A;56`@g}6 ּ;yZq͛W `*׺yaoH`&":fgoo/^xk>0w ى[nmVy_F\']oŭ7kC5ʊBͯv6'mP:Q. E}:[OjQWEok⼝}^cmnVzn5E5sm֬c'^ eSyNy#([^3'H:S6Λ&90]:¸m)8oqŢnuL?WZL[܁h.-oWu!6SED+]vALy &Xn4qJiu'Uy. %.&m!:ٹ_ t6i8Yȕ`rM>XۄtKyg4 Њ5 20rLRWV"-? _xMeyTi[#N˸fO<0;:eEiۺOYDQm]E1==0mRj8{\tUUm"z!]w-EzA]-9I@0}LkEN[8 c͇- }ńl[.k9qLZ]-_*銎ds`֪dOt* Fn#<.r\u\FiSFikO|ms`r?/ " Lʓ/R&M@0"ͧUv."#v "ckg1u`ޚDyV/+K"")t]DvV-|uU8DT +ҍF]ٽ"tO945qeeN."UBO:qB;i%kUCT9 P)tm#*ϰY9خHDImU'!3}`t#5."qǖm 1Yʱ9 FjVp-QUessֳX`CTWu&ptn]yDϲԉ6_ns&67֕k?˝@@j]Dnql yU6EvWL_8L`"M-W-[⵭5 @ښYL`B"N-;ahuК>|L`bڼ6u*7nq߆w,H׺6: H@0QE 閺l8Ded7kPkYf%\_AdU nЭvvD?ݺs&n#B}W,Sm{Y7L`wt]tḃ`FF0Fm9yv@9Atk1v091`{ks>7&I@0S aBMu6gZ!p`t37  O8@" "HDs)$lsPP@ s, X1E@@&1S'RˍG!S *JXhB@@#U6(EvtA@_t: :(06C? A@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@Q%مIENDB`pengcheng.mind-test/docs/aicc_cards/assets/obs_prepare.png000066400000000000000000001574541471400134000242450ustar00rootroot00000000000000PNG  IHDR<T) IDATxy|9YX@6IP!MpPA*3*`iB Ax&HaQq  X!J4la ,INN }}]} L={4EM"u<Cxx^Ƿ7P_:*G/*ձ7zKy]묁Ȏ k'kqRYm^+ԁsyubASo40SϞ=DMMzBd؃ FEG]%CEm)9&'HkO]ӎ˅Mx|6z_P m;r^.W?JÃ[IFE-Sy:q6?hhx7@^/[k֓!շ b 0 )ЮkE@m Te䀮z}djקdr|XK_[[ oVѡX;.`AnZ_[jVh:>-.ZxO/b-yTrlYAOآ9-oTj]qfNܴέ5CK/ō(x\ް6T6dTvrw2r 묫ʷTݡc׵"_u ̐*"Kc<hD@E6nf[),Po VR,%IT_Տb\(aםZ7#FQk@ ]~SٓU帻5+(O^K׵\Ӫڪo몋WlzSY׬a~(w)R9NQJVR49 <=,۟P^qifNǯ^П.PA^'~b>U˅2i__[5"i/V9[l_[85[ws`x < Th \*P*ǽحBeYWo1*]Ϸu*OCRuˊooZ]E|zU #1}&VT}Yʚ~lB$,QV>Ŕ@qӂZ aM@nϛڭ'*Zz$_L|@KUΕo5|kRhmV-.vmu*DC\5D>-W~ņHRT Hڧ%a5^Y)Ye:jzbJU_&>!ZO{V1*}c;3Yьת]gga<&md-*Ձsy.QkU'j\.vWn*Ȣ}51OK/jw~sWzk Hef ۦ 6u l]ڜE.2BR#x~NgDr'bSa_?բViѮW#8i[Y(DxQ+_Yo苜swiv6:WP_ӉVPojFpKء-R PGA R?u-܆,FYIm %[[% WJ[MU(*rT(J\5Jh2\D?UV8_Ng#aw|xL(*QwSލbM<~EsX -|Sotj_kҾHEY٥Uɨ")7C;j^ئH!Qu(TPو ku  DEB*xR& g[#iIHIRԤ,>SxAm wD=QR!S-}$""f~۴DZ֪G|ɰjz 6^+RbWjV!0Ժ&EBb$+KKj3tppGD8+.u`p-{f+/ؗ}v\!Ցl߃3xZ@S\sǯ+1!Q!U jxkkծnU`&NqD-Vo Q6^X*} 3+f럧h yV kRӵn5emS}Kns~wAjɾR5#r>! >Fk%I=wFrsnOP Vm] ԳV/߮Vb7qb6)$]l$i*J!}~ 45>mkxmەWbb3Is͵ D% *.xLY<_*<P:ũO=Jq̳J6xLaI:y]}4]K(RL&aH%C~sRT},Mĕ\Vv!ګw(#1V!Qu:&0ger3^rDs(EϖS?&fEeHP$*1Q:k><؈9" n'kݥBMξqZjlZU ;NsG5>OaPyxGVeFU\謉 Ւ,MZ /G$Ӓ~9xiR&*dL Ҹ}$)SվߦkXh7lkٛYh26M:IY5)K)NQӊ]A(|YkRf$BU g +(=`ȰQ|' bV^/+7%r-k#ަ[HEt{5wLEvTW]󭻑ZY-%OHܕrHEFzxn*>oxMu@zi4&{;"P[Z{A>WV5Uzm2@ *PjX_,j_FIn <|Wl.0[5X衵Igۅ+,)G5Ӿ}ךz)bUZjPK_M:^lUI Ms-}laȪOK ipۘzej;TktĪFigs-}lWXjxE}BUSPbK׬gŪw -:e5|5*O& Ð HEd{}ManHʵBU <8uN=56s|||c6l6t M<[IijVq9,Վ!JcoܬvL׮ՎToT;3䑇|[Վ1:r9U6ճ{%OP(Tu_jnxP+S@zފk'{㊪D5M_ꇶ[>$z^~MvTj)uu'5'-@nua13qU8<pU)GB5F\9%{_Ӟ+}5::Da(/6 _PwՋ۷h`z譮U]/ѡ+{qx)z~8:훮K[ъA^AC闝si~kX"*v=5RC:֎V^ U *Y̊{RJGO+'"B{n;O>>j/_2T;>T`HUDKZ:>[@PᡂIkls *(0y.%̘_ϞVgubn}nmܔ\V$W3ԩS}yOUk*=CԮ][fh78ϕ׽{Wi2uvwP<{W[vq\&.צ=UC/C@JMr|E }\C: ԐL丹Wx=qq[/E T\ʵ8&#~9z?W~ h⎬`2/?o 2w[A.kEc-ZcaN݋G6eM9+Cl!h@]KݪVXYP"tAgueL6کJ[`kbZ7vҭ{O% 2P >L{Jܵ5ugewrC*yڸ)Va}_P``k((0PI.4f *Uڵkr@7."Y<_Aھcڟvlv;zI\Paaz~tI&x%_RN =vDٸ̝v~KqOҞOZL8G*4S/eW혖-ZdՎ٤U~[Vټbl< 7a=ig+((Haa 묍j\*=8,^8ҡ|']h݆?v:k26/?_ӦֱY0IAZ&E!Iz{ζt p/]RqV/_C]bBi;ح^]FǴHEI>Mq;bVLSɁ儮ھ:յM]*UnkӦ;Z 7mS#/J~fd '0 Y, k$0&&ѢjMjvc;H]:4Nhu <Ѭޯ(ï)4?٩=;#= HZe+֨{.r]롁s¥j侍#D.4fHmܴ=[k8T.^upT Ѻݮ"iMPSR8 d?q-Ik'ǔ'I {R^{G@_9g뽵@B'qܹ^FTJk:b*[l5jȄu3u_19&4 w rp'iG4.xTxŋWP`7V߾]ƕH'NwP1񻥫4iX޳WlgQ#px?S>Pq>Ij׮A:~ 򻥫>װX8眻2JBX?٩oZ)I ߻r$޳W!!Yl߱[7%(@Qe3ʷsηnWkü/}ƪZfl߱[% 5!-_}AÞýgpIe4L&|>jٲnSu+ڝ4_U^/Ua$]g7ح~_͘1+ IDATGj{+*ZVzGWh NՏDi7ٳ:uꨀZFo~Rs/A]:v8UKR@-dW rԩS~󻕚0mp!/?_Kְq ;a _xVOVk?ۿZ۾Kx"l_ᡝչSGmt[~ԦMZrZMVk&hiQo͹ ٳ-_O2AӦLЁo4ul4Ux/ĉ$I澬ig;i(ߟ-]Ov{ $Izi֫{*V񼡝C4OjίTS:v#$IݻwկfLG@ Ed:šzʹpE>>>1e6e2ђT;׷ V15k%Q> %jfj0NCm<Cxx^x<0F$~8m4M@uNd2Nؿ9 TY!Թ۹[-5X[^ӋuyE>7OSnauhr.\|ffLJc ÐbQIIC+WhO[V]o)y=Wk˖:s|5z-%Ҵ!uC1 z{A9ڲ5G1^TwË㯬?I_~^$yۢ]!Gp1}3pR> RʻEuVlo/[YiV ԦH%{a׈{*+ZskJE:puйMӮSQO ܄Ą!eU ,i>N;wUxPoM[{Ptfp-}AqʊEmf2 èۭb~3gcsz4¿f8o=C*>As-3[΅+,,TiaX,*))QxhpjKidm=Gg.ݺƒե==yMYXx\;Д<F@sen u<Cxx^:MpN> F-~A4*<ҵ2}Os-W9M |&cݭ}q՚rv"l,viZ2~"x]YswcJqҗK<$eoW6wZ*?AUNZSɚ5U9)-IcVN%k_7Eߦ4&25P;^v.|P*V~E\Z~kQaj ;43JZf{n{AuSҴdB-\3Ӓ4&m,r>]Git-C ^򃛖k&/Wm /UUF]3Eõ=tp~{Ah溮ڔ1cm!!S%iC4seEUথm~IZ1)6SC3{!IjQ\J'8zhX !Pek!I҉l޼sCvJۗ 'Uz)gH@DkL\? k'ҿVRH] *TؗqHSE)q>נ#/%I+x^ aJ/lNwP/&(]K*THKҘ$ Y9WPŻjhqFr4Y̜>QmIQN yTpVx8sC1:5WFD>vPs!IzM[Ƀ/S>8qTrzV?BHm2ɰ$O>')"aؿ;(I VB1j*}rm,Mю 7OsW+\IR|@ |WXӒ4&\r*,†Ux;JUЈm?uyJE6o !PSz=rBI;?B!g冿~*,v鵹3o3Fn%K\{减Iу]J̲e7t 5u ͸{(C4s]cTx@3M);tSe?W{4 [ri]Dlm\wZKoP~%[>0ᅾLޭ"ʝ4c5I_/ّy8$ƺ<%@`m-IfV*␦ip07<͈PV-]n۩m|oՏzڊwhN[ͽtY'ˮ4Eƍ{Ysm'ߣߓh'Νwf vjEJ;в9@BJp H˗*>$WCx8[cڸn61Lg4f|-uSJ?gB|vΦ,Ԙ*NnVEc-Q._A9i?)m2{}ޭh_hϟO$t y7uZjlԔˊxnFOP֮"GKqPTK]BƲA/S>H׸l>g;zu"Vk`jzܞGk>>1e6e2*1 CE%%%  n]6O?WAh<4~jxsSo#^7O\<u<Cx߆am0E_dPkph=lE7Rՠ /j< MWIudgMxN-ƿ?!&РVѳw꿼ztzzg+OZPK;AfiтPN{ҪwAVwzUx(~s5z-Kcwi#1R4w}fЪРOk-3a3[^~]p.AL>^%^>V.lld40 Y,(<4 v WG7̩uAZ7_;GpC!_[Rܢ+G/%]4S÷?).[k;T][eZ~K1AP0 w[.4[}꼲$)9VIf,lld40 Y,(<4 v WC#͕7P[׹c~skmp@Axx ɧr+<~]U`W9|zfC)mm?63J!-ny_R}g(H*7хb,THREЃPmVjeTHzM9Zp-({l9 R"T얻4`0 S~f. `x?\;sfW<1/"""""""""""""""SCۦs۴t<ΫoSûC3:{쒝:5!#sMDDDDDDDDDDDDDDDژp=q۴tH0nҏkRҝOaU[m?! Vz5&i0ҲpLNm~8o3n\aBk=ֶ' IDAT2[@~&Ԗ5R?8nt0錙HMm2w<9qn6:֘u :pzUXIs~Z>Ӳ{\(rqZA7`OHְfs5G!.t lbk9'\{ JYcÞ شV~(vp'GT5>8b0&5ہ#"uC}b]܂=;äg6S⤲]pڪx)ړN^~ w@Nely;?!b7SȚk$TՐ۴0Q &PWҖf;pL)ͮ=>10>3j7,Ƙ2sqdZ:/>+IծtX_G=\nj3}/BpcCӃ-3#% hS⦦MwԴTj2u;. hܫC?moM8(,g[w OrjZ=ń~.֔ 8)o/A.47B[)qC;`w4{(s6h^/6;yם'uhvIƴymZ36x}e7W<: cn {l ۯ*/gc[o"{ i|Tiy,Dμ|+_D gȨK_~~)"8<2ȁ7=/2Fqy(j|h;-N7Դj":KGG֔w:+e[1,_v͖Lo?B鯶aoa$_/'ɖ@㑣7a"8߻ 7454L!7B?b0A+;i1ؾhBg>8k700vX]MQf9cmUt<=2pjPo#6p'qC1y4},8dݶYj˶PC:MߧpSXxD^t[ǽzmKpvlM#NG72K߷GNM nm2'mslq綴]֑B߬St$SkP\Y{2|ƖoˉIJ<FpFyD>|N,uDȘaͽ>adM1t3YQQOiE{.?0׃ـ}\1;37|ݑe.{F*2k֣y|@ QE!z"Pϟ;]cIu'>耑!òEv3<_aS`//.Ox~Yc vik xˈ1"0t5a#u34{@G%(y$)ì)~F*R`K"9,"~ZܛO%OV`aߨln#m Nge}xv:q [`d~B:=Hjq>Lzj i}MipsGϘb6ZM߅J<{~`Ɔ5L /hURӰA-}K,33]ԑ +mYvYD#˺b[*YsI!0?]AS͢lX[dn\@l<`\'|u&AͮZʽ `@ M b )֖1=O7.#D`c#EFDؠˌ "0kMgh$6]gF'bS䵑ChHE-n9=遴TRI¼s}mܩN_ 20$?exdͽ}ý_gvYbS{98$4/X^ 6%%;| 3##mm5܂=v3O-@Kv1hr.k?ޣOhnN۲5xl!e.g M1dsӜz p[}1⻘񁣚gx/=)19䢚Af*K; e[hlYm|c`ti/?|pKpF?' ε}9ƚwӫe $f4 "9R[s iF煷H=]BwRvk8)5m``Ky}Ɣ\\{|/ ڿj2WY:Fr$v9qV>K|#g>|M˻g'uV ʶ*!$,P[f#5;\-q <=//GՑGbic-T I9M.ă}rsQV j{IE˽^޶XY}#ØMR>ʚ`~q`  ^n_œ RC~; 3fRDDDDDDDLp6UqF],,_9W{~H'Z)kݟO=,񧗞K]=@'mḵ,- dܮj(ⰣqyaHggDjvΙM 4M񰛒ЎQ[W>"LunZ13%N\ePwƈ4uRD:kص?s꽩pn5aJ6SO-tj8q.p0wMwGp#h<[6Ey_ 4xBGs<&g*P#HghXyJ@2'CygN{'!_-{kn1Gc`=۪VG{S'R` "tƨV;bc$/pC l1㈍tՇ4>|`SH ^[,a,6{Sp4S(/clccfx0AMm$27Նgc5vϨUdtro5.%| Ӧe:䛄o35Ph! {uk<a~ M${om9z%b\[4)1!^[ {v.5:QY_k̰ܳbazpaӘBMœ ES$FVkCHo o@f-4|+! \ i x8y_^$ m3O;bxj5 Tz ;tMf'(m [؛:I}3S7s E篣Hk;I;\8Wg7F%n&}jWTTHveՖW][_*6X:KZNoGOfni4tF@B8N*F#t5Β)戾H,s:)S%O1OuZ G%-TCJ@Q7ESVzʇWHj,2_<8mrH.\9gf9FP478|0hTj -,Tf d?e|P\asbQO-4<5G'c>HDeևѤO!+?Y/'6S8Qlnۆ=Nu? 7G:ȧR~N% ?1:]R]ԒB]UUtoyr?q,c2<w- T|y&A 5U~;_CE)&9L- H%9kNh~4qɰs|3br FojM}Y2,!bوN!sG6@;UMQuXҜ(vrɸ].EeVNhpθkPƒ-*gM@@A5t7&Ãl Yn Rt&IPIt#/sO+B&mNnb,\[P{u5Ic\KmTF^ 3k# h{udmF{/zggAPygx~?=w[uMlg"|KVp?L8ߪʹwa)1|oǶkP{_蜃f4^JƴynSOoΓ``4ы'^E/T>}bE. WDDDDDDDTq̣UD :2sqdvw f'/?H)t3CHSk\;-̛D{}5%ڶ\ ýnEUZܫh4ZQ+_ ~2ϬeNa;nj׸;y,J8lyo&Ã̱ܫ]QtbkU5&;Z=$?`J gnhQhԞ$5%!{U^06BUa-YdxȘv9?2COO ,Ω$WDDDDDDD<4d_?_#};yOf%kbg3$9."8Of>t \uGh[kd%H] c#RvFs3#kt&ߨk#ぃ\Nbe(-r4cgo| Ꮑ9Ԇ}j30 63[>CyC$OM i 7zkDaO0錉qqeX;O ;5ӱ::v 7;j\8 HQޜsom?mvY"fxۆl)s+ВCRH0VT̼dY:|zdHBD;n6Tmazu TҔi -K{ֶ\ƱS RKC BEm!7.R ?D!ߞde}Ҳ%hl%}{{?-6SԠ)-SXҎb?FRU { F;??sk6sUO}ɀҩ k}M 3xm٘CDDDDDDԩSZf6{gb ^h0+U 3o>c>A켒Rp.lADDDDDDDDDDDDDDDѺ/s@giIpƵm wڊڀ\2J06{̿ѣx%L v]3IA.6-[ +GڲBsW:fó " KHʜ{~k G%]5zwu//yS=z,KH̀KR,?PwזrӚݶș0L?b>oU* ˹6lFp8j?Ü%n/0z/ǎ„oJjj ɽصz(`Soѣx7/ջI!$ ̺C샀+\EϞ;{FKDDDDDDDDDDDDDD,PC ݘD"Lᦳz$L]/LOqP=f ;FL dͿ}3مKgdP/3Úɿ9qcB>灺w EDDDDDDDDDDDDDDډ

z篬xs/-~տqMwb"V8qlWCۇri%1rniyˣ>@?N5;őd Z.,"""""""""""<0.o 0h`?tO?Pw]җ{g]5KMM-GOgqŨM?ז/-5Vm p)܋ˆ fq<,7/ e1FVv]}%I=vU:b(y9<>@oŷSm_ x9)! |ԛ&I sH?LBBW}M[)߸k;?]Cc w|71CL]7k׽Ϯ*弴U/׳g"C c)%GyDDDDDDDDDDDDDDD<ɽO!""""""""""""""rQCfBrz&%r)3 =hq;e[ i￿cV\a xvVWuq)],/>O%7d|qSoӇ7#wf }_-`WEU-N:ulW~ٮ9GNBʜ:uFNUQeky?6uzgk{z#V#ױ{$p;l|9o|lW@DL(]]ʔM vW3Tb &ò$X/ppkC}%M'w0/\9A֥1_پJ''gz?JOb ExV1mZ#*JArߋ/es.\2Qx@gdv{:R$XJw&mX. v!6y0>{.DDDDDDDDDDDDڄD<f?]Ţ6sy)C_l*5,֛; `h@8Vd RŢz67U1;<=)"1f0r-.pVF:gp>`)ӆ"q\}SJ^ge}j) vk)0B}"4ŧd,K=8JJ&Aa5/==/vEDDDDDDDDDD#Rtx&3`vXUF**dat/꿔 lxn0-sxp+a< fӆlEQ8 *G(s)s2(:ÿ< ÃyLT 8<'եd0[z-ޒ@[.@UT 'UC7{55I`cbݱ7YC;튊HGJCG3/eы;`Q+;ìSy3Xs)wWF'ð LD* aLKg{Hyc:k-s?P *(eC2%d`|JŻm1#Sb S S bv@u KYTkoыu)9s):Q{)s53DX5m1E3:n?6j`]_~E nuO7SKI^(ADDDDDDDDDDDV<|ſ͖쒜<ȂYWEDΞPx)s g-e,#`>= ; ` \?LJt)7.4C0/\oU0 EӂXζ`yk;2AxaL泠ZU (liI$N\+ob{w>x3]?H;퀇 W1?o™7[cZr^ϑ;@"r<]`Lo_bFS:o1> Ar&d0{1-EuaJ ̝ bcΠpp$ϙ3(y R^>yNNg0 =k=  (>Gge!S,lLv.e+U al+)Bwk,ĪvPR<$\DԹv'"* ؾu_Rz2 Fex<X<ۼFl:)/#ς, gP8$6'_ɠ >(t9y%@a)zo-YXNo<g-eQ^t{NʥQL-:6&0/[&2ٽKG1۰ x """""""""""m촧9X9S=ʒGZ.co_1%,ZXJ?WhT\Is0Z2\>H3=>m }_Y`m+\K@:>.];>xgtg({F$WG𼷎_5!qo8HDDDDDDDDDDD:NN:u:(7h#99(C4GzvD:CG\\:u )s)9y$}{ V1-/|VZsu^=9'U蓜N5f_W"""""""""""籶¢p)+kU<Ǝ؅6q˟qɐL">;Gn繧\|Ц{X.$ٮE xvsi%\~ٮy.0.L89k ZDDDDDDDDDDDDDDDqN;w *vǰ`riOiq^m^{+cZ(|o?ۙ[9u:uԩ>)ޣ-s5(CU>2/nt@5'>.8:uRԩS466rIR: igxK*OL6٘H&*[_Χ5q)X dy8߶60Cg.ÍB~>c_gWEݺu4K CBB%""""""""""""""M eyi+\܊7W1I!e.K׮].\˜#y`]\Jp]*,r=Y_„^^:: /9ז#|(!H1dS)O5^Xqր 3Ç71IF:\Ϝ5C|?șƳ]ic%wHj#E ]k""ґ&rzЌo `xV !GkqWE׮]aoLRRe/,{cJ.IK1Q~9/¾s5G7ЌpY Gꝿʝw±e֕}#8vǎF;j*֭+WFV:L;Zz,n`0مKlU u^ͧ5G~ۥ[ ߶60Czeg;f/-ן8tI7\KϤDׯ܋E3&k HHcFTƇo3c/СzzLdIA 3[l``o0Nѣxy pCyv부횙!.I+9v쫀 ]ɬ"!\fï3,JDDDDDDDDDDDDDܤ0֗Wg\6l?XSț29KuMހen6dpcNa1>,FDDDDDDDDDDDDDܥ .\<۸gTK_Ut֕oPq3̹:`%C$ @ uu8v2J<ō }}i? >\1W-(AD*DpŬ/4~OF1mmKO-xZosdg{u=o/l.R.;"!틈HYQ'A|=.C<>FνKXF}b,W$x8@Sr]>~e>Qi ʾ~[}FsrwY[\}rdg|ԀQvQƾOMk!r&>hV0j ~LK|X5+AAFV~#mjj_vL?xǙ)Y\ xFYCaoH]p#DDڞ%P_?}f0[om`ꬱgomF;l`a#L x> ,*#8$´o3;6FDx*xoSbh~&w-}f?~Yͤ_z;ӌQ'p%i^K4.s/]`D䛬T_W7/ nDjU7]P|]c hFt/KQAsW=%gmVBX[FRC!RS"-TvHIrNg3W,! yJb߯ 7CٿVXs,f3xejJxVҏo&m;2qFGkn$8طjI!Rv% K^'}. ´_ emotEm+>>}߷׀Arnt'5*H=g9Ys&OoR8PCc޼UᓵWzMx^wqwZUƓ=5B=j,֔\ϯʧz> WN#,jWa'$ TiJI!Ӵr}M|a1O)?fȿXM4ז[!Cl c[f-֊e[|/#8/=E ZKy5Z+<\8Iأ^НT϶L HxVxjm)֨YO쯄O .mVМ?m٨}Rs#oN^c_ڛ V\tZ^=lK{Js~\M@3o_&sht-E5>37Q#kTڨeibBW~۫}NTCEJ<Ir$}Swx=u-zx"Dg{,D˓ErMJ2ZvMK(eOΪ n "\@a.<%WWWEڵsSYY U/h޽[n^]3J2]{o7re'. UnGvN`JwrF/^/qZHeoFPޞ%_*~\=ixy2@?mi@ pm6p:]xK. ԃ?`-;})7ّ$xf=˗khFۻo߾2LM:9Y=Y~&]g-k|∖=;Y.֊l3F*r⫚2~Tnxׁzif}5N.ט?VY@#4:^2 VWItIj.ZU[}Cw'?xιq]ZnM۩jY;s_JG 2Z׺UO>PO4js}ǝ uX>k })^ew*:&ۻQp;y{ gG^hi6_];̹ܔ Z׀kӸCΫȗ* A{ Th㼺v h Z׀k\KzK_Iڤzovjw0ER4P[ӲGyJFhWw474&nJamfǛ{xІM;g;}eH]׹Kt Rݣj֮'X9˿ܴ̅*W]vNc*++UQQrapni\E'9My=1voծK 2HÇ;/:Yo/gֹ/˧qE}iiZF>pmeeضԽN/Oa¸1Zw-PtXqթS%ND?6ROI:uDζ{7/]ꃄ/Myړ&Iz9v>ܴY&? {ﯼ:|{\eesWu>?ݮcq$iF?36_ w/j?|~{{3ӿvXѡ)n2-;[Hn<==$U^$۲7mokU"]Iݽlm%$B C郄L$V塦d-kWYYYܓh ݓT?4ã<$\xJruqڵk4R*//ϻf GK3^KtteTYYB:᭟ZguUtX/4[N8MyړӧPЦxSKA=xwwkTHһKV*1HUm)Pۂ4!I:3|XSCoMn-Zdg(;;G~?!$[uk!j9z.fzNK_k{7mVYU!PMbm$IRa tP}}~:|~8OLh*HXcqؾZZxxt}?[oբwxkK/X5$ ߥA TZl#5Lf-U@[l,)|\ IJҌMx4uIIZ8SZ:9 %vhIRZ|X䭩 #ԡj FPT~}iu~'.&}> YI;iRaI JunC1֎A:k9g%z<[ŧjB@CSZJQ|E{I]xDI)}4M%aҴkn(J}#vr;Xuટ?UZRDsIG$oͫ-ȖMKωvKraJ\ ꤩKhgВxV]pQمթ_w /gv\3 })f%j]l%Eh^~ffi` sistͷJQ%INRV'EH}$w5ǬxU<[嘲 /La .ꕿ5ra}rMIZ㩏SuK-#B?KeG!U6=Gӈk 0.B"y\YkD?1͎PP=^G7v=9^= e.<%WWWEڵsSYY Wsl/VB#u5}i-4OU"/[36{haI9dh:榨 ؜Mͅ$diFI #5cA5nJH҄L/ )ф rH *@@h<>pO.X:upuwAI? Sط̹rxx0;<@a =a5tk<+q\n+eۚV\#SZ@Kr}*<ݤo,},<_6zdzr=)xu9hîKK1k|} |Ai.mV+JR_7a@մ/M [zdvn#6IWVāNhMxpheQ-@iCM,h"MxEuM'jGT[+4~9#6;qXaީ.yѡb>u-m5Ykqz*cXߥOC 嚶^4㲲Y қ/_֥KK9JN@Ks<jWYYYtK>/OzO4׿єAroI2No{Ӕw^%+>d o51\]\v9TEEenY@5q.-[gke 4$]Cb&O}AZ14ѣLyBw k[_ZZ+k¸1QIw$[%+e2ꉱ6m">It^'+:uٻxзꓤmz77m$[0LｿR&F ˱Jt{s <#JݛiŘg4|X=tT]~lU0UM`򳵭PVj'9M?߭קXkK C l2}S©(m~}oȟk`_%+%Iݻyig.cG+wTw* ^Ԍ ,֒ȣ‡7Z\سowI6Iu߷ _MTxZQᡚٕ! ɷ tǀc@_[X"g:W{Dj<:tРenwAZIRA ƍѩ%}wN5PmaC5=N;,~ldwRkPVvQs?]vNǮZ<\:uD&IuVxi]n^<<:h¸1:|{IRGOO](-h*+++{s<_A{6E'5}Y֞4}V)<̅*W]vNc*++UQQrk i IDAT|<Xk ?"U6 J5ge}oV.Bᙩ C'M]⥴2#ZmVBzZek.UC5 ֺHSUUAXƥdiFܧEOEGZH{5,A.F҆TMPyJ̒YG4aeLE2n5_ɬ㮾<@᡾k$Y4a.]Q_(ڊcJ2c%2IciŒ,AD,U.Z[p R[\ aVZQE$8zZ*(b!5#ֹZƄBZaNJ }L~ ZRb\g)-"TISg+q*If-p^ Tp|4'Кj;J-yuy4;.7~R?++]<\'Kzqvwi tA;ȳ4.أ*i}A`+K%+fvwYܵ\*n}^Eڤd)ѷuDK-8Vd\կ@[b9A]ErfM+ɶo7,U&ˢfl-b`mT$Fx>)T TMa9oҴeFא:i8)1ߤcFk4&T-%)Q^"8V˨}U V+ɮ2OEzIZ-孨H)H&g)Vir>.BS3S5!P")8hfX#QT[~P6ꝤnV+.&y^)K*.FV1&dzv^TtAE?~zqCu6ᨓyɤ% g<21+RT BI W I9ge**<[Q>5m令2G#rOH Q FSTub-5pSN[YdZM&uq )Œ[* ;ɼ C5"\l4B5]fYI)Y W>?&Ma)YZZ.S]y|'FpLr=_ꤩKLk㾞Uպ"9SRQr󾺑<@+R q}k]vRWJΐydT}p?餵ۊe+P+wNN-3 Ώ*--f[#".2&>Ufij*S!)5F4-hѭK*"SgE~2^RTkwhQvW)9 n"yU/ᶟO)E}4=E9Z0_sGݖIolZ ^Za1RB}ZO&Z"^x$TTٱg53PYni ;*MnG ]҄ 5XL#iZnmC e9ֶfwb?&B,-X*pTJ0%w7,&Pi l,裩3--P>+Q5lQK rHRA"VΔuRȠJ!\[{W㪎WTznQq|$LE=$5dCWaڕvUxnf}~[xڲe4y7 Mt7ojGQuARVxkݕƉ+emuP>Z,ǫm!RJq1cEdYN|}@F]̅咿}AitT-_d`ϱCGR4!ڲ2I2gHcAcBE^[~pS~Z}DIw|=ǫ*^*C5Ui5au_P5̿  E~R]{g2I*E挳*oR"7zJʔ_xr=Αe%[`HZH4|م ZC5UxPnH}X")&j "bC3*q,U:i8iiWjiҼ2teiqjå9wD'ggu4)8tkԇ\-ڵ'ϫKV(}l4@H݂e-FnC|r,\]]"k|U|ee***T^^.s靛'I5&j ߶E*+wrzՖ. [oRW .)/@}DI*ժ8:ΩrN̟lhWű% ֶ a rVk>_ޔ,-IϏ(M&).J`99oJ F1NUİ\H瞝[n@-ܯᦨGܕUC ߷4 -N$Ƕ)Yᒱ5^7Ul3[Z%q5I!kN͌"{2dR{Ck;_KÖGn^]3J2]43ԉM9R]sK4kh]v,.cyAϬ>:7W|8CniV5 ɤis{M*<}g^?k~`neO.oXނ @Z[ WlV ;Ԏ*t.I]~&m)ީ1+imriHd𤲏tRgK^K3^ؼZj7l56ovT4 k|a:6X6o_A由GmX[Z>ڼ~Z'4~z[UT6񁇫r=:H҃j|Kۂԥ(ћuV~a<c{ᰰl6rNEn@jOh٦Kem˺$5ɣ m@v )hi)÷,|cϦIs?̑]f>mC6kufW<ƮyUXs/)t};hiq&/];|-J3GkVv vm]eee5zeo5%z|J{ke]0;ԣIe$j){4%WP/ OU..rqqQvTVVB2ЊnBMxں\mu"o3R Eߧ+_N|U3GɽIf ԍ֪.l9]:vjҐ=@ݚ;sl]㦇cR錒 KQN]s@wvVƦS*pZkO'k=cM&2$ɻfYoCfY/)t`Mb"%<߰ X=MjXQT]t7v=akb>t(iw*Qu{k&][}n1zCwh:(;c)_Zz|:vQC>ݫ=ՠ-Znh>/_Ϲ])B@ (i*XZV:&a"^znzk(] sXZu>uK:Yq$]YFUlUqC*jHjHj48`:vBN:j*cE#^znm֚ʷ|L{|UC52@ Cd WLTGV{Hqvi}t.Mbw|oܿF1(bHڔMulo>)feU erñJ3k? 㮾<@Atݒou{5ɿck͙lwyvFk~PF+9n?bikڌNI s@1Sn\ kkgT=B@t>]7B+dg=b%HpNw)A_;!NoW=qp>4Q ߻ᑒL8לeļVd[3$b"j!#$?vVhT;XҾv3JRGMuO*_7],2͉feES>/$O- RmVg+ťR]֬GV&ܪGVU[p]HU2)ZZ~]̶o7-Ub^3={0*K*6 K~û^;nUOK-'59^;Fא:jXik)h߱=K -%9?cjW̰`̒*30)aHWiH/:&TzdSڷRhUɬб5t,*_j}43ЊS]g#W˘ң'La]S*zڕ/-ד1wڵmJ60 _X )evQģ]iXӇoՌQeLﭞ/VW}NcZ#X;i=%Jc3bo]?Fl<3ѼuV?+7끘@|:kA +$3TfQc;ɿ]% 3d$zI: *<ȻQnz8Ggd;_$6GNćK ,5pSN[Y6[M~]AQ,%$ojU$拓%ԈC~eeyV qLqHq?GKU w錒 a{iUQ{ P8?*I&wm2W{XkZIzq_?\պ"kI*[ՍDZ@<ҧ1K`'Ԑ/Ԥ|]%OMER tӮ~z8Uԣ J%U(k'CʊAMTT5+.D+IyϭYt모v$}]] "<.. 0R@ P~M#kՇp`%l|9UKowh`ckz>zfɨu24I(]/Z]&cWUB}]g{plWi{>tJnC+EEYKb!|BҐzdM騌]5Y]s|gŭ?m3 P▘c2xb%8:vYↇYh.h:ƜSz|\p_z =]3rG") Ϟr>]ҙ :LͿoZä_O\H=OYZ_z;a}ci{w닗?X%nm @A ]}aqoS3lZ ^ZgKkҵv=ڵα'U ^:vQ$T|J_;cM =lpñkfeMz+n%lł7vMnԭm(8k(p[8*hPov'Kl ULx֤ ;X`8U?jB UCl GY]w1x! !dD F7uSmܑ!i9 ˞mJ=bۀ۞ifd ٍMz%uh̪HFt` ;>̽b&s2ǎ'DG"lz&ӳwI&O(/|$1g]H%)ͬmw\y.wu{nn  `զUnwzt.2/}8/>>J끱#E I:.Ȍ>/_D:Icd'`n|:*[Z;c13&fVR .G*MxM }0#O=5?p[[iOړ:3)Gm?7+_r０Glq; '_ZxW&mTr*L`pU&*\Um_o=|'z6kJnrW$4/ssl>ci~ża•Jl{_6lYùf!ݯC|+qGwY|6; 7xD[Z\Ywӣ̿3zƶg_U7n1Oɓ'Օg豓=depr ]xG<֙ʬ>/+?R LmaPx]0V9>SWg--zml𹚼m&k?ßz6fV.~mW7:g'?wt=P<PW^yevVqe&rW^U@ʗ53@a֦v Us%x G(P8p@<#x G(KF{k?%c`_1kQ G(P8p@<#x G(P8K/?y.ouNMPʒ_yW̾@+.6P?sYk3z9FĂS9V<{˄IɴI06xsdzs Qx/?=ΩSJ9熙`=5HZ gO0J)gExvoI3ydʸ>^eyu>g|ȶ|ԂlZY}A }sܲh Sp5{[ctSJuvgvmx1minfmifjvJ۠{۳{/n՜d4%I͌siٱ&ټrUr>spfv=tӒ;qǔ_" ~EٴڇwKI4g̮ ϧޟw85摙]ڟYgvmh{}w Ik>Ci={K-?w<9Sg<$ɑ|ysG3j~6_ho}<!̮`l)YeqV˝ۓNtmiyi{~&iJ66r iyun]ph^q4;u^H2&ʺ9-ܖT<2ܝ8 P'{֑ܳ|oӺCI*!|We]= l^יAOؽ1[;f{/15S n:{R6k3-i;ibM">ns)7'0"b_\7xZu{^ ;p"r$3{բ==2bղmoIӲ}Oٕ)Y~bޤy0 iGs*2F`,;~0i<>]Gc^=U33'S*KN̥7=:r>~mzܗ:yb{/ʢa͑W~ Krݗ>['ǚ}=7LYC/O>8{O{fܵ7;c|g lӒGf.-y>[L[fF3';T.PbޡNJCcR#]z]K@llwA6 I3 .ևuɂ̯LMjYB~mn_7 %ELk^nw:CY@>jddlް/NJ[f~y2?iВ#lUѴ@V-ڽY`j;(<ao<9!7Opg;2p'>|ڟɴRl5 s׵LJR}ysݪi߀}9vizA[3JN%9~ _Y˿oV{?M|6G'&N%U&gS|*c-vIr:ǎ5v_P6U4_KˋRds==O{ߥy3<\K?tv]13w7U/~|ϓ󝟕oZol n?}؞?˓#>]7j 1#E+,K5摙ie0ung&rϺ#_?'u-ԯ)]MNdΡXHj_=WiNö4m,>,ˊ}s`pQ a݊5sҴ@V-.0{a.iև;iـs{"Cz‹[ 3u߄ ?9~3K0fSrGGy ST%]':`v~3}aiZ\L4Siű<%?'U%jsUM5+9_ݗy3\< ~gmx(rEAQZ]~`,*O]H2t\'{?џFmٜ 3Y(Xc\p/oAHv%iwT{$ck?aGdOkmZ\Z(Z?wԯ_ϧa53r˽սAB -1ΠoT1Ԅ xˮΊzL̊Y.|9ȭ]R<ҕ,Ny*5)7r$)RTLO%F>ԩ*zO.{$ɴks_udfe'9ce򤪤HR\΄2QaRZ5}>a%w&kܦ`]Ź=>H:QJ${q}l7ɞ>7aH5rn=gt$ 5!iw\&}h/|=d?~|WU*ƍtӧstuuwMnAV[H~ t`Uf_ w}GJCy膞)`}0LH,g_$mŞyu~0}ӫgbF--n&踈0>fzs:ç7 2(LQ(%*t~l JY.ЪHjsw(yWμ+Fy.P5 8Wp@<#x G(P8p@¹d]_n%c`_1kQ G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@$\.3IENDB`pengcheng.mind-test/docs/aicc_cards/assets/obs_url_bt.png000066400000000000000000000371731471400134000240710ustar00rootroot00000000000000PNG  IHDR% IDATx}xTo23HB 8P`E4Q/K4ڕn-,ժ(찵2H4̣DLJ!@b <93gޒIs]\dΜssf8E32"""B!Hgk+N]]!G`*B\HN'NV!"B!Do U! B!DH@B!@BPB0*BT!" $ !a U! B!DH@B!@u'y#2$!Dwү 5_nV' ؿk|>?"0 }|zs.}=q#Zillr;DrUj|2l1wM}a iGE[m4 z^7h9:ɥop:u١KSej@ddD1p@Qѿ"u:D͗`X9WZfӇ{?8JgK+N'ʲVN'Çt;/t~u_'\Rv1II 8˥I>K뿿bIUjmvvY2!/(/(zMl_MNyo齾Ft* 9Hdk ?Wy-x?Le䟻3GS&3#ֳ7MխZ[|%scС|t[C?ˏZQCGεPu AQm!53m[|?~u/Z8Tz-K$D:Pcg.3Gez  @4XW&iw=.c'&;t$7HnM'Gi uX [0{i{_V׼^'JÛ/mS|5+ (i 5Øe~9#{39t;@>G3y%kR6!:]V5ӫ@gaYŻU 7)_2'NH &M^(.jJZtT:N?٫t:q:eLw<Ǫ.2fWvp&mP1gݷ1^):#ﾋc͘[kڶK.CETGg4q^O9xMg|)g[h[.NcM4ݢccgδQD5c85xHo7P(Va8.fFsק8ZF1󮻘=!Z)vJu0&.(./`,8 >}L=R2nfw! +?Čs@zeJu;Ǧ~H?9č8[ ׵G9R~hqĎ~X dS;-{yg Y˿vh ߺ)wc>g}bJ@rD%*y_`8|G@%|㙵wڸ@L,wp`xRU* CvɌmAG/Ԥ$_ٕpAHf'n'7FiD=v^y48k m^-ΪjO :xzk 9qSM^;!G@GNRpc]1{_ǪÑ( %ET!BщZƮSI_0/ﻏ¿_,ek!x⯃c<$G|4peUM h$^=_g޽=4qPtGy1 1M8rZk?+ܳ4:8\M~co_ @~/1o Dp@Bt}I@m[ԪGrm/* 7{wb(ٙ!We"D3ruDOWHHŽ&R+neƬ & W(3D? DO|d1׷-uEuRaM|VK.π:D88PnADd[A)[:i{?Ng{7>J@% !"HD!PB;0)$ !H}&1"BKPݟT!DQ餵Ï1TѽI@B\C| W5?-|M ~FBP?(xxo^mn[̺Z$ !壿ๅO&#Cݝ[ϗ3z}; /ṫ\V!:CTVd΅O_ݩyoyOL ?WkWKxͱX!Sg:u:H"##D餥fG b !zKK /_477Hzӯ_?NCAdrV7W! B!DH@B!@BPB0*BT!" $ !a UP_A_~zܭ9 G_`ZXK-,IOb^"v׎4WUedR.?s [ؕNXv#Pk~v'$WP 3x1l}4 :c%ټX4GfNBB5@gI8>>S yim&DG'N6#lhuly?0>@ّp[bpN 3z^@f.J)ϩ/0Nux0H,XL `n?*-Ff\w f&lwױDw| <9c.9On6c!3-#'g{)bLpOv[ h8}EEF<'^VO-\_q:6seϾkTN+AUtT $o*bRX,G{JXGxzڊ<@T@2^?ӣH #>oUezEܠ s zRYffz'@u %&<W߀mP~--W]#:]$)O͸$ 2*:gTGUܙVS6ysgpwvJ5/=\ũ{M BWvRR"K{ \2)i9lb=ݓE/iSx^`׳{i+=ٓaɝLŷ,u'%lSkJwL1ygoqjjZ89y5$/>p◲)>oP{F2 pFS/Xo";ɑKN<{j&ݮ;I=wگ_?*TtF n_|J•fN^3~}"o/=\哝F({#o< u<缺2ל"03~HX7Nod&X8 @$a~g KYN-vghN5dg@>nh3Pm@ LU -=6nĘȃTyu2n  gB 7Gh|9h|t_;tE/rשT*g|.鰽ŷG6oyu,?IK٬,;Mdq<Ř~z9zR69e> 6dA}UgH>Ǟ "^Sӹh1מFDDo`n_=3~Tk-3-IKK ďetnqWv=jMYi[iw @o7oghOPW_٭jp[R_f ΐڅzj#֓ϫ4տ- 2U@V* B/vP rݩP~|f[ضRPAX}z{]G]@3ю *¥'wuj㺺B\U,UB B!Dtú?!+tjO^!PIBPB0*BT!" $ !a U! B!DH@B!@BPB0*BT!" $ !a U! B!DH@B!@BPB0*BmU6IDATT!" $ !a U! B!DH@B!@BPB0*BTqmbɳ-Y-!DJɰLҍ>OڰngJV:6`'b:}=2Y`H&kϫڰ!s {[UWB. 6>qz\\37sl[4JsK#sV c򎮃1Â%Y88x~WuKs|6Efxjt4ج;)[E1-ȱZѼB('j (mt?^uC8:a@{ Lւ:NQN޳a s}׃<{ur|ЗڃgI[Js4?<+ Sl'rw4K<~C`kY,>rCk&툕9  dF)h$^WB< 0z_+IzH[CFط]y`E7ճOjQ9d 2Z>XfrO%1؆Ţ}L4{UvG#cmLg9+;\QP-h ucg4|:L`0IZ4Ka QQ:Z :} G'iiݠL2=o-s\ۈh$irD&~VYFV5By:\rӬ$* I2zlϳP1L nvuUd2ƾF`)^Mo\DffV/Ph<7 ~;v=ewToH,1iA5G*a,ԥZ6NB\3Y޽{V뙽6gjW^F3i"c:YFM4jܬLfZŷSG8wk),o w<_Jk`PHê71ejTHu :YvRZA4#mL2ֽ̞CU(K TXe3W7+gjr|65mi.vK;HŲݷMigut}p(7{ C;PS|'}bt@ ueǶ\Z) ׊O.W>k3iܿM,U1,;1gFS6Ob]q>X.)Rwů4xQ@?`(/gH&P3X{qOHhuõBTkZP $iqO}zaa׉53+ZS{tT7fX8Ck*JWWvZk%&)8H)ͮ,*q9 w0)F]cķb˧%՘sChu 讏A4ܟ)0vhqeT7V{$TX}I'kD 5qOo\JZw$iob5BcoGo4)G9Lo65cTlwaj, $<) `P%ʢj-Hyt鍷tC jLgJ[m&z<>t{>艿I"B=E =YBM+tAO<q 42"4l5b9 jX!̢5_ T,lzn/FpSfݤ*U% zz*JES/ZFpu"xǟbηӿSvD~;+յDs,]95gfhRfC;qh{m41%lt܀Y{>C/ir-&V7AlsXW0ޯ7-dD~IuP"=g1c_%p ӐF>QwJ`F_%]S`UC4٦}z3_ko2PiO4Wi߽*k{ 7)PYd<#f:f5i݋MSMkƨy]G56Xڣ|1\_~WWF 8[hQy]o&1(e(3K~[IscV3I:ϻͫ+U}iLG4.6 ʣLf f`ʹtG{΢+%f nkQWuNM C0gf`qt|x],ay8#fe5IN5V*ǘ?gLw8NyoWǜך=\Rs?\־cA`"}Vx̮Oܬ9$Yakv7PSt LւL?M \ko'חI{ܳ*=^kmXK= _^xSWrkWw*/jcmLЭQ?tYgfuL ϗ vہ eR| =bŲݷBi)$i-؇R GO&pmdZk.vcn,wwdbPᕱrL175p[<-bq__Fχմ K1H 8srk]sC8J* .FW+3}FlTA& Ӎ1<X37)&r,3Ժd >1KL0}REu]df'(Z+`L20f2B=>HLժ:C%=$g%S(C7W"1pWg' iAk{_o=ςdWb wa׹oc]1T6wZkc_s*,JeZt6*&͊gGA5$1c|.pUl&VY:VxuԫM&wYR6-v b0g4{JQty\lUzLoVDf,Pب)LrXfY[5b:0L>]I5Q{9 qppk:|cf޻qwF*>'3gQLZ\i04VSG ec Il9kK#u@[ݧF/aeʂ8p} G<4=(u^3 : ǴXC*UJ=eI>4JDzlV+?b0{jP }w?+5;36;a :a}±o `CV#@ZFt2YX j09jggLVV2Fd.ݵYS`UyX\mf2j[;P>TGPo$@K3j`,߮]O7%NvaYQ\{*ĀcN !B+Lf&Je3.# >`'y^&w~[ DЏ2q$i5CmǤ;`@>TFu1Iwfp_D3YYe$]+ K W<ǜM'5(TP ~)vnuUi !f6` 0˯4;5Gj\m/ ?+6p1}:>E4Xw[B=tp}vtjĢtUlƃOA@) ܭcr½1~.4XKUoQ]ebQN@qef)_\*M%#jxu/\F%cSf{2*bmF'MqecZQ\l3@c~<&]ݦ pzCEjmpIN'm7҈+nLyg/AbXAi:ԫU`u 퍡zuլe*CJ{h2#9?6 ȻSo\5NZo,n8sMX&_: ꠍOh:4ڹJ^Ao$Zd§-ԙNN.2H"""^tBss3-BWB0*BT!" $ !a U! B!DH@B!@BPB0*BT!" $ !a U! B!DH@B!@BPB0*BA.SE镢3ٔؤm'{yW7zJֽ"#:TO(z^24fHWWo8Oa't/= |=Rɺ˗[o*.mjjJ*3SY籥S~t}6;^T!7*BT!" $ !a U! ۳z&d_" H@g5iokHVG7qf;jw'n&#bg_ףɘ Glg&]Vg yO x/cѶ\o\R߳:'VaX?Pg7S`㐲b3e+rk 5gFe FP^fD~o&*>+RoޠʏX%WPͺylʊ5̻g nS.aW&S50RpMJ ;pLضf 7[T|ed^9]AK'6K;7L}`CY.x={g?\E&ϽDMY=0+PW8#ٽy ܦtۖV'eŪ ODș{+ysqyMRXm-5 n'f eo*5U,#u0rv+Ip^2׬c`޸eʾ/Y2cP6iPEGCk47*CnolzZ hy=Qyp‚a\n39+zo0|V^qf3jfʏs5H|_ώ?Ofɘ:v5uPε)+6Sv//{"-ef22|~ Xse7al:/_q+&;nƻyDU||j qc#UJPC[H4{t*#y9kl6Q\mgPٔR.#Iy^y7( f7K~?nvc-,d G n9Y@^ !c^H١5x{7k~?)+65!ޠĩΞ!fmQ$}w?FN{a*z^;V7&Jc]3RW{۳ ]W㣠t3*O?:ea{<0fo!Z{V$}jok8ذϞ tkt ):H՜cc#Lg9k}\Nqc6k A|F w, f1g2`L}\wOƳ1}GW!!'ԙNN.2H"""Vr:La-@Ύd, '¥3XKәs˯ClJ(RTT&R;qP=^RuHwu1Dס !!a U! B!DH@B!@BPB0*BAW}..B!zPB0*BT!" $ !a U!.n=z?;՟ &J&7uѣ̜q+Ccth}"U!DIENDB`pengcheng.mind-test/docs/aicc_cards/assets/train_args.png000066400000000000000000001762351471400134000240730ustar00rootroot00000000000000PNG  IHDR * pHYs+tEXtSoftwareSnipaste] IDATx{\Sw/[5R F"^F s_f&/ie~6:2NB[V[7}9[?f%S`\\Hȅ~ޯ2++k}7+':ؔ:8xh!`c1F "/KxG^yB~y y#D^lLW{A6t.:8*"/ `/ "/ `/ "/ `/؋\7f%K8nu[8h¿0͙yxP9::֮}eW~<0A5>>a\\\BC4l<yxP#su `/ 6zP vT}].8z) U?D^4oNņnO,d?v=7 *?VHy Z<.[FDiY)[l;pB0eF^~&M h=rK/7Z8)C%[l9lh'^'r_m%ڨpORlOٲ=e}qgT+eW5ayi&7MZ,DaQAѱq bP͖[6.RW~anw,uf_V$&+qe3ӏ^Qy *^W?:ֺmzuʔjX4?+׽N{mzBayaQWuDTfiS]ySVM{Mijk!:xŢ?N+,U3ű<۟OVjD ڸ ş+(81%.DEUPy_e+iVsZ_~y[TQȱy`*3BD[~<ה_DD!¬cB"uJNGۏWQ)DD>y!NɫN9&q^Vx狳75UP{];vƛctWW"%n-SPO \٣|^[NꈷBƸ oUwgm[`y[[(m899qmAtQx+m+f]flF^/|t/HR,x}ۯK^Gte63(q;8%Kiц]taגl|[&c+‹~޼dtf>M&ծ]i'",Yl=:]X\_G4; N`u[!1RUU54mg(b5UlOٲ=%vɯ+؉0͟Mɫ ^T][v{Iu \Z>Xfo( h,Ԅ2w"fL/ s=&&_`CL#&&&[CG /`lۅ`6_KdtTW,hXQ|v?r4[6~a[ E ӷ Z~PJ|iuZcTIu?-Nr۞2::/x9&aLitQ[]TqE|,ٰ l̿K\8Mqi*$0BEB:"抋]k+-Vh3J/||8e5$x#I)gMqv(kҕ?RO`zB/mڱRVSMࣼO,^1 ~X6#2Nh抋]zźbuli;̓Do̅]Zډ`w/ŬIGNjk*ۈi斎M{-f-j)`KEAoZyQ-#~](]KDhU747VSѳCX'̼h 4vbO<(?K}和+B|EB)?ǽv,H=M p.CmE?K5Oͪ,\׿* |&?( (-&ǣ”{U~*6%ۚ[7[j꼊'<%)c?&6LDž|B]\&9I-!"U=}ϒ?I>6ǿh#h>_h:x\|5s^Z %qm4Q״u3^.bP20V=фjhi4u ^7["L\1p E&1tWʯ/)vrVǧ+z yY~cF|tJJ. XS*H=hzkq-:^X.|-0R8&4F ^zdqoYڲz.^^ pُ9r 5~tS7/{/`#pé˫ r{mјj>ippP5<<66>>>[s*Wk䥥HEhPbNY[)0+[$ǒ /7dֲx狳2ӶR֯8Q3:/m_+D,xuLSU]?88dTϟNz{ܝgSoJcc#8:::;;3 Gy?2͟=~Ӽwihnӄ],(;":wXk5W\Mh֋ +7͖[6.ŽF*`aL2lw˅{}qٗ]W(6n}b&.vvzcc;wUC^˯RL]T*ս{êa[7vJ1b5 |j*=IrL(QHGV,)ȏq~ 1"I:tOl=wg_C~+i΋6̈́WIz&Qe >#H!`+&=kmVo[G팢.""NydD׍M":[YXznHnA. |8];;jP^DW^Ay{ýT*Jekq1؊tO$D.2YzL 9(OQb x3"a8[]&JNO*;z4d##?"Jl)"K,2 ,krƿBe5^ӤRYM$/Q18ד#Jyux)*`/FFJb4= T+_IE7yLxa#oJT ^"ڏoع30O#/ZE]_3k"5!¬cj*>?S[<9P|s_.6tݮ][߮g <”(]pe-{K[s/M%^lܻ`G>lO=#ggCC.mk8y[b4}?~vbQj½wu[r5}IZj( 'D8Vm̍(K 4)LJk-#C.HL*(>S?OmY|#'ȊkTxA?&Ggj3#,9w$룞s8IT|'MښSYO}f(M*OlA&5qDY Lݣѓx@D ($I[i['Gg2.Ttcg*aOD>7qyŖa' vT}].8z)|[3s"ڲ9i2MQSg͎ )@ا^QχTJo$YUSRU;+kd 9c_bgpSi~xT:;"p(LQ  D'dZԞx8@#%Nrwta~(,=&Lyan.Ceu'1݅"PCGQ);HeUio$m =`:aLZLldJJDYgʒ 8"xN,K?ίI>;]gDҮZ{`uא[7(ڻ9E]O8għ/N73L`NQWIi%!3Q"H34 ˒jt9]7#Ę|{m!SS__f39D/sH/ GZb#A!D˫  0mm4A汱}G_y^""͏QKb{S7˶(|L=䞑״t 1}cS T0$ma.?x~-QSWjN/]T;wj(9\$oT1|~-ADTz?)j4?Ľ_u|IqYMcrȋ6ignKbK#tes?omZED4t]ܢ<{W1T0:Vh4GsRK}I`l_%∴mރo_Ӄ`X5fitw, fY:S-³q-³O.۠9i_.4򂏎JeU^"'On`T/*"QATI~P+GY z_CgǢ>"oؓ 2%{\JId?|vX RDe}Gr'bi~Ly7C9Ev~nggTәgz' l sgebj[|ѝg|'PřV嚺:m7%<欍 Oz'W!34~%NS_ˬDv.xC*SoJ%4iTd8JSc[t vq5ddD/b:Y_5K2qIP"?"$Tm H[@S*J(*D+.4gV!X?يORYV`g 2in_q(u2 DDcSO8g~ qvX̑}OۻRҖOl]JD^>DPqׅUo ɽ^^ږzu4~4;]C+4$J,-'$:Wqο5E#;yS/ח|q"zYfPĶƆS}[ULu ٻ]yDTO8#/A~U7>_j 5HO~~¢z9jH}U[؝Eir_E'DQNA&y} Qj sÎYdxTއMf}ʹzzˈz^PkISn̖nL;#"')KĿN. Gb?X ޒ=%}DO-9$SƦK~kno%ՓNQӍ42.MfeO*+g_P#1 ,iBDN'_4mKBN[Fiwb-u.R*4i+HnܚDů(RB o XؒQˠZ4SN'Z36MofVZ]"IV^MS7ѨdLIF>ɤx)* m8ɉ&`b#s}:ڻZyOD8 "z?V/W?zݷwVˉ|| "ъEQ޸u%gU;t 7mEw䧉>DDkI~{2Yh9?B͝C6FݙO*'VXZ^QލO+iz3;'-` _Zo)9jkU&J%MQ-װGDl+Jɗ-)WiX%}kmƙmADD~ "by SEMU=u_B8pVnT6>PWsih"/1Lծ:'k]9EDNkh_uNֺwri ɋ.TSεri9O]>L"b"UG󤝫ROgQBZnkx+e7Jg;c5 ʉx1k4y53ilW" i<*0C|K2yZDOLlrꢈFQqތ34u59/EtB {%TVwGkwz5n cdBѤ7HBa~x:29!l4Mf%aI)q -StRa Lld ѰjHWe}[/Ѝy " 6N "/0p3l?~6I+nˉDDnJN-.uݶ?yqe KW[J:y^Tl'JxeX4ye뫷=n'}heQsHvSES IDATB%%KPT5GRRKIѨ-^U]܇3 );PD3)6)/qS]f_KY Mډr9 ϨIU!oP{]mi^);đ4޸Id]nҔ3p"5*m0 z"J F[(Ti@D8sU]NDTbaၨݳe/Qck3tZi M*׈"\9]X\7ZD򃂈'fQygpٽv`g8^AIIcl1 r/W2n'*/u{liqpctGoh~aqSQRӉ. )>;#qhʼnWZ O3GLϳxil={ZS2䓓)LY%jm=.PODPW(ALZQI,?Q;_d%L Lg٨LZ?<n~} p7ӓkKj2eJd(Hx7PtcxYUe|j_h(<~"~,G5+h2ef54TAvYVHnfkJJLe܍Zv)EEwLhݱ""⬍Н &4(~"/Do&"rvqqvyP#??ߐln蒥'ϟ !"bO:k%pzql|LOD#⹣UUI)~m<Yz?^ 22 ܝcIW,ÈZ|BAw:y(؛L?h_C̨9ǥ&.Q#My&i6G/<؋#4*nDVɳ4ZNppAb8EGE3\QS©83"F,%m\j !?OIl4~ɏLla"n5_j79Ԉ9/f3x/ ">gw"Z:!SlD>r^` PXu#x" {7.)/\.wXl;,;M`jHRR?Q l͆%39aewou.w@X<8$Jv{'I/>(rq?z_"vd,z1?}ے6,R(jJh{X+ʤMkfΡI+3?z>9||n>Ϲ2K޻W϶_NR-k^z $=a6O0y?}ù@^8Ev;<" ^9'3nNq7)Od|4o.8zg|`c>yTZr)f￈.$^l>kӾ~Doa{yIɗl~5y^T)[T|O@8~g _>P1א jyTKQTx"x``H|)/q2Z~ofz57UUP n#"~iQjqNN^fFӡi4KT$ɏyYUtDC|FqauH'>5FJDUeĝs2 &@dZ~rv*ۚxnI?b][P#Bˬx{ tRtI-m-DD>~yӅ /+S2?<~Ɠ+Sф)YYw2~]lgv8;swͅZZ\P ""sgGy{?Nߦ:Rt/A-%YMb|̖rޔ}V&U|rNz)G`;]m#s&>s'ouq+]Oa{ kH\{yqJTXY9|gVF]3β%>Cm|(]"'On즬O$DLχSs)G8k78MZ1k$6T~g|ФLN7XsTʉ)VdZùmdXYyfzx fݝhM"Dza.?>a|||3Qoɾ_#`{SםzTN C;m&aU<| Ziswv~X`"M.nN$7nM-6qONUJ9ܸUh^h HJ,Qo۵4\}MwtM=P!G"̯?> Ba>(ܥΈ-Hdumӓc2i lJ4s`mb׻"K*lT[Sk$ھ3 k!Z%~7>pfu+{3LC4OC9i"$ZL|DX<+PY.ŧ*ۥ[\{a.4+s2 S.|dǪu LJ9QP(l,Ȫ*.Wx' lJiv+Wޘr=*b"/RHO*9gjCҊ )&F21T#V8e3NN%ڶ̽C4O7eEw>gz*( !{i+3 qbGk?T7hy__x=\`Ӫ3rttppk^@蠮+6}Z.'7hyr֧}_5Z?n- pv7qB^yD^{A^y;rrrRRl)tQ[]TqE|,ٰ l vPUUy!.00&[CG /`lۅ`6 0-u+ okkC)00 ,k ̶U\Qm _y0K6BdoHA k!zP"/PB$ F "/ `/ "/ `/ "/ `/<(vrrV<yxP98846^G!!nؘ+qsx899,Z<\ lˣ`F "/ `/ "/ `/ "/ `/ "/< wM 0pBuA s/::{ h#{A^yD^{A^yD^{A^yD^yssRƁUo(yy:3CVE{u睾{-흝޾!1 OOn < B 9LHIeY vi8"r|fs?S0sc"˙#ͷT5? K~kT/} {u . XL Uvtt./0==?}}uxҧW1m]e]Uňև{rsnGE_iRy?nYÆhzez0jnV3?{|{/~9+8Y+{]ʷշ=#<a7P{d3yQDXJD%∤b;o3e9s0_ͥ{S@! j"Kk7l{f_"C}/^qhHz_Zg&m@(̧8lE~DAnLʪ~cgX,o)8b}DܩsfeJcgƝlШ掞1VH,v_¹nݴvvyr z޹a\G/,KF2=n5$i99YzL7LFyԧHqʊCE˒g|D^UQ7˪/ 櫢m^NOվ Oib4=DQʛyg7o6Z՚YW/d2P$@EMR f+"  2SLuKii!34846۟U]a({/8V6:dwX!`CCm=vީ ~o786xvM}8v H V ʈWN!A!'?LEySVFnR&3x L9|⯷_OH^s؁Oz6ʛr0jhtrP%4~.TJ҈ƝgDܙ7bz]Z+.jnkV=)c||vGD_&ޠemG Ԛ@qfDI8 @$iKOmw7wųQ7E/UU]gaaPk?;Gƈȑ::M owo֔Q|uraKjkn oĪT|_?FD. Q" HOSM]{#DD,NIJ0_G"kZzghz" ݰ*BGƉXCU6P+'GgPDTtce&(@M zmΏYjrM9Ή_aeϐcw*l#gVQXǵ[1"G8xZ'qRסT';2*:TT]i%r YT J}R5Fn 9+VYYt'D}*]j4}Ù,|Oq{plYA,SJ(?s -WJO_詗}·h09:{â"DDc=\Fm\R9|Ev띷FX ̻?}]JO++TtBnjP:D,[v8i:jcG30@`͓X|O6 ~R Çr9[k=<>YȟE.8=i^_xNM0n=}Oy7gJS=3e C#k$1ԻӔ' 1 74QA,mTDrL 7*jCN%>["HS`Ww~B :qeH-C?NYBzn~ugH~DW}EThlPȍtO|j1?Ln8"-?1FqBou'"Zdd7ȥu5;RYM$/kO/,kRMaeM8gʒ_d1Nj3&56!)`qb^m'y90܈ӝI;=B*]|B| m?'M !`POKCÁ-׮(C-wıo/ z.yܹզѩ+3$ןEw/_qڰ*A?Xu51ٍADgTQO,v62]T_mVVpw]sR 'dsXڬh7SF403t2yps3_=> IDATt[#!a̱[-W=8h =N,[ _t#y2Z., pomhfcq֦5^#+]rVERw*GFyn .z:u(ڮ0>p]CzGύ+ȹ_yw)׫s\: u V VvthOb]¥1OSWO|g˝A=#UWﹳ2:~k+O߶aA#]M WwV0U]MתP`Z>#V9FzjR4/<7Zhǚ40"c":uwܺc2<݈:k9ⶈD֦5WXIsP6|weԗ0FSq LR*o\*$E!ۊ6n?8x. .tP[{c&x hS1*ǸZuUWC9Q$7!e.Uߋ]y(dEFːaGRՑ)(YCm{him_rppzUSZZg[jA& =7(-(-\HEźT#Uqf&ym`Jd`.1å$25 9/VXLwab4޻Gdsd:TCLvV֨gyR_In L=mW+]-c/tqU[Mde#r ⦳S DOG2'XrmRkXu?w"ثlSq5{^>>PݹT' QWg0y2@X̛m;nvOք]ӍAԯ0}=]\3)=Y8:3BŞc4u#?͞CWOu o 30X3p!$bLyFۺ3IXfL||LA}n K< +h`H?1qqdx_Vv{Mm쮁%nVۭ1̮$!(i( ^qɸC}bL#da.KK ꘎Ñ)z"w15 fq`cYXY¤/QKJZyTmD;f-@FYb^ WG]>fztvvsG.1ȋȝ:d6ħ*4Z#ud&I-|X\eb[f ^B qq10Uy`ڑa[5o{ jc}k]Cqf(νE{ݑXw߿6Ҵ'$by[5QergßmeB?@0(Iʩn"qrٻ˼ADDB@Zjynz Xyz2VY۝eNk[yo)iVq-xfRT0BD4? 3`;s]׉M^ct&ޯ:s94߁ٕE1ȉw-G%r^qv׹Kc !nfNxFΝn 3Ftq5QsAPkGDm <#8׎[ p(NXT$JѨ)o͘>Ͼ[R4@*yILNX氱R]{~qe1P&cj2iWuo`!u;AXϱ]%"c-?eYqAs]\۽,)l}П^/ l:ƆY9 TV\$"kRfʪGtNGϵUJ.CĜW$QuO;̵N&ۯ{_G tX?wM %mnlrma<7<ၾ}#_;)^AQ;TݾCG}5pϐ w;s=>khi8ok xc򱼴.o ;ʿi7<%/jˁ,k l!ENX}xXӑF 9qcNcZfZv?ag{,e2\wnj/ z pk8V[d;`єgz~;q^D$9)yewXR^q5aQ^>z苫uRk}͡l/LMR'mܾ6}bH5/%u5nky?kק s;gI*)T/Mw9kYqQ:8Lyn݂g=1u &]r5ύwЈQN>\("bFS%`p߽|X/wuψQwYxu,Ӿ^č6{y;hDt]e#Ʀh?"g._jڍse9\l'BW|8/u' u_3MŮȢ+pYwYqȉ&A#MS͉5!Re{]Wߠ1w:닽ǏV71>k8oh]<#"E\j^A|x]G=C"Iw<]nlih;>^i="rM2 pkh*J]S)"%ƌeE3R,gzvz sڷPQ:;/lZGv1 QXO'㼘7=;,F91Ě][$R2]i5cTKe"X[yQy,Jj8^9iiTjW#kd}|"sJ7 ίT/gy}|Tn26,(tY٦֮˗EiC\h 5_䂈 çk]+N5;'-9rnVDM;D\p7(!!|F RTr,^>]}\/[\;:wIu^cS{gAA}6:~9x o< <Vr`mjʔWVsG6K,/MiԵ Udf~U.'5"j+GEJK7ЧŠ@;O^&oj#E"Rhsh~{;$Xl3#w>se֦& \~~Wg\yki{e^lAqZNj^D6\KYVsNg~;:k^n1~t|Ł3}Z s%dp&Nc1TD|%ۇs?'gOkjV9hȠu~8jtGjrNibא~.w8dxHPC?<,PD##꿮> 14 o07ϕo'gk~rѝ1/oN6~{wp >&zG;{cG. ~ur1u 2loin8q. c& {\ ;Q; j_:cmcgfCNM^>rђʝ4~Ywqa.6>l<-1c}ןgO7?8qrxG%Mu~Bi.몏W|kDpϴ:oKvz>l~Æm>q`o>BFH<"R[i6,R?bszM%2(|7_Wmm>V[SbڋjUv7$.1nU~^FZb pZʾ則ފڨ}8{Gqiߚ2(i?ycH\R엔y%dtRb:҃F =oK^ ھLG#yaOW18;!*.ң/_G׆]Wk,/rȶp켌5]nhᒥQbIYt5/st >e?8;aQQӺ-Xj^19zყ|FwT̄sǎvwݣ8oW؄/8pw0"nJ#Y&jXVo`p̽N}ڶȻb[_}Y /?u`Q/ǙqkoU'Ĝ l\ut]|׽+vkk4o Sl>eNםl{eGo_WT:X+?|l܄ZwDŽ|:xZD%"h>.NPy: rj?k:+8:uo]mQ0펦/Km='sc""}n7%0r$ 9X/}ub1ھ}Dzyx{yxy{\i )L&%z;mJOG<65uhu 5\#c9oطkbL6j*7)[ѱ?k<| xԅ?srG~ɫrePaCzݙ_XqWzVݎƏPϞv?ޕ0>Q7<׍[Y],WneN7 z[ϕ]l1Ɋ0vXFNidtUI"إ{1_Yj 2:O쮻5oQJϨ}X-Hnf7':jpۺod{"P)ЅIW963emš>ZϘg2  \@Mzd36XO~lQ\VM[S)v]yt_q%˷[M[o\󈀀Ç {T 6$ >e,Sa}ܵL7er 1"*]֭h?^{3yUŖd1jSh\7cKof}.gs#xQ =;y{SU '/ ip[[ \~k&W+|7F-FIR^)6E#+wp܁hebSm=³C{܎_xq_eqe9慧nw/yqIOc.3˚]9 >=T86i؍$0ElK\ke!wwWOb{7h X+. VFoșso#Fg?:]廧ۆRcŕkwe˯4킿\1~S}7S&s1KKgϝۼew=_I^]H^H^:U3ywg.\[3SrX{rg۱g,w hK w߼eݓƻSՎݷtW ܢz h@>jk{=ȫc/ƿc9JbPT(__ϷF96wM&U3{cOD7 ܺ{p~)wČ8[DN_=_y5nj.7;† mCE6bذȹ}tDcÇ 鮱]p#y[!}x ;)ݱO7_}F od.=yBsctsky 0dȠ72ؐd='ꮀQniRz7-O!yO!y6pD}cO7 H^ M7 zx x x x x x x x x x xt\=m&:0L=-f1YSH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^kuGwgE#ynd[dfC2ѿ`.iogg.A l) b|^D3^A 'ֽ%s/6kE^df.v<~z>;C^9%/l}fAIɛ9Jm^ge/df\aSq5_W>/dtI"fI^;sd\^i퓘)Y`UiBR( ELV5Q( E k葑oضY S)r%$i""ڸ ;2ӫ* EVDni9@M%+5"qjra֘3m  pê^9iC76.b9'\?ҬBHD\ Rd_풪1L&Se4[LS3ƧjJVuEșa*RbXdA*rʜi.w^\bڝenk":,nəY4MA>;Wa,{'{"`gyJSi2VrV+#VY=4(e)0T_yeKtJ5ީָRR⬷u*,t%x.y9p:@og0VDXg61+s""u;DdTD̜DX2Dܬ5VKD T"2F]bR&*cIMFiZg TZ~7,,YY]RKIj9rUi ʒ1y4Cn:i;r"F)z)4ODD05g~ݪ>FED&>fZDEȶkn1.w֛" 5ò[RY1,|N'WJtATHƴ;7i(""J>GSk3M.bI+3bW@mܸBEj-7/n8[똸xIi٢V$88G퇬uȳL&d[CO2|м"".jDPtڃDHQM-_T}[E9ibPכV&9 ma*(#^M)}vRrhPUj7i"&nnieAp Pk-T_EO:N$53g>k5$/ +NWֿS~qIPY_$*Sb7%PT%:r̫mPmre,܊4CX*m13,496]^Vmh"^\513;3bkޒw:,~(pԐò/ؓ΄AL^#:G*JE'o0:]_!J Dzu@U,UA +@R_gϞG>G~a~v5/U=;[Nެ.V$Yٚ9n^^P%/ 髪 b*|\^8u}bUym_Ԧ;mt5/ FH[.%溜kϙzCf'&Y_`κ7:]tlf:*J)ه&vc ¢P9&6"MOf:}#Ŵ>[avc2+VD Z"}o#wk^ +ҵʜ i 2Ғs$/ {hGIUf 3ooX[& jҢEL"_PHs6˲ Vg駥؂>mĦ>1쒙oҤ|'騅 IR+P)kN;F/Hj!m5.,dIӸɰ")&[YP||TMrդH pH{$eaAYi~sIPĄ9?ef8NN[m2v^r%DѩҵʜܤYzQ.tciCU%vPm  2-ya*TZ|w\e'Ͱk*Y %lS/&7tt\W:cT`=MO[ rF?붅$-{ :'JYRVX|Jd@[5m^\ֻ-"~E5tZSҶ_TiDNʚz.ևd3-\Կ\՝A1zKS( ,>;=f N{ũKv(;_:l֊<\x<}v]"*sJ^*rÛ7s<^jEDdSJ }wmT&#Sslx_Pf?sפYKJjAĜX+URE>5{u#y\+?2ŚO_BW#-lۮ}EdOqz7Gyc~KiAK(b/ J{城)[>G6>"[-t*BPdZBP$nm{ũK 2"WR0b(X\&""ip#3:5IH׊h-3g=Idy&U$NY.YΒE$^|dۗc)xh7f]Iwp_QM<ۚȼHk~Tf5zd ΝSQ+"Ӓۖ_{lk磾a#7{XR|s5O?6L⸼ vː9_!GbRP(\*r]R5&d̙fi*sTM괎93PeZ ^ +,NEnV0.ыKLsmMdVb4973nf?0eJcYn֜u""bo.S2lcT̫!{4mLMz]2˾D$lT۲%tͪ:."qr0ݞ׳ٸ>wHމnn>ބY.&j1H8RC2K$ZU&9V$񨆪Rt /[YPf]3T ޼N?^%""J<ҸR4q:"-O9MώQdm7-3SbRW;z_R#""S_vNS0%#5{9a1n|ٯZZj;Hlj.kN]3jyU7b:H)貖\X]'"2vtڲgEF]u Ћ)Jw/:. j^,g}ܫTęݔ@mw4:P1 UN隡C2ݰ?>h?nϼa +k=*-ip6dUhW͛'J \у&ffgFl[NrA=LH14ȴU:R+z#(**}viRJ,T Kv~|kAN^jtYKmcut6L}>p*/uj6n9סQzFWVu3xzb?ps˩FSvmj'u"0=%(圌}j[gWRٳ!ꢫaE5gW8Isܼc򡔫K^ *WUPU"qz﹫$#"ھ8%UMwCg( V tޜ4湄} #gJH#x%O-ѕe7^RRcۨռ'H$rz.o4Uksc~M"SKK#\`9\op~%4?0isֽaSd4+O4&pn6(**UُLх=e5:qAD}RL7.M&M-"5; r9K*J TtoJN%&uϸm+3)POѤvcGR5T[9Hf-ujE"[/m~DJ\N>*rƻ̦eN)}κk Z*t}ɞ,7>a~">~ #R>8wAfXm)~-f˓8tşW\gwP}??̟]H^nN7SuZ.^W;&l؀cSmMSH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<S{ޮ+=v⥞no>> 1=w_Z_@;aB醹. IDAT=@a2z Ʈ+/]mz|Ş}x>_pt[ ބ/bѣLj]|}|b#= q_]=t_z!@ ynT/^"vpQz\@ߏ5x x xtlU}iQ6!̿O͡/o\.Imv{pP-A}o[6ԼnS~ӍP)eJFfox**?Zk/>2[IMpG񉙱,#޿[ZtƩy9˚t}KYv^RN4%!*.aB QӗfMe3&\{O07>!-ޖ7kSLHZZtrNIda|wNuT,G3^mim;o]5(ߵ nԜS}!Ls>qqvvϔ /2{f=Sbq`KZ'|oxȷJRúgsMSq6tk\"U3xe怞nG&w[{-.կ[4.x۬|%Mgt}[Tys{5xNW_:|;9U7翖" njmܴ;9véo+-lZ7k: 񫜇DkIwԑP7TiǪؿ>Z-yvN ]@7jJYcEbED~i xYp>+Ԕ(]n.qe9ߧaSy."rm=-#鿔e\u L}}}b_gnc6FDx`]SQ]Siw'U&V1y  hM- mƥg^Պhh>'DpXsfŖOP4?0jj??v9m#Pp{[@6@v0oq;E\~['"RWg bۣ'04g1nd[HR51sWMH]K#i6oQMXmK6rRPC7 =ugree)m*^섨s?԰-uJ|BT\=?-7lH/Ԅ vfB0[[j)3^:~7;uq|?=?BsqYyX4*VcyܙZ[UvQ]V\}ɶΨqBԏtp邩SLN]Z\}IZcܙugr겲KvweqgrJ2QZ\<=@DdYT\¯Ki3&ۚEhB=m@zcS 39e*럘3giݿ.|r&$>6wYY}[5eƣK̦Qq/}!ER$DݟY;K$[Ͽ%&6Tw nW-> [^67uG?67yt~%RPȥE@g<ӱOOӦBHd;<֦Ƶ ,R_jn+K~h5ZU=3,/<]13*.aq,+[DgXLHYZ:Pz)+ۯtxwA]~[s8-QSfukVb^??uM51'¯ss.-.]/\4cM WfѲ剁""[i@r;>zݥIϋ#"7[_}c ٺmk-] Ƙ'}ms"?S*m\5 bggՉs-]FĈs{kH/4iXW6"#bHUKrdwm _޸mvâaD>[ziӈ.hEy~""Ҵi.aZ?n|1ZOn[bW27) Wm.\ϟ~dga>?S~{ᛟ7gysooٿoT]ʺqmkgI5{nuxs,Ew~E&gY3=ug[J$9lP=hҢooٿ鋲֋""k{s?7}mJ6c!W̲a#qO]a@萄 saU>5U[<xߚݟ}yܺwѣDU>?S*>zm+e]~[3&¥LY5w۾Uɷ odx~:Zd֤{lykV}%"穥n2ߙd|}@rwl&?OG~o ?h Gߝ_F>睜KmZlS:=>򗛜#wGz}]']mpjwxzcf|diuw~?#Dv~K՝1=13.'/Dƫ@G{:8鯪 vO'/8܋借;u3Jͯ'ͯ73>aqwdjz͗c妶 WxGݱsTӶ\ٽޮs>bwWZmnmm{?(gw|(6>ᕯ춏|=;|2#x'vQG09UvO2JŮܲQ7eK.]x… QϨ6EF6͎ۙ_&6:˚‡fnI[ǡM;%ᇝ;PVWb~0c-7R_FH@ra#DNֹw۶.<8k..+ LNx~ىq.&E%K%M}g:cFŖyXƥgJ &x>!aENȥ-H`r?l֘m)=ڄ/s|Ht[B؈"Nt#͇ĵ!CCE6Vy}G[ɶKΊ;z9MƸvGsu&L<{vu툰`9Tig'"9 )-r k/tɮŁTNb.^VajQ&MON@Ȏm.Kt w^ac#x۴E1K $~8tgâu ;JEE ɒkد:zBW ~N1j*.쏄 X.a;V-jEfP "À<}|0k>Ԁ2sטnNPM;Xy8T$;FxR;in2̆mM^fl6mр*r X=0v~0Gd[JׯS{FD|[ u"ELC-yC` .{u](ݟy "#) \iV x^ƿjRD!""Qz GUoȨ ;%:7tM⮗JjEju luf5n=w/2-riW5[ QH"ѰܶCl/lټ6ܳo J5,IܷTqBE'n^cW])k,CtԼm[O6G{}EDq{ɋe9qmcg{m^"=xΛMBtSƝ&2l[d@۹U0B7TVa"Ju_I"v*;pVnBb'堋 5I{5 ؄✢xHOpgWVK%HA.M\Wy1 (^uD{*DEmsf/o_"}^zIǩ\{ٍJD䋮 || oBr9r6 /=3u'RHOfTgƒ_I%yꥮmAiTSseOJO V\v,   _i9`\}~~<,biitA~5$"b@⦒ נ D k]r _"K~{wSF9 \34#rDhwR}=CJݪEWLƓ[DG}0C0l˕W[&g\x7f?ߘ;u.Etqav+4m 3BBZuXWVwHF;)ϭn\KsuQa)]ƺ17LR̘wr&Tr\=To:lxe>z"/hY0| 7a[qunqnuեe}֖0i(?^~HrdI/Y.5ÇoGԊ(gRGcUQ5G""jڟU룉)~ZXJQDlyEkN_!`_4ԝx.5~&_ܴJ'nj95qR!it#4rԜc(OX.UN ޤBLٻs6JSMm[XcZDBthu/.ޚ١ wvr\G=̯_%|6PD֧ṋk_-FKmMi;p*&"HΖgfk=uYiK~Ux^Fˉ}{w.ƺXx=׼dݦ6ƶ1eSMQ^iOmz[D6k [Vn\y;`lL}5џTSؔY{ԐmzxlሴeE +Iool.|\[=%}zcK/sb]tp9H `"=YY/Ao SױFf3[siYe3/<SEU uD9/Gpqqi\'ޥ3fSqfj# Q'&?ؓ,k.Hz]mԟ2xGC+7ZV$kmpkfIߨuHaM<;n1Ar pq-R;!q ??9{ﶮpǡ ` x(C5{J"⦎JڽaD &WM9[WZ"h?iEʟj\3 \~'^<1':6xɛn߽!y`oSNqt`RS'9"0QKgXgɥsf[M@/2gs ]^H^^H^vGWdmowrR8Q:hk*n*g8 H^* l )^u& хʎnOٱ `Zggk0Jg>|G]x _~_]8y;7.ޘ ظ;y ؋ 0zg>|G]x ).IwдNG`4|PUȢ kWQ(O\Lq ܏ޘʔ!._npt!#E7.dܝ IDAT"y&۷;x`S:;O;\`]E`/$/B`/$/ g꿼kۼo>p@E W~/;~uuU+Vs=@z>h2[5c|k p 0G܅6/BbCNշ(1 ɜ∝8j\ĠQ9S5'O6T9B`r49Z,mRu{MDa܈;g>Aa."2kw]0H^&kʭ7[xdY9e'Hݵ2_e1NWg>E7~%=a`B47A]?6$fĘ!DJ Bs߹Yji/u9 M:!!荅u1?6$2:d{<$rAD "b9.\V'ŞC4ON8fG)#yeD䂐+X{k#tyGul,KR[-bL\ $e+ySE <.Ѯܐej=95$2J BiǤbK1L:j~f."cߒ;2_:1aQyߜu?~U/UE{ 6UU9EN\5'r6 sICLB>Y/fȹ5s? YidԙrؓPYQ/UDDZM[KQ Ya=z׵MZ09;y VE0 6V&x㡒\HDo<Ӭ}S(D\?v'C\-',L~^yؘz1穃t`K֨} bffܸ|EDm{ֶ8Ya~H^)C.02 s)R絷J]/=i=@U:O_ӉE--ɍrѹՙ6+Z"u9VY{`rBD"4U>&%\DYۏ6o-ʍ66lyPД7v|ED-{ 6?*E 9YD=<{A3JVlKɵ6yR ,Y"7V{H+""^_68=Y*+v8w.""gUW""HcSyG{T$MDDLeFDD̵\woTd\ڽin*2gYeٯ?he^kFo$/Z--._b]aњE|DDD!\/8޾"-"B{2oݛeW96 $ʰ-nQw#HbݨcGm\TP{wV[_ZnTxnij:~jG5kȭ[suGƣ&`jSDlHl4 ܣ+kRꛚ7yU7 j͖ԄVio5xL>.qu~ޮvO;.9Y-=Lnn]kgn5+=Cvl5䧭7:ΙK:\pD?vYĿ{Ѽygϻ M_`b#ih4{1or1,u]bV%4'}Kol.^*I}1/t`sqqi\~46*""LmqGo#S_yn5+wW3`Wn z  y0:ꤢI.9ѱadzOvG5cP0 `RS=硗bc#wK^$/Ieҧ8`0 `/$/B`/$/B`/$/d䤰; p$k{UmDm`lU@oUgk|]Ȉp0M60uvv:t˗npt!NNߥ; `*tm`zc$/cFB`/$/B`/$/B`/$/B`/$/B`/$/B`/$/B`/N.۷;]8" w.dh&EC:;;]8AuGW!Ύ^uF>)s0M6p?zc6&˗x`*S:;, |хmShcn<)Np\҆63"s#\fȬE}$/Nΰ{,hlN=C]O~j:ڛ51ur؋X/n?q}{W#m3u/܁6/DD" 窱#\>9utkñ[.go6ns^E RxF&lk3gn(Gȥݵ?~W=; W6&&qݿeB%j 낸ÂƘZnںvw݆dTsІo.ڭ)E"⡝^QmjZ|d_DbR>͋lZ21'ehݷde晓jDsuFt)ֆ' Ec+DD s|$dEY- 6q:ǗT;{0u=ZrrU"sfܘ>+=r%rCnw؎T\""7.=m@DD|<bڛsa@玛.JxLp`;]teCE/ [:JL9Q%"xe ۺ̠ m^nřIL۶JV6] }--"#hSdsco+cUM-"Ācyx#v4~X޻`1&;]hIYpd\H`S˯mCܢ9Zݳ,""4y8#yMI)9/ l]wJ~ɋA]}i)oFBBbCC}ZePBv0J)fLK7c\?4(>uG0\Lt~|xUdnϲʲ_lK9>,"20H^*2t~oti9Ŧj"ԁ"b64Ia[ܢJnnJ^4SPw/q}cV?ڼuo_5!/>^"ڹx9u5n?`sY~ѵ51 0eT".~Fs}pw9QR\}31*55"r*"JMjm{[DDt]`=[S;Ul gOo\]پȻ-_&L@>~""lo*"|2#ɞ{Ň7~(Xhi1gF9_6n?Oъ"b+ZsJ I{uÜycBK3~r7ED"k&SNzp1>~"jnnNQV7[ \E&3Q ]E)rΜwW-mDDʍF&J<&1yMؖh\&t>jF]uqY%5kȭ[|t㺈ȣQ 0u&f}E{ E.Q [m}|ε& s:Dk #vS~Ro[dQzCWtm4ijPMkû7ݮ7FqS?_r9sɶlWC?Iqլu8u=/ĕn-xPۖ[0Du2hѰ<\aMqÚ>0ؼȋÿM{%˘a(q]qh0Ff?ie -͛8{enPh7RDdaMtt FWߋqtC1l}0aMaR'ؕ.me?:pQB ѷ6;Wʮ,->1Kg&1uHc""Fw60:u9u[rG{ǻL]~>~rʵm]6SCA$/FGTtys |CA$/A}kzjC/cF'Ι9n5H^˜Oqt`a^H^^H^^H^IamowtHv''.0."y&ߪ50eY۫: `\m`lMtt F՗/7ܾBKvt!#ET6.,=H^ ^H^^H^^H^^H^^H^^H^^];A ow8p''E@ǃ]HxKW.!o.u]tt!Ltt F_, R:;;pk{{b_ Mnu}#,lƌi/x,=mLV/7ʔ!._npt!#3.2YygZ]$/duv )N*;^hLnݾq_0!~ #x8)O%W*)tt% [e8/ѕN[2mZbi<UncLBz<8n[--5Fk*H~!vwm3cƌQJCm뗷ʇ;] B1xrcֻ "mEJ˦Xױ?vSUo[Y"Yip?p<q|""k>~}홿oiS@oc"`OGƣbLm$/5c VTz&Ҳg^c$?q뫯z1) ]rsEإ(%>M^ˉu'K[cǍ͖g?v4W%r\ GSɸ9JuL_T=P-">m:׋Z>f۷/h+?vBD"||47&'}6.'p`i[6<C4~l7eVgg ˊ"Hx.&lݱAߝhus )ylwgzsm}S̙"rs͛_̴e͛7>b1BLccsU[Y󍩯_Y,+[Mido*\W۱3]qYǷO=˩ߚD$w7u6^r2_ G;ۿEܶGED \}S~]"} }.Q [\u=}=~e:ʠ{NHʐ\% qQe25,' 2v#Ǎu]g4mL.eik^Bu_RE.ScKr،*ɌKC<+0"Z絶~q=Ե 0nH^)cHC-9fuaYFuSYϽ(%"s }w[칦}ʺ ?o.0&%m=Xh!6 :S{*3 ŸJDo<Ӭ}S(Dմ5>MZ09;y VE[7RUDDDyuA\aARcy-7m]nic)Q=4,ǎu3!"b)5|/pߖ˙ ۜWs?_QUAI|`¡ Ճ IT9Uٴ-9طœӗK^커T}7 bTm?T IDATuE9ihVܚ'͹1Ykvjhj*{m5+w}ʽyf'W<:no5-Do:ڤ-|O4t_X3L)ئٮ"D*7$%ZR{0';97/&ynwI2b77m96ƽPNNt $Wh1n2zop>\mnae:cϒ{K<p uޜ5VX۷o4΀_$/pVTtoZrrU"sfܘwܾW%VD)>zPÿ\&5'ޗe"RJn&*D$"JXwyocR]<$PDW\C_:\VHgg畆O&u2PU!|MdWn)ݷhs0K*""B!wO5Q8{z{(ED}Ƌݺ梺zC 0gJ!"n^=)89}>l;H[\*H3g7-M(Ģ 6NuRqo1{xrg (g( 锯ٍ7|}#0v4xZt//V7r&"JDՇ{ޠ,Я}}EEDtN ="D i(?5x1̵\r7*2H.~X}#)zz-[*R}FDimXL˻Vdg"CEqE Vdԝjy: UbtOzV:*ˏ. ߯mŲkEr>?US];Yޫ0b޾}I=@7]\};8Qf(~Sbu:8vr?g >\:N?cپUWi.bl^KTtJjN:\7eVy&&{[ǁ{=t/?-o޼)"2MO m^E﫭 "#) ܨeTo.U/ 1uG3wT'R6`q"2]zCF)9kw `ܛSamSHOhuw8%hjEju &vqvC,iljW?W[( ;mK+1Ė^–kÇq¬ft >8^龡^S(ԁԌ"""C3=NDr}O$5u"!5[~5//L,1{dH|կލ-Y[^Q\ﰻ]\Df?7$/T|,r)FL,iȔpʘYXP`XE`>2(!;pZi3&zvR=FsŎM*dYߤ=/>c 8Pϙsf=w8l*uE1+\3WI6\82sqc\sMUr_-7љ~Vwz߈9׿C)?To^kӧ۞|s$/%rF{/R{NDݻYKDDJWEd=qpՏ<3k?eE]Oƈt|SSYB7E+r[F$XZZ0oݛeW9*)P*WEB=:jϋY\IګIg&k7=Qt;OjvۡGX]mkI5%}ϘXnA:P}F$(XKLAgi-=Ay\T"MB|M~=/&el$:c$|hM6m"?Q:;;oߺ-"V͜q^)my2aĎ̲us}wK+eV5Sǂӊn˗蝚Qs=mgkjn]S]<}_D() Tn*ڏzTy4&:?RimH%ԢhRb4v6;7MK,~=JMe&(ˤprs" #z׸j5a**7ʉlT" XZ' 5ڹ""J7WRW;_us^I[:k{tIAQ7Ҡp7HMŚZ]6ù^>9mU{]9ԈcHef|h*i27EhG 0&lK4M]x:N5r6$lԾRME4We램_!9JUEiwnFhW,hr+"rz5o6h+K*{E{4br'/5U=9sg|╵땊leQbҳ]IGԊ(gRGcUQ52'Ru֯:[_!"r\pP[XYo*F%߸FxZusϪU9r"&5´ K}bԃ㺦Zׯ\ӑ5&yX/:Po+Wh7ss93]'mFŖߛh+qo>.">bۓŪ5COtܩ K6/wZۿMD'Y7D<H^ma_e0ty.`o?hg%-y a=ͺvZլ IXvզ;p~vg5jmzۇ;Eakl!OrRs\ڸE; zƑ0%FߴY:D5w~oю|VTeXEMp`sl2Ɔ"Cv'"R*^]EDMgN4[_M1~M6M6Ci~<3:[TUAIHnqS6k{'[o񶴒gM.cכ{Cw2 `r];y%E{=OuB䬍M*DH!_ 674^N^B7v߶HFjkgc\5~[̶ADmr19Zu8Ozll!G}dۚ :1ލ&L#/$-LܷkE4U{bB ehE*֮N)]ށxb/'QW΢t>/oܼ~"|kz,=&wRJEUcPz 2V(]Uw|լxuq[siYe3L_'j]D 4vh̙"mm7{*{c~KhGg]uwu9u[rG{5S˱[qdacWmm7VA5ǎ]^E*t7jcN0ΚK*5/$56{Y& W{jVǾ q_lxd̙]x6e^tx#y0̏/wtꤢI.eEʟl,:u8wly:PrW qy[42\vMD"!>ߜDJO)nBܗ[C϶Ә6mZdys6n]H^=QTTVxsՎ!Z@3~ﮌXwfssR0M6ks6х`]{!y{!y{!y&+''U#Y۝bDRNoU#~OxU@oSlM@ o.u_LY_tgg\Bҙ/_n}"''E@ǃ]HxKW.!o.u]tt!,=H^ ^H^^H^^H^^H^^H^^H^^H^޽FEyQP ` KXP\6IIfEjhg8vcGac1^2J4\,/gңFKK \"0ɪbwy*y"]vZf-h@+$/Z!y VH^Bh@+􌎎={޽ԩSrd„ 3g|E m۶ {v9wޤ$m69}[GGǒ%Kϟ?hРݻwWUU=zj=zfϞ= ?%K:::ܬc6l6|Y]]kX #B͖۶X,a;b2L&S?? k׮y7_B}衇zo]ʗӧFX,gy橧 ⇘L&+]e+&dΜ96f%''M!yzڵɫWs9ok׮o,ctB$Ng4+**/_>jԨ)STWW_9Iql^\\---~oӦMVUq 6l|MmmmOQ]]sdYR]]]RRLYT\\(-Z4|~zŋz>((HdJRWW'=x`0̞=[/^[n&Nl2!]R1Ͳ}d4}Qvvvpp`HHH,=:))I:tj9s%&&VG}XsYYƍCF;wB 8pРAr˫j$%%)a2E,-----M>M6#B򹠠S|)Lpp,l>|xvv~06aÆ+Qˊ+_=q_b 9Ḑ3}ػe|{z-!D@@C-αdZxqVVVII ^B$/@_;cǎt:v.Y,]jjjv#nܯ!v1:NsN#;BUQQQQQQ{zz_~C W^ݳg'2D^<&oVOSy]#yN/$??ԨQf*((u:]qqqKKKvvfKHH͕'""",' Ws?~|˖-v];uT\\2b۷lr^{M#ʳMnn>^\\,tv]IHH>|f޺ukrr2/z :}K/?i͚5z_oG̱~5kg͚~̯j׮]aيFl?-绊EE111ArGn/B{5/@tҧzj…-r8v$zE.\SO-]:ȑ##Fљ3gjjjN+=_rrr0H6ʕ+)))s؆>/3ׯ+((Pz̙3fu%ho7n|?䓍7>3?1cΞ={ر?B{3A&ҥKwzcі-[BBBƍ'_ZV&{/Uy_j>IOOMJJRˤGbbbY}҅ >ޘԃj/ƽL7k+C?iӖ,Y-ZBFPZvͩc#ݬ!R픹?4ay*+i>ڜTA>`ٻ(Kun+pk&''^f.~~~W?sϹ_!tHJJRZ|=IL&SNNzD~*-----:))iʼn]%41LBltNp M6zF`FF:qxj* :yLxxmǻLpwR‘Ҳs^9BSXGF'G)ZinnQ9&w377788855U߿>ӧooT՚ӟ붋+CJ$J Q>(%%%/̴zk޼yFQ*iNbbCwu#>BJOOGa0nOc}^O.0>JNɤ !Vd646546}yȐi MY].RZvk]d[>trwb̘1#2֊V`dd3,Kii:&Iӥ !nܸb˗2hiiPYVq.撒:4n:јLޕ˥<~^yxehq٢u97n<"῜.W&()*:,n=(ٶuW]~J:蓂C'Opɓ{1xvvvvvl...޴ib޼y~~~6m)ɓ+WʪSo#OɓVr8S4bYpa^^^nn:dC_ ^tu,Blꓗwh:˕ M뮤U+䳯РԔ|.w|Rf!n6lX{{XbŊTLkkk{{a/iGWwUWWܹsѢEq?cDbI,bccՋLJLLTIKKZ]%Ų`,7WSKW'_lE϶5 W>qvhPyEVۣ:ёWYz3喿?_N<@EFFtcǎ=S?vNsh뒬p:ө#F|. !^~CLS5VQܬfW/$Ue5ꫲWgYFW_0{si94D}͋$۵SNaϿ]徸i.#)D5/F+?V7dȐ~n}={ȏx!CuETWWBBB]tە/f #زe5/BUc6ׯ_uVrrpps=+˚nXw`\4٢ё᣷}!r6?:>JܮEM&2gڠ/YrfaW5/zܹs.\e˖Et:9wԩb۷lrᬬ,^߃{~j֞&oP45l߱KIտoݸO۾cWD3/T8׼Z{饗?YF׿2|9r2nk֬INN5k333ŋ_vh4lيFc~GIII)))999zEޣ$nݺOOOWyג,EYjռyF0lѸt'NIZ =rzġ<Z">`~*:䜀}ǮA ncΩru6Ʀ5Kr6o߱K!Ҳs7ry^timm… O:xb#!իWWZ'-]:2B#GX,ON9sfԩʈ%''G]!܈\K駟&%%uuo~󛤤?XQYYi&tultNIz˗ !:dd2;Wsb#Eo}U+~tkUʳO+* `^˥Gw8$hllnk%7\z}@ֿ,XTl]y]f!Ĕɓ_FGn\{篣?<^qO>ٸq3<㏏3ٳǎ(HIIy:d2JJJ.]3嶶l2n8jl6EBZJ~ו+WZZZdT[o\?44T鸬d[l*++/e`dZ6m޼yBuP{tsmYfֆ#G !d+_L4 ʈ^R#J/v_/nP/ٿHDDSRR};Zʓd֭[䌌 82yqn.g7{s8+d6eَ>L)**#.{*!DJJpUe/Jbݖ'++i#L07}Uz3ם6p:vCLL<@ws7W;9׼n>Fh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^Bh߽M\ohhji~F[Gg_0h#i=lٲ{.efmy'dnThmg[Ç drW+^=ec~[d5k裾۲ ѓ/^kNo޼Mm o яj^WVlo}/)/_{˿>́ёnokkw`DLfZi9sѿN{+WTYY1ew}W hֲ~ÿKOըŽUU5ǏyxT7SKle{͉]+mmVd mnnQOpSTl]a7SҜ;eTA16:r}}}_e>ޜj&rC!đIU H{gt 'T^V^䞛FJ>ni\d;p9|Oϣ/J\9+/9tutM>5>/n6}^@g]7 7nظƦkZ|<{B7 o!1ve{ }[CGODv#^lEŽ7=>,wOyF#jL!]]/ݽO\dpkJtTm{tEEG-9sUU5$/d1y&&JZ!Рeo}~qӶ$8_Q~hg}߁?{ 7_k0>J=yawuuJMFw$4dFWutv^9?P4yqoO3g7gI'D ,yx͛>m0>PƦA}KyhHS$7CpOD{#ij%/c#}i]ik"rsi#% jhl:_Y+/u{ø_&/Boo_GqI':6rpK˫/tUQ7 /v3ק3_=9P6rnyq|eۯks,)-;g\~g]vyT_stw6$xjZUx{iUM_oL^䩲n}^ws_8hqtcen޼9 v7[>yzkOl??_VOgtljg*/T !ƄΔnGB8NQb~to7ۆkV_GqI'lPV^퓃v6u e`"y1(e/O"OѮeL#Z } Їo1b͙sw™sUkzz_5~~jo{xTp#l=6qv@E.^=}lGgڀc8dE۳.TU!c". \Wrƍ6Ro//??߀AC1L@+ Jh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^jW1ڄIENDB`pengcheng.mind-test/docs/aicc_cards/assets/train_entry.png000066400000000000000000000463631471400134000242760ustar00rootroot00000000000000PNG  IHDRz- IDATxyX'*.$GM-4SNt,Ǔv44+V'c4dVCǒ$⊢"3, 03?ƙAAy?+F_gSǎ !ssyI$ppB88 !NB@'X][{zJNjv!@t緑AtC_^B}rJ;㵒>e!Z4e#wezLgB0ܸ :N^N:PQi^'HˇMnis!Z|yO}e˫Kd[_]+xp']0TRo8[Vy-[aQ97x8qY];3zjx:`%n rEӃ Rko6b׎7酡/_կk]+ڷa㝿orK9TRAFVP/B\DB-4E=ЗdI6Ȳ33;@O:RD5MbٲJΖdLW+tl/,o"DB[_1 SNx TrF>:۵Z Ϸbv sZ]S| !n-nЋ:e0,>%~ƛYCWZoܜ-¼ uin)z[++q[iQ!0?B|(,,6μ)*)g:BV^)>o7VՊJۅeh0o*ddd* 瑜AFF2ln(K 삾8KQ< G9%.pI_1lWuևH}-Š70׽gvq eቩdddXZ0 R/m\.ތ}|ŝń@`oB.dKH>ތ`_7"u]?Z\.ykFL!1{M2 5Q׭W1kcHe,&>2x7!5)-a0r?[3x22ċ>v2!z*ؒ_h6G9Wn]qZ݅KvѼ/\G0<1UO . u.ቩ$IΘWhb`▦?Mn,aӡ>X?~ESyo:OYrgbB`H/zm%Y,]39[Zi^ZJW*xˍԥ|W\a^J3Å+7muZ4[Ϳ ](/ښ[[y_+c7B*0|3u޿G塈* b27:On,ټ%)U? 硩נh<]їU3мm˕rf^!{K*[Rsz0#b|׿V`0dbi8< v*.5<1G1(cQqL+l;$ (AK6md3X36`#=ں@H^Sif7/Œpz?6U[^EǙr)(nPDQO-& ՊoGIjrx ~ qn.D:Yi`#`@uo|AobҗX\ w R!|$zApP,R'[^xƶ Ǿ'k&Q3H0iAE=w.UE%DxTo鴈@WZ_<]imՒFI׵Wp+n*ze] q$ddP/Cԫm3DFƤz1^, u;t&"M,C q$9/^ōhGǾG|`jTL%M/꾏35i! e(uz])8_ݾJ "ܜWV-GQd62&7_W2[-₃i{޸8U<âX%U'Rlfo-"iѕUڠ'!—ٙ1 _ӝ`@I]@ѶlUn+Doj  2][ԹF{r*N  qkq!p\?b0sAv us6v,AsDuYzUÕ{<݆T4U:hW9;a0Jg{7 q{hqmUʌNSgsQ9;,nΌp%B,df-:rsfB+q`ނK*H/|]]uw1_j:.[k!`ʟrv'ph?S`7W'iT:_iur†&L+\rs1 2P s|.@;mupqBd,h iZ̰ᆺ3#ABjqNB@'! I$ppN/d^BD:O+'7''2חUL!$ zs44 $z 6/! mBL7]@B@'! I$ppB88 !NB@'! k;x|}XV[@l#h() BBƲspqqgg]HJB88 !NB@']-V[c'8}&+MvMMځ 650\eZ-R~ %ڄax{{X//z;ֵznnn :Z\EBoo߽!P\\}G^^BdLjp RK-Ͷ~nhX`0IX p>reנͣ'Pt3gαj>m>޷lEs6Zsرkee˯_/wQRi3L姽P7u?WiWO?s%:}ShA:p?LjG~Gne'Ns/V/)ѐ)|Ïh4x3t...WΟH6 j9s _z}i[ɹˋ򻇇p}Wt)qwwˋ ڷٙ 6KvW9q:oGAu_/紃Lyf~Z1~lN J =?DT. J68;;ƅYMOqqqwF͍ 6y,~{9fvia¸'0 'Z10Ngk7xEx(;kjY^#?EӫgwWg{}(?zWT~ܝ:=GAvqBCZq>K iMqI %$$n:"!pۓ ^J%]{ɹMOX8^o`;۝tډrN^K->r##?C`=o{u2m: ^})b۹ps.^0hLDqO6n<=y,~s?!|¾ags'.B]ӻ'.f~X a}ϼȣ7@ii)iS^^~zvŅ؇b^vC5b" N=GKxy)8wV77W\\E眞~_~9Nm׷7m۶|?ص~z^4a >7&;99]Aʷ;P(y g ;TjѺuCGs vCٹk/> sMBAD$<,^On^t:ڵkS-O63O589Y>UAW,ŕ9yA.eӱc6~~P2ɉA0O|Y_l?̚1Ik*8L@?'O3_~+u]quuJrNlNb Qݘ> ?lA~Wu>"{Ov39Ӊ-N!0ПfLAJw) ɉN ߗ/>d3~,mx{Ssiʴ ?:gϞÃȎvM3(hP(ܩ4mڄ奤XC7FEu 'yw*c16)|z=:]ݓ>AXΝFS*!p //%1Cѵk$ղ.`萁|.0wG+{T^>]:G9zZЭk'ŷ'qriL:w7Wpr"%i`PL~}oc'8zՙlӃcO來(x)>{:]{()@B&ۻ޽i_4mB[[]/f fNDun];Сmڄ$-_} nnnTVVRVZFP`S'C/{+=qN:ҶmΜ9?RG6 зFccÿʏ?4ʀ~}Hvxo;n~N;hƀ1xm"oFqtt?z=.fR9.0 \tٱ155?G5=w^ly~ڳw/Jytt;yiYG痣 zztJFAl~kNxX(]:GZ|j.dRD%4yIqq {ϙ3ܹ#ÅY>r¢b:GŠ̥Ѽf! !ncz1!DI$ppB88 !NB@'! 9l8[yAB4! AB88 !NB@'! I$ppB88 !NB@$hO;Bi{}ۻ7aDKzzR _cײ!-g6H[*`1 Xꀽ'ƅ. 1,)?ba6W91s߰LYuQGBI"ԝ&q|Rr&IOZtag㉪vV\5네2qAi1gK яIsqb$߽:!$G4ap/veVrөe6g`ujm&dlCs9$RtmDi@8 k8C1|^0W0DuDe^ړ38נ3BIJJ|Z ƅ?h|;û\|qI{˜VeVhv + t彤(j &7+K^Dun!줁m=8Fˈ;};_ 뻏H:V^VqSf9}z2FghN@a' " hi/bٲ0f;B]HptEh9 &z&,u%|i۱#ǎgb\6:LB@8IB@8:{O@'! I$ppB88 !NB@'! A)%a:T\2:W__ZcYuh4ޓd6'%%kJ'O9}^J%~'TUԹ3yy|uy=0{椦>u!EB{XjϿfy<:Ǣ%+hJ/ jVHddݺv2/S*=Y 4п7:3mSlNJfˋyG _{OOOJ[h-t`ދsu5h4,~g%g3,Kӧ=mamBFI>N@{Iyt A|%k̂Rzs-2cCkh<<d_QQ^Q @Xhnd]F=65..\tfBp*NNoJNGk65d_UJX`ʴ\fDSJ҃ 5][I`@+_P7~~>RXT@/>^s (.1!(o/OДJ6!(\VQQi3  ՕKY9zڷ ٹƿ;]B (,|L"*k w"  I!ztX IDATAߥl}SR8٠w@8f;qex0c]̚ذBG ᭵E̘7Dt.CI۽XR}O>8IP#: ^LB@8%! I$ppB88 !NB@gG452nޓF#̞9'ϰo5cJyF㸇2xT\@^^AǺfϜTsBB kSWiMӗb`c uڇu!!`;$`ZXL:궦}ZrT.w{ao4x` |ݨ ѫG7Sd?ŐҸ/" h4Ѯm'TRZZJ`M{O۶([q,|vI7!)2٣+N3_ڷ5jn/S_W" &8(м^s$I)%m=|ӿ;{vvaV Ǎbk/4IBV.\|5OǓGxzҘFcOFK8& Z~j4 a#K'GOںj@ӿON q !MB@'! kAmiRwgRB؁}KH6?@ss5z$!دw@w K)->_g3mbr4@zhQkF(zLa":` !q!%uF gʏxU}G`̹7l4SV@kԑv4:hNWiu'-`D5bzrG׬Sg g.$oPNX 4 0H{u`CHьƭ]l Igܤ+b263kT|GN 6>@bQB4NB)u,OL9+Hx/-"ػ|x>>ɤЅXzҷR2E ӰPkEp'2&Vm" E϶ d^b9}z=rpqv]**+Օ`rP&$R-9< ZP\R:8g'A~_QPX |ͻNQVx+Q)b䏗҃k:WTWTYQ3ȶ o /]EFء](.tvr}P*u:._Ņ)/ q+am)+r{z:M* 8ȟb yVЊ덟yKJ ˓k<4ZBTp%GV[@@ w.g4ʥtz=ۆ\ĝa!ҕ?#(>$tD0B5&1X/b Djp`aFg1IysBz1ں@)2M#|0(]@1;>1 $/^2ƳeaLjEIG!`9 F@vȱQKn$k!М$kQ!nOB88 !NB@'! I$ppB88'pl&>06|cagaI5ٱ+iS"a:?luߦwI}łWfd|;FSwVP æeg32Oӽ<-GB:˱w?I?d] 6cuԹ|+`?6sb9dvI`IlNJ&<,dيuv3EBI|u[3yyX)xc{V=o>l :F0l۾y/N'oٵg^|~*'~=b. Xoayz4+ 'Oڰ9)aP?żSѰ|zfN@pPEu q' Y,߽'"7c>T=i*IG@ .>8 ck~mw.S=|{ kJdRZZ:O^Je:q [Y3&Tz?:F?eg {fçIڷSM?ø'GQ+ 22N;psPԨeu]:2}+8JץM64mSR{@&?g%!Pô)Oєڇ3?ƿoRv}{IcXbaVIDS^4b9@Oj=h/P*=?w=hB%+c(5YI@ ֺLwp`ދN=+&]G.<:o&X- T_Myf%چ Z6S@C/ih%U^3 /wTzR璗_@6xzzԸ m FG, \{\$!PMnyi4;_jӧrUҢ4MN"##V fHtΥjuCHT|G(F-~a,~gyIۛΞ47Vg ғ`^~IXMiiY+]a q !ՁbRW.uMEPяa~yHa?-x\{ ]N{SqؘTqK5:71}i#f^Q̔{>0#{X!ء:ŝ)*mϊmq(ԟB؉(nRPqb!-#B8& m~.!_@ƖHrl4 ђ7tIzw dd@z3)-CEv=NMP̚vgK]C!Dsu@Vcmy9ZiESAB,,tѻYy(!Qb.7ǮB4-' I!dhavqx^MMV?MOh)Am6v#|H AZμ0 ]v&aP'! I$pp2a5YL$Z|52bq ^mðC˱SV)74.g_5F RL7w@d#om_nƶ;Ա_nV͡SdD#UzjѵHBw!ܿ_'ϰ^3@ʳ_\Lscj{t֙/ޱ('^=`.p}pHs +6'%s)-UԹ|a3Ǎ"8(:oܥض}'\HX2Yl`iw@ۘ=z"!NB@'! 5I643)ćwo!%$N| ?˦a>']k# !~v]mw!֏'.Gd-jנX3aE|m !q!%uF(ά&q],}i~4 Pب !!;M℧IƳd~z)s6f$;ӴKOfmL槯_ ]WBCв#ޝBE,ah a }n +m/!DiXnw dZrO/( @>\tO5y_s2/^1Ѿ zY98;Ӯm\VJxX0Zm9WrrP(hDiUJO(.)Ek|C'A~oxKAAכܼ{ h:EAx)=ȹKi :WTWTYQ3چŅ6ܡ-.]ɉBȺ| Wچj\ kLY׌ÝցhPRzOQܼx+ hb χB J un%Ƨ'*M)$8OOWrh DprJIxX0n\Aоm5MV8IF^զ" R74pEߎGI?ߠ3BIú|Ĉly;b'jVxgE5#U3P"ܕӢaϛIu v^Dt@a lȘI\{-')jU !ĭށ`XBjN^LaŘ^a'6\NF pWWQeZ-ɱAkߜ G'3IעBܞ$ppB88 !NB@'! BחRZ,{qޛ8yLBTEP.g_X)2_b&b~XLjG1=i|^PT}طpOkcq`!6ҳGWfN9}`cl۾b^=L7J~rVCfދӹp1R6)vI`IlNJ&<,dيuM!8q $:ӧ3߱MT\@^^m`@0I[vgďxG,}0DURpLRhBfڔ~9)su0jdj@=iPUځ#66'%mL @5Uv. l۾O$;;bѣ!8(<`4 VI!#USu@.|՘ض^'G0_'NCa} ?K}ڥŀ:XHHX|GS/Nҫg7x߷0odjxzz3$ғ`^nkBEJB";&! I(I]./Bk !OtIbɻ_'5Qv4!>H;YADB$Ձg3fA m6Ava# @cq[#n/'3*:؂$Rz$!5v~x=k,jT hՀZ DKeV NBHv [ՀZ DӸP#eջU .1E|~ ud!4*R;z{E` #;sh!4*b{{H^;eem&)ao9N7l[,Kdl%QߞQh$X# _EB[ί xRqkv"P@^L{^i1XM> =J  !ZnhQ R6 !Z olB{׋ $ppB88 !EXJ˧mIO<Î]5)O\ 1Cޭ3M)Vc#[=o5cRwI}łWf'(5iBR{1Ϛ\U+ jп7m4Pl9)<5ɅY՘ttqtґyy|5g~rՆ^e{ɦӓe+B8 +c'9qMhJ&e'O٣+3O(MT|:F0l۾y/N'oٵg^|~*'~=R4扱Z,hؗOBSfAp1RI^^oa,f "}Ւo-0Og5 GJowf># *WoM#آNW@̐h:ouklS} י# WJcO&=6IXP*=Y .fϜJRinH/CVu~rwoR65&|M)]tdQ rM'p 44bpǮTۚ.^ӀwwIXMiiKZKEnyptyH'! I$ppv'$iE*}Uuw"}(!ح$=cGC=jTqKe@9s'd$[Aj|6C2N@1Np^>φZ ־Ϙ >R.>:mMEd}+DsU^;y),_';&wGD?Fwmv@T|#X'isz1b^,بrgWt;HHld)3)zi #/o`ɾ*#/o#E !!p(wK,v DQTuI }{qH.WT\mKriTwc IDAT`̫+\sQ|t!4KE|;$>>mC!Ձ4x,. h@=C S#|/jхv҈B=yijbV5r@az#}~p!4j`۱Ou))]\p؜"olܱa6B:kt7v1#gZ!H(!w?szc3D ; hqB88 !@SjY4%JOGgkGgI?]@70n'nW[.qE7;W>*WKN;ۛJĂǟďTˇ%L-_FJ^yYgO)bHt0ZS&qېDwv[J4|U~ɧ{g_Kg6vKY,t!th/<[ĢOX/ ?\e߼tioqE Ap|Ӆ+gn|=H+_7̎"8׎C pwASc2d֥+5J%JU.di,Q*WKAa9;͑&!ЄJ4,~$Y~5s8/.Du%z9 jMJ?>g*W˞֋]9 Ȑ`vZcҕyqxCBsL!~6CBY$ҕ7_YF:vAE邗բoMʮGiFMm}{o.ldwXABSi)+\eAB: +P0_'Y4~0^JOtC5ea 55>SIUH"eZt ^ʮ^m.qcDSIE48VerzϾ;JUH"?Q1e]7-jў1#ڣ_8*C 8qbl?q%\a3]kXW Lx.o'`$n+9eL}A u05&V`v}7_iwKP%b386 $;HA`*L2TV)``t"VNZ*du(3ViPD $\r#w<0cK H Inowo~8ᾆ)MBش`?UYx'wylQwYbt*7c6%sw[f ߕw+ @W3%q&cDJ^i˖ 1`$KI sMtvuׇ`Wl -wZ>pp5gyG+eK 4 bl? !Pͻ6c'k1d*/z5\441 گ_"/ +7~l3̝3;& "~1nI@:zy {,ϪKn]&!ƦfRT5OWbHMMMŎ*J̢ErDJdE`ʤ׬\^_f)g;spu4c~$pXb V7'N,1xT$?@vܜ,2-InS:g5-?Q=3Jbֵ٬W=#;7.iV=DKgĴ ZRM[-/n6*ukFy$CGǿ?(G5qCP+Ex+VB} b>Zˁ,W0 "aB蜄:'! IsB蜄:'! IsB蜄:3IENDB`pengcheng.mind-test/docs/aicc_cards/assets/train_logs.png000066400000000000000000003126661471400134000241030ustar00rootroot00000000000000PNG  IHDRG IDATxogO-zl6֓!:NhiҘF.dGE" E^ ΍hy,HŀAPvlAzD<'ݝt'}Fsvf^^TS$E۔~DB>O=.1<]rE0vB+lNOJo_-j)/J_|U??]}O~?{YOnc=j=G?盯'?+P~'2|`ӯW4޻_cH?zE_}+ɿp"7=y =J^YoW߿Yӯww$Io~M D?XFg<w?V ǟH)P# LX_ʟ^gNcK}0ߙY0ߝ1G ]?ú>__c>Q;ϧAw8]_hWWzQ?GЯu8?/|Wvu<\hϧ~r,I 6YXxIEs9,:(/R~NJeX]-KꓰaGkwcAwϤGn$}Hs_~|{mrý7$}򱾘N˰NG.9>^x_{Jې 밤FP$ֿ~?~ƟO}7X{EwwAX?~_w_]YrGX?V~ ROYx̿wt51s c"fMc:6]lʻv!K]y@"` 35`&p?= L7`\~O{Cp\ytzQ0XfkL ` 35`&FI)#R#8()eU_5}b*v2Ym>ϧsndLq۩|֧x`*1nGty5wkQ$CX7t練vk޻ZM_t3TbɒJO]FeSY~P-zjFVZULuCٝ^YG'qݐhjN;է$i,ϛjH·KE51p}ywc7MLX҅^cgX_~Kw`c]_Gt=_{/h˟"S=isU Ė{t;%0$=)nd}Mx{rtGEY;\?IV4I<5,i\\ZcOEyO&yTF(yw 8V9VOZ55>RHZ/i1uMjU[ lèB6Zݦ{E5ۥE6Tbp斫l;޹yB]zk_6ﳽBA ݛ?K嶬RzQa! f64䔋hIMm8~=7@گ}BAzNYE瘗9蓞`׭^0zWfYk12sJ+ߞm5G7}Dp}kҧvt CFFQ <!sv=OeIvJSՔ7תfzT=hZh)kGژV}J |2w=ySں!;wJY4;!S󒔐 @ΰtVWWۯWjGǚ>Xt{:M5>/{B' H,]Qb=arEpa, W@Z-}J ]j5%tN凶E>9f 'ͨ̚/u7?A0H 4JJCwsye/T^Ժ֍1wSnxgvÎE)M:ROviZ-VcʖBvXw;%ch}vOY>PXz)oTjq/ YkjuZƘݑd:\#}οzW81ϥk O9c7l9':}թ4m}9c>N鏓B}h&w;%#/yP)vK!<#uCE5hL踳sÓ5U\>wSʪnr 읜}$(L翖i\ٴb+XC#\Ρٻ&X8LUn8u!Lգ,;2k<@_q׆3'~(gTNPDt].1zEe3m7nґK\$a Ö́Sa9+i3k{}a-z:2_,jJה,DKK:jV˯i=w'-ǧz/ [NXaVRYm(c3 :l4sդlmÍ2,W~`qg_ 7^`=AfofmVGFud ӼHh0DZQOlbs>>V,ltILÚKLd7eR뫽6Fd.eųu:3t/c٫*}hKon*6ϟ^X aQ`Lɨ|V&+]rҴP6nuZ_Ak~s sQ[ǃw >;JdV kLhE5`&@0Xf멫)oJj2R*5CRF^![c'^>JXj N0g=vin^1x >ytOJuP'+T踧=hgTbR⪤̑eTw>_Lm8ژ˲zrYx? qJ(<()e 9wX .Y^Y~Gs]'~yc;3.5;yO׽c+ey;QG?t55ws]s;|~oh ` ^xXbh_:՜^~%CleZtmkѵ;*YInǚ*>;N6_ {k2dGrr6 >{6q䑣߹h˩O+$ٱ^Y9iQMWTTne/S~vqFZ;(#ّrgZ[ lj {r7s3Ӳ7suܓ~90(C63Vjm頠dyV4j;i&PteבV]&#kMFsnl]XRv6索+0Os9-k'7Y *r`~a.eUk:ئQv&,`塜?'hokNsfߥo}o!H7vVb?דwHG.Qr%L<5vO酉`7ffY/\_x#nhN|Y2gud-?Qה뇣κ/0ǿ9^{Si]c f*XJ\=Yߋ~hKk2 ;)t)˫Ix߆eu`VG_; x^A ➬ 73FdIՑgYñ4eL`3c#<[YCp>Ϟ1eXʅڙ7F u{&,w<kn}5eQ8Vz),2= T[3&\O[js=_X{`vM>V1'RѱN+U*=hJ%-Z\~%EJ<je6ދOx\T]ZdCGE)V^okvng;7[+(E%7)(HkQgz6 qDξʶ$7MU͟Qy ÐQTe9;9b;_%;ڠ3mY Jٯ[` sGimr8a~C-ǒW:V$o::7U56VS_g3^ɦ._sq=&T\G=yɓ'r03jCRUG+_!sv=O穹e=MUdȫ/wO!x{ S'nf_ _Ӟ޿3]?|BL%%]MU8֗jtE]Rn}6R<_Ad}vJaI aҍ!~ISʇ~yp1$HIm̾V&ojuuUr.g g_qYVO.n~a8xY]FI)còap9|V J;zg_穼s]'~yc;3.5;yO׽ҾLeV) X^C]s_[=5^ w+-Mo4/[هyT pS[OeV.(% 2k^'nd}e j7Qnj &T[7}\TӋT^;I97~vV&gRz9 fenG5I^]RQNAΎYkAg&gGIjm5` ' T9݈\N:P0AqOFy X=am^fb EWvKiۥk2$a4WyO=劥\d9 \NfVdg¬ Ԯ/7 U:,jPT=Cv<Zoۊ\.~cyX'j"rWPʣ}0Y6WS_=i2):azE$䌲 b%W{Ӗګk~]|N>ytOJEu/tXRASRr/tc,.ZApMblQQWZ)J0nѦvMG 3}7pZݙkMGѼ6MSհ~vT^0ddex$NNX'$ogo(6hcŌr[VtE,HRV/ïg?C$QZvܭAݑoXJJB!ؽ_熲jjVnf[d`._sq=&T\G=yɓ'r03jCRUG+_/o=ov~8wMZԥq/kqžnk?O2Yf*dIW%'d.,?8l=NQMy#+{*n躡NNUޣ/FIr}Ͽa+qgէ$p{<]ʀ{֬̕msk/c~Ck;GDj߶_h.cMb}hGEY;ٞW?vvh킒a QRj E٣}[vJFSRMTR9TMΎrI;jܽrCjS~#@d~jrvV36p@܍崬 dd۞gLi/ʐxC[:((fflml)]ud/nZ\=8[+rv6索+0Os9-k'7Y *_xj&תrZ\F@T=Cv<< `מּ7vVb?דwHG.Qrt);K/L<y%7v6sO=udwg8rye<2vLYY˃2;9F :7J}pYF7Q7kojܐqϬ9vEb Bwm,%Ĭ ?wlr r9jR6QpΕL{v+AܓUdՑgYñ4eL`3c#<[YCZ~g1+ra6vBal竉(]A)Z8~f̖q>RX&q۲C{/$v gM-㟶^%X{M.Ƒg `}szpɣ{T*|tS}JEJEa;yNJ偎} RQQy\-ܠqݦ{ jKlh(+ b zu u?ecXMG 3}7pZݙkMGѼ6MSհ~vT^0ddex X'$ogo(6hcŌr[VtE,HRV/ïg?C$QZvv9sPyK䕎 ۅk3B{7 eUԆ׭a|T<=sJi/E9*.<ϋ]ry!uתm< RKQk\Zg0ѦLkL ` 35`&FI)#R#8()eU_5}dMq} ()[өmoxXP'oڄ UJpSq˒Z]GSJn:meupq, W@Zͤ[Z]]ͷnzթj Vp1/Ye'(rY%uX*s<ڌ\!J;zg_穼yHzRT}dhjZ(Ȭy^@ǔT)vOmPqQM/Sy%:l&yM픲4\eÀc${'j<Ue)opM2w=y^U9Y~c6v2>LDsP3Ux?*X}K)PW=dǺw[4Q/zTcԐ;eU/vmTTrn&D}_r]:*,8θǤyU{O*{U4,5GJϣt?n_yUWY^-';>ha<`L>yw-qIV}\rJ/iVMΎrHI;jܽr+m3ւ,ATMΎ=8jfSNuUsuPW`㞌́)E٪ٝT{hK% ،-UAu31*û쥴5Yk0'>dλQBANtDs? ҇ݫwf/( J9U@JQA/w11ῢzlٜr>hU?}?{6omg'Ufڭ:rx;7YC9x]?NN:՟7^O;V}Or6R~'v\" Kxj&잒n^M<2עl f{UuB,][KYTǿ6ƕg;s6Q7ko9:>yBsz ɖ:صU8zV? +Ж,efX!/&enRnIRQQ\mݦ{ɨ㢚"?*J>Ŭ z禁 u?ehs#d;loֵ(3w=UbFѼ6M3* 0 EeY&ɼS.VI5ہ mQn˒XQ+en1z3A2*Uo3pCJ̜_/b˝HL|~Pe=TdT9x˞YSpTjsy>/ψaJ?0'Rt$zE/4S%K2GnQy[TAں!e)+|@ajL>7en& 6o&3qQKoiuuZ|U΢.) fuMsX}q1_0{ d}vJaIE8 mVbjOJ4P߆o jp{E]>WdzԆyϚ.'u7?A0HU~xWay0<^ن K++}y̎^Զ;ޙInر7I@)ΘBAle?l9*,oebYјG{=8|Y>yxɊطUbozѹYcݟv_;56ԧs>~ 8#d^kzΰ7kk0;>o[LDsP3Ux?*X}K)PW=dǺw[4Q/zTcԐ;eU/vmTTrn&D}_r]:*,8θǤyU{O*{U4,2G cv:J4,iwݚyxM-.'}MP?n]4i/aK!Uc)27rҎ$w.g_hm`[G⌬ wPF3U#nCV36p ܍崬 ddgLi/xC[:((fflmNN#{)ttMF֚d?*>ߥ$nPpY 9rjϩW] ,uQc~=&c3wE^9zc+wğ0U3x& Xy(cdJX5 }JEbm7n 6q}a Ö́S"a9+iYf,s'den5Y+(zV\GXKY@:jJOj1Q7koj\kfZ\]} ˃rR Wjgb5jҚlX)9}y5)?pl͕zܙNGPC'9͌[#Tz΃yw s0+-EYnXo7GO{?3{s>X %3o<AaM{Mm|PI)zw|Wɐs~K#o+g${'AhUʧ-[ ? j}> Wum8J]OյX7A4쫪hԎ*,ρd)+$rk6V(en_x~= Ҫ!%fίߗk yE{$Xn&>?2{jrJ+߾_3A&T\;V((RũM}qԔ7l%{̄6]`XfkL ` 3Ք7 w%5JJ)FI)#ڐvFZ /&-ݧޓaN{^֠uNհ|Z%EÁjrv,]FNqT,HmmWsF֚giבV]!#kMNcI=9nP*>hEgiYriAKţ 9^QW 6`~F6xAF.jgCEd/u89aɄ.T_q/ͧ;jgv~3w -%*Qr=yUmgW;Ұg(9cVZ #߳v(`}c7'<ӷ7\dفճX懶f) 9}y5)5(kwexȴkO;5ܝսb)wPW=VOّr7pGͻ֧RD]z4Zyw^=UC[z;2%t34=̞u0x́+OFeҨY/Lt]M>V9Vo=U*U>:֩aJMI_^T~~K? \8KE%e2rva2wJEelQQm 7#u斫l{CzT0d,ԕ`Z}7[(ȑțfT>J Mdӆa(k;9 ~xOxN;,79S?ϨOCRzQa! [okTAk-IkQ01vVb ui$U_.tb3<o25br*2"0Ȱ@0XfkL `=u5 C]IRFJFpQRȫ6klCEV[7dX%Ϩ?w;%ںg+~ٹ q<[=Wte-Եh>PaC]--~Q2S%K*i>!sv=Oed pjYkUyw3G ewrze>r}ruCv;_g!sד7_)oJkc7;L[)˛ļ$%dB܄/֗q~: fnX6G;M t%- )ݙSB*K܀J&}46Htȃ$uO-ƛRUp}<ۛ")by~X-rs+\\{^aw{q s߯ݲ u/vXhN5wTznmNA?GDDDDDDDDD0^Iktg׻KϱgF%:?}ĝz˃ 6M/KfTvQ5C U(.mKXڶWn)oh3ڰ&JQKN`6+)?Z.mzeQ"TVB>`nV~aH X@ 7IǻzVoYPCzOD)t+He;ŸQfrM X}fc/ X, ~=pppyX:*oTض u*jl^9mP 5(/#mn9wNy 6P? we/ T(iA. ?8mvT`Y2զu'մ'i&j &ToYmJ oPilPgg)9Z*F܆mh-`v,TKSׅ6uwt 7;y߉(,sߟDԨNtum kka?ڄJC˫<_Bi:5`(S 7a0"h;Y}L%*ef˲ߧM3!?.:Q) j6 bY&ARVJPOhZ͓Gsu.?Հ~>duaɌw""""""""-`[a7}?=x pY:Ljng_QD7~T [E8 %w""""""" S7sZNʀ\/Wro>WHGTEѰ8c~XWQZC5>IL?Zsv_xS}X>LNDDDDDDDD+C_˷Nm꿟 qkU9.O7lb,ב+ЌA^:z~MQjө1?lwoi bh6pTEu9`}w9֖ V{{9wZƧO\x%Az!}w9aoDg/YDDDDDDDDDDe#`}ObinQu9`}OiOj"""""""""" d X~@H)`}s?u_!-Qs/ޕ{\ϸ_3!""""""""".g۶-: 5yfx%Az"_.=}_~yK;Vw>f(3,>O_s,9!S2e9ߟp|{{kgmHh*L>|hfe X>VMGwz{9`36g*S7qww}?ݹ9!aMDDDDDDDDD42԰>G;`!~]wpoi wX=gwx4Z;ְ&"""""""""y9۶m>՞)1Pȟ`sBDDDDDDDDDDQ2dX$ tXofhf}!DDDDDDDDDD42=th&"""""""""5h&0`MDDDDDDDDDD3F"rr"n MCζm[|ÇxA.ޕ9||w_zk0'9^t7-`ӿn wcm)}_`̾Wz%:\]S?DX:ռ2j$H5:X.=}ǿ;"+7gͱKDDDDDDDDDDk%:? #oՑ[IDDDDDDDDDD7%Aԥ*ó5/#'i3.Vфm5m$?ρo|ka k"""""""""+C9՗ocsp{KkxV/ŵgH_DDDDDDDDDDYζm[|Çxu'[D+_wSh2ܬ"r @}j"""""""""FeXu#2cfDDDDDDDDDD4&"""""""""5n*r"ݬ+0P \.w 4: S?jգi6@UhOnÐnrUKO=Y_w"""""mC?\:K9ƹsrCin1X6,uzG H…}x d $:?䨚|pr0D)$=cT+;OB]&lZ+aö-eR+O C%c=CAL'ۙ[^h2F8qϝD .moM=up{1rM`9<񝈈f9`}w9?,.7#:m/d}aW!:oӂ\\c\UQVk(G A"#mh.6"*uM=/*R!k/=K4b jYmB0Ķ{m3It{nQ'DF핞z ھ.㒿ֺVf[蟪?v&z+{e/L^޴qhB@ThX0hc> 7- UvBDDDDDDٍW,Y{as`WN넶Cܿsrf2<s~5_$j( ɫE?2dij>66)(01l&# 5WbZp2֜`giEIMdfǷ'$3oЌ`?y/(ٕS u( 4}.Čn\q(eb]^ Pp̼A9Twݶ#2GNۃg g8@w]vZzѮh~T/0Z&\cXb&=GUT5Mč:ٲ{l3<)滣 v2I(Ւ)q@Fxͱw"""""[d%:?G*Ⱦw\.WmZ::UпԼp)(nx\XZ(c2AKp]Z"ͺGox*+=4W?#6>,hPQ Z(['Nf~.\΄*foP@ζFqZCP[0-hnc먼s`Nֺprlnfu*d(Y +Y A?v 7չ'XVbn`>u3uZ:t/yfU4G5 &vB@X4c<zoB0=M9$cگ3@۴P =n%g;U){{v[ mqe ƙWJ`N񝈈v|k'>&d_-:F5PU#lu6zy]h gϼ z8BuDK),=Sm׆N/wjFVg_5+=52haQA rYn2rz9ma:3p1僠Ǒ)W،zs?Ȟ³cVޓIpH&W,Oe0Rw 6 (W'~^IJTq3rT͙3ֶ>΁J *h#vJ/t#8v6ڧ* q5&AYOǜcG&JTo7ΌGE]&Ně ,?Z񝈈ԁ}-2}Ÿ kV_TvNֱtd$A.K:@5#li p<ũ ,gAFg9B$b,2=a`?$s?ZvT8L=`<A'hC6ee4CoN֟4.﹙/X}.t v=_򀵤zlV/CuʈG#J7Xx\^)YMޑK TjPhPW٦ڱU|N'ttQl1t<y>hQedLL IP;rQFN#HܦM'l_%n h4'\2g%cmd|fÄ++N)!""""" l۶>|\W{mhĠ2Dc`5 k"""""""""" HDDDDDDDDDD4i XL`fDDDDDDDDDD4&"""""""""5MtTE.WDuU5BQ_bBC͡z4F m aY q[ɴg1+[Ư){| s߉h|{簃Kg):8|!: IDATFh w@ $^g. :G mHtڛpvT". {m&jϘUyă86a'؜ -4^ l)Z>7m . b:?IؕeL:7q;68\N2n{z;=\?cpMc@g\޶gD^o@?Igun<Ԍkkk⟁{OqD\LǽiAE.] CGU@ ~PBEpZQ΅Р]lZET.nz_TB֠_|{ Η@i@3IA ղ>ڊ`mPG Uwf$!ݢO VcJ}m_I?UtM޹QWĐ6Mwv ] ?5nՁm}f LÂn%P{V̟n@ 1""""b ` s+?v ϰKy3j| L9ot`]/O5`te %ȠޒB؄Rr֛ۤ/kI@~J6PUaYRp]P0L t- fd ++xCum7b&9*p,?k8cpYN&}Rjüɥ nw}yl"~+#^̶WuӼ稊ʾutuǞ g|wtA=aND `:*Q&TK8WDę9.m5gm}W:&ڭ:z{ۊ-W}DmS?"""""1֗,b+!z2]r*X_j^ d7hB. }-1Oap|Dڭ˖@u&3}9m{~0D T[B_:q2s9r&T!K.6ұOmvڂ nAsgNQy 3Gu?)+4r) /8rzx hFBЏ݂MoGun yyUḘO`֮}]ΛFfiD#cIjQa'EϿ8/k:M #۴C9?{93 ľͨو*r8~{qo?&LgՅ6Jւ]qxLw+%yvҲc6v-Xv,T@n>=l`_Lj5r=6>˷&~ s>KkB" %R `z96zy]6*(ᔏ$ LIj_:ܹЋ:6Ϫ0jW5o=e{7jd"@nA.e9yaÎu!/f#dc>A#S)?s?Ȟ³cVޓIpH&W(Oe0Rw 6M&J ~%Cu*| YndEstf>XFT8K(M(8d\)zTVJҍmvho$Zț姚;LaG&JTo7ΌGE]&Ně ,?Z}(1x(z8ɴAە6߳ FR~{%[S/`9^ ~ u2 aJ, Ts:g5- A8,,8T\DE&g:L7Luk? mGÔCDq1tֵQs|zAG?3˶ӖIdb15\(鼞?8 P 2oƈuÿYĺJ h~1XfPGS}:6MԎ-s|3Qg "LB5Hɫ O"""""<yQ_ݶQɨf=Lr֮=C5݌ @iOvb .vkN4P5'8 p uD/!d +/tn_7ƣ4nn|6nZ]Ng0 m݄94kҭ.LvjP0҄=Nm&uo3=ʳx7߬"*Cޞ| GzU em C1b^-JI UQ'g%:onخФ3:p$sj69ѭ6L)c=Z4g&SX0mmS^?]xƽk&eX62 aybZq;^ݢS78( z8 (OLOjJG<{` rTs>nT:=AyXF!E !NRE"=8]( 1m&!{NM /+JOCt 3nh^E)uq^ctE*R2-9?c¶ xbZދ$4vf@,_^*oԃ3hN(8fSKnĔqo/eRw0 C6l=1dĉ8p?KX⦀JsE(yVB[8Ng6Lؿr*BŒ> Zǥ$>4D:ޓf2e)9&DDDDDtrm >|=D6P| 4Y~F &""""hH4{,4^֠ g!"""  )=FDDDDD3h&0ÚfDDDDDDDDDD4&"""""""""5h&0`MtUf]jGUr9kMDPmPsMc{/ڐk\ 㚚7xlC?\:K9ƹO)7]6,u{ #}тƆލ`u7 %7:ⅣVyfPnLrOj}IBͰaɖ2xsS: CAcG m| ]Yo~'Z\k^X;5gYc`g??"Oԍ) l$6q^ODt͢wcmmY3p"~>B֗_}WO@w2 *z}VAp訊¶(x\C\,uT Cb]nh.2"*uM=/*v֠_|{ @i@0ueZG[1hzm{m3It{nQ'EZkb~X蟪?v&z+{bDe\ Q BcWoBADELK^ PُXq+$~&Z@qKjOsu2)y=E?Y4O*ODDDD$H5:Xij pϿݠ];-f2| L9ot`]/O5`te%ȠޒB؄RrۤgL#""""%X_XY~{?No{wkW\ 8Bb Z Lr?u2,hP\x ^ lNk?j &T_~Y\K8vCg:Y~SaWiڰ4);-8F #P`*ciQ1pz^^9jn`>uJuZ:u1#pT4G5xHo4 7*lS^7cx?afhv}_6!nu)'c:t8ByXv_8lR{}m݂vgto0S@qMȢ[9 x@cmo}-`XH~0;GUTleq48cم?iYI6󏈈|k'bx}քkϗCI" %R `zV6zy]d6.( 5LIj_ZĹH6Ϫ0jOo=e{7킛}`t~w LbuرN  t4(\rUͨ0C2y{r? niФ{Ѝuǻq` J ~%C}u*tL^m(ӒYv igm}*%"q{nH7Usg~00)cRPg~\GmqFA;V('7ǙfH}76L]n `9R/qq8 lp5~ʙM= Sظۊc03OOG ft߹}e'8X~&yn`M~͚lOр}|'kق}- \dKٰ0L#vtϒk*[4b4fq*/Y{Kv.A"}vbr{ث! t|cg%iOt?b>*%0uECEz6LpFt^Oz^Q㈗]*g]o!J h!-7 S r(/[| &}X ٿՅo,I. ˗sBYkX_C*moP Ε2<y],=k;Ճ!NFmD0+%#gx_Z:t)'\Bz0r^On;dOJp`x kJb =Cۓ^m8[ u=\#zZ[ET:5]Gy)3_i`lkZݠ˦LOW3pkClrW?]p'5tgb푶0PŻfQ4P '?G̨ nl:P(8 DmBiEA NaAx߻~hW}?t+vGK?]!m}:JyuniР|KQZQ  s_c2i}e'cqKLߧ""""2!=~>=sqxU`kآFNk(D=hU \ 5#^F0useĩNk.ǭPǷ'(/Q(ϸ𞅚4y,Atim3 Q~71̙Dst*Jz( ;.ҕVyΣ2=q:sg>͠^T=zGo8ń:rm cQ*ݤ}Nt }[ƥ _#?#A}wMa2ĉDf|2vupQBE~rֲSV&S|Zc#34 ,DDDDDٶm/|<@%МDD_u#+ r:/ve"""""2?thXhA(BDDzB3fo冔#""""ۊDDDDDDDDDD4aMDDDDDDDDDD3k"""""""""" XL`fDDDDDDDDDD4&ht@5*rSܵ&P[(6PL}Xh9Tl۴vX08ssr>Ku7Jy%""""˷888;ts(.{@t6,u/c\lz :G mHtڛp1zTb@UhSn3IT{ W\Y&ߧf >VhJfذmdKo)P34=+˹uo=wZi?ľ}cu-heه T#n>6P? 7cDykT{ {o2{K1rs?=Y]oqGt]ov *z}VQm-/P𸆂@\DuTrpQn64t~nۦ)5%P1|j_V2^[ꨡtLݞ~0[IpanV`Ze/@}<*򏮻;7{;*r0Q{ӄ~g۱Pa3QYgv@YG&l)/[nc#oĬ6i^ci{<>#~&25h53vuGD}Yy H0XbmJ p_7V0VyB3Oa۔;n,:x 2D+I_o sz4G&8mB &tTCBylb?0CD?q_>a@j,wJk%d9m3~ {Y6H0FR\] {ܒ!?X}o`qi 6z;@mj?^Qy 'nh0Q'k 89teMʈ Q@{Tf1دfd)h-hvTf [.̧nrNkW.fB4'4 7gިTϿ8/k7<Ӕo}XB m̋Jeeٶ !<"*Z1Y(ywto(tǴ}Xg, ~򹦣F ~bfg72c3 T(ci vA 4_ ˳:}T֥YY-o,w4oܘQQ?sM7eX~kp@++Ѝi -o]tǙ̎L%q u6HcjSUh6ZhT3v3M{e{S`6#""""@_ox}/pZvw鯸+SMZE'X R:R6>LjNG,^A/`5fp{" >?1AW=a`?$XADeh;YiC_8x !5K&{ ]WLSO:'ANmuB|=Tzqě1"g]Xm>WeFPVJ)ˁk5AyѿP_CG!<2nxegnsFz8B5OɫMV"""""y<y(\scu\>88';.E g{oĺ^=*]zkV3e5}< u2Ο _:p`x k܊TeCۓ^m8[ u=\C XUN5p)y2ƜFx ֍:|5z s4)jn tsh/Ē-Տw&5cAmٙfb{- gnYET=<ɏ o$b3@uh{5l'R(ucyE1lsF"~դd3\7{Yww,q%f[mRzniРe>!-BuJt)m3mr""""2;Ƨ ֩S}ppÎv=`"`WNAo/ŻPS:KmSPN:e!Ni~)+=Ay89E AA} =T]2;m&!{b&@¶=~9h]E)u]yұ0"Uil.3LZm¶O9?Zhҡ6l9R&/|nJ}b;d"Ư8nlL# q(D:m7Q#[~2QD}9h`Viޛ66'j~'Z憈Kϱ,dwOeixH۴^gu`\UQV.P/\k]GEJ8wA/ȶv\6PHY~q+/҈^ סjYmnj 6D'$.}u-U+[ys"-+u2BTEuc6yfoG[N?_&j/V?>I+4vv,T ,4 `dB?՝޵ޤ썳e8nB]GDDDDDDQ+ {0B?o7p2;"e u>&_7:ioA2ʫz*pя *Y-i=,M+%M J+*Lu2as+[r7-h1d%e:ҊrJoOR?H@ Y4塞>pnEe6 YRvEa/z*ae 3o:{Ю}F]-߈3,Oa^k\kNkS9Ve3=hsۅ.'' ӏXo 켷8E6C} 5QI}3`0[<Ng<.{ȄXCOu_"""""/Kt~Ks1ޕ} \57 Y::UпԼp(mڋM2p,vrRyA \/N1O2gǂ~e;H瘸yBmӟ"XrV**071[| ㌲+y fŕѐǙ3h(c  VB%8l~fVQ7Cpĺ ЄDV1|k'>?u`xt7iXE<9g lu6zy]Ȯ ׾NyLmvJA눺RY\{@ rS"npǶ5?)ۓ 8rL/e2Q/3`@x{r? niФj eV!uǻq` {J ~%Ca:U-&%df!Tϝa?vP A"Rgeڜ\9 S­ۼ. q&8p\~QBIn<2/ `\tfhy㶜Pyvԁ}-2S?}&eRWv܇1v98^MZE'X R:b 0 _,tϒ3c- a8,H,CSBT*A"}N>3^ s;ʣOA.CQ0D4,4^Q xثnи0_ 6@v:ۓ? ;p`x k܊TeCۓ^mD쑎Bpd9hО'O~-BF꺉quqSeV{nge>%M)K.pKjP]r&~4|Cgb 4voǯpᛓViY>,}ן\YD3phU&=M|87ك_J Hv6>G;mDiO큛I}Îl7:pZC!ADMv, math 5#^g-o.\ \Y@qk: KHKqQ?* $=]=m&Uc2ᔭS%x*[Ml3[I0ZUR(P:Jw\nW޳(} k/hwOק &iky/dB=ۯ-!卺_vF,l9w.Ug&J}49;e"eeܛ6QQ_Q<+|?Z;wJyϩ,37qrV%eٶm/|<@%мAe"q""""""2?thXhA(BDt$F)L!""""""Y k"""""""""" ̰&"""""""""5h&0`MDDDDDDDDDD3k"""""""""" XDGUrE4YW`~Q\)Zh(eUa\wCfq b<ˍ0޴IDDDDDy|{簃Kg):8yqˆHs.pjR 5Pz_.Q ]tnk s 6Ppڛpq|T HDL՞Uw够`,X_&lZ+aö-eR+ #LC^>:iܰTN`M'<Ue43!k |?(Ȣ玱guэ5rsMop9dA2stM}fs٘HL 4 m_WO0LoګdX{3` gQENI}/,Jd^_W={,{[3/Wⵟp9oa`2&yطI!>|A=t<ОGaAKpJ1Zpb#@4A%M:Ý+/f´j5xZ؅m:n Ё3HNOf=bm糙h[Fѻ 9€ӱmsr`!'KCE|цaI_h7Z+Z6b_3l#\2`l8?B&  X 2yK<7'xR.ѷMS5 :z0cb$JSe)|&:cdՃ⸰M`:8$\ֻϭĸG?s92>5ꕜ뢊| ސBhz\٩ m ~`wyZ FI.7PFwUձ+m[.:p 3zg6G'8Ј;vuOh'cE6mLL3m#B3Qvj@a.䡢j>+\\~ׁу >7 G &.})8SFwEw̪#g>'ń(vsl8- _#3G*ptsY`̆3Cedvr` o߉Tc`v DSh+R`ʻ`c5Ԝ`1`Gk voo+7{]Nrn%|@>%vi$@a˂-<vZrnUǑ=a]uΡXr `a 60Ll(1J~#:HFwLEni܆9cN`ˀu0zl)\ ٙɁVxNBvZ'MWiY>쩕9ܹ3Y}çLOzw{N&%쇾wTJ^Yg 8En1, |yENOvoąwu0,3P'K*Wbm4! XбE=} 㨕=zX;pJ\%;Yv` сӆTge=X\x 隰TU]{ڮu |I]񶼦KFہb}Sl'%|o5a\0=3߶AAATsX| _x}w|'YP9_m0ձdDi"thڅ]s]j}ς*#q\ũCېU[P',TDRrz{؁l:s?tʄu`=!/zP#4wT/+ȱC*f3XZ1cƪGXPqj/9"б܈+1M$ّ`LjNAp~|yN!wcy^@=?<ܴ]02ʠm#   TtX_(G;\Oq?>OlMjOL CWw8; 5f5Wb=Ws<."+ꈎX]9T~~ knE ͒Ko{78>.١?_Uٛxe&2[2zh0#. mUAQb.GxXJU(tX!UxBh'Z-8ve=3mtx7`[SzO*Yo6͛4O7E☇3)Jpa/L:]`@JZL%Ŕg/!E ΈV=vf"H+DG]L-yuW.)s Ձ{yfUFAAA0jl&~?/|h~ q)y   SEX=|_t³A|(,SÁ   CAAAAAJ@ k     b% 5AAAAAÚ     X aMAAAAA&     VrX}F@\.l'6jZ H(A|ب}жj;-$n$ٙq\;wm#I=)AAQSa+.]+k\7H< Z I|&Pg {nz7cB~b+M3HJOU`aYZZZY)f][ }_:f3w/mJuaG#'ȭ{}?t*Vf/;M x-;VodgvqeN!{߻!J?U!wW=/ąXG]؅L@Ǯ7ЌeTl\md2  (oٳoϿz_~HpYO^{Gw$hhY=?:sB IDAT/GVhخɁ9φct w ܝ`h1<`ۇ.:p9Zچߍk_e^.   /$5.ԃI=wOxP7 9Em3ia==sudnZ@0an fT}1:`l5yt4 x*baoZV7Z; ?-3 S'DžmȔ QO}n-%}4>aѨpU\U;^|B FpWۑ1Old(h}4LD |3rzS|L`6a*1.؁ȷ3)Y6q}NrDYn0FjwhOaIyo}'#u!REbR󮞱;#BxI|?kE[h7T,vdEaˀu0zlH F~">9ϋS8]?@AAwNe77HvW_%3A} q7x ϞOsWo~m~$Km \u"8fhQc_WEa=BvZgLOϒaO܉t")RO/?>yɝJ}9xyy:t@pċ bjGZl:dQM\xGG= i!ENlMxjOYCΟ]NCY*;btD*ꈎX9T~~ knE ͒KCjFL~TǷ̑+,\1SpM㨃cZM0c$8+7n}JH'3   ט}]OoxcD㳯 <%g?**db[h8 " Oߢ?'hrrs9gRD%Ñ Lß;yPjB,뙙oe ߒP{r5Yxw.hnbN<Y2 cΤp+EeQZ62 t0W#4S vu W"MV},4'-uz,"TK(Vfm.z-u?ը!Kуv|p&k8/LF46j6\pP#4,nxNxB- C=c0KLC;QZz{TSs#  b9fL~_~yW!x (AD*uJ&1}d   > ]' 3rfA'Nl3əN   cAAAAAJPE     X4&     VrXAAAAA+9     AAAAAJ@kب菫FV ] %V }#5ۭ. wOoϺs;cuoSȸlme:ﾪFmg5[eIqc{kH7}c:fn{EV/߉wo^X{uw*._^:5^;>I8ҤPp(Őg9,(,3MUfUW@O~A{BgY9vuC,wٌ鋺GmRPꔻ2R_\ޛI!3o,:;%0hwCXs> ,L b|~`5XMOG6kk8990}NeJ1_I{7Et츏K`xh]Jjt}O}Q"!Ӛig>ɠQG2l;h 78g؄6к4MлiI/==%lA,C6r7l6AB݆T.Yт83z͆\֧Ͱ. B…ށm3;P6>Gw$hhY]9k΋HvM|nimw>,s6a-Z!Hwlݯ}ޒq?qwm!CXGNŸAKe uNZLnciqaׄG09ӵѹHmjYg#3gͨ' tw'\8%>%+sD*K֏mE5uN-f8o޴g"4o`.`1 7We18_s"ㇴwIUE/$5.߄N'x"8??kw<_4Gw65 :d}$>0KWP,%"봛[A)p=XDW#50:xa$CX{OÂԅm IgO60\CMk ~r>e*}4j6\aPZWwG`[pࡳ☾m6Pqt"|'U!7PdEӕL+m)^a r`{^,]|WwddĤp@5|͜ISq[:zJQ/;#ٓE؊ Ȓ}Wگl=B]%OFxL娪ءUFkXLccfGqu1]F*α3je}<5} &nFYg#5ganG?ԁ<^tūp΄{mh}JV))ƒwɟ7>i󔤐tgyc؜ȼ)tZ9vQE'!u7wp0M}ōU~a[7R$kޔ=-ܻ O塾OؔQWZb25^ u{O|]I89Юp.g“&lMovq,rAxw^=s܇}"䢃&W̬{0fCX~jԭGZ/8\ ^ a*Pɳ  SV% `ce9',:FG+ :AV0?Tq;Խ  o TrO1aCN<8BXGS7o"},C ljp+ :wN!%|DxW ]}ǗpoZ6ғ :!:&# U`릥 bHFZzĸ{jA)#擑t]~]LDe)Lm6/6A11&/ N1Ui2WPcEGZ4x?dúmI;U~P&דhR>.f*1k3V3RrMo']2\Kxڙ;b[0ayY柎vO\Yk: B_*|~~ĆU\\^VTvuܴOWdJM,$R,xTd!lguut+.HƝ0f t\1f|q]eMzʺ<+mMbchA@ 5az3b }NE^36C%w6*!g14a)%qXX'(a:L韎u֟~?2Ps{s J;%u 0%Bf-zZ ݯnOs Z53GX,UvT屌>~F$Ra%|OG@:`(rHXuahWٍD߳`+Uj ^sK ALYa^apC^ A A2a (Gc+N[5dv^4b#lYe2)ndPc|/doY]-!l.w]tP_S3_ W$`$e* c"c3E6`)Uu͐lFbQb(&'}8Gbn}c-pduu9CJw_ۅUd >'s7W>Z{:o|kΐߞ-$32!6⟧sZ3o8Jup0v4t/WQbXb)pyc*븘BUWs;I~KqY]PN+U|b_%G%_=Lp:Ko}|shZIZ9WԞ-.Ϝo`C9_AUl9l+=ՉVg\kc??6  V~'T)+*IqRӂDs-癊Sp֕P DWԲbVŐva=x[t>[Zwh/nØc:zͳbX`Yl؏b{ ΐ$sdeMЅK':CZ"NbWl]R.N1ifm(vtآHYJ`3s# GZl v70Gu/;bVLƬVw6A(B5|/~-~|exI'@1XF m0=|}Ѷ81'ZbhktxMA_ߧ#m`HA[p``"lR.n}P&=Dv 1o[7=w`Xz( u907gB;*.?eoQ}ߋ· =~Bi gtRŸprQ,Wb{GJVEIꞮ-Sj8%LrnEB~C(Dm6~|wq2Z@ng!6<)pߝ=8Qr3Q1+ĉ}=~GBZ|gSwJִI+`6;ڢnҁ0=}Q7mSj>JqwxU2F{ 11X${dDu/͆%|.ZbS]Vn"};9y31T8@RWMH,W>eƟ9 aub QuZMk%NAqO(oٳoϿz_~¥?S|Ͽg00NUطAى!>|A=ОGaAKFf,0 nz}/5]`:j4{ <:9@F -f331T82#59+wDhGN̡PvEE:N3lmeNAqO/$5.ԃI=7W_m?Mcm3ia==s:CM 8DŽ)N5&Ntt|N=[M&-hr9=B+y1J}Xz(Kͭ\8,7I'bd{)oHq1HHpNq3gm9}4>aѨpU$8;s+0Cg=I(2 ԀQ߹;5];KRc ,m8ysh[.:p 䘢86 pݡҎ+w\9XBQz\V\QzU(֊7>/Fh~8pb\#4]iԱAy'ƶ)nJOž26WSX{$d^} #9Ŀ:z=ic:ǀs&dCc+/Ce#3R35ulCڢm!%y>.&7iG\gƂv(;/vHyPx]9{^Y t4?*HeuNAqOa6\_&5Sg?1О}_8204a:H;Q%eZ4O-E1L/dGP."DŽcVX5=wm& v%(;S`^im:UYtrp02'Kᢃ&W'{0fCX~j˫4ż ɹk g\E7U]iKRyF)+gG-O)5 PhbɁ:ܝM6#u"c`PbF 1Uw5o+zG+ut"<^y->[Źjw8jOR9q0R bQ ~i9vc#BQ䠩(:qй`v4r.;ڤ 6V1>W=RmmQ Uyig"֙*>nB~56fq@U89/kLEkv܇}Ĥ8/=]8vk,jG#YalY6:v$6,>&KW=foN2zRm;AA Sa}pKgſS?2:`!!J0YX3p0a@*;*|R% Jꎴ,qʝа- [ dw60]W-y6_W&ϑpY a- RjaD"3B bXyH '`(҅w$gaXCD;k4! XHѱE?} Oޮ mAK|eD%}wbܴbYbm ۭfa<q xi q1O2^.i R؈h0[c9hY :/S5!|U.'6j&0/4 bM {|>OGĶށ'91[=(47-^Jƛ0kut..+ZXkYQ_hcUEoho V|_LaUt EBș!':h;⟎;"ךhnx˃"/TΗr9;AAĊSa%|?ԓ.^ɟdg5S\OOͳ?Ƀ7n6X3E4 ~ Oڊ®9Ю+F㸢+=GR5-J v>9/0aQ9i!:L'L!S9/Mf)售@ 'IVFxŸqS2[ʳ"б܈+1M$Y`JBanŢ*J:/f> *ս\w$CT{ӒɶNa{`uLuƉK 6gO͝^A{C6_o3#VY-hƟns̚qwҮ|6,)"*p 6D ':bH/D,`13\z>ej߅n]Bp<͙6nd\/^q[;}' TrX_0G;\Oc?'*栖KjOL gΟd]NsrjUASx\S<=bXS-q&hK-S0 tcX(h' L5W<_BW %7_EP21Wvq w.":v![ F< Dmac r@5սusl._L/bA+ 0 ITU,O}`7K"[=u;QW; `qr^g?OA;L݄':%]kImlMGV[: Qmܟۡg\`R &ԧr`]XmΎd՘l8/,sX+_e)ߟ(i|NAqߩాj?~zsc6?kƏ#:> \V|W OB[|^Ge=9V]]ݓs-1IOO^B:#<|f#mo{[tpM C=^d s lq!=X@_i '+.F`H3y*l-$ÙЎmR H:-%aͥ!QxMecܖHݵM ;в^Qeߤ/vjZԸI ~Q_"CYT :rEXfOXUX Pv^♧wnZR^D!,, `s9N/ 3'`sPY W:ɌPdwU aq!0wz]o,6K` <HzgA^Fղέ[ikw  Bm6~|wqDAiu27lDsbvf*L#  с|Fj kla6}[>=ČAADHaMAAAAA]$     ECk     b% 5AAAAAÚ     X aMAAAAA&ȉZ \OljP(mVuB2`i `wG0TYf>12!gp%E9SwΠJTjZ/Nfs->/X ;u{6v?w_)uj*6YF=OX Q1 `.ɋqlR>uഇ yP[9=_g9   b(oٳoϿz_~(pYK] ??ӇI0v8Q1 &;L7l;h 0̵&z~MiMKWu;^͒!i=raN Hl:< \ 9sF3tb~ °)lIu$69:/"vf10=t&AۜQ]:k'KhخɁҝM>F pBw?`hPFAA3_Hk\)s:X:p M՘ժ:.#Fxzn(̱:cXp+QhF[TUe1sеX၂6]5럎hvFFݻW(Y*Z`eƱ1R㊗amVۃwE# B\X0Ew%)PՅ9Z6ǂ˩rw}t qFal%x*Y/O Cm,x#4XsaKjq]S΄${Yɱi>YȧXٙ(AAAטS8, HpȢA@>-P{Ճ!: L!*s|˸t \VݻObp0Om4 +< –BLhGebY6XA;aKIyXP=خx\S;0Ayy^'lFdKdF:-xmnŖ.wY@/fCgy⼳U~f{5 xa#{2,=j\iTu>]f'۞Zr`C^؋i[AAAwLm6~|wq A|lhOa[X {H,!   %]$GEz,< A!#4 !   bu!5AAAAAš     X aMAAAAA&     VrXAAAAA+9     q9Q5W [Z5J|(݆P?3QبK츏FF'}Ho"=o`{k)Q+P\<~:ZWGw$hK KhѩخɁ֭8 a]tZ-KD:w(O=]OW`e~W IDAT+{ s[wcɃxb$=.`L謓Z%6zќ܆5_0!!RTm{3 G" bՙ/$5.ԓ/w3|{C|=~w?зh2A_F8`07-W9L' [_O:>F&OtЉvOhl>;sTB`VPJ.}Vz;%s;0uB./@ Ò 3c93`l{q ~r>e*r<zc ފBhzC٩ YݩCo:e)_h7"ǭ7@oÁwc6A\}WXW|GhUՕz+Am~=بvwHq'ٶV?#CM.(QOu/#o>%.WH$[VZT]Sufgғ?IR%3ն%TS|b.? #;,;U;0A]UQz6{Q>~Wķ_s/{<|3RO @V=`464IU6O>8D1oW!=ت{!J?‡v|YvCRWWPUQWwFS޵O0G[}s7AD;9Sm+Dȑ%m '6ZB{<ȯ{ɰIj"iovU䪏y_uՃ!Z =ˋav>/=" 4xi7.YeQ-ڶ'Zj}iwCo}Et /fv+utcqTv^*y{bRzX33}\E}4Dm>#=E|"y= $) ˳~f>g*,Ҥ\\tPOqg?AU|>1`9UU *;߽𯯾w61)< paXHy8fhPUtQX ycҳ$}S+׉ `ba'l}("x`NRib t3BUEua^$we]=V0JXyHj'H* Hvhʻar=,eS^#D I5k:vho5ap%N9ULv]m+e2.51PWGh^-2zϨ>&BT6xF X0:u1y9Ngzx,ĔPSc1TyS{Yg&S3O}?arë$XJAZZH1IE-=23ǭZz `ۀ%,xV&m4\ l˻S$/'W b5䰾5ĦS ?_젊PSV l1:p MBԘ\$xz#(8kc?? fa*iTU&ǰбA7lsT'2I W<V^Qqr/yI8VeSDZI.Cm':B`{v$LHS{%uuA[e}h_t}ӄ ZkQ`A牟γqq'c˂ /%Խ<HͱJim>JI|1@}O+|ǘx1z uLz*AA, kL)(Xc\s<)wl>Ńɟ#SRH8m]BbPnL?X{,=R D%s l Q a4lYpȢAҶŨ_L_&p".CA(@%ŪS9uЕeB\c * r"6>lGv$%U2IʞRA \> ^Ss>q_Fȓws.G6-ϿgH XyyVT.U\fA Bҋ龈uUTN~%= r?A]Qf3?#>U cGSy"A$@tQD!4Ȫ1O`M/vnRhK $.L2)2۫e^]QH;Ȭիor]AADUm XxvŪAgZ@یJKkl0\>X 9yn0;^+wqp[qO8sU7  bP5AAAAATz"AAAAAA, XAAAAAk     AAAAAZ@k     b-5AAAAAP #gj&ਿ?sPբnD"@ߌfq9y=|l *J9/q%j}}]ە˙z qYypݸCCgd mYdILu>X~'w߽oFgqKגH):3}* Kf L(^p4,Dص}0G@U62>땲3lvE]0@ ۛa6c.E]ޣʥ$ >Ly{\v@*sd>y{o>6\.g>Bt]:fn{[QjM"=3Yl^hm3unv;6gnSlc?'GZc\D.U:( u[UL  EM"w(W|4˵dyzVf%W,ܞ!Ӗ̿\Kߖ#w=V~o_-"oj|ܷCa=#sUrbX,aPכºOgog>Xָ߳ybk/@(oٳߓ? >?nş}n.x0{>-yfll\C'3C# f uЈ$@y w]f *Tm߆۶4YÍ@ '_![zp,܅>MCA)lV,8y:}7&_I(#JO!*X59 о)w(X $Gxp[TQ߽6+ 6KӯqΡvSԬ@ouƎ;rsa,\6\m͛bg#3=PEavxȳ5OpC]]W)u%8oOڅz=궏4|| T>)A<=SUt MݽdvbTb`Z^?L>/Z Fǿ_ٳgQ6Ǝ;u1vh`ug,H&܇m26៏zL]n7}K^';o/{'vXЇJFNLcOkqM~r?i3,}4k$l u`AZnєwf0LN[/:|*^fdG 9hcX!Л.ܭee)06FZ/ gV@Q֛I1iO%dˏGhctAo+؇+߳nS+uYK!Ϯڭ'KϠ/> .3y>UWəW3&l>,YH2lf>\K~SVY3pp+{V'C cW4T>xʪ:z>ͳ,ho,cT1ebؠMy-/*xGwX OR]g#Y /0`,|iL;\9@VU3Ah{R-&R.B&Sh^P[Td9(BDʦ^ LMl3m XhS Ѝw#-SЈ–m;)'m8#rρAQ&G&[t6._W&&(]ն&f?B??taO9:svd t"f7\잸kR$q^wߓGY#>SBf ɶ[3E3W}8-LcqW՜u,[$̸8OW (bG f<4$g&8Oؑi%=)l㼎fQ!#.S֍6d=bAU1a4:Aɲ4/$Q&)|) Tj56bKAqpB- MpOcw/۵ jE_m.)I=SisgC{Fت z@}K34, Dњ2q`s~c WuyLb=)bs7o'ΉȺnM`9_GXbĀ-r{tm>~sS 8i&fW|}9T K>Ô{"^Q9`;+j1%AR}y M4[vnKx2u3L0ŝ m[1R J;TYY>{H1ܹ32٘n |>&QO{ڐjP䍵!_LQE=v|9c-?;ècYN "[ pJȺjyxA1[0yZ}^g\XP֩{^*!f''f(ZOmci,v 1 ^]1ܨL=4Xc=IϮc .Y1,`'fG ٱCBkK Gf1uvuֺz/+HR|_VJg=vD7Gl93%xHPsűjgt7 @=aFU=-֖5k~\U:}֮Ed3qpw!}6| mh}_` Rc4,g`>UN{\fq) t6 h}jɣsp{=@Hq-ƇoڨÈlj#fCS(&;.-ua+Vӧ'pgJO^Qo[h})]ΡOu6.3j2%Mڵ\wO/WQ $=ƃJ7 M")dC2SsQ;>Ɓ ;Z Sv;VK(5_nr;6\7-Gf<uW̰(9rP{=|AѱNKko &*ᡛ׋KV[ݽ԰>&[p,ة2bfg'QHq^/:Z.; O_(ޒxBrb~?sѹ I}t2="ga0U+{JPp1Nө3ϏdNbċ/GxNۭ}FݔJXбk0&9m&+7.E(ZkDeVԍDVJW> pVc;FlSnEJÔws%c}g""c U Q`-50CxI2%[XfMXme"{u a6 V Ϯհemb.vxrnX <-hłgBI3ZZwYrc\\_VR:D\W62Ig-*2-V0bװVuTTH1ÎG;ދ$,OjEk˙~aRU $fpejׂ:Yr{";iU^{\ZZ…w}a{{fDŽ-0QqߙMrC:o,֩|cL}LߔZzp_j0/|t]/dOߵj} :Az^g_Wo#?L(^TX`|iwzWO',=y~7}wCG{>GJA8h, X΅c@ uӮ;xX.|s^]k9ֆ A |qϮ#o[l>xњRf3?#>U cG0Q$ 7]9tp^|ium *Ǚ˫0K;n꼲?-ѢܭZL B>#zY71!>>^{ײNAۉćO.`Q#,vu],,#mbzf᯲vua3۔#8{hMxLyl]8ً!zFD>6{OKyeXAAAAAkeXAAAAAk     AAAAAZ@k     b-5AAAAAP #gj&ਿ?sPբnD-$1AŹ^M+H  X5NOO~3?{_>ӒHc#4@6jpΊdWUY&۞q͊^pt&X(tT2ړeۛCIƄj\";fX_f3X=ܔdaGS z{|\sʼU} ͜yga_oM"v}3usԧZN"o7X7 fXw( IVow(o6f~'=nL!2d/ymwYGםg  )oٳߓ? >?nş}n.`;}&&š;g< w]9hsφkspno۶ddV e/nEmj:gc.̃DЁ#AmJRNoO^goO!735+@:lE C'RK/Vݎ$#2y8%GEF0Rt(@N 늼s[o$lrI^k\eQ@(uؙzO| t c%]>V{F':S݆]%Gk#8D:< QGfLRAAD:Fǿ?'gk;Vla{[5`N|`ש4ՙ~d`tn6hmGEÃ>%ۙw!o] vQ=]RY홧 [!Ua Oy4G}.6#wGŹ?DmLS21 _vſjp-w23⳪Ykkþ-ʝ`ޖ "=afު:C"N.ǍC8i'3ДFUʞylIzcډC"[s#K[|M}NM4Xq}5ٳgµf<2&'HLcUs7{xpj 8Qy g.:6*L*yۘ" ڝ/mKdIi)%rL*J5k9Hpw}ΓwFM  b;\xzoxُk|'< w{Lo._a"4aH s1#'vSp&8j%= U~-di]Ncw1 LAؑ:9}G/;>ْ>6( ?x φe()r 8@).ox (pew=֙Qu1 :BimyO J`4ڼJڹIދۘA u~yk\K>,G@X3n- w(nMD>4טɓL Էn0Ptiu  ærw>~W)J/q[?g M4[vd=85VB6j` +8@j` JhJX)#=Kb܇3s̡M)tS J l㰡8qxtBt!HuhIz`b`gv.[H\_ AA7ֿ\g[F),X_U VOmdR)|G~wvƁdvz<aGbg}ܞwGJmQ. Yx60 9x6}`pw(o`׍[h|"3gAk7)d_˂ʞ׋KVؽ԰>&a6uAYTE9v${֛8#53Xq>ܓxq賠nF ,t꤀>j) enJRbtB|ʂeP[8;qR!I!}4z5R0Gݒ2gR彈bVf]*qz2:X9S+@jrb^VXTm>x9hCWܓU*E=ΆE(_EP2Yӳl}U>4GﳲRUkW)/Ǔ1\,To5 α+0ulЭZoOBbW~ Oyx8n#; :9tG3 Iu8- fb UقRt8 )8zgNoڦ(0ֱ3R/] n4*1a kzFe6R{&sNk]ocw훇"k\Z]NEmX{[W*:k9f\Hb {:k|   Lz36kVW?}'<LO>?=}wL?ʏv#>›ᶗyTx&grqܖp/{Ly$0q~)qh?^ $GSo kCe>M}VyBF ɣiѮ72$ҋtkMԺDi 11g](̄1uu΀_d\jsREuطRK*;"2vm m]D8&b YX/1؅^tZ=KZdsmVG?{ׅ{dWa?Э)>ޕuwї]梑&ds<(c3z  C6f~G|jAǸRE pu bnJt"WDBux,zMx&Y$  [ubqx>`5Aw :l$X/i4ѻ`5AAT(Ú     X *t            AAAAAZ@k     b-5AAAAAP #gj&ਿ?sPբnD1no Αy_sO, jfJ{>xǝ   Btwqzz*{}wk\77 IA 7W  LepǼCګ 9%E9S2ў~/DJ2DU`6Ƣ.ʅ0Ƣ.~Q$>JW 52wK Q0.0 `-΋ql{ՍYKq͊t]H}HZ(Mm;_ӓg9   b(oٳߓ? >?nş}n.`/?߽M{01 pā3Ca}A#r0ԟᮋ|f *&ڷ=ަ zmn0Z%0 {03rA=t}tҸ D'߳ 7ۀ'ʐw5XXfL9VK/Vݎ$#jǢHn0laQ҃M\2M7NeAAA;Fǿ0 f?m+βڨÀ.kFP lOADfvV{wGVnuXfy4VFߛqM~r?i3r:!W Y{~TB.|t6('1 3vNExr@`f~ƟϪfAF )Ph\%Nz,.?lP]1T+,%tOGɞ'|i QYRn ϝűfFϟa.ښ7*aߨ ehޓ:KqN's4c40   w𤥆o7_~"5gt][=;`z 9#:L& <\XvNwDpL{ox?J&-L 82;/;xanI2ų<@*?SjJUYY>[R%M5(֌ %,^=U=4~ymF=e.o5D22pac8s =v̂5x='bz03XxȆ-_ɦl4!(ܸ0Ncg=0> cMÄlE@W;K߉2pޔKF _F]WV|SXp>/hct~ilNuvz$+H9h4[NB~/OS}%Y:d^6"  XOR !1 |ɗxKʇ)&:,#Qr>‡z4\ԯHƁ ;rq2@}K΂ ~qZj >Lnr;m- hu_ )[\|T3)w]{riV+;K(wEAAq?S_ը>==O7O( J$޺e 09 #m :#.)/v 8zŲ- VwVC֐j.lhX*+=qy T%^fk5 /Wpc`2 E6   o^wU cGSz"A>Ss×-}!*uJgAAA _HGDy >5f!=Q+`,Jon=,fypxns2Z|||,Y_mk [.% w߽oFgqKE&opz7EoH)R`;hoeDoO`p,c\嶇;U 6^sv?O^QЁ<%ÔH}JM:a)Ι$Xjg`+7l]G,h*ʏ"s͑J(6ca\pV?nޕ_c(1$)Bu0!͆Wg.Fۃ }=oa?Ì\AZs]Aʦ"XЦžSӞ|?܊?\l |s{01 =Љ3CБ{A#ZԟᮋXf *&ڷ=ަ zmn7H^jhoyp,܅8 8Bݦ4.Y0=qܯ-bW;9:/1ȍ5dI!DUc= @cJ` `:59 о޵ ӗw)g.F۶̣}tW1L}5qW`3ֲϕUc|,}pO'vڣǃفڞ&ewęvT0G(Sc>Ջt_Ѵ ؠXEA*Fǿ0 fp >-qcNJ: ~{f W u: tl}-&mhnNhlg } x)Y;ygv8JC6{YT BOL'wGOڌ0K<2L=14pᣳiT,̊۫fşϪfk v(\w!b w uu*iPF'H7#>KK..w'Wl0N}4ά1gpmEGEߗuNi-UD"ٺM,!'Le} UNy|pZ S|~Vȼ3ӕe5lV{x^zuC7QOAsc7xK_/pw>ϟ p!p[e /{T6;q^*g>XwA ld_ a`N\H{A>)9cSإ2+r)vaΆ|EYY./68<g[Bɏa!l)6OoѶ+bp҆AQu`rdJQ._W(5bLr11(qJ#M0a$9FKFq.덬M8rw/_6yEmX'GRE+:=0]]-9l˅ ؚ.n`>υކ}ghd%V"{ gRDm9A[i~|KhRYO.R8U}תLJTu^xS6LG$A ?%摞E7֚>؉hQ%m9/k.Ӭg?V]gyUwOԻo6y$T2Y=sP{U?H޵LWqM1C]״ȜO%OޣSNa[9Sa&߫ÃGq }Q{e|U :O TXo jxxIm\+E `%68oaR ڀ{anI,tqYDȆL^ج,q5P!DNj@ AaC`ƣl N=mnTu31 p+ac8v =IEMa4fֱ,'8|%ӫ}s|5ǎ݂3qeUׅر3q!nQ΍R׿̦ Z׋#;)4m}ڧsa%{:,o?݂y9( Q=p'aƂZ{2;P-jo:0.`tn~Ğ=006old5tQ/.AF԰{o堦^NSXb]v4[IY@gav9/9UUSMx`lSznz:H[Fgr׸j#=Kj<696~N=W|&,de<8y6Gh] A/$ ֎j_3-'S=ƃW73~)L26YֱfƯ?\ԯq`Î8pzHfr$=Zj ֘&Cnr;#xmeN{ ֮km[.G&:0:0yS*-'GgZL[-u $lS=q6b3zs>7#{oAJډ CkϬn)@VWn/\>NY*gqpQDAA 7{w/c| >ŧxY gˬNqlO9S&\aׅ[ QkV󌄄G՜!lXSuY4 ǵK,ݴaj0VxUqIAh.e1*:ɺ`|X0n\~l_}s9:K^I MOu9^n9e8d'r4.PW|V65(\t— ;L^):pYAT\/ Ap>Ԃ>hUE2{/] ouٶ*2AꃐlݜxNVʚqK=ks]1S䂰Vwƺԓ%Ҩ.] C'55d8Y&m)c%{a=$STZkY6JRBkX;5>JfŞIIjIڎzxhȧ% b ?pgQO'<ǺDbgjǬM~H]y7.X_{xGRtKH/)p3,hH/BiNu̎4ȿJ=DD8V_(C|} =V,(xֹP඗yTx&̣2-_Hz1PF[fbo_m=nkO m *BIP5Yː]Yv,ĥ1֐n\`('io#vkGrYMeLy,#e^DZ舲Ѐi0ݿBE9}^tw1Si.>g @kGg3?a X-V>ʞRA \> ^Ss>q_q:Ouϙﻀf?hۓ,T&WIW:&Ҧbuyr>Sɵwk10z!O!w" ]WԷ;V?y=5nc= E6s>=9#[@` WtSJxGIޞ9||?#Dbg ˬ۠ARf3?#>U cG0&.>9B3'1?tp^&dT m1w2)2۫e^-0sn ͌z[ +UAATVX/:0ș!ugW_DAxˮt,ϸ͈qMV5샥w ]93z[uC  eXAAAAAkA.AAAAAĢ5AAAAAP     X (`MAAAAA&      X}AD\gjZ PHR蛱\>V2=852'<ߒr=㜭AE 7%2tXϳkr9sP[ٽ>۞!3˕=u},[ݝEt_ݶ~kwtwqzz*{}wk\o߽MћҤP f(E g9,)(a,8涇;UQ^"=sJ*ZEy)rm嚙a+<^3胬^);fX_f3X=ܔdaGS1=.)Fz92//N!˙y%. `-q8OlLGH v{bc]`M9s1,Pf#o M}\ kqp,BV)d/m(kJ:^wWX$xWŵCȶgⱞU3 grȴ5OkG>fG|{ޣ뙭&?u|?܊?\l̳ZFAmhMлmKB[5>^—Zuԃc.,(l:p>Mi\fѰFbwmj+!DUc= ȌeI UW]do ׼!Ġlc.{Szu4>]P2 ZBVo-i]*"WdWx{<(]cNJF|Lgu]3Z6pGg Q7iztl}-&mh%r{Q/ylg } xٙ l\944GɃ{f%rî.Sm!%9W`*s{c-1iя\'mŸf́'JN3N ,ʻ6B7ϧ0yz&k1a$΍Z[>FSJ)˳uWoғnKgmeMb71]L3gqg oU}Nt~NV>_T߻ɜmm`+l]{Õ662춅yJs'7_ 76Q*v;LoDiBi_ > \cj+-0W1elDVvP&3Ah{RmL=Ed)MpOcw=+۵ oXsMWOʙhL:SFXhӰMqNpd)lT Xjf󥈭|IG-m.̵xQ=L0O sƾc a~YGpb =ZۢρߵToC!XjAU=> ͩ7 `KO [W*,*}㷿xxُP:/Z&-LJ;7H<ո &uWXL΄T붭8L)q#=Kb܇3s ϔsgd61T !18l(Dz3T|_LaWqlJم>Z b}E=v|Yă"g%vwB6YNbJu/fd\ ٨-Tc/Թw随f/uvuֺz/+HR|_Vg=ȊNz^W.[7H#<5`vF.k9 ȶbydTgV`31%$6 ld?yUOdY>L=CM~hׅ;==#ص;{X< &>tL%chOѯ}/O1l4;mBVʢ |c`S_i98a$np{Om& %Jl^49է>Y;ۄ׸ܔf7p1uj_3-Hzï:>{ dQpH\ԯ8aG+vpYɝn ^k[=Xr{w܁-m\2 IDAT#3: + nxE2^h'Y)i4-iWH vQc1#{DWe"0մroſ+5-]X9:ȩ.]B}XEgWt)3$K}H!]1RZS8ZB\Wmmv_vXjeO .fxkCC(fvd֑XG7DƆQ7sd dj+g &j5{fQŰE+vjGx{MatLA[9e7O$͜kO\5>L";0%Phg\XotuJL ms^8!UibvTc l5<}Z~l_@&]~؇]0ΤVEH;uHȣR7@7ĿL9l'#%f{zoźa]|t]GYw"xjOuDejצbGccE0wn͹3.Y(.vJ2VPZD/wK _ԥt&_EP2Y̐u_?2_~[ԝ{ׇ{dױAAR"DrV z>BkQvf?@z&Oz +2z[naȄB SS_!irwj٫3fx@kEgK:}Z~OyB}x70G/ؾUMRjV'XW>VIBӟL WkVW?}'ƖV8*-K|46.0D[K~͸OC^39Uכ#Cę±LXꠦa66>g+z :g]]+6qA:^kDr Q:e0[m3b(v Lj|o{Q݆>/\̶S> bx5L _on#ͱu/ǎL9>Z*߻f!EDyȥNw)v½dbOL% u> ОZѺ*k#8lu*Օ:Z[z҇lmpo &턵Boy֥w -omedźVY~49ɞs\uQ<]rS6.bD!jl&|hT~ Js}EݿeA Ŋ6<8ʋ"EE/8g?,¸/TMŀ^ {*9Z[#@ti1gX/=1+X"gudecbK$B.fl,(?A@Ȅ|Ļ˭]y(KQmV3 + dp63q9he8lMxLyxqyR/ଣ3֬Lb'(P5AAAAAP5AAAAAP     X (`MAAAAA&      XAAAAAk     w7O )~>OC N9s     ⣤6f~G|/p3:=Wg{oYf&     WdWx_~Ȃx AAAAAD%Abo++ Ak|'O{?Zӓ-`-U+k?g5uNJSҕz֚7_M?35qܑ_w:?TW$ S zOQwsE ==_%95Gdi> ~?+]vDe"牀5`"L@0X&kD ` |k8κjה 94k|+pƂr|Ts ;`}|W;;;ݶN;-sjuA};;:4DyԐF)`9\N!p~[jxQF縡 ' 1O& Qzwڙ X_WM.ruoKVk U}y-7qC >ռ[v}rX 1vXE[_oڑ㻽Ƕ^ǁZ \Ѽ/hTxx~KרHcm$ܡ>,7fy| l;/6UƤ{B̤;`==_ҕv-{ݝ:n0{Vw|[~VSSϴTRj4VОf?xCOyYQIwam,hӦ6_.N>,=+7hk.O:YhV1[$@uu0~?]RM\(-ZU[ֽmmv]ovg4<{+2R﨤ݎgmhTA܉fu0>΅]^|/||ʒolQcmڨ~~4il!0Nx 7PoJRi> H-pkoEFό:~H{/M#>1>կy}p;%r]M:L<$H@mu'&_A0sk]](F<7pAw.Lm|fH+AVPVUzl:'U|AB$E߬rYj=5U]vTPvIֹfEU\{u;(-J=Uc^ag%|S7ti-˙+|T)CUܾp m[UuU:jo.0󸡅RC:Čr55},>}7䞐/ʐPƑoٯ^5Tp{Mlּon' mQ]F>;^9[ڼao{gǯpqL4'N?Y3NjóVB~'|MXb|w{Y6,r*Sǽj7Z*7ꬍ-l4d:e43Z|,k3љqwj}ݿlԈq״3?GjoW1F_oЌRDxxYEY$32}Cj(V߄GJsO'j"˳ǽ~vWK!W%S CMUIЉyd7t ݔf2u NfzՔzGeᛂt&Js6FM1}'0/4 CjfL,uk;ot3o:|श)3Vu, >׼5NfwFGun9Y7L{:k};fUᝐV|%?nIv{Bj ۞3jzo Njik hSiԖ%g'F>ܚok32o97eZ_LcJcF8ww\s͐~t3qkcA{w 6`jZd,}oE?,3qlmXS5UbS[VV`P<32̵JΌg%{rv]ZI?fc{~MGyoW)0kXS``U}A8nhFOUߌ2+0Z|V~Y1 փ ͨKֆZҏ]mt}7U`6TW"KGQc<~Ǒ(w~q7LߊY̸܊ߔupܐ88s<Ay'ᛖwApYK->'Mj$P/XՍ}"T-(5nnep3gxY퓷`:~l #=両Z:l}BvNz7kz՘ /0XTX9mbJojފ}1" m`pAF t^`ۼ7}zdbє8= ieoЇ*~쪪> fh,Bi jz'e#TU}?C<龡]gZϟ%VUu.3KX]Ɯq6˟S 2Y0{r\}CsQƻ؟ ?d]ƳRUuyT/iG/TTZu!5G#p9ilWQi,UF=eΕUD6}'\븥֑}nFp{wY5~$w>osYk.$:pIS\Y~{=^t.Yb)pa,T*08ޖS̹8+P^-`-ՑgsVSN-}`g>#2hl _$g`VUz ,7u)Yd\v]` DyPRAKV'AQ٭RU'Qv֟}2Yٶ7ƣ4AS^/6h,<]cx71-|S\His"S*00{$8_d,}%;?wï;\_m߯(F K" l3]Xf(kÐ%\!lXv ہp`t-W{{pP]I5'[j#NDjljk8[ȿG^-B_P E;u}fc_tHyZ5nMK@7nV-}Ƕ#qx^X;n'(٤{װI; ^KIFm)7d֍ryt8հnnץ}BkOC>|?r=1ߔ ՞cISׁo\Pڟ2{%UMw_Oi^)>R~DO޵Wl~ӿ#ʈ$N(o ЍgMYЮL%m[^+k/=^y͚R{5J끩gfyϢ i9;TFK:sX#+X5A}v_[vUL%i&ܪ5ԬzJoeS_U2m->u>`L[+70 ]wq%AAzofT bzef7tC3cQD#~oeGmeenѽ-JSTt}'gԺsTbgW[|nT1MEHtnU X3|I5xU]/yd:Eǒgr%$?{黯Hv㫋 t,3!e$$;YTO |B 8_>nc[[zJ׮]gsiC *̽!'̘HcnfO ?F&x }oXjr/ܟ5qQƠ\(57[v'wmy?=@!pkQ!CNy߷\ fxC_fUF%݉ŗN|8o`z U|AB$E߬rYj=5U]vTPvIVL*`{హqm[c? `g ڙIّv|;*T" Ĝ | 39seoj==ۚM\|qO* grB`SbF] dOg*7Y2d~#*#>4S D]sm8m&6k޷\7{]QRU.# ;^9[ڼiȼ "NX|,k33G fq}@J_6]toϏQ#4ׇV^K̄9b[ݿm gwÍLZ+]7}3X$ϬS zķO{CqڿHl~vWKӃgT_=NO; {kÐqsS{dRo*Ro(Э IDATfw8ӫ2; xK)|S@QZ>"K{ڨu:[9`pn("3}=UGɂzKJ=U51oh ͪ5үJ~ܒ n=gԺf*f1Hz"O-73][ꮗ1/F>ܚok32o97e 2%m1K#jq~.FA?TcT uۆnL{+o{vT|=P 3z c3 G͞9nh"/><[g_l𕆩.wΖ/S~Qg䶖x3T? \#θn;Cio8əYQ1u+xޏ2gkd@sSﾋ 8S= Է>yv9΁RVdgMqT-gOy×}>+a #jsA#˖=7SUFDY~I5v[[=f͵`I^FꬶQ:To^pg]toS`|pAlwj5e)/ )^p8^a:{Um4d K_yAewu8-=' =]l _߻u~5}+3J[~\U \c=m (Yޘp ۳v}Şƨ<]]WV3č!s1`2e Xm|sU)YMӟk?fjv$q.kc:8a3f _$g#VU{,΂7Yot3|%HuςtEN`miUK ƃeGd;WZ9{aϔ4J SVRKyPWy(hGƾxvEQY1|A$)c}\ZMLp{vE3U|֝;6uN>/ԩ^wu~ά8 g'5ºUYT'zjJ׳ DbzOjψ4n[v%]E[\#*\}]zggG߯Tt*.J'O`8gtTL͆mn /jWg֑tčg֤- r@ ho3_++=^yMƸUFXgYjܢ ߴ}ɹp01eZv-_TLQ>S 7zzߗmnU[:óUoJȋGl}QrֶwmMbZN=勯[?{K+,Cƒ2R8gaxc72OR~^=g̝kKo+rbi?+7PٶT>mzόqZ"_碌Muϟsٝ 3p/qjOxZZGu3[pr~իWve8,3Ss9սI߹ Jp}P37]&ƃ CY||B7 ώ$2 kD ` 5`"L@ה-quM߯) 6sh(iq6ԼV;`}|W;;;ݶN;-sj)?tD >w/5=kQ ޗ˩~=4{[jx-F縡 ژ;(ޝviA`@:n &RչZ#Cf_wp^ ~B7;sľ0} QTq)<΅]HjaOr&]Su}X>;[I*Luq键 mY?~}f½a߹ 2o!i[U}*|Izً *YYijzjt촩̧`U68asU1ڶn~Ej+椌8;YoVk 2[j3WVӳQ a|W騥J 5<3rZeN3΃n ߈6 e]sm8m&6k޷} JUգq*WcN6o2 ^o?IXv}]4}LRp.- eG6 77UuŹUmdPw]:ϱ>5xmR*}o*Ro脽)cTU/;UC-P(,|SQHvs6FM1}Cىֆ!5SNqwO SPd5TAA:[a0\8ջWw]0UAyCd_hVe^~Wd瓻٦3jzY /-'=ԚVOU5}1Fv[mWsmF-'LkÐf9﯄BƘ5]?qjXƂ]\x9MrR3lw/.i@W>yl\jȲ%v0%" }oUnDGxr並6!hF*Ҟ5RO2To^P{ fT+f5e>fg1HZvxqSc;0讟RiݽPЪ *=^Pw7~ʽ6oF1U,Bud=_q}O*HO}vj󅳬xa}]ʼ_'+<:;o9kḥ֑lnF#to+bl-(-`-U+k?g5uN^^^>!(Ƃup3pT-g("93 3_4il ,f8a }vb\v]KB؁VT]/,5T?Fu(;>"JhFv]rƕm(M9ׅd?fyIy(hGƾxvEQS\,&P;LXo/뺢ҊSiP]h2;?wAmםN}/";ynszEAoIލ"Yl3]X6iYT'vM`ҷ';/c- a52򾡺*Ƕ?ufqKpf^qT ͠ gQ렐?"T_iWplAֱ;մts2F%[4J]r&j?G!8Og&'Iʸ_j7T[3,jIҚ3#Ӂ!c g{>7q==0i2;ybڻ;NɏRq>y;xIwJ娒"Ú 騮ZRwL_7uzxgMЮL?9PǷ+//spugQn0(7e]r. nLVA]nsK:sXM7$6l%i&ܪ5]S9uvW姾/7#/GeX^[}늻̚)1ml4 4vٙPq@g]cɼ_WbS;&ި.5!e8U)y$q G+^zk׮]V{9nhၴ!,ZS X.RA],1 H40+@5`"L@0X&kD۸o8y'STtuo`?/LV>:c痴9S3OE*gvZmi~I/B/N5dSO?tןٛhdX<7; ԥm֒o4D_tGK*~Zﭤ/ b>E֝;Ww7ZԴK*mkP*-7dFW:z'*~9Դ4eXw}JW7>c痴NcjZWg3wut%` :C V[_R3 I>ynwwWwgwug&]DVm[曫*'MN0=Я e\S'z]-S `$F*OOh=dvwZJ j>5֝;WwC7KVqiYE߷mJ%M;-'`5 $CS<:.ʟ}E;Ա(^zk׮]V{7]<L@0X&kD ` otONRQ׽x<+˺3%-{wZ;} Vm[fTJxo@;ԙٱ ؜$H@mCYZWgꀓ{*=O )CD_R0iehu{vs*E- Wl3Oz]G@~v7.KZ^^~΁:Yd˰~ۖJO>N{k~ELSWzܑѢ=]t}TZ ݐqW) V)`Q)]>* =❭ww"@}^ҵk.=OT.pX&kD ` 5`"Lqp|WOfuw\z\=s|Gc71;`==yNkGCAmSi*j ƂqC rr5ej+5FQ߸<{eQoRK皪96.kuqTgV|¤c`o2ǥ|ZX: p%C3TvޗޖCΝ-56ÇZ/GมPmwш}!< QNS5ܪ~2jIݢ}b$iX7WfPVxӔ񴬭/ Z5;}.aH-%꒜uqQ=UޗKZ^^.]i޶evO=S~I/GnŦVkY+h=^R/bhc`72*N-t}֠_|{RyUojc;}w|'wJusXSy8  Q~P}~W7Kڼ?3;*)ec2}^@f c .[O ]uK%h] -{eq nWz׷o_E!gKg_`YC(A{ݎ=-7uו1iTF]XYPxOkU,qTsv&=7pPm^Mܨ8Sb )$;[I*Ό~z D_L)UگphU޿ i4ܪJM/I/{' oV`9,*.;m*|$i\ODqc.=ںIjfD֤J Q;M*^6ep-)[&dg7Pj1Xfz2*A}(C72FBmfx׼Smi5Y`:"2T u/p[x̵̣͛f}{ {߻=}h6=|QNJ>xۆjCe˅l p6]Y}/a xAЬ}~72eh1(w=|83=UyP Ew8̚)=^.q:a8cDxlYG[>tgPU%wܹWRU);Lb7<*(6#7xfo\V !&=Qv]#;87oE&sx._Ӣu,j7Z*7F87|>QX +ydP5eɷOgɠʒ]=zvn26,7 $f9FiF|}o&Pr{`l#7>T6{Jvק#"]:3¿)q\3}*Ff;U/6UZI#x^,{ lwI ۆ~?s^3:[͒n}9x5Is.T5J3_{ru_SQ{Y㙑q7P}>=iH C8x2z*9w!NAvvY%'glu*1':1sOAle@hs?~`_IX+|Y4f_{4;_eZj NL\YAঢҊƞ]ʖ]ud fT?rϫpԓ%KU.atRww ?eM|c[!,oQW8׽ǛςAʭtŬ4}/;bV5]qt=2zz_ 1Iq@gqmmn$}wm=5J5s ƎPUu1k-;wKP g/Sb {߸l- # jYxIZ,Ա#MDYQz?D#ۯ%36 ޅ _?K>_`2d Xm|sU 'vOvdpc/jJ#;5~'TAwSê/901a|;\z_99#؞ܮL[ƟrXQu XjhV7^հޮ(:#̞1̺qo+=ɞΤp Ϩl&A{nguB~kW .(ӐRzIռzTw*w=c̖5$ 7$^6ևǾ,qK-w߂*Q3b-[I@#xf~&=)ĭc7SrgqC1(9{vḅf{TM*8Jԅ̨U&Cu7-jZ%fjjjVw7V ȉ~{s+_φ?lfgF܇=2n}S 7 NB5wq]v=o(X tGlJrl#jXlnm,>Xg|2zkMEx4~u>y[I[=[UuŔy4,`mT?3=گu.]֠O1>mɼ[}[ ݛ֏mS#NHcLգ=& /u`=mp3ջAseofQg{aӾ<;N(G)o|! Mߋ[y{y%Sm=r12n9̙Qmr'GlKV:4d m۩~W7C.=3ǔSr I3i~6`gMGϧ=9Lֻku4tܮK1%Ϣ %ZT榺Q %Ґfmg`<믒ʇxu.m5iO̚"ݕCo {BOi7V]|{z˂7:[7BY:-9ԡ9޸=Om=C?8n5A3[ֺ봦r,vW姾R/7.W2m->S(pcVSfƮ.w7=Y[#jގ!S ęJvK׿`߉ `gkT7:0 %R9uYȮgpM1^.WnS6ZQ*4d .j:S.1\SS@Ms)@*S ZuŬľ"C(Ss~S_ߓ"cjpmpqy!0V$m|p?o\bXʶҼϬ(MOn&p#wEV[ {[NP_H<߸Uؔ"f>?=|~ , |8K{.\bi3\_>]~Y 1nnfǭy߾a夎cYKu~s2L 'ہLK3)#cۘإ8nh_Z榺E/'p\=v_ [|Vxxnbz?2!M8o5`"L@06NI4Ԭ.5~ݛ;:i/̸|lXO/iyQ`Zo2ҔYY fPjZ{ ~zxuڇuu!|>QX Wijg֒o4D_tGKlN-}L)@@uPyZťe*|u߶Wr>RJt}JW1~2"HW^ڵk'*M8O5`"L@0X&kD۸o8y'STtuo?Q{:^ry6CKZv:۶4|oDiIYvGl%Aj;3;:鵮φ)M'Gʐa9Q,iehuK*vd#銊wDAt } 8wgwC쪭?9P'klo2\ۻc( 0!P)b;d(l Ę`H9ݻիA3,n:R:O$q;,8CGNMYSn_LA]ek:eGG+ףFn" 0$h,zg襋kN` I~P IENDB`pengcheng.mind-test/docs/aicc_cards/assets/train_nodes.png000066400000000000000000001304101471400134000242300ustar00rootroot00000000000000PNG  IHDR! IDATx\uab. +vI)ӡGR[eq18Gvq5ѤLvHRkÃۤ'XGS1]ƒ"jA% J|@ˆv3=3s]~_jjjjBq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BKtl\OPͷ\e ۸'ry#mvO".B)ygں""""".B}*R[wADDD䚠BDDDDDDD\B!Bq """"""" !DDDDDDD%BKtlX63"jma-'IAŠ}le;"ՑWtIv܎a\6>ʚɋ`nںɎy#Y'zJ턯`R"^?Brm~O< [Kl[k:a=ݓ%` 6ɜEv#z$3EYGy#ܛoXD׶oV#""""NWMyblfLL l{=Q!$``C'azo'wiҵmQՌ ڵU0y}$fooXUbG.s!yl.<5;)&B5q.z{_c͚UV)xh֠ jU|}|l?8Lzd= ݹ m?6Zy6>/O\gO+lǃGy6r{x2gQ_Ns DDDDEjjjjjUd1c C˦'xf_aQ5ZfܞAxlsyzդͪ !l>Sdˁ;m;u֗V⤚`k+ lBrd=Tؖl:|W]lʊ5Z{T?ewGXuxgۨ*▛;?plݓ9l@*y^UpEAK.0*Mj q\AX}oROՆJ rkȿ,"""rpq%D9߾N[o}B.|tN FK\6La6Tb{z{6$`m~czԆ%mVF]lo>aCOa-mj"m`; %n܆!#`;M~>.;e69:g$__F |VVrhKruN^꺊 u|=8TYaWN\{:Љ5acy77lЖ۳cb,ʸe!~='1AI6}>ogܲAD}>'#@ɏˆlkcq<>FU6$1-+7 +&/9#Sjr\syV]Eơڊ{Ǥ+cl?4A'C% ϝxΆ[I_tkCsnVpgЋÐOz;t rX7s||_~F~1أ>'szmDDDDŵ!{]/ﾥ0kJ<7Nz{ZaWJ|V+4*a[ȫml`8x8Ή`[-$΂0oR}T(&|Lݺ{V+4*>Z4W QۿG)>(ׂimUCF[_X a_W=V!C%t;c%G}h/(kyO[LLyE.Mh:a#tg @<|mUB+j>?O9qr:B׼sTEf &h=OpG +k lWXV5 H1g!x zԙnn.'㍇_)?toTа{eDӀipP7WPqhpPmߗ 78qBDDDDڧ6 !*+pd 8Ι}fI1Dogq J8QꉏB†aF8ε#n:H{sUd*5'Qe*+~fđ\GG)>WߣݟI8 Ìam>m7aɯ6:Q_MP@Cq[w7(rN8d3sB8[g4ʹ{5eal;gcO9[ߚ  nG]U!""""?MsUYOХ+u8]=ΖeDi9>]Q8'Dp<쳅,iogTws06c-'3'r%""""XTBtO7熁 cFˉw>eKg` |=v.{E #l xS5,d;+lp7iW%۪&aThpI2װZ㾑^_a* ?GQ6tw$ּʻGZ?ؖl"WY[X$^e{0i^g4go}^׫ݱW5[7졢sW+Dvg sXVBwg$nr}7~FsOhrvsxM,(ɽ@~ ~/O>K^J>Hky4ܲ'm3'ĕKc\'b5l_T' >޷q3 ޜϞD}P? fd}Cӗ{N18l2%D~1==ηMcCeEĮ9J6a; FX U1l\ sXt(}r( :"I>};Ww;ƾξ<# Nznjk\Io%5aGĐn78FWа]?gG*Tr[w9 r᣼T?α睟$ē3v}?Iӫ{T˨5555m _VuGr͝{^/o.{m0'\BK(P!""""""".BDDDDDDD\B! С "߃WiBMoP!x#ۺ""""ׄ5555m BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bqmɹm£>D\L;BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!"WűMOgަ}ĥ"ƚwf, {MsQWE%u={lrl:\6%ٿxN&"׼mimJf;|iԠ<4y!0 >HxһK,".At^>u?όn)fy WҰVompWD߫eoKĭ*hpUӉ[U;{r6?; dyY ~qn6 ~24D]S!"?`K ^`)2z }>˸&=k?8,DnefZ^쏷pT }#~+!4kdؿkO}Wm&p8y[9 pjq2V@\)F~!j{= H{BDcy`K%$[A=p'>p8j:q@M3IS#"׆aHl>P޽|_҇7^Mely"aG! OԆcHNv¡82slS=p2y\m89hO1oUCzK|Ы')IBwjJشN0#z*䩦gN_f +V4?{?d?k/c1LFe%O` ,{3ccC"9~F HSKT3:ln6±}߾m IP|3?96ƷɶEڡBDF|ƑVٳ3"Ҟ58ʹ&U V8y>gĘ졦 /սO;T 8vs:]+- ?p`qe$8saGd~<"59`為3*lZSNܾ}j+)Og)g`D}jllYӜ<UNĐ|~z˴k>H^*LdX}Ơ'N=}C%v5x_Q!Vz?{(MS˓ާ=y D*2y vv bl;YE˓ŒaF àq lW˜ }7(S>/wvr<h^rB7EOBKx0ҧ弜ԓgj ledCwoهf 0l6ѽ`MW"F״bafCs 9UH8KvtS c&Oo ؼĥEqӉHd&O'mmuTb_6ʩӜē W|ν: "BDJ7k$ 4n~fļwCьWmbH{ lUle?bM5;R Vh=l8Ahmɱ1w3d{o.Wnjal!ųFjy4' rdU"SBDTt{4u`;[ ~W7A/cq2~ {0Axz)E*0}уYG'aSm\^bئ|`+ WQ*:"ם5555m Źm£>D\L;hNq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\CMMMM[wBzQ|d[wADDDDD(p q """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK( ?gKܑ71|un.^M 555tlޠVM\@*؍.nt+l[e,[bh~XT,έw14@Ё;2 ]jrρUD3^ a|û'cXɌ?vAg? w-3$XzEY mgo' %a0~fْW0`D ۪z~ʖ>ҦFf01,$d9(2JTzq>ȔQ& F=ʶFUENQX*?ȌG 3+8) ڕlI1a!Hڞs4_I1CpQq˱=ZÀ`&5u#ն,3cҺmiys4"C00 A*ad@p4Rltu0`ž4lI!`",HDϰ#ÂD%Y(ҿ<Ϲ4֝ca& Sxal)mRd2g lB> $'w &,@؄wQ!Ҏ>~0Xi%iN1m-8BںS+ sTg[wDDSbw ~ǃ7:]9o:tId$amZf2ϸZh'nԊB̉w."?%-KiڡL{1+L~tT[Q:Ř7W WWͶe_?fI٬F%kg3c_ .TGrF|6ja͈dxg$e t("k5іqd`c {yKvsXfcVoYlo95$cfoJL#kT2GڅȄ`An=Xdg[D4=l}#ag_zq b{ۈ>&gy1N]6pt_7 IDAT:͏^PqabaG %䅲c>MP?m1=%71Q׿hK!FL^'YVyD "mO7oƒؽ&.JU2LRȋQD>RK1>- hيqn#+LH\"{Ծ&((*=SeY3iil1n} e9)i>;-tyW\tŋ5lZ?p@>;U?))ZhORϚ\ ;YbK?\ągI6j_)3)ji3?5OƱsbSI{j4"b5ì:aDݸر ڡ*h\ aY0Ԇ1ȓٍX,1Iv=H" BZV(LD1VEPVX5zl E)0j&!53gf1n/ 򦕀ٹ#K+Y9gڡCǑ]t``BF^|ʲ}Yv_,/7t/Ǻʉ2Q!Ҏ}奺`_c=n&_\ mF9V,c`sm{ = h/0GM>b9f:8 y2@ /"WaolsI!'/UsOR}W4.gSr(=I$m6mBE)e#~w;> @M5g{7>wo/ٓOXI{,qb,D'wexRTf&^vBv掜.ui۠_Te_Eeݺ@]Y*ESeT|8$_$ W0$ko)*.l gBPgIօrgrCgrO-7tKӕ4Pn% %^Zli%Ê1{1QW0)um4Dg1iљ>|$ [2-4.3F`~/- tQdh@]axrS{͞Dfwg! A {4 !ZuصDYS<dy8^KF׮P]h=tDŽ:fp. cYelMPƒ[.q?#mTA^&<9BmIb[w|mo*ht{#mjKjDS?=4޶Su#o|dsdlCDeex b|* /4J*N1%{Nms(b&5;$-3Y'ק;\JQޟ3ς;pȊ,Y}2ɜOt'?ʶ1؇c\m{Ex9!2*6@1Dڵ7qVxcF_<ojYϛ_E ^՟ؿ?Ha>ڄ[x0~h.qp >* i33]n fR{+X .]`⭝[YUKD0dvK.tKulc꒞nT瑕I?H2۾gaWOO #xs  kө9z!G6bS۷Rv:= $.lvzyxc'dg=M[yUne{^Pډ+$pY\mZh@p(尮 7,tҺ,o9in@tohsʏ?@msl˲PP{G{]ҢlKf)t\nē<ǍսUT|9JN؟we1V &ۼ8iJ&Q-\-2U,3C0e\vK?de[1F:O#`NO[]9}e\]ZF~3GOۡ0|O_-B>؃Ё>)Λ["71}TW ~W3[%Uwdz-abiF93KUөK jj4VNas]ܡLbى9oG !?^ <-g3VV@keW<է*忞샻Xꤙq*Do/ rzPh%; 3)JWJG%q,˗3wV{mB*ǵ47@|!)]]%8PlǾ $?p[4XNg7+y)㮰 J,[)m~D-/C(K̼ 3/1aL?pciJ1~k`{'fSOCҵ;婌٘ZSX:3i <ŋqFAnIL]WlǶ~w:r a]L7,We?yR7n+!#az|SgHmۃG%"O]KILJypiK̳),g@^ϙvSŋKv֝NtfzYخ2ӄyn16.gدĨVn&`mpwB ˏMe}+2 (fƟV:q#),?5,heU;~?oCEU^CX@ PS#O9/"?ow5<>mnZo*+3f܇UTS:ŋM'Լ9!薈\g|trnO[0|wvm^23"XYNՍY2cc#%m#ߙk:CD̔<G)V9um3zQ'1 p~ ܮB_kNrs{ WWÆ%~?Uڲk?9-EG BDDD$&$˼JďgboEs$p} ~RoֿU3>n Хҝ.rX?```@عuE$?r}o.{n7"""rDOLͭ#>~|%:`>Z}t/oH5tЁnCtFqܙr-"""""?6"""""""QP!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\c[w@zR|d[wADDDDDtiNȵO1DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(=v|FUŶ:uw oOB:g ϻuWDDGgؕO!H34Cą.*ywU@!Bq """"""" !DDDDDDD%BK(h'!D κOe?Y7X~>;\;dIQ[e^zCXfƐvѝ9I imHVϲw.`f Joϛ'ZX"f6k\ҦgXaF,8n}9P e@x&-R@1iѡe&"]!Q]H)}ҋbeFkTs$m$,2PYW-+36s o{? @,/)NuœGM RIS 12,@Lvd-O3#&E1Gi;п & фL<⠽m?)h+%[Rd2o lB>=w` lߕAܭٻ@|9JN/%4>_r#k̢od>B0qzvfqv>ٖuvP5E=O7 B W[Dv>M?ⓕ +P\#^FiG:oComߜlsĩ}ı%msd31S>O;6u"GyV"W! Zl!ܭ;2KJ"gg|efJ F,&k嬋M[bL-=rz}6J~;?1 H9Kl9%M!n7; y$OϢ_b|9uv?rUaaI8p=Omop(9d&? 7ƎxU;%i+L Rvƽq㙓55zWVfD%QTH^}1bi$p$LYr& $;57cV ik@ud;=*JN؂żc*;ܫŬqqPcZ6knd{MRҵS}Ag~Ɍ+q`B/3}<͒;,,WeͅG/ y|-7{; ;d39/ 3xfW[NZVvô0g#m(ْ´%ݝΗ*(17ڝn-(Fwa{ g#_60#KeTȶ~4u-0$,'`d}8``]"2 Cvr,KV9ܡ{dҚʤV*a鲱AQz 3.[s9Ŝu1vF|G ]ɶO;=nm=k +@O`Xİó狴_c^w `Y"vUJoϤn6 TÎAD^$cqĂy( k=6b MXGUZ{HY1ASè-X'` Ï-Y1`eFt1 8V.P񖙬> =u(w*ϚIK73vפO`3}D7+[vo'nD3W<_ow(A,e?,1@a;'N~ 8O){&a/0,yв~qRoZ5ע`Wv{c/ |Rj_3B W'i%Joڕ.pCa}`]`Slh༹q9Ǩ<ͥOOu |utΤ\ߦE?8̞8lfL\ x(bH;9ۏh?£Ls=ްK 1s-କ996{sIaB?_{MY|@3V[B鱼kaoP,Y2W2%,Yfvd>X/4?FbYeU˄_fDZ4Jʪ78hQ~V+@Ln-IFD攍wi5Ϛ3frs7dH^[?4lGV3ҡ[I{䯘A=LnNk':ca= E hc _- *X`|ɒ'H{(ё8]sɩRbv_c}Ѣn IDAT*֙ɻÀ7G +{o﯑KojTqL5Ha8RIM -ķ_gŘ]dB,33 H*̇L?Oۇa8__ɺ#U-gݿbtڇ?Nsl?{e%ijj訙TW?3]]Q\H\SL-R3kZZJ"x(S7M@a30h9Xu\ļ':mM?r9ڙJ#cpYN uɠYLf+y u)ena^ ^v}wP:;rnKx҂m3xke*'JQM7]Cݤ #=ǡ0k,2> >M.X>:Kz6s4IߢdrU'm1ã9=c yuLBG'vQE" sxZtDWUgmf&?Bwٹbf`ӝiH62,;J aoB[ ȱ^r4nb!bD':Lu N184{=!ݔ o'ޛ"}c\'yle'(=Ϋ1eѡmH7£8M"_+u9Җ31*yi7}:b5P=TM?9bXcU`00Ye4O>:p6GiSxc>+Q YCl55@!ќe}'9l)h0u^jPLet[UkXRH ]41[/݋Ԕ20αh37=Ϟ7FhZKQȕi0w/ǥ "\;EY$N|eKb 䙈+FHkPl4}$2Lڟ(!W杺q7(CΞoYUCOoݝ^>;='I{s42,qo)TMSjԳ̳Ԕ0&G٬\Oj|Q ޛ@?SA@D'&39k-J0UapG&0`,i ê]O}˵4=YDzy `:t L!];=ݬzG Jo`RzW'P[=vfBx(FēKRb77Lެ8xڋ)Z:X:XBG%0x mH&=XLrI-Ͽ)3v솄 KTsd)NID]vJFSHn?4=z]njjG^K;qS0#1jVnKa}q5ǾAuSg?FSC'ЇRXv?MaBV=gd2p O6 _V}2Qpw'VF^f%5U/glWObS?jC8l֩ւۆ?9[a'F db]s-m*ymG s]ԩ؟RLɽ܇ON۷CDDnzXʋ;w4n4Kѽ9*ϔ_R9>23L3hg"? )ya L!y7Uвe~ڹjY2J eаly+c.Q[qbG1͠fzO0~Ɂ,KxQqgzV1%2q$lߦ1Pe[mRhF߽y&(u3 %txNg^} M%n^`"~ٝF[ϧ֙tO0Nuݼp]zW'. &~c ASGWӋSn]zbwJ9d~I98g:N$7:.o d0BwQ}jOE!Ҏ׸L+YƉ?%!@ھ1ppHԹ<!_L˪dn9W0no&x(} ý_roDJ"( CoX+K/KrlZr-gЁ :^cZJ|=>G XGъ -bԒ8pπӮ3=9eOk] 8RoWf0}_Dsk!lz>v/SxXjERC%:Ѕ`"3>xi׭fpd/64BDDnz @ xwHnv3a:؍A@s6D>߷M+73r9#oB+$5=a 5gЀ`Rr=m/! M`C820`ޟ^$jA3f/`4i̺DЂG A`"6ljfՉwrlh].qO?\ۣˉ֕*2HM fTŒ ɜٹ?*^?,rGrǎb\mo7 q ?'NMOh~8Sz~^ϞфngY41f/ 0$x+$ x*e=5nr?_cL{x1~bFt]myN#Q-?O c? ba٧'U #~:ގը0-i>&OYCYZf%.ӕjN`LKğhCp$z֕~֑v\x4eu!ƿeuNcdACn$D߮rM3ZAlVMDǙeF`VVȪ'3Ycp!p/glˊՖܟrw |y _O1۹C^x#O^`C yfs}7&V=GAF~3?d>ZHt"sB:"rE$$_G;|xx;|BeP=xY.R%(RNDαtNŰM\tpDOҵ]o79vlK[:5a  m۲q:ƚp] !E`J&<YTGNg11~3cğG[c1h39Gar0F{֜q~N N0w9~l~""RnBD!\coLcEՊzF.I_kṿBWw{:Ԫl}$k""""ӤBDDSwL"""""""r+R!"""""""BDDDDDDD4a\ϫzmESǮů50-ek;3u{ORxr _^k؟/N`x|j=#˸rf,jاdgZj9<~8jmsW_?ph;*vNH='tahF߫N?go~}TwggZse&].CΎ,${5^ODDDBģР4l//TU&qW?ˌnԜ"j 9.:QEqU{cklih=(MFصx{I(Ink[;VX:WU½T6@C6󵍎[Gޡ86Ķ>RWDžڸPs=6lެ:^X v{| ϧ<٩o |NdIA4رu.$J8ngZ9+;е4ZTQNA-s{,x4as| M߸wN{)MLꡁе4~k oR.QyvY4W+BxgH Q0w4jn&DXKGҁw :Q ?vqm(v?y:\`Nd 1~ oiF)Uam]g9!x^n=qDY›xב4;'܃v.qI|m34J;\ٜt5q/*2:%o-Ԁl6Su.t20j[)p U $\5걐F=R$e{|??i+#M- ;{0%S,^w#p zE8o^*Ak~*""""7"(Ԇ71۽b'c/śvM4J:pYBg*:Sv\G:UƵҨ7*w@Py0% ֱcWt՜ŮW< 3Ƨ!W75֩:vNt0>.0k k.F~$IB|>!v8| yM5~6x,$y뀥BZ.Tca(h"`<,Ϭv.:#8*8}{armϿ"S&=TGi؈"܉O3/&'ߤx]Wf)7Uɪ){z$l;L0Tt龅Wu,jCWmd7mgo6{BֺWP;s)I6Gi;φt{Ms ҸjkHbZaL)O6|<* Z݉N GPf>)4.q"졲|οCY!O =Kio;K3nW5!DDD'B!w\ծt6$'$G7T}2jt8֞|;i8qSMDv:hn ,\DIB8 bidwޫ-:֤8iAG|,JF $!t+erB[`x*Sh9]-v6qegOmS G3@;\I{z|>Dcɦ`Jv`(Tv\w4|:CIA[ިªQ,d'NEC:7[M̕3iu˫DŽCIs%8}.8i0؍NmXOáy;i(I2IǏ\J\h=/8.sp레x(m^UtΡab].ݎd >?:SS:͕#(F,(qT\W lyrɵERuLl*, ƨ yrR*qy~oŻGG*jb4zrׅ/h:Qy&L2v2BE7;3Y*Ө@l'S6O'-smSC4j rMTXW2cL+o(߇Fy(W5lhԅ'9-~$T Ka8IY TzLPOkk[p!NVG^M{ef_c(Fl2{8/:4&aQ+[!M4W3O!@y;xY `Vu${<odM*ߤR#:ZVge2R=lB,tl`HދݧΫrX}daՔӊjp ]Kn:WdS`k=4m)ϜfKmL| wdMHR3 ~!mk a0׫m2c+؍^(SDDD@!Hrtf/L/֨_&Du/@ x܃w=׸ŷT1j,TFY@P[Xdvg5\v[>we$ev =hheՍgO~p(F>weLgȪؤH#{$".CjXG:'|G!q||:=(9{l֕u)+6g];l)\<+ߨMDj[j[^ZS\ȭF!U||bBu at,E:,')[bv؞GٺEFFGQ 7׼:Ɣdb5n) XՓFG\<`wn|1I/Ogլ*`. q?-Uiޏkh1=C%,`Av)dWM!h&<Ch\ݜVz ):v[jJ`J2©eFX^w]PֈH36FaJ%'k%&:%'ye ˺^x+ i-֦r@7u=e&qex,nV/ $dkz2fDFp(`h ӧ"""""SBDDDDDDD<» """""""""""""" !DDDDDDD#BG(P!"""""""BDDDDDDD<_F0}ㆷ؟qP!i:i͡Y;^""""""" !DDDDDDD#BG(P!""""""" !2xebm{I"rc4Myn!L%KVJ|>.d811?qJ&0-6f\#sg$1f5\HVp%R}7DDDDDDjӠ>/~\a wN~bx%#9sߪ|{_20!O;EH3a+$|>"30&b8ޛIǹ?DbJ+"#N*ί$㕌[TC!w)BBDnIr^۝G3#"""""?*B49^A82 $}g"nb@-eNL|)#9A^h-a Zb ;rT6=7`K0>;݌ 8.c&)!і/,@A-M6ҡ<qv)11+ \ØyvSD^gb9E B [) SnE`-mssyآKcQ/8>AOx_ƉHm<Bw硲;!3}|:ݍO2y&^ByϮ߿oU !nAu$DH7F_[4ȉط*DFs5NFc&&1)nȜ|!AȪ [ Ns[=t:݀&nn|—惭"xT!_hN;<<YԄ.~R/^#!n= u9C%Ʊ-5#`)@7 7FwҶ>O+zl-z2m4KDDDDD^BҔDlOçј{=.íz5!)(4"ㇴoU -X[FK$r-!""""""?2BSy|GMdq~Jj !`^ +ݎ.8R|X%)(tbUMHbLfO#H".?Nj8)"7FVQPӊԻz!h4viڽÊҘ?1%5]`׌=İr{YyH'1+!udEړ#DzV¨oE3&)D?yhyNQת5F,99G]-}Ä\b$_>ۆ]23(I)ulۂ$$a[]^PcDH=#hX}7J`(ۉ6Z.FDDDDe6[ ~|JounG;)>7mrm.m!xq/c"/¿]p{W/c~/&NQ#3$ؾ^'e罿"aH t lۿ"a^^@6뵶9mwgGP-Hkq+B&G2wǖtx9kx;"s=H=2 չwg&m%Տpz'~η/PN#B92V!ng6ObdSb1{"3-j:6bF:H/ /ӈPS ^#Bz9swUm^+8׶ƆD@XbwOړfӱ$W%a]slW,]F̐fŸDkjzN"""""rS0x/F%:BSU@` .lVfZjAA9]˲uFvwuz!"""""!hʜ +Kc9%e} 4OaL|,_]Ge,ے[zJz_+ vkH3m;f^FbdUHu.[ 1 Rb GC 4*0LjlCHn񃍑iO~_"""""RnnD^ƄO*Th򍉼16.at[co@-yK~mXA֘5eb.8ئ9_US=(p,^)t t%w鳜9-M XaV jE&hԤ5ZѪ;\ػ]Տ~HjEdLhswq:XGOH{z$_+gTM1*{,if]ݕQnVƨ5!"""""7Gaug7ƏߠW) msnoLc.oa?x.Ct~XªEeqa{kqDSFD#x2Vmm=` Sp52iQۜF|}ՉCe@UMˑvKj:Mǰ.y4-#IKvi۟a%:ʰ1aLpV5íHC9dP('9\nޜj8rqr/ZATGiz-CZд_4yΡ30̃{<%s(= VXİ;˖jSKnM]GI\LO4l=r4\F|mL:,(ؾ!^;mY漄GA@kQhWl__3U(r:ue4CUȭXK_vyv;Ә&a*8֎p5c !""""נP ONY#Th=Ѯ?*8[,}| ͒\P8EOư<ہ}wil}ih.$cʴrn%f0QbӞchbִ{yj܍9hͥ)#_EE~CfW?omkvokQ_0+q3jNc$mF3l&r1;e$W9J(l?zi1^x [ity9A1>xҍ]ް[#z0"aH6\DDDDD ɓ^YD~aDw|.`!y1\cg,ՉcEDDP!7gs19oGChфc0R#m,;s "-DzayAW3mqEɟAY4uVUۆҷjqë?=+x+G E;bTqwg/fWe C"nj8ŋH"ãXq(&L[`MaȏZm:Eט͘/kcS1AC1ُW꣚%""rlQzYɇ\-&1߾xy?(S9 [Z5 Ey/eJa2{d2fnF5ݨ,AEj$G) ` %#2Jeio IDATd1OLa#Ss"m7-$ҨJU땕\71/^L!ژ{43i2a3kE&R!k-#nLNe_PLM;ΠǺpb^ [5gC>L!I`VGbc1.NUL+47əayPCYbBYcWg|J]Gb,fٻImi rD~NOD8rMFSCޏY.gC4-L#F8a"[fφդ%I*q%tFGPt"bh'Fd|5wRNyQRFۇ1i?wiiX=r?_oL"z&.*hWEm/a>-lOMr;_G*rhx $?!Zx},S2~?;_ (#&<IvS4-,Ę,h݊GSC4'r # PRߍ;s'Nnۙqu0Ajv.gNpbVEMeu({38fpLtGN`Vr0Ͻ? O`AYT-8˙B<EoUpgۻ;o$ +7D-mcԴN\Ѭ4}DĂ&ے2/f YGK7>(}:}3z-f{b.n<ေ'gpﮄ>84<^=;߼^y?u'<msvw313/?镧(:u5OZ5dNQ%B "OY i:z):_UPC7ޢߣƴ=~xE+"""3A|)w'CRIna͌0ntĞ%5阰0FN[DsW3/vuCnJ/B`y}Y>U&?}5K; "" e>H䷱{͸O}~nO|Nn9g^;~GI3 2,A:i >Dtk+}|eM`>^!MF;EXU3x5qـ?~>yxj>q׳<7\i=3˛ӡl6-rMw?@ݏk~Qɍ06Rxה9dd1eϿ퇕R%A4c;hK-ΣUϪ31&_.wyJqUY:5<kpy_xU5਺?Y~SBQ׫Lno~ͿoL3p=Ϊ5nܸ{?G׼хW_?>I.\J>'^~n3Pt'_,~59Vi磜<]g$u[ϲUBqee~_T2!DDD.S'EcidfF?XuISZ;\T`u}T4yoil؊/Ex'ƄXs!CX/9'_;"fꪫꪫ_]Ͽ&~yo6]'{+RC;G~'ܸĉEyIcpg[}G(Z^E??9w?ήL}3@^>h~ޝ<܎L#]=EhX>s'7\1u-vc4 |M;"21-ؐݫך`ЏΝ#9i;ym‡~ȉ{~5܆O}f[-}3(_4x){ ׿;.kǎ#.|&p߼7cm7>q 9#?%_NLs1(1ɍX&Gg~/eq8po1,Y2/_?#0ZYtc;So SgLn96[ZPL%cg^P0"""" ef̝[ș3yGB:y W+;cM)&x#7[uPB24m4 fSP0;oۜ~5\wݵ,t֬TWW1ܼKDDDDD. BH֦_s sq e 26v8OA,,(r!\suȕIAiӦqxc+US]+Ɍ=1+ f̘~A]8"""bP&H漙:禺("r1c:9ou\\C1L;nEDDDDDD䊧""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""yqT@D Xxё>sdӦM`L 沰tr^ɘv>WPw=3yYz#DDDDDdJ(!2E 0o^!XPf& 1v N8FFD.;}POUH^\AVHPuj9t/Ҥ8˫N^<b+ʹEM[ټG[gU| PlX\ӎ*<+7Hʄyu-Y{m?}e6eXo7_ɦ6ōy ll'Pνo5}\qU>FGNxޤq!ډIDDDDDb hfADLE{ra@/wG3[f)}BfZEznl 5I3la+p-WgĆᜌ YNL""""""e6Hse#V8YCW٧\AFfýɄ&" 1o!"""""""~"werak FOteB~x"̚3:5u];u:52%jtϡt@>!/"شZy1{*W¢>C9/H>\A+X&|֖)}BdXbIK٧^MM& !$ K&tRCtL""""""p!&'ĄŲ%cڲ-6h6Aj졏Pq55,|dUT,w3<~kN)"""""2!D51o^!XPs\]ӦM`L 沰t"""""2j """""""W!DDDDDDD$/PBDDDDDDDBA !DDDDDDD$/m:S] q TGDDDvT@Jh܉D@$:sWI2= ?mP_[.XvQ<}G n\%罅P9vmx;+wKtGk0X~sn-in \,S5^+=W'g o,*́Hlc$( JOYBe,^0BźU.3}i5 PB%lUx2/})u^( nؔkJ8XnZVX*|^.Q*a<<1z\̲&?Oٲ̫!0kfB Fo+ c-ʼ;6%mK=n3Pnf07}'@$;M Nq웥2zr)if,qmO< UjPC}OmtJRnl@O,2gL0~ZkgG+𸭥\;5[;"87-oﻉ,^E)qvⴻeJ$!a9?}&dڽqq0o֮~pعo^uX~%@.*Fwpzp'fNlGrnvQqoIg}{6R6s;eV=&u7[ ' #nI9DDd)!o.\dtRFvR8DGR\nO[Ի\8k'<%Rg,EJ%=R d(wY9J$H|Ge 1џVϨ|/PV롪'9nwyg]LILp  lM2٤;^k>xzY9/O~bAl0Ҵ [FU2A PMIH~;Ƿk"VeJ_] dVs]f9RG9pT'ь4oP1¹֠'{]RKtk=0'䥵n,C'#?e YHlTk,31?9caXn>/^&q?WmLm!Ba,z7 5ߗZaRs;u0ǣ8~Fp#cge[#.q]tj8 fL| Hލ2z"yPm˗Xi w@4+ ƻ.C' f0c4]4Gin83 C=^&6`n^N!ʶF"^i7䬒~kfXJ%߬;8fw'^ G,б(eI7 @ہ"V'tɏ/!+K;;12MDD" pP{"ApTrM/u͐JX*F݌,sStr\oSaW&LC w?IziVZ{*9fs'c'tqkmj4Hrf'20pܥ'R88ʝ8wyF + .?K"De |2탨 p?s/Gv盈\ɷvƃń3!wa$oi;q'to9"ʆeK?Fy}B,:`Bn A~QVư[oG)M/nT]~B *&u\'[tQ_ _4\iX_ђ/dq {i' 5I=MN&3"䎂"6’N:4BdnUrZ7yV".<hz9} =ˡ=0J$v0*[F~_+m̕.Lk_2K)K<[dhXn2(mQo5@81"LVb}q$coį}wkh?l8 };+pqb vZ0Z3'ļ>&"vY@&5i3!<ߌ ϺQ|{Pn,ED"f Qls[DjW%0^&^-V>)ы'#/޸%K7GppT2[o`oJg̡Ni҇$CsKe} `7㻔'kjsu-)欋WJ@tXPOJxa]m4L`FI'}zu-ĸ^ۑ~":ڃ|ow˗}j (vpHy X\Ƃy\{lNcph~Gsn`o:EDDDDD2QR{Ļvܬ8XN͒Ln@jJDX˗USUx{@0eP BDDDDDMAKlۯrl55דp7o|?u,_2{}0Ek:We,wFDҒb/(eɇbtW7)_ǽ3qE{1_˅xqKY& ]+IGV]q[p:/$F4&\¿^\Ʌ ;}y 36SU8}D>C(ԧlɋ"PӴ/wnXC9>{߇Zxtc3}5 \tl9{VS&ZʟNVϫ[LNFrcݟdѧV1xO?i??o-YIN PlX\ӎ*l]ۚqoKYyu[y}GE+m)S fZb[i9^~s޾E'z~?0c/ęsΝe*n=,r"9y^Dž,k'Z& Bd ѹ v 2z^6]̏}i|L :$;`|2^PR^`ϴ| K.Z0!.aϜ 9hDDDDDD.&MVkBqE5 _Oh=ڗD2o] ?~U:%&1t/N^ܸpK_|%5}w=fN_neP t{b¡``="ADq(쿴̚3:5u];u:qP fDDDDDDҹlwL tJqhע=tfKsE֬j[qKĢ_f_T9<_S_wߗ3cnrݼivb\ 2v CÓB  L""""""p!P1}N삕_UMLX&&/Yf5?|%:BտbO_+u',3_`FƸ-,]@I :ڟ2eȺ9Fh]mli+;yq+l]ؙ 9d=\[2}Γ%y*LZE"y0ãtp hN]0KsH\6Acq-f4y#Z:tb6Ŀ%FdYP0O}mb/ҼzϷ{V1b%{?dJKZ.<0EDDDDD$_?TBl|?n㻝97k>;Sd>9t3P>Ƴ2~GX:cFDwќ ne5s{icz0E>{>˖sz\7אK51o^!XPy0ce{ fq-KL !D.Pƒ겜6mfRX4 |) US]2(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""") Qh,IENDB`pengcheng.mind-test/docs/aicc_cards/assets/train_outputs.png000066400000000000000000000250411471400134000246460ustar00rootroot00000000000000PNG  IHDR4?H pHYs+tEXtSoftwareSnipaste] IDATx{TTdF:`p "PrVihNX'#Ĥ9ҘܪiIlRly_t4$rP"5#PG.'f: tY,ٳG˳'tuuɐʏkkCK*00`ڴȔ+(۾lmowzN i&L8QvmL6ph=G(8Pq @!B3g !B(8PH/9xN6` 6ođ3g !B(8Pq @!B64tv2`@`@dDD|| &tuu =rt%&邂|ucU]2>'@Y~;imhDDgmhQ⬳2 (yաgI^)wE~4Hwy7o YrR#a9*;"RcmEiy&!F S2{"'zěX|yc5>g~Xsoטln>ntT~g~孜VY~ #􁬈aF!ۙf5&+z}s6lfΪ .@EzCGBLo0|7몊Wm\kޏq~+θED?'w򔯜[L/ԹE 9ygl6a}e_X֚35rtד$"'sD,7no1?ޗ_8:sAN }sqW8gy[nIu/eW1gz yt w{YqW DDuyW kED )oQ }nh=~{G$8*ߺ/nӇ"b?f [x]qٹōCf^HT\>\A58,l/ ψ,""elgMxlbzn΢]܍+:hfo1b+qƢ-/wReӝG}]D$zzf9WLO[5:c|Y')l{VrԛZ"شluGJQ>|s#%/玅Z-"qq{}w{~MZ|M$,a,CE$$:f Lb++x{A^|_3Ϲ!L99/OUHTuavtԂﾬ;SXx.2iŒ5""!3w/訬 L}8[: Y|m7[*v>Uu0#AϼguN|PyD\$5ظ|%"ML2mk f]y\dv=:(8۳Kguzs'DĎ=k%gq_xᐝT;,jfg=秠=i綕4xhsr'333W6GqKBrRg-^l<¯rjk[M#&*-9~""2 ޷9~'97${{]pqr ؋>ȝu\]ш[TiC@Hi %gڪv>_hNY~6!r)YI:khA[ )Gѿ| nE^>3I.?!7dO]kC-><ţ-+q6Z5W=ab[V&ݞ3Dz?sf }ю0g͑-ǭY)|[f"2$ggE|P=Y|Prڙ$fox ?EDkJȹ}=[;Y\޳GUvrւ'{8Ozp[H+w,bNZ˷nݮqLZ5}a^o+ËwYřg~NRYrސ=xsW/5xDD;֜9[Svxg%G`􂄈"suI󆹒,y^%|SF4 >DZNcNlMIhb?7<b9R&)Փyq |rk=Lszޭ1"n7}Iw[k U'D4q 7CD\|1oa;/5 " z_u!UFtl+x ̭gܹo31gů≠ G|eܳE5n2#sr>YG[W'-4,oKjcm{xGiY(|{k('wqVڕ/O WѶ;eLnG|9&Ҟ/)+}>h!3}W!=+)zd?~mtU?~k(C%r]z졧dw 2\ӻ%Wjrxv[h SE;>End1oopGKLj3_]vq}Edne&D* wEĔ(WɎ>DfnKmEDD"f&q;7~& ^屘Li$=VtgVDĐ5pics6<9饍ng|wq۠ۉHkyaQjv|E#nߎe69c+l\p[ϼ\l7}enfMZ`i%ӢFw<q3cmYɫn齂^0~4jM9ߊѥϛRHZj׽TTwIDdr?A? @r/Pad7 jZDDp&?Ws>[K3?[f.!mX|~[4Q9?]e߿?/u]""㼕W̾͢DD 3MRy1&wx>,?K1Dܐ륷UQmYqmwtZ]Շ~bCo' ~|QMgT7LpׄyN7Qvh 1qoqVYY9A|3ٳǎUuɄx/O*v(R%T88JkCCg/oA|3 ? !Π,9Pq @!B3g !B(d93 vp9t|0OiMg !B(8Pq @!B3g !B(8Pq @!B3g !B(8PqL9=2J%kXshjXyc~Zzv/ޫh8X>ZT%bt˞O-r\#!꭭ ""-6 Z|W_]`.}(RDfR[2.Pq6ޫh?ݿb}J>}4\(G\-kJ}hM2rdQ_i{Css{֤G6 Nke~Y6cd˞{MQX9?]}uYn$C:pl-.Bs?&RE"ݽ[#EZvo+^Wg9dMPq66C߉˽7]cJK~Ydfec9Q\&"&"]PBTzPKP"wrv2;| ȼ^sV_i{4b-7V]uf[pTg!cM]{Y*"le1zk"ޣq6Zc{̬;bwaļ<Yr}%tpCpv M떧mքɶ$vmz歱""%q8evknJTquOk[Lzk=\b'Jz礸Bsˆ-7֕žӗHL-9ز6D 8=hVp/N'ob)A]vQI" f,nYS҇­WZJD S҇Nmtɘ,ZT$vmHuovLp"||MDrK5W3 6&m"n5X: 0y-DL\h8=Nqo'Os5>8fՍv?~;cGgYMaibӹZTURz6kB[8K$HXն5m"^Y X#Fosg}{W[>5Ս+g IY{g蹑c퇶Z0[}U[{nz2YB(n\刳gϦ=r]~8^q=yDo+7DD45M0zl?jEㅏl}3mļ|wv?QJt4M\x|vs?U4f5yw;Wxjbc꫷%!꭭X=LmeP#z]~$DM˞OW+5=unnzqKׇ;pu1}{"[;:r+w^2 wkBO-} >~,!Ց'1ֲGZ}D>K+H8K}~X‗,{̥z423n2ٽjSnYSW05gc\!n8{_gu2> ݚ !B(8Pq @!B3g !B(8Pq @!B3g !B(8PH/[j.8 "=&B3g !B(8Pq @!B3g !B(8Pq @!B3ˏkk>l0mZdJR˜ &tuuˏ;;=tAA>l:{gU '~;Y[kDD8;:/̺邂|;%:6*O@p!Ǝ1rzٲQ061lAϟo |@q٘2F9pa}m %t` 5g Xp:qipDh96ŧ(M˛6U7N[wK#Po*jN;'?n8E4x=#n2Np#FmD4\Y\4ٺbir?hzU37/$""ߛ-4%Mِ=9˿~ظ%\Ht]OrE˞Ok$&fj]R>"")I&ӯœnVoկvR."a60:f{G.3-9}zliLqWDzU |\ߟqײđQ9QMFiOՊ\k9Oip{$2iHikiq, nӒs_gNl!0JhMpJp l̘sr?{]5G#1"3Gk"wM"z15"/}0Pq6Z aîcsn}xLJs<CFNDf\#*87ɧqZsii[橪w FD#S'-Y󶝛6}+lch*8̜D/"bXhZG{[FO1;'G0=Bg/aVXN1Mo+hj꤄u;~鞄cOB _KwMd_Jq8Wô"nqˏG!3;qGQ~jClJ޹/+Ef6H;`xKzQ*mnCBT?0&tuuˏkk\0mZdJR˜Jyo|q. !#KC8p8oPq @!B3g !B(8Pq @!B3g !B(8Pq @!B~Rpy15B(8Pq @!B3g !B(8Pq @!B3g !B($Џ]~\[kai"S|T~4/o\~I asٳ> g`Wi;:nO`celw*"Ҳwӡ+Kck9Sȵ".^M<ߝHg{qIcN֠_)hb}Նt߯"bkD-_2s=Sĕjeq=mD'ƈhi:Cth\eJHvkK$bHW4#}#oυJI ֊ojD/KmAVq|\.I"0"οX\/!s.n""2E)4(҆=詪wytӔ%Ϳ3OO*X(mq6ɚ 27*М7]'9BOo4d^g;,!ˣ{J}õ)]F/㺨HhV\@}in-{o>OV'vkCM.On8-]GO rrK5јOo9 Ě2"Nt~֏<":MԐoyD4"7?oS{48= 1$k%+|KU zgii+mKD035&Mzv;'?gLiCIh/><-{ۡnA "2/og{˖~[cw=s1xuR_i<ۑٳ :{60o4|#TH'hM\86-gU|3ٳ>6-r6rډD??g)I G̅&0mZdJ/ b(L.]s 1B3g !B(8Pq @!B3g !B(8Pq @!B3g !8˖3Ȅ.=8  @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3 c)$!IENDB`pengcheng.mind-test/docs/aicc_cards/assets/train_task1.png000066400000000000000000001610421471400134000241500ustar00rootroot00000000000000PNG  IHDRk2 IDATx}\g`$kPP˚aL  w2Q~5yU&N6Lhj[LB ^L8?9pZ*q͛\8O_.]5 v.Uka iqF#'a%[y쬩P;wY6u'MEEECl\ a00@֘ԷZa(ka{Κn:g a !#5v;BX`Gka !#5v;BX`GZ4tnOvN@X`Gka aapo]WRIsKW9[m&pw]чghحU7MO}[94Xià]sa-8k5Tt֜LՎK R@cьšt@9kN'>!^$Zm{]ؤi jW|`B8M4haJI677vWFI=5aP6LuGZ9a2$iAI5zVhOeX9Ʌz˰9N)9[E O. q~T]^NUwBBO0ݿxXRlּ$OF{9/}^QvgGW9P;ձhDmXӢ+~7*NjlIR%hyG/ѹJl!WߣjiuD֎Y?lUJX1 aG,$)W)һ]asj2< j`'M7zki\5miN*Mz3t:Urv:mzMS䞒M^;JtA> RRFkI$&LҺpYoJT!XkghXI'7)t5 -_wv{F=Vh,Кy %)kR% WIi+F%9xbWR/)u PhcTѯnұrI49 ik8FD$=.z cDE_ V۔|P3QkVLhJLtc|MY:t0_o-?46^ـ2-gZܮX9cUPSW[ʊ/xšoUvTC]{S.dԟtiq*}!>jTuS|{3Z9Qq0(N}u,k儣%+qGibw% f`wҔBq?(z %!׌ | 6۹g9۔PAג5wrzm@z;I ZS_ө;,rRHOnRCvZm{/|jN֭M|Zl?ЬG5wfBOVdRo{/v$T}Q"Su(T.H'7iZeE&%œcGS>Y5%oҺ9_=0&7zu7hd#v-S)y]5fo<ԯ3S/LCmbGd;U%!a jcvlBN)20EYj3 ?EJr?O6|Կ9]`7Ha$QAȁtW4vm/aϔH:џUzƾzҭh,&х5%qM~$gJ\C't*%YH*/ԧe*> S[Ij$\Lr#V쳬Av TV&֥wIW M4W=0jgs#)<6I(IJdp&ˀMS;s=m5pr9>9MhZH9;ItI{0QZV"3@"zϼZOWӌKJ%rB O֝0hڐ[!}z(ai2c Sl6k-^S1I3C<|CcjMI.KVB=&H@tw-wF***T^^2ytcFr%:AXmUdy,7wԤ+<$7jp_7tM߉uzv(gOOWu4Hum􋪱ScGUN Д]Qn<=%x'%yp4T6\ӒJgּQ:9uhidͷMgkdA4m5vcK]TehF.^*cˆ.aaQ}~X7̈́cKhCݥFѨ L6dmàa !#5v;BX`Gka !#5v;BX`Gka !#-';'O_J7̈́cKz4t) a H|y+ QKE$0a @=cH P}s/ PyK@fsM!'lU7{4*YI^S~1Jd)<]^Ynw0uj4jUÊ7S~Az7H S[0}Gga._$K '>o9/Hߤ)̴3&=7QӮzECp/OzsC;_wulߡŝ T%;9ZŗqprBPSvz*r*?iN%u_5S_9m|yNR#ԹWɁ2lsQdXt- q>r:Ncj sǥIwT:IJeꖱ@9kq9PA#hmДg-(V(UiZC*ܤаuJ8Fzwwfـ=;ENOtqAz7LsS:r?Qh@8uKJ0ZOJ1z 3Q! RhL8vBצikT ^#JStښISBa5S{Vjxz i+=k@,3)zgQCkn6ވsJ^h300DcUvxZ;Q1۵xl"ӋєLO0-N4dfw6$5[{_rl =\uSAXim@h:x}x/D!z6&]%5̰9l-u EXQ F}yiGib>SL:+2~Wl?Ir2COQPW[ҵ/]!a>Q Iӑԫ.]5Z^bPXl?ј{~m?ߟ9~iUo?qFY?gogx0q\@{1aT;{qFӧ/l35)w\6VfqFƔNq㣌Ƣ{>>hؑg4Ƣ^5G"˾1~mu_[l;g?`u 6Fg_Wfew=8r}aq'VeqόʌF0L-Ӡ?7⏌ FĘ2oqFsJqШwO|ݪ:>8uFUƑUukL|V}QͲ-i F^T2ݽXφ-5W#5sSl^b~|T?ᏗkV]n4g 0`+Fce;QqQ(F"?j =w#k|S~w׼9~?{p74Κ\%:*ߨyR智g[߶qxN+jE>2_sꚀW*uBLf@? Vj[(Lt.ZyN/$'#Β$'UBS7fSNv,tdwJFSH[]K~\U\%]Cu|wZekЋzzP"'Wy"g^yZSҔn䨠t@,M35M㻙4n}IW6>t]INN5:S)DRy>(ЙAw_1:5ՠ7;~DJ:|%)@ mpт o55;aд!B"W9VxM?FR)N-$cּ F'wH2~ޛ֮ܳ|ݬV:>u\kRI[*8ZGƛ6QA\c EL443D3:h?ZPŮ/ToS%NrtZBAs(VAO}$Ƕ=t~æc4w\Jg+6&K^ F'GI4 j(HzMB$9xbg}.H]iJH7oE7ՠv^Zawc>.KW;FQ*//WYY<:rO$O) EpC;pWW.Y!# #5v;Ң f%R:P(j;MkVR/܏c':$F^:>yEhO~}qD_2wi_Ψr]<$]ۥ*_*oA<KG9 Yoy˩P[h{J.X")YH*9k<'7RU9>5Q{WM9LZ/H7rwv)_?'ՠ/µ.%Y|/X^ަ0]~>:LAyA{:V?կrSǴt񸦜Sߚh=]uUNR7 {஠rW.SnvkUFsM}WmqN)U;Jcp:Nb~WzYia v@Oӏt]a*9~K*Tr9  I*:vF~j2nҘ~.VWGV ӽYCq߶eM;t=3thc4UQQrɣP_36):}W"'I|i+@]Ѿ?*zjbUtۇt/)%CRʱoU">= Λou0 }:<0a ٿhk>_ +J=LPЄY`G#5v;BX`Gka !#5v;BX`Gka !#5v ]h @cѢ p{Xۓ  kt$]+=h#GI|zCC5@#pjyܡƆ.45@#2\o2w B[##5vTv#3%%֡5c-D?<'ϖ [ )*uz'P7Z@aw ]2>ݣ=JSԞD5MMy__o\9O?34Pc"o(/VVM?7BM`;KnW45KÚCQ$*q2%}cGo6l Lƶ+FIծQu}7iRs[k4z䜷YǨaW]flnk ?OìJŚ^6J_Coqu zNJ}KB5ԖI7-ݖ{fs6թEiZm^ J7/6S/h|XQѳQ+t"*B5(J,Q7|{Vix}e OoT^Wʗ[2^Rw|w2be2_,tC[oFERP-~&%Qz^Mr;^h#Dm8/5N֮.KkJ4bwt`|JZ}87OW3[yۨ(7/ҩ5W;JآIտhz7a"n ՂiZpO:;[^C}%`wOFhXtp$עIٲTo|eOZahz:Q3[_+[c4SM>QP+#GKV)8 YZΰPc!heX$I&tyG8(qAѹ>-GuGCj遯wM-oIz~$_+vT?{_́F*]Ѩ(/=&䬍.m?]/kfTb)rӈ"]9mwXn][jv4W/EҾœ&inΊ/#lt%xGӰIK0y6 ]ҞupMrj7AK|5:5"0)@9>gsoe_6{QX{t+<^2TELL$eSci?1!Cr=asnQHN:wwN:K)odU0cA|z?=by+DӼL\n\/:@7Cs;t ONJ LwZӕmsR^+zeAanWn/4Ao-K[Q2[v{=rW)wmbHiؙ-J/ӗG4bQZm۲uoWXM|s*o3[9tg%3MPiV_ܽ!MD'm+ϩQۻDswכu U/hftgM1>}z^d GƘeXd[sk_!9c+fl^q\x^W$ZQcǙRM>#d/5gu[V_[>nG˗$dKO=?z@s-Tdc4w#z9k핫ڸ43At4%搜K ktט\6݋ 3vHnZĭj+WqYfh֦^Sؖ IeMCTJU|.k~5 NjIzw M;%ݯ]'ir%ZgwĮutǘTKLsUS٥-I/hKn]S dkؤG~CId-F <|:bcW]OW}qJR¤%&myV9 ʹ0[ k:ᥬnS88@כ_P֤{P kRQ2t^`hԡ}kvԹ4|;kͱ6e C|kt?,wH|%{ʷ1eKUހ?HOyrNFT~rt{qgtȿղOO~iMEtUA8/^a^Q%k'=go>V&ujͻf$XT\TRNU]ɰc'ֽmv֘MUf c0u䍌UgGedފHǹ ՘ C=heE(IrJugyvv̵=_sX }Pu1}Zg+W_21-}^3iWTw[gZlSwʴkYFEҩÚ*碩v-Ssdg9qӚo1Ie7OhmV5j$~4l~l=y)m<1I 魤e ?gx6;kj>w<9t)qAߨ2q-C] 1OZ<}Nz7Nww~(`$Y y"Mvv{5NzBzicok<9pi0UvTJJoN [EzVO)ۻD0B^e"SӰ ʚHU}k%*ֈ!9k۲i츨paj"^PDϭϏ/hNUhTEEUVV&Nm2po6Gz%yV>9M5o,7ja7mS *S y\%.\\WF U;[ t3Pƹ`W; םTaH֫.YR]?):MNMU+BY [ziiʱ`zUTcLۭVrUrZiD:>Vϳ>]]tzՆ&ZeY2u2xo P׭:gL_љ-ؠ̛HP[uT?pU&vsXpt7\a6n2^j#GI y4zm!Sw_=K3 k|]w uz'߇R7tKk?'i*אa 쏽5k4aP͑CG=HC>SRnأw̝-?PyaR4a-=hLiBm @;BX`GkFr@CC-[5~c.pkmKW;FQ*//WYY<:my;BX`Gka !#5v;BX`Gka !#5v;Ң p{s屯TZzKL8:TޏʫGCЬėǾ ?yȭKL\TO3ADi Cnt4;BX`Gk { nJ[ʙ:n 6z><{DߐniB~ʡJh 0 :ΐu*iq5I*~FkNjǥ h5ޣ:$?ښJQq0(N2 8=3&$yVhOWUe`{VWo6G멪EVN<,Ҙx9IGykhA7;7wAI677vOשhdmXsmt]W~4i`dX9[().2l3&~v,PPL5 ]*xEMWxy$/Sxr|pfV<ުРœ IgPRZ=$EuLh1rJS#?t-K9 ۥhNKWԽUΥ*(Ԏ|zuFִhmuT=Y%cyU"%+gOj9SZ9a) 1tHYbFެ˦J֎X~9щ3g[uNK5IR%y_ы}tg%W{')nAqVNhJKh_.kh8;ԫPΩTm^kyQGݪY9jDaMrl$>3u8G-1YIwNꜾ&]+%PyKze_:kBLAGzisNצּ(+sGy+1kF)2Ԙqݱ@iڀZY?lUJ:z?쨘Z-O2nerhA5Mپ9ijvv1].U%I=_BtMyuKrDx ҃ô(W~N%n9;v])|<_MY%UhTEEUVV&NmJ*;W,マrp)T˙RuVSG5 +>qN9aM6欱ݕR%czir}rQJe5QAJ G9JJW.c"ɊM8!?(wJ1àho^Vɞ NVp}v>uarI>i r!àf/sI҃zc[djy)Y{5P=yOhOgS/LCJ3St-!WF]Sˈ@rKej1) 1V/P^hUNJiyNы55/|.YTyΕ|ޣp$;(vtTM6)4䯚ol 2tL(o"]c{P.IW{| v Ӱh߷[(gB96p/=>.IW7o1TNdHjLF5P١Oes9*U7Q:ܵ{$uHx+6ha8^.Kp~fs+l2 fLגgcyfY֩ U_EJʈ7(<~U'N3$N廼2f%)kddSLgI:D+Vjf uA2,L/&7QRU>3&wUw%H[" P :*ԙrl9HNW`eDcN*gW0S h Ģ9h" &jhsNv|B+s7#۷`%N3-KKvSevyysscϡt$IjZ]Xsak&Е_85_\4.IۘL|RV2^I)y&=7uVo_YS? Ծ cC<d#e,@MM e`y_JnO!M8EV:UmR<:'cc*zfsW_XzM ?_f9wm-1HI]׾)=\/ ]?vŜ$IZV֜ t ]cn!vAN*5l~&w<џ^M@HI| J#E%} IDATm`[G1lՠKqL c4Tߔ|7M M~P(2?֟Ձ_O)}~y7šRy(=dbߚ(.RS:Ȕ]$k>51:$xyLF/I t#e:}gim>cX2ȪI]fΞkFϡH&0czRy,z&&J \ܟ'6fdϚ*iKش6/`GTԄz4P35*MWpzM`t5Hē>9j=[h c'8 CY~N,~t=n niX;e[cYIjkqcwv,NSJuɽbg)It),\a+BC׈nfeϚ .aIU@&kzɤ*fBjf1sJ԰ e̜S@(3jOW4VJIwԇyma{Ry߶h{~_JJ;߶@$IZֳt]ݒIג$I$Ij95$I$IQİF$I$)H$I$EI$I(bX#:ueZzڐ?S-= I6CK@R5[z(ڈN:2zԈ$IRdYyuh**++ ~@$I$IR4(I$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$IH):Z\yKERѩSGFA$IRbX# Ա\۫gKERS.wa$IU4(xsIWյzZ'I k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)e\!gSH$I$)6$mR'\ӹgM3[͞'x8xlыō<͋?^sS;˪{ܬh6Cܬz l fIԡx%ߠWv_$IR _\[Kkd(2!3 w3edr3Ic=;:,,f'UZwWJz&- >'x=kdlR~dPj9ϓ@3<_sdam<:M,Ӄ 3({6-1'Rx8n?$sgrڝÙoL{d/MHBv\H \þp^WvB ɰo/EǍ`4_/LI|D<^%ISFS3HXΗO0q kr_ލ\:ʙ57'suߍ; ~H$IFNW6 iɱ} ټx+BHaPj@)SI7 wd ~ڏ#NߟN.ޝEōLOks2?wsEŜo]OmN7*yJB &ljRYyqQR[' L|RGQR|3I@d/ ;t}HK,Zz$}߳M 2"5 V2tiCSjOсޣG@iΆޭ E~qa{~5uMϪ#&(uǝ룓.)-a؄[0q5;Q gXqg/ةLhmdmbdlﱳ(RP%Nge$ҾQ+gaUn l$IV+$K&~q6&80g"eu:/.#~/y&=7uVo_YS? Ծ ffd3: t6Xj+B}uƵpw 7'|:.%BQ-ePF^/oTۇ=tC~!76#@?=+Í\,t۷ծDXBvc2ihgE` UuPEQ`4(zqdns|-1cI@!?5YOǽh)i4*aCF?s^$I$]Q_ᰦ8,%U)iJ_60̭Tv@Me ՔYΦ{"Ƥ5h6)n&C)&XV7ԥ]ό?ޤX_M7zF&4 {47*Rаe*kw2?"j AU Ћ3i,},]u)(ݽzeA|Fo$qcr7B5 zWFXqQ1{;bIb` K㉥R(}.5m7NIbNNKbҔ@@v,+ks?a ;y2s؛_!"I$ 5 T~6I&U( sqؘh TϚ2Ӗi!l^̏ @Mp&fץAtV%tBܩ<"79))=@:sҕ5-R/Dōc{9dC߰Ǥ弲~z10 NCGxx/I`M\;4MFYMӰp!;_.$vT=xa\_ag%p%V 3PȻAls.i^w5^TI$IkΊPcg'5⿛1tٳ&Kش0!0)V0v6SRo:A͟NѩS$IRdYyuh**++ ~@0EGK_p?oHj#:uQ#HC$IjS k$I$IӠ$I$Ia$I$IR1$I$I"5$I$IQİF$I$)thH82ێNfWߒv?ҧ]nI_"?PAL]mn>`{Rq TvǤaL$I2$(穯3gl~vejb9Gg(yQ!^t*#[3.>@o|ALJq+.T$I p$IQ PƠQcy mc.pӰsNp$3^\Ӌ|\rSgѹSLg1G$ItUYY#IR+㓱ӭ[WNv|'>Ĝ. >u={+Pɩ^!돕Wveȭw_{AY >/)>n_Ĭݮ J$)a$IQ8Qbv7 jf>z u}?y{"\7Q_w_񥿐$I`q$I^?lϐ./o#=oƄc:=Ow\ˠ1|~tI$]]5$E~uS'kg\p5Cf071mͅ]uо= HkNSc$I k$Iz}k8~S7n9Up3ts73csCVŇ}I}32 9x8!\w$IRHeաjMPuݛ_TB l E|W &nVXk1uDv Nph˰ot3$It5HԚ}_8g`KE$I_ AIԊ]x͠F$+I$I(be$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQ$bXjB$I$I@c55$I$I-AXӮ]; l$I$I&z֘H$I$]m{ִk$I$IdYyuhJ*++*C$I$I!|#kw5ڔF$I$ig ꚔI$I+AXӮ];kl«m-$I$I$I$IrD kB g$I$IKwK$I$eH$I$EI$I(dW$I$I"5~Xcx#I$Ite5kƒGC$I$IheMUU$I$Ij4j1I$I$YW2$I$Ij56$I$I"55$I$IW[&PQS5$I$IW_I$Ig$I$I&{H$I$j3$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE kVLyeKAZ$Iҡ yv=ͦ0t]9ON^lᑨ9 s>kᑨ9:uHsG$.VHJ?ÐVҊDIH@lА$I(dYyuh**++ ~@ۗjϚ 2Hs͋:d/{mdܔOIh9?QnTVffAt7j ~C eV")G/mypI_okxrz.-&7{ݽVң{;ϖ;w+hp<$~S*WڕCGܻܾݷb[4G ;}sYwCv{_[Ľ]KT`c s>kq&{K.B!L)9CH*u8j~Kx@bv߯@(̇oD㑕=>^sEEe+w>y;P01iW*#1oz6;[bGqY\yQQYSON&y;PЉo!X*PM{or-*a"T$PW&ߜs;>`5M[8Ij IDATE5|hWЩo]? pNckryzN>)Ӄ?M`Y4H,*u*k9jtoGYݟ gŮoPSGͨ^y)(l,c8VYMLϣ?%Xp"åwaܩ`<4tMLjI6EU&?Zo{@N NMbvM䗻T\1nWщPa>j̆'lWˀhՒ V϶;gܠ@N -s7T֛]k-\Ł&HJ$r * ~KŎ_q[cQ˔8HUyG 5cL%{B 2n? O 3BϚ96-.O4Yu25'62ոS@{np6,΃c:p7dʞM7KYD֦4es0Vyy%l QspzyQ&;~h3a};m٨44.&gà["9]ss7уWwǧuI$&f$ >mPL58tޏֵp9/.L 5Mg9idx%J̬ۚGH[_S2nа8&q ovV U0x25 5/ړ1β RK3+ktww!l9 V V cH,* #i:9/xu$=741-5Isױ@#YWؽ/_ JB4H9.Q;~-s^nۡ]0 9~[Ug/r+I65!|NŞ7wtb;9B7%J '8%S&z҄*kPUbJՆ2Ӱ L>%(z5{T%DYXǃgY~ځw44rYԬ2+y=϶j@vz@EёX1w=sӏ #M Tİ5@/h *0{)|kNo8MRVve@_SAV/уá|ɼWMaJ$V\ؚt)t&nMǴ"y*?EB YI)y&=7%/QYq?H ~`y0޶5R͝Н1~rΊM<O]>|\FeMJ\> 8YaAN m&W2hD_呹Ø!Y}b&=V[Kʚ£Eqkذ>wM rfhtdt|o'n^ɔtO·f nM]tۃ8wx$IT5:gzClZ8_bxb`>4ߠFQӠBӗNgu!5KN`iCXnUK5Sy''|~L?u7N^~LЉܽ-6*rXFsX.v$NdE {4K<8r'x{çIKA X81ع"xVڨc" %0O)܇^kس]Ns|gW?'qLxS`So }-L~Me$I5/)?yYuO[Bl-} {6-L45Mϓ=p6jn}i§bAfMɜyx_i+]YJѥӠN K%A%L)vГ;+C-XZg`&ڄs5IJ)#;sꃢqak'Y6W2aYK4OLRf'NfB8سN5+KnteNUk#NfDzCy4Kk:iz Aȉa~F:;v 5<4bϚ:8.} C0ŪfzVXz0򖖒$Ijڝ,+mTWWSUUEee%ӒcfZzB֮''/HsH\]vi!H$]ۼH$I$3$I$I"5$I$IQİF$I$)H$I$E8S^f"I}{#IZ#3vt$:۷o!H$]v'ʫCTUUQYYIEE$I$I+k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1Z3-=)UU$I\ٹ4~P3]9ON^lᑴMݻK-= I$ FjΔWoٖ*\ I$E kV [{9ÐZ?w.$IZ7I$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQCKj۳&KxꞄ˺ed2}fxO%lefdlLgeNrNa;^Cb{ϖr,˘H{<xdIv:AxS,* +>E,?xuر_e泂<;7Wၹޛòo5#1onsw-e_2PI$I5ZmXsnB3y/.᭥uw[̭;H[ټxpONk/Mޜul}Y@j:b\/L̋+㥬jӇJx|[px;ŏ^8vEw%ypIO>]UKqi%sݳVʬDlؘ> 8VnϽ˄u@s9]7Y6w cuLH|>047@I o$I/Ն5Ѯy+)]y!]$2i"U@"s/ z}E~$I$55NPQ[*vsh{j^lA;nu,#`fVS7# 7Y gҜP Gj \2CX;:Zk&nM!ޮഥѽ=wCwc~B oN ʚSW3u H8Vpnx0SĆ_1iisH$I51#]OPn܆b9Gǹ?~ob]AΦ@Lv:YSJ|JaLɱRF BpCi+M?V TzϜK:)yiJ MApl9[1h4>85=tSʬDY=,i:vϭs{.gщYabmןc]}|ņ+I$ ' s*I|:_+*MaT LK&\OtSLɱ8>p< #>/l'qVÙb6O 5rS;s]ݘr9a`);g95l+Gd\(i8 T5j{[=i4 0h]V J{Wa:eED#I$YZ]Xsak&Е_85hX\D Sl^4TVJRIҦ\y$vm)O['x(m+aj›< u!|=g6cL:U^|8Z*Rme5 iiخWe«aOdن"̺͝3ZU#I$ZYXs3qƃt9R|HGIivR_AJ5iI7)jk p)ПF rŒOEf^.?57Rt Tt%1=GWi ʚ^8ծ@~4@MC6;~-ıve߰U,?ǺC-`wSӠB .-I$/št:{=^9=7<졔iEѷ9gia.OyHjټxV*9N #xxkfij35^xrz7xcpӇvIJfVDVKOD<,Jv֮iP͚¹wt6' 5&}wJrv̄e$ItI,<{򎓖WԌiPek]O8ҾFR(` aϚUdͦ% =3/M3yHfMɜyx_i+]YJѡGrپӑ^Ƴ6?czvY=⇰vL@ga,ĘOj; jG!4}iWN p 63 <ΣKKCj:v$IԺ/Z֞0V!ưF$I\eաj}Zrl$I$Im?~$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)thHj%/8ŋP/S5A g\M}J+ϰFj%cWϖ@KѣUIt9 Jj%.^ob]۫g3GS_$3$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$]yG~A_p%dѶ^{?ik /B__61I$ICK@ҕtgL'kOxԖFEӇ} -:|:XHYvOL{i'S*:~6`؞YȬ{Ȍ^[~{oo;?A0< Om穌Y) >[¿$IRc kCzn8 99LeRP&מ /4{K /^~aPAmv$lϯ&GV cIC<Ɣ`ǘB!I2$IaX#}?^Y&??ML 6W޹Cxr2Sx)__7nf"rD7gƗKڠ2>v?[.})O"l+)â I$m9GjI*I.a$oMbXfM`?y3;T39<`#? .\B 1wR9 3a&Ζyax &dma?qLCM0\*.#68˖=[G=>VL!ļ<n '$NTx{{,}m2߶2oEwX1lyl ~9`{u+?gcCֿdʂ$1+צЧ=f$]=l]Ċ͙N@Gho[YpTjNCo pxAx0SHֺB<G*I. vNX௿ɔO%CIb熟@P޶}7Fu D 3x21;,^ 4?rx\@Ps.y7wpu6ٳ4%-9ghX=o{g24ooeҥu ҧ޽E}y ԠDI$`R;gK7SUb7TBc5IUX5QIwla . ѴhFԄ ~>-]fs &S0[Ԩ(愈A Bμ3`nŶN DƎ=cf4m9]]C5˲ IDATA5Vq`gMuts50řT/[&wǸP Ydf}K7̴]'0#{RlcdNȸ`z1M!S W3;ي%v[N%F^Lgf2PB9ַ|*f5Z^e"O8vTGE*a0,Y`|,yw>EBE '$d!Yx,dz20`dVh0wu"?712ƜD*:i5%1BȺA|o赌"$'WhPs 0%jmcx~PwN"B1Mr_ KyCaK{j.$ExB"bV9feiݾKMn{5еmZK .S5h:Q=V91fڏ$""""2䳋Wihh:<&"'̿=XhCɋxn֋,gg20ļ˶Wȵ)SYAyHb5wjy(l;Odп;"""Gkֈ8 g)9O,k1/MYM9\ȸ)MU!""""rcֈ8I0Œ%٬"""""%5""`!7A-""""" -0,"""""""@ֈ85""""""""Daąw7Dnʅsq]E?rr󩩩ﮈ0gg'|}N?""""|vJcAcc# SWWDŽ17ێA85""""""""DaQX#"""""""@ֈ85""""""""DaQX#"""""""@ֈ85""""""""DaQX#"""""""@ֈ85""""""""DaQX#"""""""@ֈ85""""""""Da2 RSS]qhNp$nv{("""7NaQpF}WwEDġ]9974ﹽqEDDiQSS{~hwu#bn޸"""rֈ85""""""""DaQX#"""""""@ֈ85""""""""dЇ5a6'ϘߖqPr` W). |e5UpE>-sy>*:GE'),:IOZ^&>_ "Na$dg@ ) duTL0>}uPWDDD݁u?% 燳;.tƌ98y\{z=G~fp9M*K%ͪٶ'"ط<^H(e˜;wŧxdøe;Qe!JVG %Ʒ^VQeo-R9){pi[-Eg ϰ12Crz"қ l.0BQX`6jMd=L=ɀ6؜F|vxn""""}h5|q/ yh6`?-˿TRv`<&SI|9 ,"P&OrA[_R#ًyY0nʡ4L;0oj d DǤ1V$`ģ '>O?RosNO^qqm W2Q>KS l "̞/LC^`匙_A ¾‚?<6F3iT}TɁ5-+Po%oAD<||I^]ǣxrBR[x-֓򁁕mJzv;޶0LO> ]&=!i1y.!8 Z8Ȑ42TLO&=tX*->ygeFW|dG].|JS^Tgu{!ٻyKSZ|BO=4KҤQ>wJܵ'ǻybЇ9~Etzљ N5̿3Q则aOa:vˋ@6p_\ _PS]4r%1múTr^J Sڜ@Ɂ?P6[k80&^@r C83Z2FD1{ ?3 !  *o!ѫ gSw"[P1)kRrW&2H0b#9`8Wc\6nVa FD|]w /.!A]/$bM޳w]FO@;Z?"r 7AHR&9J|x  #I9O5+a-4ZK#4N6dQ{4w<'\,C1,|;v/G1cQ3xL {6gJ)i`v(! .v1ۮW\͔@DF|vtoyE%tmAZ^\%%<>|""7TLns23l_V֏h5S=Ӆ%氹iU"""2p ڒ̗?527>>.[2385kvqCyA~Vsjș&xO-SCu(F]pK`l*)P/u57Wfi:z0$>NI+)&c:gJNQ*)(G@(26]@7}2 9ei.=6"uWDDDn\ q]PĥT@۰Ro{x1/>F+1m1[3!F N48e+>7^ʧ t^S+g2wr*J¾N鷰KBÀa>I2rB(abPV9Go~*v* Δ"1)XWkpw@x¯|Ɋ ƶ޽PDG0M{^(3r@X޶_ ZNV|odAӞf5Smպbf>qYmZ6Z%""" khu3CsR6Gb ^$7eFSs9˓C)Wn`cCς;>mO`0`twT0DmM$"q=bxh cFcw$}ʗsX..?ėo>Wd%sod2c!/.{yF6p!''\h e"G2 ," [>i_ ^樑5"""ې.^il:hllz0?+o[C%0xK l_ykHiLg7)ԧVֈbeb[[[uBxPmN#& u}l$O=N$VѴw6"wDDDfЇ5"-VGm۷aMEv*[mN#vjv\nvқm1htDDD%"ǣ.z9G^cBx5Xr/Zٖ X kDd1C"""""2xur kDDDDDDDDFdpvv""qvv6n޸"""rCNn>55섿Mq=7䳋Wihh:<&Ͼv4 JDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDtϔQpCsvvχ&yp{noCq kD0+""Ϳv{("""7NӠDCHOs=FDDDDDDDā(q kDDDDDDDDjKl9KHwNhwԿ(Nj VR{;~C8QK;Mrn6"x.*{wENc#466hFi]iEa`S_6`čcO3_ aZʬaӮ=,9ֿlKm\+(%%H&iշxHʎc@9We!JVG+jj$/#I1#a?`+f^DNf>5"""̀ kΐznliw;e' ěsYwv:'쇭F 0;~LٔuXl5|N|^%<Eq_y9:!ۖDo 8g/+0b[vgG|zԴU^IcBP\:Kcm-daݑ5Ykr|2 ,1#)6}*^dㅈH@aM%_ C#bRi_ opFț0S O<&#oMy?6? &e79wa=۝V'ؼm͝5r4G}ak8O[e-W/Zԛ̌#}A%Ln p'ᩓ߽|C #diE24maO?Jz 7^wg$BFQeP5:{Va ԟj6X+Ņ//%},Yo ,oBWbP;VT0#f¬Dƾxoy%KVMEĦuN$~W>Wbw*EYE ɼhX2;ŝ=ql+Kp,_62~`oȭQ<7f/e v1θ)§A52cҘg f+F 4MDDd08aMy}wz2*khĐ2ZWt.[RÄ]& }G*e)8 -36{/;;Imusm1$]ycHn.N0N9^O[l6b椓'?xbfc|%ef)ζ9w u$}8ǒqʁ,.mOpB 1FVꎽt=FI u];',z;2/&dz*62f=] HL(,N x5v7YNvip2 Kfް:=Y~Iu"ތcV011;i<8CUϞH6?[wz+Y "'/sLtfӠy(rL밧 [(LlEDDd 8a/_/)RӒՌ6a (98 hlאWk}AKm,XhiQnq 氟F/B|iX㤮5si<1K)!;ùNKZXlީ]TtY]y`.˜|eS,32|8;1i|Y1v-|5E1pf4|0~^WTׇ5X91JdetxeD|םyy5X{~pyA|>ȊpaTRNY5|w}mdsEq(gsFD|]0vR.;7e1yA+' e &M9n)j%>mXO ]DDD k |g=|;hCdӉ1AZu@r84΂[4(ێS[T{i_ ^5""" kzS-cXF7ztÀm-:BXrRw@ о3TӵlLf3c(Ό 0z9 _]C'0'rL-e߰iq.x6z&#{x%ql2 4:93əښ֠zJoiMaqGet/ŋ-_ ͶmS hܔ剩ӻySwRSx-;=q򏏑w4e_½A<Ľ%3 韻_1/O#ھxժ9UO,)vz=úϝ̸{̝nxL.ecbZI{A2ݰ􁞼gE@@DD#kazK<>fAe}gfx!&;*<9H׻4͏|7W1tPak㒫՟?}! h`1G]wĒHwBDDDD(Ĝ55섿Mq=7䳋Wihh:<&Ͼv4 JDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDtϔQpCsvvχ&yp{noCq kD0+""Ϳv{("""7NӠDCHOs=FDDDDDDDā(q kDDDDDDDDjKl9KHwNhwԿ# 2{""""""s ksk};NR1d aakhh5^{x\SP֋f/e 0F1`i-e߰i_6FHmmm;aQ1|REDDDDvkC+9Jٿ2-]MnbW /eOczq؃ڊ"'vNxbdr\+;:KgV49Wr/Z:r;"1m/':Y0|-3V{8ÓojdNHk/w'0GrV?3m4 ǿꊈȀ5`.KrgwZVY*[BK^yJٿrOZ<5!M' 6Fe{ĂI:w@5^bA7Щ-~K^`dKf[t [jG̺hM)~8s.zL]]]wiPjhh'ߊe$Vنk"""2( ذf Vd Raɂɭ3fP#)u1[C3fg3 1Mn1k.? qKx|+&T'6p?~su:#CǷ-dY|'~L1 z˖}ye>-;eiS:H1[(?_eN^ccvYN.ˌs(Q+Z}*ПeVfG$TӳJJ];\J[8A PV ]LMq> ``sQ3mB `bAO%9@$4O%ËtCChT%_|/?CB?-|jⲓLlޝ0w<~>cǂǁѶǭh (cM@1>f=,\72UZ1Һym|Y˖#+&3cj (iۤ |'k2u/s829>&at푘X"C*+bg"4NGcg$mzZzA  g"udNeqhmMu;c™ccVP8Q{l0l 7c4Lded0ZYN<(R>jC{b1'$9!^?a%?w0~{1Ƽh9Hn Fe'>Z?'/0#=lىBle6ٹbƬ@#Kv\҈ƜpBBKV8+Dى a1D5d4ތ#p6~`޼OJ"f'ʬ~+:q!cNhsѵeneOt8>ImqM55ki*G.&2E9ecagUvmi{ &PŸ)BsQXcg :$G&"""f>O?RosNO^qqm W2Q>S;”wucR_l`t2M)3BP;-;bTB@)>dtMgu_oN#sl6b椓'?xbfc|%e` dQ\ n6n^mecɸ Kr8#QK+yu^O̞h#[ᤆ HpdR\qGY[d3;`Bv*g#2ˣr8yT6<r߰gQ8N,,Uyú8f-w_%։\z3Y"(NN? \N$YW1w@>{#n.V{l!ܕSd-N;ݟY&vc>LbSQ/NdWvnyGCcNvma̙4||/YK0^V|E3`&y?Ě2% 2/0y/Wl8*ތ0i}I?7?`0z~O/6>œ‹n |2 (V:{= (5[(O5AG?^}KX;]MЇx}m`M":=L[ibyڄ9%/bG/l?kĶ[^DDDָehL*8-Y͈iΧA0)}Aun̞/sx[%L4;&:5;ٔe~xybs3BzqRך94Z݊kJ+ ܫoJ:kዯ)΀33E$Q">5Ųo8U\&+SU@+l.D|םyy5dovSÊ{< SdED@%Qw٦lM60w\rV11iD` c'sS37r"iYkߟK0@PAU[^x[)>>|snVKGfQqx[FHF%ovƈ(fgW<$٣C4^n .C2p3>ŊvbM 50c0;ާxy$skؚ[Š8n..\Nx%7ø)M5erCćHdI*ȡ~^AoqEDD Zqʷq&çぎӇi*9R~fudecfuã,LF^cR eL'flӫw+E@_ȯ*[tAMYy%B/O\ztBz_^jNY.܂Iv^]]oQxws[ޓ b!Q9ө,~NV&nm5x- S:R#lѡf$@Rͯ+*a+n){,nSZk/<Q\L\N\%%<恤k|PY)^fȻ`|GO6'3eũlQ3#I.jQ0KQXbN26šjK2_}2wHl-̳xZ,M;>]_f[CZ( 0s.x+W+9$!\մuCnQť*TTTW xCs`s=mnIcNNgT\Y0ˡ &t]a ^u\nqO{p3F0K3fU5]5}ʑj+4od&{*M?;%M9DLd{Ylȹu[g-0 p*pZ#Fq]U͋>Gm Ye$//l99(;[i+PIYW;E_1׎{L7qgKV0RIfv5͸;r;w"kf2PB9j|)jP Yd+[_ 3/mA5Vq`gM5c/2ѝMP`0s;10߶n=k9t`f7o^,Օ/eXȜhvdEO Δ_}BYg?__Dʮf KP`MEŧm?#;95M{›TW1o 7K.w'<&Wq4ld#nufbhtv1 pJmnG ֑8ecl;-e=$Oa obGDiT{ly~J~pSk?ϋfd`$ӆAKU wUy{3r߯tn(9lur(?>/'Qj y+~KI Ԙ]ȏlu g;Vvݐ|6>ո@(B%cنNf++Y3`>a`OZQl~6y6짍s(Lo}&PzmFֈ nC>x頱멫cB_{dqkiP֬iX0+7`lk܄Qf?lkXe6錙xx6w/e̥n.wb >6:&jRwebV~)=%gn {i}Ʈ=2q;C aС89  iTax;ރn;B2yx5 G=/'HN#z굝:4oI&6öS[=am臘Zn%w7{:D-#iʾ{"x.*{!i@/1/O#zje{5\eС j W !CZ+CfV}v< |g#CUI?|ļ7$Nei<@}2?VmG k3AI ף_Ѱ1[r8/׬F5#0dziR2p'wСKG9mN|_ͅK_Xǿ?v?5y$G1!Q7m&.#PBո=ˮ'fP#"""2)ĜiZ!y/K/휟`ὁ; FDzMw EDDDDDDD75""""""""Da{- b.~-=5kD?rr󩩩ﮈ84gg'|n=7ܸ!]tHCC1aLMDDDDDDD䶣iP""""""""DaQX#"""""""@ֈ85""""{ui~0]+pkZRN4d$Rfat3+&Ql$côx֠EHd;g2'2. 59+C~5.TOF|=N=ufs8y# kDDDDDDDD?FDDDDDDDď(# kDDDDDDDD?FDDDDDDDď(# kDDDDDDDD?FDDDDDDDď(# kDDDDDDDD?rL "ϵq4RDDd $rR8lK(di ]3]L>٨FDDdhRP#"r ])""2Cֈ5""""""""~DaQX#"""""""Gֈ5"7}Xt""""""r1L#`e=Ekn,em78ۢm<s6rY,[ʄ=fcȳwz&ww",{)mBp=o먈/v=EI׹0IšI%>3* Ô58f'ah_ѩ[ }*.ã䵴<fggWԵtxc)ĚL5-feM4m7N',@x'/pz5f*YNU֒kTDDDDdrfuX3r&Nڇ>f~a6 Y+޳Y qb E?\–V?eY9ɜIwFK5`91S]GS8p{~>;ZgR\M;jY?CW(ڛCB04M'J:Ow ݋WSLsa>B}F:nP!GV SH1 ևݘf:t3~3pŦSҽt1Tc IDAT"Ԉag&ˀ RVڜ&֦}@V&Xڴ wd7ys~@ h (\aNB3y3i5\ PCj.![(:8b5fheCVp5q,?)dwr|XFn6옂g9|rl?K!Re"Nv;ıŗ4-yl~Yl9OPWGz5gHx"˱)Gt ol!(:BA1%`n$8F6/hsv`. ytr\Ӽ79ȩ&BS9?CD-ĖPsSj$Dn! "vS[.#rv$i:Lhp?\L}[ ] tu仾FaYp.7d%"F2~PNS oauwNzlG FہcU cZ>:qa[u'6t .:ylg;!2/)}; _2^DKlXObS7ؿ c.""""rfVb!4=NV YZvYy̳rܗj .eV:9kMpچNZy&>ib=gXr [< .:' o|hH/DZp/,@Hj\S$Nǜ).q3rg1 9.%, e1Z#u9;4RC|]fDZȫɅx10=b q˜1=T_I dṗe esu1 wPTj04RQNz" 1`vg1w~XBSs&b_~IDu}vb,2ߞ22 ɴ]!GX&Rߠhy| KkvuxyH_%*q[/K0eFy_盵<#DDDDDn[&9g'mZB+y.>8ŗ2/׋@5ǵFcĽVma˲ҲMdY06XGn,%E `㨓1F&|c3f I_@{b6DEΓuhi ˠP/vL'J0c_G0d 9 rj20#Oo&`,Ĕ횅}?ikqĬf{&!f5毮!8y4p628/Cé;P=:`bdX蛯=9v7NM>Ï`GOIlLl (8ԟb}#_;H1o 7kvGkU!{3;zh' l!FXMY1zd% Ĕ%z֣y$: ! 'M̟I>wt@4D|]ߠƈ@-*^|9XWM8,5^55A{<"kX|7˚ZhLX9'&S~:!a3ނ DDDDDYִ4<;Y {XgeFdrx~)%`&l#-^@X#nLU2w{!wNIJsh3Aq,OjYN)ܥIuM C&:o6dP課S{&q$ ز M\gn9I2s͋:0$5\VLMcMxb8R0#"j}tम! f5 t;餽%C\y܄c&b)3NEjv$N.,91?oc _ .-4EsqlY^]FFDDDDd6HOEMbmZyNײ,֦BJ'b Ota haGc4©訥]Y!w?`LC;{Ȥg2+)I%aKLiqc>} JRq–)5'Y&!̓,Ɵg2_XkT44|~k+HcP琝ϫ!bTŽ}hob9S;`3E:}oYc `[)lwYd?dᘟeFAL)Vr9`61¸2H6"0.eP˗!b$2n '=[yv`¾<{%jc[o.!`.|-po.|׽K1dzZR!e G)?!@Ǥcrz2珟}yիWʕ+=ߠo>ÚQyϻ7 v:?,*hb2<BųƿN~8ߘ{̸Ac{h< ޾|ԥU.QEr2ZN\I;FYcj,Ya1M*sx#{6mmEdzrt˅5"2=f/)!cwYgº|/(SX#2fgˠDDnT"2DDd.~9wt"""$m0,"7D;L"""Sp'˗t"""9~σ~^J__W\!ЙMDDDDDDD䖣eP""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""GDdb~ML""$rR8l}ꟹc EDDdֈ$N6&>.5ӥ}N񚂆[3zLA$z{t~iu3bn1"""2u kDDDDDDDD?FDDDDDDDď(# kDDDDDDDD?rLpc9y-q-'`&+%LњEy+)Zh(emi}`6[%KC0o, "PALK$I'o5HI+d'@ϙ5MdU`.Hnd:Ͷr2?gp*'Ca0-m pVDL\Sck*F1ZFi5r}>u8WZ糳xv/ %մk=~59$CtrtG9;нx5%{4 -tߗov o{y$leq`=4S%]BDm1[Ρ*Pl'AIض N͋Z]! yo: .iޛNmTE`vrzd ZbKC9MR\yق)5mS dTyw9;\4&C4k?.&>WݭUd_Eks]PѰ,8T ILZDYL#?(70&=^ϱR1-}8v簭zzm:x6}ї\ȿ//B\H'v)fy_{V1'3ͧe`ӲC؞8L9޳ *'(9yfȌRX3f~89a\ulOhH:ՙ;"^kj \a)exa 6;`YyaqTfxql%*2MnK,e'44_a8[i%9C9z4'R֠F^M.7F6Nj=q\p &5T_I dṗe esu1 wPTj04RQNz" }8~G7;  7595H&NSDSv4cy*ǒ7=FgXZc:xyq :_C۱c;l]F:ذ'c[zDe~6{u ܡ(|g=c(7j|,gΚu5&pz?aḂx}/-G̉5(z;1VuVm89ά)kn7g,,d"Z:1c v }܌tLC *r "3HY 0TvtޯaHW3#!CR3 $/=1`h"SIBC\4Qg4wCTd# %@@9@8-kf8KY(3uD&I]P< ퟞ'_YtόϗS| O <3X:9c ֪Br4gvNٖCF]ҵc7J0b)#vSeKh%D,a ''xW6`x>}7u`>ZIj0VLMcMxb8R0#"j}tम! f5 t;餽%C\y܄c&b)3NEjv$N91oc _ .O-4EsqlY^]FFwPi6dA/-F Wg93r\>t76N֦e =:]&s1fN$̾+O\T` P넿1cx*; 1ő`~{wFvR&!~2y As];jN}H@G- ˰0I|^9x3ڝSON#4Dѷe~6IF:9ZNMN?ec ba!=؎u=LFGs ;,z82;Czb|LwӼ7}''ϲp"(#929-6P?3s/M);fLY[wbd9Y8V9hca멻0[&~E6eydMkF @L- |Έ5L$`3E:}oYc `[)lwYd?dᘟeFAL)Vr9`61¸2H6"0.eP˗!b$؎v=LhΠElݾ~Gp].Pͪ#kI.H6(/9!zs&IKMZf?/_g3PuPOݷ2˽h&s:ipZfֈns\z>\BݓLKRs6rm (^4&c1^xl޵ ٬ܗjhuiܪl^&[yWeOvdby(^GE䡛KIv+z'-f |fވ}qlJy}9iT*ϭ[FL̬kDf\Nr2EM, 9~SBSuKYY3z x}A2 kn6eP"PuplGtnE6U+o6;̛kȱ)دFDDDDnM kDDįv3Ǝ;"""""%"""""""Gֈ5""""""""~DaM" N.~L!".~9w^SgC:m0,r\zg#RDDZ@D._zM}ꟹc EDDd/=z*}}}\rCg6[A5""""""""~DakIDATQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~gX3g`5JkDDDDDDDDfĈfΜ9(vcYFDDDDDDDd޳f Ns>^u= p4vs j҈Pwz6)fֈH#š9s?0{MֈPdaRIENDB`pengcheng.mind-test/docs/feature_cards/000077500000000000000000000000001471400134000204505ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/Chat_Web.md000066400000000000000000000074161471400134000224560ustar00rootroot00000000000000# Chat Web ## 基本介绍 Chat Web提供了一个网页界面，让用户可以通过类似线上聊天的方式使用MindFormers大语言模型（LLM）推理能力。 ## [Chat Web支持模型](../model_support_list.md#chat-web支持列表) ## 使用方式 ### 单卡启动 示例命令如下，将会启动一个网页app，使用glm_6b模型进行回答生成。 ```bash python run_chat_web.py --device_target Ascend \ --device_id 0 \ --model glm_6b \ --tokenizer glm_6b \ --checkpoint_path /path/to/glm_6b.ckpt \ --seq_length 512 \ --use_past False \ --host 0.0.0.0 \ --port 7860 ``` 参数定义如下： * **device_target** (str) - 表示要运行的目标设备，取值为`Ascend/CPU`。如果所安装的MindSpore版本不支持`Ascend`，将运行在CPU上。默认值：`Ascend`。 * **device_id** (int) - 表示要运行的卡，取值为`[0,8)`。该值只在`device_target==Ascend`时生效。默认值：`0`。 * **model** (str) - 表示用来生成回答的模型的名称，取值见上节。 * **tokenizer** (str) - 表示用来编解码输入输出的分词器的名称，取值见上节。 * **checkpoint_path** (str) - 表示加载的模型权重文件路径，此参数缺省时将会下载模型默认的权重文件，默认值：`None`。 * **seq_length** (int) - 表示模型中的序列长度。默认值：`512`。 * **use_past** (bool) - 表示是否使用增量推理。默认值：`False`。 * **host** (str) - 表示web服务运行的host ip。设置为`0.0.0.0`可远程访问。默认值：`127.0.0.1`。 * **port** (int) - 表示web服务运行的端口。此参数缺省时端口将会设置为`7860`，如果该端口被占用将会向后顺延，直至找到可用端口。默认值：`None`。 ### 多卡启动 暂不支持 ### 网页使用 当`--port`参数缺省时网页app将会默认运行在`7860`端口，如果默认端口被占用将会向后顺延，直至找到可用端口。 通过浏览器访问Chat Web网页地址 > 如果服务启动在本地，即配置了`--host`为`127.0.0.1`，则访问`http://127.0.0.1:7860`或`http://localhost:7860` > > 如果服务启动在远程，即配置了`--host`为`0.0.0.0`，假设服务器IP地址为`12.23.34.45`，则访问`http://12.23.34.45:7860` 在输入框中输入文字，点击***提交***按钮，稍等片刻后屏幕上将会显示LLM的回答。点击***清除***按钮可以清空聊天记录。 #### 配置项 聊天界面右侧提供了若干配置项，可以在点击提交按钮前自行进行配置，每次输入将会实时生效。目前提供的配置项如下： * **sampling** (开关) - 打开表示使用采样；关闭则表示使用贪心解码。打开后可以调整下述**top k**和**top p**。 * **top k** (滑块) - 从前k个可能性最大的候选词中采样。取值范围：`[0,10]`。 * **top p** (滑块) - 从可能性加起来为p的候选词中采样。取值范围：`[0,1]`。 * **temperature** (输入框) - 用来调节候选词的可能性得分。取值范围：`(0,∞)`。 * **repetition penalty** (输入框) - 重复惩罚因子。`1.0`表示无惩罚。取值范围：`(0,∞)`。 * **max length** (输入框) - 输入与回答的最大长度，不能超过模型的`seq_length`（注意：多轮对话时，输入将包括前几轮对话）。取值范围：`(输入长度,seq_length)`。 * **prompt** (输入框) - 提示词模板，与输入拼接后传进模型。输入框下方提供了一些样例模板，用户也可以输入自定义的模板，需要包含占位符`{}`，代表替换输入的位置。 > 每次启动时，配置项将会设置为模型默认的状态。pengcheng.mind-test/docs/feature_cards/Eval_While_Training.md000066400000000000000000000052651471400134000246540ustar00rootroot00000000000000### 边训练边评估 大模型的训练效果需要评测任务来作为衡量标准，而当前大模型的训练耗时长，等到训练整体结束后再进行评测任务的时间与算力成本过高 本功能提供了一套在训练过程中进行评估的流程方法，以动态观察模型的评估结果，具有以下特性： 1. 能够在训练过程中执行评估流程，并在日志中打印相关的评估结果信息； 2. 具有功能开关，根据开关状态决定是否启用边训练边评估功能； 3. 具备调用间隔控制项，根据该控制项决定调用边训练边评估功能的时间间隔； 4. 支持多种模型，只要模型能够调用Model.eval()完成评估，都可以复用该项功能；无法通过Model.eval()完成评估的模型则需要额外编码适配，暂不支持； #### [边训练边评估支持度表](../model_support_list.md#边训练边评估支持度表) > 说明：边训练边评估功能需模型已支持评估，并且该评估指标可以通过Model.eval()完成 #### 使用用例 - run_mindformer启用边训练边评估 描述：通过run_mindformers脚本参数，启用边训练边评估功能 测试方式：`--do_eval`开关启用边训练边评估功能，`--eval_dataset_dir`指定评估数据集 ```shell python run_mindformer.py \ --config configs/gpt2/run_gpt2.yaml \ --run_mode train \ --train_dataset_dir /your_path/wikitext-2-train-mindrecord \ --eval_dataset_dir /your_path/wikitext-2-eval-mindrecord \ --do_eval True ``` - trainer启用边训练边评估 描述：通过Trainer高阶接口入参，启用边训练边评估功能 测试方式：执行以下python脚本，其中数据集路径替换为实际路径 ```python def test_trainer_do_eval(): from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task='text_generation', model='gpt2', train_dataset="/your_path/wikitext-2-train-mindrecord", eval_dataset="/your_path/wikitext-2-eval-mindrecord") # 开启训练，并打开do_eval开关 trainer.train(do_eval=True) if __name__ == "__main__": test_trainer_do_eval() ``` - 配置评估间隔时间 描述：更改评估间隔时间，以控制执行评估的频率 测试方式：更改配置项，将 `configs/gpt2/run_gpt2.yaml` 文件中的 `eval_epoch_interval` 项修改为其他数值 执行`run_mindformer.py`启用边训练边评估中的启动脚本 ```yaml do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. # 修改此项eval_epoch_interval数值： eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. ``` pengcheng.mind-test/docs/feature_cards/Inference.md000066400000000000000000000337231471400134000227000ustar00rootroot00000000000000# Inference ## 基本介绍 MindFormers定位打造训练->微调->部署的端到端大模型工具套件，为了更好性能地部署已经微调训练好的大模型；我们利用MindSpore打造的推理引擎[MindSpore_lite](https://www.mindspore.cn/lite)，为用户提供了开箱即用的推理部署方案，为用户提供端到端的大模型解决方案，帮助用户使能大模型业务。 使用MindSpore Lite + Ascend硬件进行推理时，一般都会基于预训练或者微调后的权重进行推理部署。当前使用MindSpore Lite进行推理部署时，主要三个步骤： 1）进行权重合并 2）进行权重转换 3）最后部署推理服务 ![推理流程](assets/inference/inference_process.jpg) ## 设计概述 Inference采用的后端是MindSpore Lite，为了兼容已有的MindSpore的前向预测的流水线接口，对外部开发者呈现统一的推理接口，保持调用推理的一致性。Inference模块按照任务类别设计推理流水线，支持任务类别的可扩展，针对同一类任务兼容不同模型进行推理，实现模型可扩展的设计。`注：当前Inference模块仅支持LLM模型的生成式任务。` 具体设计如下： ![inference_framework](assets/inference/inference_framework.JPG) ## 接口设计 ### pipeline pipeline是Inference模块对外提供的推理API入口 ```python pipeline( task: str = None, model: Optional[Union[str, BaseModel, Model, Tuple[str, str]]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, backend: str = "ms", **kwargs) 功能 获取推理流水线 参数 - task(str):任务类型 - model(Optional[Union[str, BaseModel, Model, Tuple[str, str]]]):模型参数,在backend="mslite"时，是启动inference的推理能力 - config(PetConfig):微调算法的配置,包含微调算法的超参或者需要实例化的预训练模型 - tokenizer(BaseTokenizer):文本输入序预处理器 - image_processor(BaseImageProcessor):图像预处理 - audio_processor(BaseAudioProcessor):音频预处理 - backend(str):推理后端,支持"ms"和"mslite"两个后端 返回值 返回推理流水线 ``` ### InferTask InferTask是Inference模块是支持开发者在Inference框架中拓展推理任务类型的工厂类，主要提供get_infer_task()、check_task_valid()和support_list()三个接口；支持用户查询已经支持的推理任务类型和获取对应的推理任务实例进行推理。 ```python InferTask.get_infer_task(cls, task_type: str, task_config: InferConfig, **kwargs) 功能 获取推理任务实例 参数 - task_type(str):任务类型 - task_config(InferConfig):推理任务配置参数 返回值 返回推理任务 ``` ```python InferTask.check_task_valid(cls, task_type: str) 功能 获取所有支持的推理任务列表 参数 - task_type(str):待校验的任务类型。 返回值 返回推理模块是否支持该任务类型(bool) ``` ``` python InferTask.support_list(cls) 功能 获取所有支持的推理任务列表 返回值 返回推理任务列表 ``` ### InferConfig InferConfig是用户调用MindSpore Lite进行推理时，需要配置的参数项来初始化推理任务，具体如下： ```python class InferConfig(BaseConfig): """ Inference config class. """ def __init__(self, prefill_model_path: str = "", increment_model_path: str = "", model_type: str = "mindir", model_name: str = "common", infer_seq_length: int = 1024, target: str = "Ascend", device_id: int = 0, rank_id: int = 0, ge_config_path: str = "", 参数 prefill_model_path:全量图路径 increment_model_path:增量图路径 model_type:推理模型类型 model_name:模型名称 infer_seq_length:推理序列长度 target:推理硬件类型 device_id:设备ID rank_id:设备逻辑Rank ID ge_config_path:Ascend图引擎配置文件路径 ``` ## 快速使用 ### 模型导出（增量推理为例） #### 方式1：针对Configs下已支持模型的导出方式（以bloom_560m为例） step1. 修改配置文件，在配置文件中新增infer配置项，在run_bloom_560m.yaml中添加如下配置 ```yaml infer: prefill_model_path: "/path/bloom/bloom_560m_prefill.mindir" increment_model_path: "/path/bloom/bloom_560m_inc.mindir" infer_seq_length: 512 model_type: mindir # 参数说明：参考InferConfig的说明 # 由于配置了prefill_model_path和increment_model_path两个路径，因此需要导出增量图，因此需要在模型配置中打开增量开关，如下所示 # 如果想用bloom的后处理加速，则需要配置is_sample_acceleration开关，如下所示 model： model_config： use_past: True is_sample_acceleration: True # 后处理加速开关，部分模型不支持，当前bloom和glm支持 ``` step2.执行export.py，完成模型转换 ```shell python mindformers/tools/export.py --config_path ../../configs/bloom/run_bloom_560m.yaml ``` #### 方式2：针对research或者用户自定义模型的导出方式 step1. 准备好模型相关的配置文件、权重文件放置在一个文件夹 ```bash test_model_dir ├── test.yaml # test模型的配置文件 └── test.ckpt # test模型的权重文件 ``` step2. 在test.yaml中添加infer相关的配置项，参考方式1中的step1 step3. 执行export.py，完成模型转换 ```shell python mindformers/tools/export.py --model_dir /path/test_model_dir ``` ### 推理体验（以bloom为例） 推理模块提供了一键启动脚本，帮助用户快速上手使用Inference模块进行模型推理。 step1. 利用执行`模型导出`章节，得到MindIR图，如果是全量模型，则会得到两个MindIR图（bloom_560m_prefill.mindir和bloom_560m_inc.mindir）。 step2. 执行run_infer_main.py脚本，调起推理： ```shell python run_infer_main.py --device_id 0 --model_name bloom --prefill_model_path /path/bloom/bloom_560m_prefill.mindir --increment_model_path /path/bloom/bloom_560m_inc.mindir --config_path /path/ge_config.ini --is_sample_acceleration True ``` ```text run_infer_main支持的参数详细说明如下： device_id: 设备物理ID model_name: 模型名称，当前支持bloom，glm，llama，glm2 seq_length: 推理序列长度 prefill_model_path: 全量图路径 increment_model_path: 增量图路径 config_path: GE配置文件路径 is_sample_acceleration: 后处理加速开关 add_special_tokens: 对输入token化时是否添加特殊字符 stream: 是否采用流式结果返回 ``` ### 推理APIs调用说明（增量推理为例） Inference模块是由pipeline() API来对外进行调用，目前支持四种方式进行调用： - `模型名称进行调用，（以bloom_560m为例）` step1. 参考模型导出章节完成MindIR模型的导出 step2. 在configs/bloom/run_bloom_560m.yaml中配置infer配置项 ```yaml infer: prefill_model_path: "/path/bloom/bloom_560m_prefill.mindir" increment_model_path: "/path/bloom/bloom_560m_inc.mindir" infer_seq_length: 512 model_type: mindir ge_config_path: "/path/ge_config.ini" # 参数说明：参考InferConfig的说明 ``` step3. 调用inference API完成模型推理调用 ```python from mindformers.models import BloomTokenizer from mindformers.pipeline import pipeline # 定义bloom 560m模型的tokenizer model_name = "bloom_560m" tokenizer = BloomTokenizer.from_pretrained(model_name) ge_config_path = "/path/ge_config_path" # 定义推理流水线 lite_pipeline = pipeline( task="text_generation", model=model_name, tokenizer=tokenizer, backend="mslite", ge_config_path=ge_config_path, device_id=0, infer_seq_length=512, ) # 推理 output = lite_pipeline("I love Beijing,because", is_sample_acceleration=True,add_special_tokens=False) print(output) ``` - `模型路径进行调用` ```python from mindformers.models import BloomTokenizer from mindformers.pipeline import pipeline prefill_model_path = "/path/prefill_model_path" increment_model_path = "/path/increment_model_path" ge_config_path = "/path/ge_config_path" # 定义tokenizer tokenizer = BloomTokenizer.from_pretrained(model_name) # 定义推理流水线 lite_pipeline = pipeline( task="text_generation", model=(prefill_model_path, increment_model_path), tokenizer=tokenizer, backend="mslite", model_name="bloom_560m", ge_config_path=ge_config_path, device_id=0, infer_seq_length=512, ) # 推理 output = lite_pipeline("I love Beijing,because", is_sample_acceleration=True,add_special_tokens=False) print(output) ``` - `模型文件夹进行调用` 针对于灵活切换模型的部署，我们支持从指定文件夹目录读取相关配置启动推理。首先需要按照指定格式准备模型文件夹，具体如下： ```bash test_model_dir ├── test.yaml # test模型的配置文件,只能存在一个yaml文件 ├── xxx_prefill_graph.mindir # test模型的全量图,此处需要以prefill_graph.mindir为后缀名 ├── xxx_inc_graph.mindir # test模型的增量图,此处需要以inc_graph.mindir为后缀名,当前只能存在不多于两个mindir文件 └── config.ini # ge的配置参数,文件名不可变 ``` ```python from mindformers.models import BloomTokenizer from mindformers.pipeline import pipeline test_model_dir = "/path/test_model_dir" ge_config_path = "/path/ge_config_path" # 定义tokenizer tokenizer = BloomTokenizer.from_pretrained("bloom_560m") lite_pipeline = pipeline( task="text_generation", model=test_model_dir, tokenizer=tokenizer, backend="mslite", model_name="bloom_560m", ge_config_path=ge_config_path, device_id=0, infer_seq_length=512, ) # 推理 output = lite_pipeline("I love Beijing,because", is_sample_acceleration=True,add_special_tokens=False) print(output) ``` - `自定义配置config的方式` ```python from mindformers.models import BloomTokenizer from mindformers.pipeline import pipeline from mindformers.infer.infer_config import InferConfig prefill_model_path = "/path/prefill_model_path" increment_model_path = "/path/increment_model_path" ge_config_path = "/path/ge_config_path" lite_config = InferConfig( prefill_model_path=prefill_model_path, increment_model_path=increment_model_path, model_type="mindir", model_name="bloom_560m", ge_config_path=ge_config_path, device_id=0, infer_seq_length=512, ) tokenizer = BloomTokenizer.from_pretrained("bloom_560m") lite_pipeline = InferTask.get_infer_task("text_generation", lite_config, tokenizer=tokenizer) # 推理 output = lite_pipeline("I love Beijing,because", is_sample_acceleration=True,add_special_tokens=False) print(output) ``` ### 如何配置GE图引擎配置 #### 针对910A，推荐的GE配置参数如下(config.ini)： ```ini [ascend_context] plugin_custom_ops=All provider=ge [ge_session_options] ge.externalWeight=1 ge.exec.atomicCleanPolicy=1 ge.event=notify ge.exec.staticMemoryPolicy=2 ge.exec.precision_mode=must_keep_origin_dtype ``` #### 针对910B，推荐的GE配置参数如下(config.ini)： ```ini [ascend_context] plugin_custom_ops=All provider=ge [ge_session_options] ge.exec.precision_mode=must_keep_origin_dtype [ge_graph_options] ge.exec.formatMode=1 ``` 针对于GE参数的详细说明可以参考[GE options参数说明](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/70RC1alpha002/infacldevg/graphdevg/atlasgeapi_07_0118.html) ## 自定义开发（text generation任务为例） ### 如何支持新增LLM模型的MINDIR导出 在export.py中，已经完成了增量推理模型的导出流程，由于不同的LLM模型可能存在不同的输入，因此，为了统一的导出流程，我们支持用户自定义输入接口，添加到已有的流程中去，从而完成对新模型的适配。 例如以bloom为例： step1. 在export.py中首先实现了`get_bloom_inc_model_input()`接口 step2. 在export.py的增量输入表`PREFILL_MODEL_INPUT_MAP`中，添加bloom的输入映射： ```python PREFILL_MODEL_INPUT_MAP = { "bloom": get_llm_common_prefill_model_input, } ``` 完成上述两个步骤后，则完成了bloom模型在export工具中的添加。 ### 如何支持新增LLM模型的推理 由于导出时，确定了LLM模型在MindIR图中的输入顺序以及shape信息，而由于模型差异，因此无法按照统一的输入处理，为实现不同模型的统一处理，则需要将模型输入接口类似export时一样添加到任务中。 例如以bloom为例： step1. 在`text_generator_infer.py`中，实现bloom模型的输入类定义，如下，我们定义一个通用输入处理类： ```python class CommonInputsOfInfer(BaseInputsOfInfer): """ common infer inputs of llm models. """ def get_inputs(self, model: Model, input_ids=None, current_index=None, valid_length=None, init_reset=None, is_first_iteration=True, **kwargs): if not is_first_iteration: inputs_tmp = [] for i in range(len(current_index)): current_index_tmp = int(current_index[i]) - i * input_ids.shape[1] # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp:current_index_tmp + 1]) input_ids = np.array(inputs_tmp, dtype=np.int32) inputs = [input_ids, current_index, init_reset, valid_length] lite_inputs = self.get_lite_tensor_list(inputs, model) return lite_inputs ``` step2. 在`text_generator_infer.py`的输入工厂类`InputOfInfer`中的`MAPPING`表中添加输入类映射关系： ```python class InputOfInfer: """ Input of llm model. """ MAPPING = { "bloom": CommonInputsOfInfer, } ```pengcheng.mind-test/docs/feature_cards/Performance_Tuning.md000066400000000000000000000306661471400134000245720ustar00rootroot00000000000000# 性能调优 ## [Profiler](https://mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.Profiler.html?&highlight=profiler)数据采集 MindSpore中提供了profiler接口，可以对神经网络的性能进行采集。目前支持AICORE算子、AICPU算子、HostCPU算子、内存、设备通信、集群等数据的分析。 mindformers将此功能集成到了套件内部，用户仅需修改配置文件中相应部分即可开启采集。 在配置文件中添加Profile相关参数： ```yaml # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True ``` - **profile_start_step** - 表示开始收集算子性能数据的step。默认值：1。 - **profile_stop_step** - 表示结束收集算子性能数据的step。默认值：10。 - **output_path** - 表示输出数据的路径。默认值："./output/profile/rank_{id}/profiler/"。 - **init_start_profile** - 该参数控制是否在Profiler初始化的时候开启数据采集。默认值：True。 - **profile_communication** - (仅限Ascend) 表示是否在多设备训练中收集通信性能数据。当值为True时，收集这些数据。在单台设备训练中，该参数的设置无效。默认值：False。 - **profile_memory** - (仅限Ascend) 表示是否收集Tensor内存数据。当值为True时，收集这些数据。默认值：False。 修改配置文件后，[启动训练](../start/Quick_Tour.md#方式一使用已有脚本启动)，在根目录下会生成相应的profiler文件夹存放各rank的profiler文件。 ```index ├─... ├─output │ ├─checkpoint │ ├─profiler │ │ ├─rank0 │ │ ├─... │ │ └─rank7 ...... ``` ## [Summary](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.SummaryCollector.html)数据采集 > ### 使用约束 > > 使用SummaryMonitor时，如果使用Trainer高阶接口启动任务，则**必须**将代码放置到 `if __name__ == “__main__”` 中运行。 在配置文件的callbacks中添加SummaryMonitor和相关参数： ```yaml callbacks: * type: SummaryMonitor summary_dir: None collect_freq: 10 collect_specified_data: None keep_default_action: True custom_lineage_data: None collect_tensor_freq: None max_file_size: None export_options: None ``` 参数 - **summary_dir** (str) - 收集的数据将存储到此目录。如果目录不存在，将自动创建。默认值："./output/summary/rank_{id}/"。 - **collect_freq** (int) - 设置数据收集的频率，频率应大于零，单位为 step 。如果设置了频率，将在(current steps % freq)=0时收集数据，并且将总是收集第一个step。需要注意的是，如果使用数据下沉模式，单位将变成 epoch 。不建议过于频繁地收集数据，因为这可能会影响性能。默认值：10。 - **collect_specified_data** (Union[None, dict]) - 对收集的数据进行自定义操作。您可以使用字典自定义需要收集的数据类型。例如，您可以设置{‘collect_metric’:False}不去收集metrics。支持控制的数据如下。默认值：None，收集所有数据。 - **collect_metric** (bool) - 表示是否收集训练metrics，目前只收集loss。把第一个输出视为loss，并且算出其平均数。默认值：True。 - **collect_graph** (bool) - 表示是否收集计算图。目前只收集训练计算图。默认值：True。 - **collect_train_lineage** (bool) - 表示是否收集训练阶段的lineage数据，该字段-将显示在MindInsight的 [lineage页面](https://www.mindspore.cn/mindinsight/docs/zh-CN/r2.0/lineage_and_scalars_comparison.html) 上。默认值：True。 - **collect_eval_lineage** (bool) - 表示是否收集评估阶段的lineage数据，该字段将显示在MindInsight的lineage页面上。默认值：True。 - **collect_input_data** (bool) - 表示是否为每次训练收集数据集。目前仅支持图像数据。如果数据集中有多列数据，则第一列应为图像数据。默认值：True。 - **collect_dataset_graph** (bool) - 表示是否收集训练阶段的数据集图。默认值：True。 - **histogram_regular** (Union[str, None]) - 收集参数分布页面的权重和偏置，并在MindInsight中展示。此字段允许正则表达式控制要收集的参数。不建议一次收集太多参数，因为这会影响性能。注：如果收集的参数太多并且内存不足，训练将会失败。默认值：None，表示只收集网络的前五个超参。 - **collect_landscape** (Union[dict, None]) - 表示是否收集创建loss地形图所需要的参数。如果设置None，则不收集任何参数。默认收集所有参数并且将会保存在 {summary_dir}/ckpt_dir/train_metadata.json 文件中。 - **landscape_size** (int) - 指定生成loss地形图的图像分辨率。例如：如果设置为128，则loss地形图的分辨率是128*128。注意：计算loss地形图的时间随着分辨率的增大而增加。默认值：40。可选值：3-256。 - **unit** (str) - 指定训练过程中保存checkpoint时，下方参数 intervals 以何种形式收集模型权重。例如：将 intervals 设置为[[1, 2, 3, 4]]，如果 unit 设置为step，则收集模型权重的频率单位为step，将保存1-4个step的模型权重，而 unit 设置为epoch，则将保存1-4个epoch的模型权重。默认值：step。可选值：epoch/step。 - **create_landscape** (dict) - 选择创建哪种类型的loss地形图，分为训练过程loss地形图（train）和训练结果loss地形图（result）。默认值：{“train”: True, “result”: True}。可选值：True/False。 - **num_samples** (int) - 创建loss地形图所使用的数据集的大小。例如：在图像数据集中，您可以设置 num_samples 是128，这意味着将有128张图片被用来创建loss地形图。注意：num_samples 越大，计算loss地形图时间越长。默认值：128。 - **intervals** (List[List[int]]) - 指定loss地形图的区间。例如：如果用户想要创建两张训练过程的loss地形图，分别为1-5epoch和6-10epoch，则用户可以设置[[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]。注意：每个区间至少包含3个epoch。 - **keep_default_action** (bool) - 此字段影响 collect_specified_data 字段的收集行为。True：表示设置指定数据后，其他数据按默认设置收集。False：表示设置指定数据后，只收集指定数据，不收集其他数据。默认值：True。 - **custom_lineage_data** (Union[dict, None]) - 允许您自定义数据并将数据显示在MindInsight的lineage页面上。在自定义数据中，key支持str类型，value支持str、int和float类型。默认值：None，表示不存在自定义数据。 - **collect_tensor_freq** (Optional[int]) - 语义与 collect_freq 的相同，但仅控制TensorSummary。由于TensorSummary数据太大，无法与其他summary数据进行比较，因此此参数用于降低收集量。默认情况下，收集TensorSummary数据的最大step数量为20，但不会超过收集其他summary数据的step数量。例如，给定 collect_freq=10 ，当总step数量为600时，TensorSummary将收集20个step，而收集其他summary数据时会收集61个step。但当总step数量为20时，TensorSummary和其他summary将收集3个step。另外请注意，在并行模式下，会平均分配总的step数量，这会影响TensorSummary收集的step的数量。默认值：None，表示要遵循上述规则。 - **max_file_size** (Optional[int]) - 可写入磁盘的每个文件的最大大小（以字节为单位）。例如，如果不大于4GB，则设置 max_file_size=4*1024**3 。默认值：None，表示无限制。 - **export_options** (Union[None, dict]) - 表示对导出的数据执行自定义操作。注：导出的文件的大小不受max_file_size 的限制。您可以使用字典自定义导出的数据。例如，您可以设置{‘tensor_format’:’npy’}将tensor导出为 npy 文件。支持控制的数据如下所示。默认值：None，表示不导出数据。 - **tensor_format** (Union[str, None]) - 自定义导出的tensor的格式。支持[“npy”, None]。默认值：None，表示不导出tensor。 - **npy** - 将tensor导出为NPY文件。 训练完成后，summary日志文件将会输出至指定目录，目录结构如下： ```tree └─summary └─rank_0 events.out.events.summary.1596869898.hostname_MS events.out.events.summary.1596869898.hostname_lineage ``` ## [MindInsight](https://mindspore.cn/mindinsight/docs/zh-CN/r2.0/index.html)可视化调优 通过MindInsight可以完成训练可视、性能调优、精度调优等任务。 训练可视功能主要包括训练看板、模型溯源、数据溯源等功能，训练看板中又包括标量、参数分布图、计算图、数据图、数据抽样、张量等子功能。 训练完成数据收集后，启动MindInsight，即可可视化收集到的数据。启动MindInsight时， 需要通过 `--summary-base-dir` 参数指定summary日志文件目录。MindInsight启动的默认端口为`8080`，当默认端口被占用时，可以通过`--port`参数指定端口。 通过命令行启动mindinsight **可视化训练结果** ```shell # 开启 summary-base-dir指定到mindformers生成的output/summary mindinsight start --port 8000 --summary-base-dir xx/mindformers/output/summary ``` **可视化profiler性能数据** ```shell # 开启 summary-base-dir指定到mindformers生成的output/profiler mindinsight start --port 8000 --summary-base-dir xx/mindformers/output/profiler ``` 启动成功后，通过浏览器访问 `http://127.0.0.1:8080` 地址，即可查看可视化页面。 如果要远程访问物理机上启动的可视化页面时，需要修改MindInsight的Host地址。首先通过`pip show mindinsight`获取MindInsight安装位置，打开安装目录下的`./conf/constants.py` ，修改其中的`Host`字段为`0.0.0.0`。 **关闭mindinsight** ```shell #关闭 mindinsight stop --port 8000 ``` ## 保存IR图(中间编译图) 中间编译图又称为中间表示（IR），是程序编译过程中介于源语言和目标语言之间的程序表示，以方便编译器进行程序分析和优化，因此IR的设计需要考虑从源语言到目标语言的转换难度，同时考虑程序分析和优化的易用性和性能。 MindIR是一种基于图表示的函数式IR，其最核心的目的是服务于自动微分变换。自动微分采用的是基于函数式编程框架的变换方法，因此IR采用了接近于ANF函数式的语义。此外，借鉴Sea of Nodes和Thorin的优秀设计，采用了一种基于显性依赖图的表示方式。关于ANF-IR的具体介绍，可以参考[MindSpore IR文法定义](https://www.mindspore.cn/docs/zh-CN/r2.0/design/mindir.html#文法定义)。 MindSpore提供了保存中间编译图的功能，在使用MindFormers编写模型时，在图模式`set_context(mode=GRAPH_MODE)`下，可以通过设置MindSpore上下文来打开存图功能。 在MindSpore 1.10版本中，支持使用`set_context(save_graphs=True)`来打开存图功能。运行时会输出一些图编译过程中生成的一些中间文件。 在MindSpore 2.0版本中，若配置中设置了`set_context(save_graphs=1)`，运行时会输出一些图编译过程中生成的一些中间文件。当需要分析更多后端流程相关的ir文件时，可以设置`set_context(save_graphs=True)` 或`set_context(save_graphs=2)`。当需要更多进阶的信息比如可视化计算图，或者更多详细前端ir图时，可以设置`set_context(save_graphs=3)`。 可以通过`set_context(save_graphs_path='{path}')`来指定图保存路径。 当前主要有三种格式的IR文件： - ir后缀结尾的IR文件：一种比较直观易懂的以文本格式描述模型结构的文件，可以直接用文本编辑软件查看。 - 通过设置环境变量`export MS_DEV_SAVE_GRAPTHS_SORT_MODE=1`可以生成的ir后缀结尾的IR文件：格式跟默认的ir文件基本相同，但是生成的异序ir文件的图的生成与打印顺序和默认ir文件不同。 - dot后缀结尾的IR文件：描述了不同节点间的拓扑关系，可以用[graphviz](http://graphviz.org/)将此文件作为输入生成图片，方便用户直观地查看模型结构。对于算子比较多的模型，推荐使用可视化组件[MindInsight](https://www.mindspore.cn/mindinsight/docs/zh-CN/r2.0/dashboard.html#计算图可视化)对计算图进行可视化。 pengcheng.mind-test/docs/feature_cards/Pet_Tuners.md000066400000000000000000000052121471400134000230620ustar00rootroot00000000000000# 低参微调 MindPet（Pet：Parameter-Efficient Tuning）是属于Mindspore领域的微调算法套件。随着计算算力不断增加，大模型无限的潜力也被挖掘出来。但随之在应用和训练上带来了巨大的花销，导致商业落地困难。因此，出现一种新的参数高效（parameter-efficient）算法，与标准的全参数微调相比，这些算法仅需要微调小部分参数，可以大大降低计算和存储成本，同时可媲美全参微调的性能。 ## [微调支持列表](../model_support_list.md#微调支持列表) ## 使用示例 1. 修改任务模型，以GPT2为例，主要有以下步骤： - 继承GPT2LMHeadModel - 定义替换lora算法的替换规则`self.pet.pet_config.reg_rules = r'.*dense*|.*linear*|.*mapping*|.*projection*'` - 调用lora适配器接口修改预训练模型 - 导入预训练权重以及冻结预训练模型权重 - 为了使用MindFormer的训练流程，将GPT2WithLora微调模型注册到MindFormer中 具体代码如下： ```python @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2WithLora(GPT2LMHeadModel): def __init__(self, config: GPT2Config = None, pet = None, **kwargs): super().__init__(config) # get Pet tuning model. self.pet = pet self.pet.pet_config.reg_rules = r'.*dense*|.*linear*|.*mapping*|.*projection*' self.backbone = LoraAdapter.get_pet_model(self.backbone, self.pet.pet_config) self.load_checkpoint(config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, self.pet.pet_type) ``` 2. 修改训练任务参数，使用MindFormer的trainer进行模型训练，主要修改模型配置yaml，以GPT2为例： ```yaml model: model_config: type: GPT2Config ... arch: # type: GPT2LMHeadModel # 替换为适配微调算法的模型 type: GPT2WithLora pet: pet_type: lora pet_config: # configurition of lora in_channels: 768 out_channels: 768 lora_rank: 8 lora_alpha: 16 lora_dropout: 0.1 ``` 只需要将arch参数由原来的`class GPT2LMHeadModel`替换为现在适配微调算法的`class GPT2WithLora`，并按照lora算法进行微调参数配置。 3. 使用MindFormer的Trainer进行模型训练： ```python from mindformers.trainer.trainer import Trainer gpt2_trainer = Trainer( task='text_generation', model='gpt2', pet_method='lora', train_dataset="/data/wikitext-2/train", ) gpt2_trainer.finetune() ``` 至此，完成了一个微调算法适配过程，最后执行上述步骤3中的代码即可拉起微调算法的训练流程。 pengcheng.mind-test/docs/feature_cards/Precision_Optimization.md000066400000000000000000000124521471400134000254770ustar00rootroot00000000000000# 精度调优 ## [Dump使用方法](https://mindspore.cn/tutorials/experts/zh-CN/r2.0/debug/dump.html) 异步/同步 dump的操作步骤基本一样，区别在于json文件不同。 **异步dump(训练结束后dump数据)** 1. 创建json配置文件 JSON文件的名称和位置可以自定义设置。 ```json { "common_dump_settings": { "dump_mode": 0, "path": "/absolute_path", "net_name": "ResNet50", "iteration": "0|5-8|100-120", "saved_data": "tensor", "input_output": 0, "kernels": ["Default/Conv-op12"], "support_device": [0,1,2,3,4,5,6,7], "op_debug_mode": 0, "file_format": "npy" } } ``` - `dump_mode`：设置成0，表示Dump出该网络中的所有算子数据；设置成1，表示Dump`"kernels"`里面指定的算子数据或算子类型数据；设置成2，表示Dump脚本中通过`set_dump`指定的算子数据，`set_dump`的使用详见[mindspore.set_dump](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_dump.html) 。开启溢出检测时，此字段的设置失效，Dump只会保存溢出节点的数据。 - `path`：Dump保存数据的绝对路径。 - `net_name`：自定义的网络名称，例如：”ResNet50”。 - `iteration`：指定需要Dump的迭代。类型为str，用“|”分离要保存的不同区间的step的数据。如”0|5-8|100-120”表示Dump第1个，第6个到第9个， 第101个到第121个step的数据。指定“all”，表示Dump所有迭代的数据。PyNative模式开启溢出检测时，必须设置为”all”。 - `saved_data`: 指定Dump的数据。类型为str，取值成”tensor”，表示Dump出完整张量数据；取值成”statistic”，表示只Dump张量的统计信息；取值”full”代表两种都要。异步Dump统计信息只有在`file_format`设置为`npy`时可以成功，若在`file_format`设置为`bin`时选”statistic”或”full”便会错误退出。默认取值为”tensor”。 - `input_output`：设置成0，表示Dump出算子的输入和算子的输出；设置成1，表示Dump出算子的输入；设置成2，表示Dump出算子的输出。 - `kernels`：该项可以配置两种格式： 1. 算子的名称列表。开启IR保存开关`set_context(save_graphs=2)`并执行用例，从生成的IR文件`trace_code_graph_{graph_id}`中获取算子名称。详细说明可以参照教程：[如何保存IR](https://www.mindspore.cn/tutorials/experts/zh-CN/r2.0/debug/mindir.html#如何保存ir)。 需要注意的是，是否设置`set_context(save_graphs=2)`可能会导致同一个算子的id不同，所以在Dump指定算子时要在获取算子名称之后保持这一项设置不变。或者也可以在Dump保存的`ms_output_trace_code_graph_{graph_id}.ir`文件中获取算子名称，参考同步Dump数据对象目录。 2. 还可以指定算子类型。当字符串中不带算子scope信息和算子id信息时，后台则认为其为算子类型，例如：”conv”。算子类型的匹配规则为：当发现算子名中包含算子类型字符串时，则认为匹配成功（不区分大小写），例如：”conv” 可以匹配算子 “Conv2D-op1234”、”Conv3D-op1221”。 - `support_device`：支持的设备，默认设置成0到7即可；在分布式训练场景下，需要dump个别设备上的数据，可以只在`support_device`中指定需要Dump的设备Id。 - `op_debug_mode`：该属性用于算子溢出调试，设置成0，表示不开启溢出；设置成1，表示开启AiCore溢出检测；设置成2，表示开启Atomic溢出检测；设置成3，表示开启全部溢出检测功能。在Dump数据的时候请设置成0，若设置成其他值，则只会Dump溢出算子的数据。 - `file_format`: dump数据的文件类型，只支持`npy`和`bin`两种取值。设置成`npy`，则dump出的算子张量数据将为host侧格式的npy文件；设置成`bin`，则dump出的数据将为device侧格式的protobuf文件，需要借助转换工具进行处理，详细步骤请参考异步Dump数据分析样例。默认取值为`bin`。 2. 设置Dump环境变量 ```shell export MINDSPORE_DUMP_CONFIG=${Absolute path of data_dump.json} ``` 如果Dump配置文件没有设置`path`字段或者设置为空字符串，还需要配置环境变量`MS_DIAGNOSTIC_DATA_PATH`。 ```shell export MS_DIAGNOSTIC_DATA_PATH=${yyy} ``` 则“$MS_DIAGNOSTIC_DATA_PATH/debug_dump”就会被当做`path`的值。若Dump配置文件中设置了`path`字段，则仍以该字段的实际取值为准。 - 在网络脚本执行前，设置好环境变量；网络脚本执行过程中设置将会不生效。 - 在分布式场景下，Dump环境变量需要在调用`mindspore.communication.init`之前配置。 3. [启动网络训练脚本](../start/Quick_Tour.md#方式一使用已有脚本启动) 可以在训练脚本中设置`set_context(reserve_class_name_in_scope=False)`，避免Dump文件名称过长导致Dump数据文件生成失败。 4. 解析Dump数据文件 可以使用MindSpore Insight的离线调试器来分析。离线调试器的使用方法详见[使用离线调试器](https://www.mindspore.cn/mindinsight/docs/zh-CN/r2.0/debugger_offline.html) 。 **同步dump(训练的同时进行数据dump)** 参考[mindspore官网dump说明](https://mindspore.cn/tutorials/experts/zh-CN/r2.0/debug/dump.html) pengcheng.mind-test/docs/feature_cards/Resume_Training.md000066400000000000000000000051201471400134000240630ustar00rootroot00000000000000# 断点续训 ## 介绍 在长时间的训练当中，如果遇到意外情况导致训练中断，可以使用断点续训的方式恢复之前的状态继续训练。 恢复训练时，从之前训练的checkpoint文件或文件夹中，加载网络权重，并恢复训练时的epoch、step、loss_scale_value等信息。 > 局限：由于数据集暂不支持跳过已训练的数据，当前仅支持epoch级的断点续训，即仅支持恢复到某个epoch开始的状态。 ## 使用 ### 脚本启动场景 在run_xxx.yaml中配置load_checkpoint为checkpoint文件或文件夹路径，并将resume_training改为True。 ```yaml load_checkpoint: checkpoint_file_or_dir_path resume_training: True ``` > 注意：如果load_checkpoint配置为文件路径，则认为是完整权重，如果配置为文件夹，则认为是分布式权重。下同。 ### Trainer高阶接口启动场景 - 方式1：在Trainer.train()或Trainer.finetune()中配置 在Trainer.train()或Trainer.finetune()中，配置train_checkpoint或finetune_checkpoint参数为checkpoint文件或文件夹路径，并将resume_training参数设置为True。 ```python from mindformers import Trainer cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16', # 已支持的模型名 train_dataset="/data/imageNet-1k/train", # 传入标准的训练数据集路径，默认支持ImageNet数据集格式 eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式 cls_trainer.train(train_checkpoint="", resume_training=True) # 启动训练 cls_trainer.finetune(finetune_checkpoint="", resume_training=True) # 启动微调 ``` - 方式2：在TrainingArguments中配置 在TrainingArguments中配置resume_from_checkpoint为checkpoint文件或文件夹路径，并将resume_training参数设置为True。 ```python from mindformers import Trainer, TrainingArguments training_args = TrainingArguments(resume_from_checkpoint="", resume_training=True) cls_trainer = Trainer(task='image_classification', # 已支持的任务名 args=training_args, model='vit_base_p16', # 已支持的模型名 train_dataset="/data/imageNet-1k/train", # 传入标准的训练数据集路径，默认支持ImageNet数据集格式 eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式 cls_trainer.train() # 启动训练 cls_trainer.finetune() # 启动微调 ``` pengcheng.mind-test/docs/feature_cards/Text_Generator.md000066400000000000000000000107351471400134000237320ustar00rootroot00000000000000# Text Generator Mindformers大模型套件提供了text generator方法，旨在让用户能够便捷地使用生成类语言模型进行文本生成任务，包括但不限于解答问题、填充不完整文本或翻译源语言到目标语言等。 当前该方法支持Minformers大模型套件中6个生成类语言模型 ## [Text Generator支持度表](../model_support_list.md#text-generator支持度表) ## 增量推理 Mindformers大模型套件的`text generator`方法支持增量推理逻辑，该逻辑旨在加快用户在调用`text generator`方法进行文本生成时的文本生成速度。 在此提供使用高阶接口进行各模型增量推理的**测试样例脚本**： ```python # mindspore设置图模式和环境 import mindspore; mindspore.set_context(mode=0, device_id=0) from mindformers import AutoConfig, AutoModel, AutoTokenizer # 按需设置模型类型名，高阶接口将根据类型名实例化相应模型 model_type = "glm_6b" # 按需设置测试的输入文本 input_text = "中国的首都是哪个城市？" # 获取模型默认配置项并按需修改 config = AutoConfig.from_pretrained(model_type) # use_past设置为True时为增量推理，反之为自回归推理 config.use_past = True # 修改batch_size和模型seq_length config.batch_size = 1; config.seq_length=512 # 根据配置项实例化模型 model = AutoModel.from_config(config) # 实例化tokenizer tokenizer = AutoTokenizer.from_pretrained(model_type) # 对输入进行tokenizer编码 input_ids = tokenizer(input_text)["input_ids"] # 调用model.generate接口执行增量推理 output = model.generate(input_ids, max_length=128, do_sample=False) # 解码并打印输出 print(tokenizer.decode(output)) ``` > 注： > > 1. 首次调用generate时需要进行mindspore图编译，耗时较长；在统计在线推理的文本生成速度时，可以多次重复调用并排除首次调用的执行时间 > 2. 使用增量推理(use_past=True)时的生成速度预期快于自回归推理(use_past=False) ## Batch推理 `text generator`方法也支持同时对多个输入样本进行batch推理；在单batch推理算力不足的情况下，多batch推理能够提升推理时的吞吐率 以下给出测试batch推理能力的**标准测试脚本**，仅上述增量推理测试脚本仅有少数区别 ```python import mindspore;mindspore.set_context(mode=0, device_id=0) from mindformers import AutoConfig, AutoModel, AutoTokenizer model_type = "glm_6b" # 多batch输入文本 input_text = [ "中国的首都是哪个城市？", "你好", "请介绍一下华为", "I love Beijing, because" ] # 是否使用增量推理 use_past = True # 预设模型seq_length seq_len = 512 config = AutoConfig.from_pretrained(model_type) # 将batch size修改为输入的样本数 config.batch_size = len(input_text) config.use_past = use_past config.seq_length = seq_len model = AutoModel.from_config(config) tokenizer = AutoTokenizer.from_pretrained(model_type) # 对输入batch进行pad input_ids = tokenizer(input_text, max_length=config.seq_length, padding="max_length")["input_ids"] output = model.generate(input_ids, max_length=128, do_sample=False) print(tokenizer.decode(output)) ``` > 注： > batch推理的推理吞吐率提升表现与设备计算负荷相关；在seq_len较短并开启增量推理的情况下，计算负荷较小，使用batch推理通常会获得较好的提升 ## 流式推理 Mindformers大模型套件提供Streamer类，旨在用户在调用text generator方法进行文本生成时能够实时看到生成的每一个词，而不必等待所有结果均生成结束。 实例化streamer并向text generator方法传入该实例： ```python from mindformers import AutoModel, AutoTokenizer, TextStreamer tok = AutoTokenizer.from_pretrained("gpt2") model = AutoModel.from_pretrained("gpt2") inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) streamer = TextStreamer(tok) _ = model.generate(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) # 'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,' ``` 上述脚本不再对输出文本进行统一解码打印，而是每生成一个中间结果就由streamer实时打印 ## 分布式推理 **说明:** 由于MindSpore版本问题，分布式推理仅支持MindSpore 2.0及以上版本，且暂不支持流水并行推理模式。 [分布式推理参考用例](../model_cards/bloom.md#a1-模型并行推理以1机8卡推理bloom_71b为例) pengcheng.mind-test/docs/feature_cards/Transform_Ckpt.md000066400000000000000000000472441471400134000237410ustar00rootroot00000000000000# 权重离线切分转换 该特性适用于：1. 权重过大，单卡无法加载；2. 权重为切分权重且与目标网络和运行卡数不匹配； 此时可利用本特性进行权重的切分和转换，以适配目标网络运行； 使用场景：1. 分布式恢复训练（恢复时卡数或者并行策略发生改变）；2. 评估/推理场景（权重需要进一步切分或者合并） ## 方案1：源码执行 - step1（默认已有待切分权重相对应的策略文件，若没有，也可参考以下方法生成） 在config中配置`only_save_strategy: True`，正常启动分布式训练/评估/推理，生成目标卡数的分布式策略文件后将会退出。生成的分布式策略文件保存在`output/strategy`目录下。 ```yaml only_save_strategy: True ``` - step2 运行如下脚本完成权重切分转换 ```shell python mindformers/tools/transform_ckpt.py --src_ckpt_strategy SRC_CKPT_STRATEGY --dst_ckpt_strategy DST_CKPT_STRATEGY --src_ckpt_dir SRC_CKPT_DIR --dst_ckpt_dir DST_CKPT_DIR ``` 参数说明: `src_ckpt_strategy`：待转权重的分布式策略文件路径。 若为None,表示待转权重为完整权重; 若为切分策略文件,表示原始的权重对应的策略文件; 若为文件夹,表示需要合并文件夹内策略文件(仅在流水并行生成的策略文件时需要),合并后的策略文件保存在`SRC_CKPT_STRATEGY/merged_ckpt_strategy.ckpt`路径下; `dst_ckpt_strategy`：目标权重的分布式策略文件路径。即step1中生成的分布式策略文件路径。 若为None,表示将待转权重合并为完整权重; 若为切分策略文件,表示目标卡数对应的策略文件 若为文件夹,表示需要合并文件夹内策略文件(仅在流水并行生成的策略文件时需要),合并后的策略文件保存在`DST_CKPT_STRATEGY/merged_ckpt_strategy.ckpt`路径下; `src_ckpt_dir`: 待转权重路径，须按照`SRC_CKPT_DIR/rank_{i}/checkpoint_{i}.ckpt`存放，比如单一权重存放格式为`SRC_CKPT_DIR/rank_0/checkpoint_0.ckpt`。 `dst_ckpt_dir`：目标权重保存路径，为自定义空文件夹路径，转换后模型以`DST_CKPT_DIR/rank_{i}/xxx.ckpt`存放。 - step3 将`config`的配置文件中`load_checkpoint`关键字指定为转换的目标权重保存路径，若转换后仍为切分权重，传入转换后的权重文件夹路径即可；若转换后为完整权重，传入权重文件路径即可正常启动训练。 ```yaml load_checkpoint: "{转换后权重文件夹/文件路径}" ``` ## 方案2：高阶API 执行 参考Trainer API使用 使用`TrainingArguments`类打开`only_save_strategy`字段，其余步骤可参考**方案1** # 权重自动切分转换 ## 概述 目前分布式训练/推理，当权重与分布式策略不匹配时，往往涉及到离线转ckpt，大致需要以下3个步骤： 1、获取分布式权重策略文件； 2、离线转换，将权重转换为分布式训练/推理所需权重； 3、启动分布式训练/推理。 以上流程对外部客户来说易用性较差，因此设计**分布式权重自动转换方案**，在分布式训练/推理时能够无痛转换，提升分布式训练/推理任务启动效率。 以下为原分布式训练/推理流程：模型创建后，只能加载对应分布式策略的分布式权重，再进行训练和推理。因此用户需要按照**权重离线切分转换**手动转换权重。 ![img](assets/Transform_Ckpt/original_process.png) 以下为改进后的分布式推理/训练流程：中间插入**权重转换**，方便用户无需手动转换权重。 ![img](assets/Transform_Ckpt/current_process.png) **权重自动切分转换**会在`output`文件夹下生成两个结果文件夹，分别是**strategy**和 **transformed_checkpoint**： - strategy：保存**分布式策略文件**，主要有以下两种文件： ① **ckpt_strategy_rank_x.ckpt**：rank x的分布式策略文件； ② **merged_ckpt_strategy.ckpt**: 所有rank的分布式策略文件合并后的分布式策略文件；（开启流水线并行时才会合并） - transformed_checkpoint：保存**转换后的权重**，按照`transformed_checkpoint/rank_x/checkpoint_x.ckpt`格式保存权重文件： **注：**每次转换结束后需要**将strategy和transformed_checkpoint保存到自定义文件夹**，因为每次拉起新的任务，如果开启了权重自动转换，会将这两个文件夹清空，然后保存最新任务的转换结果。 ## 适用场景 ### 1. 完整权重转为分布式权重 - 适用：**使用完整权重进行分布式训练/推理** ① 配置`load_checkpoint`参数为**完整权重文件夹路径**，权重需要按照`{path}/rank_0/xxx.ckpt`格式存放； ② 配置`auto_trans_ckpt`参数为**True**； ③ 正常拉起分布式训练/推理； ### 2.分布式权重转为分布式权重 - 适用：**修改分布式策略后训练/推理**，如：16卡训练，4卡推理；16卡预训练，8卡微调等场景。 ① 配置`load_checkpoint`参数为**分布式权重文件夹路径**，权重需要按照`{path}/rank_x/xxx.ckpt`格式存放； ② 配置`src_strategy_path_or_dir`参数为**分布式策略文件路径**，如果有**merged_ckpt_strategy.ckpt**，则填写之，如果没有，则填写任一 **ckpt_strategy_rank_x.ckpt**的路径； ② 配置`auto_trans_ckpt`参数为**True**； ③ 正常拉起分布式训练/推理； ### 3.分布式权重转为完整权重 - 适用：**分布式训练结束，使用单卡推理**，如：16卡训练，单卡推理 ① 配置`load_checkpoint`参数为**分布式权重文件夹路径**，权重需要按照`{path}/rank_x/xxx.ckpt`格式存放；（注意：rank_k文件夹下可能保存了多个ckpt文件，来自不同step，每个rank_k文件夹仅保留一个相同step的ckpt文件） ② 配置`src_strategy_path_or_dir`参数为**分布式策略文件路径**，如果有**merged_ckpt_strategy.ckpt**，则填写之，如果没有，则填写任一 **ckpt_strategy_rank_x.ckpt**的路径； ② 配置`auto_trans_ckpt`参数为**True**； ③ 正常拉起推理； #### **注**：分布式推理的并行策略目前仅支持"数据并行"和"模型并行"，暂时不支持"流水线并行"。 ## 自动转换案例 ### 一、Llama-7b模型的分布式训练案例(物理机) - **完整权重自动切分为8卡分布式权重** **案例描述**：基于一份完整的llama-7B预训练权重，使用8卡进行分布式训练。 完整权重如下 ![llama7b_autotrans_1to8_train_ckpt](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_ckpt.png) **步骤**： ① 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b/" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 8卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 2 model_parallel: 1 pipeline_stage: 4 optimizer_shard: True micro_batch_num: 4 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ② 启动训练 ```shell # step1：生成rank_table，rank_table_8.json为自行重命名的文件，原文件为hccl_xxx.json python mindformers/tools/hccl_tools.py --device_num [0,8] mv hccl*.json rank_table_8.json # step2：拉起训练 cd script ./run_distribute.sh ../rank_table_8.json ../configs/llama/run_llama_7b.yaml [0,8] train ``` ③ 查看训练日志 ![llama7b_autotrans_1to8_train_log](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_log.png) ④ 查看转换生成的文件 **分布式策略文件**：保存在`output/strategy`文件夹下，由于开启了**流水线并行**，会对所有`ckpt_strategy_rank_x.ckpt`进行合并，得到`merged_ckpt_strategy.ckpt`。若不开启流水线并行，则不会合并。 ![llama7b_autotrans_1to8_train_strategy](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama7b_autotrans_1to8_train_distribute_ckpt](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 - **8卡分布式权重自动切分为4卡分布式权重** **案例描述**：基于8卡的分布式权重，转换到4卡进行分布式训练。 **步骤**： ① 将**完整权重自动切分为8卡分布式权重案例**得到的**分布式策略文件**和**转换权重**保存到**自定义文件夹**下，本案例中保存到`llama_7b_transformed`文件夹下 ![llama7b_autotrans_1to8_train_output](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_output.png) 注：有`merged_ckpt_strategy.ckpt`的话，策略文件仅保存`merged_ckpt_strategy.ckpt`即可。若没有，保存任一`merged_ckpt_strategy.ckpt`即可。 ② 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/" # 配置分布式策略文件路径 src_strategy_path_or_dir: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/merged_ckpt_strategy.ckpt" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 4卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 1 model_parallel: 2 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 2 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ③ 启动训练 ```shell # step1：生成rank_table，rank_table_4.json为自行重命名的文件，原文件为hccl_xxx.json python mindformers/tools/hccl_tools.py --device_num [0,4] mv hccl*.json rank_table_4.json # step2：拉起训练 cd script ./run_distribute.sh ../rank_table_4.json ../configs/llama/run_llama_7b.yaml [0,4] train ``` ④ 查看训练日志 ![llama7b_autotrans_8to4_train_log](assets/Transform_Ckpt/llama7b_autotrans_8to4_train_log.png) ⑤ 查看转换生成的文件 **分布式策略文件**：保存在`output/strategy`文件夹下，由于开启了**流水线并行**，会对所有`ckpt_strategy_rank_x.ckpt`进行合并，得到`merged_ckpt_strategy.ckpt`。若不开启流水线并行，则不会合并。 ![llama7b_autotrans_8to4_train_strategy](assets/Transform_Ckpt/llama7b_autotrans_8to4_train_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama7b_autotrans_8to4_train_distribute_ckpt](assets/Transform_Ckpt/llama7b_autotrans_8to4_train_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 ### 二、Llama-7b单卡/分布式推理案例(物理机) - **8卡分布式权重自动合并为完整权重** **案例描述**：基于8卡的分布式权重，合并为完整权重进行单卡推理。 **步骤**： ① 将**完整权重自动切分为8卡分布式权重案例**得到的**分布式策略文件**和**转换权重**保存到**自定义文件夹**下，本案例中保存到`llama_7b_transformed`文件夹下 ![llama7b_autotrans_1to8_train_output](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_output.png) 注：有`merged_ckpt_strategy.ckpt`的话，策略文件仅保存`merged_ckpt_strategy.ckpt`即可。若没有，保存任一`merged_ckpt_strategy.ckpt`即可。 ② 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/" # 配置分布式策略文件路径 src_strategy_path_or_dir: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/merged_ckpt_strategy.ckpt" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 设置use_paralle为False use_parallel: False # 设置run_mode为predict run_mode: 'predict' # 打开增量推理 use_past: True ``` ③ 启动推理 ```shell python run_mindformer.py --config configs/llama/run_llama_7b.yaml --predict_data "I love beijing, because" ``` ④ 查看推理日志 ![llama7b_autotrans_8to1_predict_log1](assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_log1.png) ![llama7b_autotrans_8to1_predict_log2](assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_log2.png) ⑤ 查看合并后的权重 **完整权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama7b_autotrans_8to1_predict_ckpt](assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_ckpt.png) 注：**transformed_checkpoint**请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 - **8卡分布式权重自动切分为2卡分布式权重** **案例描述**：基于8卡的分布式权重，自动切分为2卡分布式权重进行分布式推理。 **步骤**： ① 将**完整权重自动切分为8卡分布式权重案例**得到的**分布式策略文件**和**转换权重**保存到**自定义文件夹**下，本案例中保存到`llama_7b_transformed`文件夹下 ![llama7b_autotrans_1to8_train_output](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_output.png) 注：有`merged_ckpt_strategy.ckpt`的话，策略文件仅保存`merged_ckpt_strategy.ckpt`即可。若没有，保存任一`merged_ckpt_strategy.ckpt`即可。 ② 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/" # 配置分布式策略文件路径 src_strategy_path_or_dir: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/merged_ckpt_strategy.ckpt" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 设置use_paralle为False use_parallel: False # 设置run_mode为predict run_mode: 'predict' # 打开增量推理 use_past: True # 2卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 1 model_parallel: 2 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ③ 启动推理 ```shell # step1：生成rank_table，rank_table_2.json为自行重命名的文件，原文件为hccl_xxx.json python mindformers/tools/hccl_tools.py --device_num [0,2] mv hccl*.json rank_table_2.json # step2：拉起训练 cd script ./run_distribute.sh rank_table_2.json configs/llama/run_llama_7b.yaml [0,2] predict "I love beijing, because" ``` ④ 查看推理日志 ![llama7b_autotrans_8to2_predict_log1](assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_log1.png) ![llama7b_autotrans_8to2_predict_log2](assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_log2.png) ⑤ 查看转换生成的文件 **分布式策略文件**：保存在`output/strategy`文件夹下 ![llama7b_autotrans_8to2_predict_strategy](assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama7b_autotrans_8to2_predict_distribute_ckpt](assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 - **完整权重自动切分为2卡分布式权重** **案例描述**：基于ziya-llama-13B完整权重，自动切分为2卡分布式权重进行分布式推理。 完整权重如下 ![llama7b_autotrans_1to2_predict_ckpt](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_ckpt.png) **步骤**： ① 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/ziya/" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 设置use_paralle为False use_parallel: False # 设置run_mode为predict run_mode: 'predict' # 打开增量推理 use_past: True # 2卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 1 model_parallel: 2 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ② 启动推理 ```shell # step1：生成rank_table，rank_table_2.json为自行重命名的文件，原文件为hccl_xxx.json python mindformers/tools/hccl_tools.py --device_num [0,2] mv hccl*.json rank_table_2.json # step2：拉起训练 cd script ./run_distribute.sh rank_table_2.json configs/llama/run_llama_13b.yaml [0,2] predict ":你是谁？\n:" ``` ③ 查看推理日志 ![llama7b_autotrans_1to2_predict_log1](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_log1.png) ![llama7b_autotrans_1to2_predict_log2](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_log2.png) ④ 查看转换生成的文件 **分布式策略文件**：保存在`output/strategy`文件夹下 ![](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 ### 三、**Llama13B模型的分布式训练案例(ModelArts)** - **完整权重自动切分为分布式权重** **案例描述**：基于一份完整的llama-13B预训练权重，在Modelarts上使用16卡进行分布式训练。 **步骤**： ① 配置参数 ```yaml # 16卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 2 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ② 训练作业配置 ![llama13b_autotrans_1to16_train_modelarts_inputs](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_inputs.png) ![llama13b_autotrans_1to16_train_modelarts_params](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_params.png) ③ 提交训练作业，查看训练日志 ![llama13b_autotrans_1to16_train_modelarts_log1](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_log1.png) ![llama13b_autotrans_1to16_train_modelarts_log2](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_log2.png) ④ 查看转换生成的文件 **分布式策略文件**：保存在`remote_save_url/strategy`文件夹下，由于开启了**流水线并行**，会对所有`ckpt_strategy_rank_x.ckpt`进行合并，得到`merged_ckpt_strategy.ckpt`。若不开启流水线并行，则不会合并。 ![llama13b_autotrans_1to16_train_modelarts_strategy](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama13b_autotrans_1to16_train_modelarts_distribute_ckpt](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。pengcheng.mind-test/docs/feature_cards/assets/000077500000000000000000000000001471400134000217525ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/000077500000000000000000000000001471400134000247065ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/current_process.png000066400000000000000000000276171471400134000306510ustar00rootroot00000000000000PNG  IHDR9J׀sRGBgAMA a pHYs&?/$IDATx^ y{K/$/I %b gpvGZSلweR؂M@ v1//q,َ-B-@ kv~g{nݧqg:5=3=ݧO|;,/8YwB!MOQb8evm%m%ڼpYQEQEQEQEis*Afc>P%l|ѼjQ{#Ae潢(Al^5"P{OQF^mn4"HQEQEQEQeB((((A EQEQEQEQ1 A((((cEQEQEQEQ*)(((TREQEQEQE#((((2FP!HQEQEQEQe[!苿sP?eJiON kJ^8v~f)YCk)[->?}"Sz~2vŸO}%ğZs6XO DQ$.s3a|+8P`lЊBQ5qR }Bn$Ž;J?ߕ 㧘E{Ċ.IU!϶m~04c̪~+dztrYRIw`` |BP{ҊBP n*e`wJp\[,ͻ_GW;,"|?]J1dg AYϙvh6r@&\[ԬxOl&חҐSyF'd^HL'ZIҞDaU+4p,=Qz{};1Kq6.mzK?)7k_#naidAr63/>o9}9i;FThFyv"4f.9&v(oVکsR9M.#8),ƒd@!" GU{\EQFN\0`Ϡ;y[(bXYJ(6m1B?+6/dɩ$""3T>@JY|7bHt9myM^޹k7lYlWz7@WQ r@5ks#)h0LD"TlJ4К;*"|3dӣG:k(es-f9 ]#&#tuY:[]"2I(T:7}eㅄ -R}`( n#}pMЈ'fݝkba`a}礵Tbb-օkF{᱗b?W$2v u-C-9,"'fm>${'Y uCC+hj囥?sK\R,KÏsPKt !VZdidB#!Ξo!)׷#DoGCR`EۧmSH:Fby H睽LMʼn>塛v2U|ĸ9V!ޗhBlDbd( {"!tv &N",zg02㗉qYG$L擗e_,3B# [H_|w)ٻ\O+BP~!hBwm -_7Ē{/*{Sm2tn~В"-zfZ9ά XQàXzS `vbiQ^sU"wJ$<04qĎIcvTLq3i0+)Gf}͒óY$t3J=EZQ!(1'"2NK q")c3H3~6 }#08s04',vE^~ǞRkKf^9'(tN?fyg."Dj ;( *DJR"T4B嗊d HmH;'VM }|AÍ=,EcN"pFqu̺U Dľ)߷02p IxgewhM'*RXRzցh;/M&yq 3| .K ew\cJ|;Oտ)فD"txDy@Vt}\@භw2:i`KƊ","O^]]Wh!E͏EL!Ro_1xt6h'm4~z fhX, :`9UZ N1 aD@ܣ٧"SEH^"J8Q="buZ(BoʛNXTG#Ip#ɴ.EJz&fpb(V!S'B IQ1H{JP$?iO'$a/uMޢhNO)$VZHcYKxVl"~B9mK5fƖ`ҞȴY!R B6:N.Y$cs;^6A<=ȅ'[;a3)7T"$5;<\FM.tIۏ%S}ybߠHkA&L߼JSH _kXKh)!T UIv5>:M5QCo̓# {v+~g $#30IgWe(VwKHâ\1$}t7}(R(2N<-53,چ} AHΕ+%Gzf aʵ;Gjת ߙ￳aa_{ =G6~T]n 1vbQ I7;ن}IaG}+ɓ}m42ΞkmGm>}-`&}^9[`M:#o޵0#mw|qI~}}}Im\!Cw05ciCoHO;"{8qҬD2G-$C#EJ+!`R\4Eă\WΨr'XUw^\0b\Š$o֛d{I ҰTs 7ɌYs #%0 5nzy)>\S!4nїߘe7>,!E_ٖ]>3Łx=d=߆{ *wJo7~37B#8!|>w]yl%tl~%⇎h &;Fkٝ.g,ΞKkYl}HCJH4}>}l{oiא6ގNݵ1-1@Uws5JօG%6m<gǔ#q:ֿ+>&k!In 0 H,,ꡄrCL溩l AqqAh\]sҾ$5"qg{24.i,7gs.Y9OܬgD ?RZ+^Xz0 {.ai,BDaJ"|aS=REIχm7U #ZIz7{):ydho{g>^Jfp`x]xw_2NvvGtvق<})΋DNcc~%5=:<mZr²ҒUžټބU!E+}jn%M/#\!$#qFSʆBcKyF%C#ɥt~T릒pݚȎ!T,& koN@}/f[^XIH- #a#ۯynMbd,){JB8^j (^rޭ^6մq A;?{a {aqBDH\ND|;wgpJ;< A۵k!_fkDB.5p뾗yډ@hoBA o޽_v _p6YD,̫ ?"}ɶlJNir]E?[ j.x[ o i^dGOُ8</gG$D` F Ai9&YD;.>'.\<4od."Ds(]_g 7S(J8` qKD zębywNs}ɕIc^e,4"upk5~{2@r.υ6#=} dG\Ң]6]Y׎p'~q j./(ukDQ2v$Kŏ>3lR (,&)?M_jg={ĝ7_YeDq_a~\V"-9q\ !bY>#pI㧞-=ռbQ S NV1ӹ!H;@ 8D೬" yqJ 녴R B"C=C>W-("G% ?.GW "Qw|#q)ƒh<# "GHi'tmRwZ46^}&̜6μfi+|'e[@+DZ-i\@#e߳diX&t5 PĚ$m a$mFA@`@0+<|)0)äڍg\RjAnq쇫=29al(, YȲݹL4t&;w Kd{mVE#j@Q:Aooi>"PX@#!rSObG=*W?W>yOFoED tW3Z/% +}]^]lr5}eh %-YS-lW~3gvyo(v, o[yÏSD@`|i qAA>h.?hDjJ6>$Էmɓ:+ڂϱ$ApB/|P EЃUJ"P{>Yw˾5|& DCtr0mhߺG6Dˈ@WB"Go"" MQ.N 1덈@z 1$|M0س_^]J.~6(+8!rΒDDPR++RF;Gf7"RIS8Ǭ"Psar߈HvTfnno|n hQvz[}-́`6yxDzrBPw;čFdrs`K&+mɬԬ~Pgj4nїE qB1bP3@0æ># O+D,W/{>ؓehC<̜6iZ9'&}u&z +H,I~A:LLQp"qxC$1,E'^]듊 Wq41+IRYt :QO)DzJ >E*oSo+JJ m7mza>D ʔ+^g?2=m|5CFH1hG{F;G^ͦ .v23čƅǴx\`z=OS} VшMB}J>"4k_>?(np!LDCLK(@A.Aw-) Xv}'_g?6B={?s9K.* 'DYD~R8wDrDi)X 3-.KNJW1l?:l,ER\ؑ:1P\OHc\z^[ bΏ ݣ;tR%i7%'H"H}͇\WA{ 6;oKԒ'D~O^i%uqm̆"_| X1"G$1ZvsF*eׯ v:K+r9Γ \EBn.:z ~d,0ͥ| YӞy}V.>~a@uÁـp`18o%7yS47OIeߛZ Dh Mʿo6B0nReXGq_v,$aDb/ aI.QWZYxbD1!` Oi Z{"j#Ap 3.H"\K3 YrgQa/7{K}Pwp#iF0tHݴ1Md.K6FFB Y2 P.5o[V^w];ޯoޙVf{W~*o(((Uz(((aWGqF"Czz@@!==D"C% Zx}g {(((͔єՔE_tppͣ x(((ZR6SFSVSfSv~y饗M)((Ƹٔݔ"/!BQEQEQE_)õRO_"e+EQEQEQ)e8e6SO_"F(((nO<ߤ\'/K (((~6իWo ruKYNESSf|O{O'ãW.+=n(((~ڵk-֬Y3/\n))5^w‹W{W^~53wxΜy?g@?Pyaz=EQEQEQ=gW칮wq7އ~px7P_歆# wg/}y򭷇_:xtS?ORUjިKv[6v{1ˎ[鵺^#뷔gj~YT^c˩1?>fǩexW^ͅ~˔7m?o?K.ӶW(jt7{)|[Fu_՗+Qr]˿ 7; ۿ/~uF"a}da(?ԳzsA^5m~@:I=Op-s߃ߙ7M!G@_T IZOkZ]z {} z>[8r{~UzmÚQTѸ]Vվ)))|JN jPEu%^N7ܰ>lհsݽgo6]A_6E5އyo:v8v$7>v~O,a sǚ6P$+BN@u]RWji>VukN}v)O1UiL5yx\9nuj,T/F:}tl(vCaʮ5#/9 ݐ]O-Xz~sn]Yow#M̅P?=}2量?; kŹ}cM< jjz_yc}YطTO-W>揲ɽUUpvkŎŬx\Z7 sTu ^@_gʂ-c6mͅo+mڑ<=VS|>nml>0_lO=?7mwח_OjoZ5FΖ-g~m}\n}khCQE Yҗn̂?9\&. @0OmeREwF?wlw@/e'v>ɿkuP_3LzfU_dTc}sk[ujY/:2uOۑ* amʖ}XeӴ|<=^McW_Uam|cm4Y6jܯ2=RuIciu6c6~]֧k}+^ޯ_&ۭ򯽟~4FjgCRc3 6O~lFZ/Ee^ok۠ݶϞ۲O*MK=F{l8ٿ:CY  /[Pӵ*[<8d[lkzjcg-_6.v*M+תL*M/ Z+gzіUɲqu~˽ ZvMNXu>NZɮNN=V{cMɴsYqi6ZY'v* NomsMb֙vvoX[=ײmCmy vW W\n3}r^+&mIm)+-cIm||?~?M՟_t}|;緻1P5DAenj~ߋJ1M=:qyoox,SYVufOeO-YpWC?AYehWZƦAԖ|m~jlqߚɪθX߶>0}2;?}h{mEQ5[_ϓ5z^/?+ ]ku©v| ꫝ(k4tҫpU?ZN}i$ڥY'S}~՟mܿuZF>:m?}[UVchh]~l^\֏kos?-6jk嵜f|M[ڸYY`ѺRUzlSoWߩqx~l}oJQQn~|ԷaZ};^>ϢJ'URPϦYemmY 6_ ֏zOmUӪ_?ޏ/&_blVZuibjKU}-x\q6PEQ{} CG~7_5׾z%KyuۯNTuOҼo=dUm_=lgYa.O&}s=V>y*VX(~|[UX{ۨo{_oyJY⾋9^ZVml{UZ^T_&/h֥+ޮx} l]m܏}KU>ge닷_i;zowLUYTu-H-U}n!Ж~>j^2LmLY&K|'=aeq *S':V}e':R?ZNN}zo-+-[T x|?'I'~[mQ~n־vkY[q?EoIQ_N{ۨx]uae>ucߏ\mY[>JZWǏR'W_+.KԺ6-xe~}_vAexq~mYjo,A/Ki~YpUS,rOjY8dmiև#}=qY[/UejM6o˔kjY4N66|{n}:(U|׿y'_^}n媮UP?˯so}g$a^JOrR?ZqOӬjSi^ՉJX~ԇowꁵ2f˪[^?ޟۭ4׿q?EͷcЙۨx]ua]V1}x{Uu1K/k/oa^Rj\/g-Pj^2SӬ}<_oj~?EecEm۬_6gm_5_}[?/x\(]+?mGuv ۶7 Kwc 7w=+ux0󩡇y9_d6= x$V':as}ՉN^-Pɼ([X[MO ]z3aEQҮ/իWguƯg/`?}髷e%)*LԢzẗ́+Ktr<5EO-=us,/"SE-O+7}{ /հqaڵቧvϛ_UrtM藈^{-:z(j-cUvr)w?nA[Uv{j(=EQ߼}bAKͅVe9eK{I)((-e8e9eK~sgɃ(((Rksu^KK/\(((RfSvSk@;eC SEQEQE-~))))2oK~BR!EQEQEQxlwczz@@!==D"Czz(wIQEQEQEQW!==D"Czz@@!==D|}<<^ '9`PC oy4\7o/+s׼cȗ}YGA?C}/Q[ok[|gcTMtߜ nϗ|8jtBmaڂnJڏQjRڤWo?r\ٶk~w|A^ӶOoge䧥Տk<}mjl;էӎcR#R6qi{K-+6loR]Zr\hQ<N+7ݞ_]M~qxzON6N밓ߪ~NFt4!tmg6se8L1P2m>-z8X DRf髟_mض OuJmt\5R5ϏM|;#OcMcЖc۷i(;nRRcڏ^gULQ@Tr_e'HsU>'>l_?va6of'qln? viAi'UU뱐1v;h?˶?UO~}~l[8hy]u?=:֮ [wѶYұmz}]Z^RCWv4a\֭}I=qorHjZ܇-cbٱ:Ǎ~~c\mtqZޟD؉mٳg9'vBh'bj'k:'avie'sbZYڏǟxvA۸O4]'*E'h~Q~c1ԯj|8cѾhĎ8s[\5mLJmo[϶o8vZF4/}q_ElԳu~Kj~Ѿi6FuճcE fRdӴǕsܤgϛڧkdXm'8+veDNer~;!e'svNDϮeg_mce[ک}l~Ѿ zB_{QU#v_-k6qTuʯ'dz2F>շXim靖awˏgLmooǭ-h[~RD3JmC"/For}2S{چmW=몣qce4[jh{l|Lyܟ_?@t;y;Q*zml׎d'<;SqjOtBh'C_S}2~{v)n2m뫢סmIIeCnmN䭴q@2~lM=2u)!;ԿDmuٶj6&6F8ӍQ+o<_5jcה^?ͷ}cƖ59Fߖў>EFl615ZO:o=嫓'M*n_W$,uf'bvf'v"fm=:!կ~SxRi'/#u$IMԉmW&><-iZ^hƸ Uh6<~}TXپ~i9AAXuzض4->~m諵Rlw_/m۝ m_;رeEӍ'X|cY46uV1c?@ɿZ~eǏ).$UvZe|MuvWRǙ-[Nqyy$xUɖ?QJ<;AF0kiYN6'eʦ NdSmRl N<曺K'mi[M*ڟTٶk?b}hͯ΁̖\m4Vkyem'羽kyb۪鶝֯JAضk{ĶUDŽ37mO͓zoO[xBۢmטhJuk_?]ϭ{\v6ZFˊs;yEeSll%F-皮Ɩb(^W$J?KIog'v"f'hvObS/;q9_vkOxmyۆx{5=._*۷T?KU5߫&e'u>WJn]׾lM!B} 2?*]-|^5}EۤR_c_FcWZxLb~L/w5B z6[X~w> ^6..mk̶ӶóL?x۲}"i;nw=־jm<5/_9Wm^? ^ru|S'A3zѣg˛AQM^Sc 'NL-8h<`Uqcb|z/*)ãu(ԷyN㜺GJWRaƣm~scvb^˾Q uS2ԷAG׊`:%m~7Ƭ{0p>Q' cx0nQ"dͯAq_uƍ10 eI;M km\PXkEȚ_u =E?[cܪθquWuSΣ2O@IՆ[Y&_;Odb쪰[j57d7mٚ/aZuRkקC?E?=TY~%ǸUqX(j|O.@ŗ+vOh2nئ[yAH j~/V], y}AiTcJj޸UqŘ-{02<Kɓ±cwy7|'O9_miBmU׭ڑozovBطu7{{8VgL)m\d4@/7WY;}K em6Ve^+W]7oSԟFYHmkUjml[SOx^~t8O|2/[&'v}v\-;*iS)5Oq:ձWH˪M܏[ی稫q àJo/5\;4w{puw|'u;wxne^:jٖyԦ ^[§*_~ŵ7ʞ7U7k}$&*+睻¶=@PmUTF:Wj+. ViiƨmznSKC\>|8yާ@_TMC;շnH[{0賱)'WghÏ}/}x>e3aɶm~'Yh֥uïgۯuh?;Q>MKmosQ_ d~{ov+GOva7oͮJg_dۆzU~p:\+Z# k?\rEe.}ܴx>NXLCǪvUGq+ ei5N bWϫȂ[W_ q/+Wm}UQ׺S쎄UW!hj^>6_W6UuRq1I͋ˇ@|tDi`UUP՘I^dž/M;Wz-USnj Ӱq à&RU06˂)Q͋Y,Ǵ ,ZKa>_FۙCkY}iǟ=n[-Y}6sYY4[5¶Lb[o?Ju)kwN>O=ş WZwL`,x4766k!=yzxv)WcqӲjS}Q]j=K `e?HA} Y b[M JMMt^AS?Ȱ0ZV`URB*2ZV56>ZΖsM8UA_Rߗkjˮ+*, > ޞTVC mKj~Q8XPmEЯy*.vȅy3K05<. un~cC^?t]Fm,@ ~0H䲾,XQw< zKI׉uQW5v ȗ1͏{z1~|ml5~E?p6X_qU*duuYl}@K->Aɸ2[Uo^6o)z7:˪,kLLqW<7$֗ԭ) >ǚP0 sѦiL` :x(+[阶ꎂ*?~7sZmmAf?tL69]WyБW7))r|&9M ()x4`^E_Fq ָYWV@;_O}ƯG˪t'@hf_UZVhDתmz7G_܏_Ȟ@i)=oS,Y0~҂W\ronƁ&6@_`U_nW譴~Wޞ_MYieTAeM(+HWwϲZ®&[}jz\?ahF댿rM>[n#{F-oz-Oe`ˈYmyho3ٶYkS:VYxaO)!З wuOxzV})2u&4ԕ{WOy~Dmn}7+oμQ$׭e+V,U BX?ծ4 ʪ]MͫSuRq1J͋ˇxn~Ԙ]j^z;>ez*KO~XR6) >결ʫ?©}.h6fm,Pk-HSA֯CuUk ߢu6amSqخ6)/x{e_-ߴonr{ޔ^" c|=.ܴ }N'q('Vw<^&cⲫC%5]DU4n6_a(ۺ~^u×msj^Q5y]V,߽O\U7Jc1YpW0K}UEsunPj2nMƤIk9-_X5l\4*=W7)~=N-Ӵ0 U:k櫍ª(\-/ֿC4]-j{ۗmR_>klovZn˨ct?)VK׼r*=#ZDm,z/jAӲvŹ^Ufjx47M ԺSAlaB,v]WU~hYu=tͯ:h]u-Sjt}|O}P(-VZɭ@_TZ_V&cMX~+~W=W? }*x\ovޞ뤯XQ{ ?״^܇gQ*M+*m°>.շfE޶Ch=U{n}m6ⱱhG!]^Wu%އvs6≶遇Uk" uK+ qMXqx4:^qʖk]c*(j#~PTE^}krE>^@o۩uYJתiȲSRk&,Oi@}UyYp|nPj2n:}x>jrO6}cz }ש:XkS/"X0 l¥걙g UW܇sQ;]ۨ/@,4]5: ZhDm}/V.&c:v}!֏i!ԷBw%e!+[ kf_[ԧ׫.-w8yjῤӶ=Ԟ41~ӖaJˎ"?ΪEGqӋ]5A]Fe/8BUs jǵh-6ƴl=:VMCW6Ze m?+ d sjBYY_q JMm^dϵ~<@|qB]_U8y) >+V5BoLWW;UUh[®}4^}/PǁX.cqVMZVm,빍|M(*O ^cm_S,kFXA׾ʷQEhY8뫞ͷz>V?8Y[Gm{W8_Sk[uW.su|~U"ZxKEh2nke?go Z +d&㦿/?4J˧3nkUy~"NXV}-ba[ t ~9=V}ār8ک ZV}ik4kz|/36-m=Eţ׬ ]kBm臬x0nQ"dͯAq_uƍ10 eI;[}YnPbWqc̺) x{@?u_+BUgq 0^F$ЏG׊5%m~7Ƭ=N`8(ߓ1Uw<ZWݠĸͯ:vnNk;Xu¼=N`8(ߓc6Qߖƣƭ??p r,ƣ\7Ƭjr0p>Mf:dF^G[?kλ'p+~˽4:sUwnq EO齸mǮ,h0KOf:`"Czz@@!==DPSEQc\|VSK8)jkj9TzeV|VSK8)jkj9TzeV|VSK8)jkj9TzeV|VSK8)jkj9TzeV|VSK8)jkj9TzeV|VSK8)jkj9TcyxxDJm6'EQra|VS88E-}M-jcѣSMIQKR)j8E-}M-jc>GUO.xe˿~st*/<\O^}WYmOv$z]፷KΧ(ji׭>55?UyoNLjzYUZ.ޗg{#|{<ݾ7~oשEdT{|-w)*UM:gqn񽋢R~HU>7M=ִ[miNT/<_/p~(gIjz';aFk꘷iqY ul}o{"Տ֕Zmo!e^Om6U~Cػdjz?~uuSTj@?7а}}h_Zz%u }~QTc܎Ϫ|Md}z6ꫭ;.'|Rm4O>=Dzs> OٟlCQMz/1>}i%{i6}7T}7,W-1mHS)Z%驲e'Fg5$uu2gWc\eǧ|{iee@ଶnǯ+Ӷ?|6#~ɟ7uZV[&UMlyRMZk/^q0s5j@?IJo|oVpH)jy)Svrc! jT&gWcuˎpj@ol =䗳{?zӖz~>߶gU}mՎ3{ߨ4M!ݞ1mǷVQ=FQì6C.}諞[;}CDS)j) |0ZN4 R|Qq59}َݢ:R}:}cW譬3cMSiTc}hzl˩?}.+/{od[jVʿk.~cǞϳn߷TZ^!?SmT!RVԾt=<чM UH}qKQK$5NCS<'3^v~>iT\Mq}>>WxS;>o ^ʾ[Xs>-Hh]~ϳ~U?ϲZe68.M}1h[jo\}?Lo5y_x:RW5~ئ6֏ԇQm;.7D5F }o~}tUQU>4,EQK$5,fGU OTUgUVMqzN[Z.V֯^* *|jgض6X6ml9{!l?F<}|{l[|޳@U}Lώ{*jZmT޾!kY޾ڇ}Vշ.? {1H)jW>{ɼ}Vs+}V鶬?!Ҵɪ3*&Ǹ?]cn˵qy=ky|^e/oci۠y@?f}=KmoK^͖o?`)U:^ux{q{ο{O_5j@?қ>&k׉,D'hk󁢟ow.7R{Z|՗n7IIQT?KRNrϟ:_EiTY59S׾jY8e&os ozϤ4vkoo1z՗k9ۆ&UO_Uo2~wx/}5c(gIjzQ百/; giTQ59|k!ٞ!!.^õ ZF?܏c~ #@= :m{OBL4֛_UlU^PVM*vc\ǽkg}6B~aV!E웿kЇ <7n EQK$5=U9?RZΟXtXviTqɻi;uu]˦t{UqZ~FhYW[?Hx;{MVS;^GN=0OQuoG{Om6'EQr.IMR5< ޤFmS}RE-g5ESqS+r6>SEQR/qZzZ(jR/qZzZ(jR/qZzZ(jR/qZzZ(jR/qZzZ(jR/qZzZF~F8?C:͔Rj3?oѿ\Wo{'9JqS+r6:m$ISL럅pC@] MSK?h?_&7~< m~o>1@#wLZ#X<|&,> @kz5=g3h@`M56.(`>O:6{p{R] W>l‘V23`=}"yu{p}W0?zX>QŐ qoy N m)t}i; ̯d пxpV<&uXFZ|&ufW>.xey/C`?#?,@$ʓs p‡>Ѭn|BO~.[Vm+_o q6Oiy?.yw}Me9=R{{xt'.XIݾQi=ƥh):}>RlmUGmƫ۲YW^WW\zt@?>ʼiV/ Kxz@Epd>Ia>:g|BߴLN dX'qh¯ :,Ǹ΅RMF}S_P}}7Z(c?zϩootN}B^e)eW5^PW-V5Uz1>۶Rپz{2y~נxԏ_mRǀc]>S"ЛoR?z- t@?>ƕB={W>\Lh$ϟNNt8I?\1c'JQxLD˪){m޶Çq~{bUd6TUXZV}\ ~\OfF۫qԶ}MuѺlkW:*Z+A]`ZXpu*sr vh[Sw]4W}sgFK'} iD;w=VM;'vxڨT_mZ0KqcQRa+{+cSԧuaMTS/]:^֟˔훨>'fwϖ>}[s*ЧDl;DZ6O?fm4X>[@ŗ z@?9[^xz;':ɋO`Nb|Ҳ!^lmc(~Qz_轠͋#RӲb[,{m߫tl_mZ~XJ}6<{svxq.[Oh|bSieRZ%5=Je?5.=wߋ?:?gox_Q[wiY!A*ש* W[ v>`@fU6 *pOcߏJ%eޫ:՗=~GCg~6^ej}s!!FjZJ:.ƏCǚ6E}eRmnd, ]\WMcR^}l]]z@CCz#f4ݮZr>RȌîMá깦 pܯ8It/?nޜxIO,#1u}G<IAm->0Z:ԔN`J[&ta],؇T*bǴTH{#ދ*z5xLcYR$XkiEՏI܇ki3Nx}?gcگ1K-kW+.*UY 5Nj"cC$#k-/4Mh2qoO}s 2-׏Mjt)>-omRH}P}ujyۑo[Ѿc+zL_~N>u2Wt++v"MɪBA܏˪ҩ>.(`똴fǷ>wM|#:CޤHT8e8sAOW;Sem D_}Կ~XjW ?ulՏuqh{U m_6%#Ge6-Zml=W2_c/FZ|&`įvdEc'~=ZJ9_XǠ?-0vVu_$͏Uel{ޣZ_*Lӫھ-J~sRDžW}^O?zh~_*_x>k|<t |{-믪7q׆MRz ~^\-|ŁO4O(lTSN|p|-v>kpyʯ?ޞ8>m,Tnͳm͏jxl|mw;6E1k#^V1V˪?t;O-klo~;ki񙀥A'}7)`iFFWiSWoʀs6&ooB+ԗ?a@?yHuI./ @kz5=g3h@`FxLZ#X<|&,> @kz5=g3h@?y'E-3((f X<|&,> @kz5=g3h@`FxLZ#X<|&,> @kzDlڼ5YqA4~Бȶp߃ڨbٯeֿ۟Ο/>L a̫;ޛcѣg.T_|?7`8tͼ%̴LhKd5HAXh=wg~} oㇳp0ޯ23{4r>(q+[׬{G 7qx_e<'W _>5LMMO+W]>fu7szzsٲj^ [x븶yZN˛qEλ0[}h-{Mzs k=p>ʶOJi<4.EM8YO,G15bm$>hۭm|U6^ؖϺ ү.ڬ䵜s+!]WuO~Y7_OB.7'HD+G9>{wgFO'vRh'E':iD~](X1ga%>ݿt.j7z[9~rEq{N}k}~Ħv3},N)۾̥hxoqZOٶE_5ZCu~:k:L0Z:';rOFW|L*(XRS'Ѳjes~[=-|ܿߞX5/)G}7oczXKj=[N_9Zokcϩ@Mk}}iYVuN1WbtZHly?O?/Rm^{)ziY~1E-EmvʶW^:^~-g?S>~G|8'V4>12uUC _%2Zsmom`ۻşZO_uٳ7 oY-zRC T+]]ۥXW}nJLSǷU[TY?~~Tj/t-ko^ >֩[$?*:S?*߇T"v[x<~c`ۦ?#C<~;TZ^ۤue*Lhd0>!I^YÀ? %>'vOHկm.XSТ,z:&(MOZ{hD>1~Ͻ enۖO=.//5&^O,8 1_+?݋!]_. ض1~xl{ZTp˖,Tf0^º/Zx2]hCP7K@/ЪpÖQWQ2X:(~5W,T6YmF[>/f}-_տS6ݶIضY_\Շy ֮mWf>O/e|U6iʯg{HVTӊNSk9uxL;LˎNC9.`ziZmic)R$E}m_-U>s!!FjZJ:.ƏCǚ6E}eRmnd, ]\WMcR^}l]]z@CCz#n-h|pCRȌîMá깦?GY~lR$UmxsvLyO&5>x$#^֯@ϫ)j__x=ֲ Z|&`tB_);M'LKկ隯䏻XkҿS_}xcjH<αLӊM:cCgJQ&$_EcZj=}W\vUXp=k<E(臠I@F<B[8l)Yc[&X:סzn0~^F}۾I4.eۧmMj|bʾxYN-vپx;R⾊;[cc'vv򩓹]t]hNV ~_VN}|tAA,խ6_Ǥ4;}FnY}[_,{ϕ槶W}q[=_f_ۦZq_0&UG*¡,[ޞ(>x\PUjoNU][mC=/W1߿7ֿi;l, Ïl;ۦtc}i~ܴ=^-gM/uv?Emշ~73g'~unwX&[֟,;)ѲW 0ǚ?lɷs* i~=/cSRogj^U_l9Uj,ۭǾox':.|3zcFTјHcQ_l=}+EWSkYU˯6lڼ5%cXC$7 Pj+|yZFa(>iS;5O}<[NoOy_6*^vX5ݏS<66؝oJ/ceՇWƎ)Sey}Zގ7LҠ> ]) HL`R޴ek`ҲmV~T)+0m1.қL@ #}e9v{7[ˊߟǰ}Lv/E~FxLZ#X<|&,> @kz@ŻBP>t6wc_Em?OQ_mySπ}#CCSEQEQEQՠ ==D"Czz@d~¿Cn4 ΟD+osB2bZߣ Z?NB \щV>Ή_iնMN֯~2ߝeUYk+Hcxj'Cf(yշVY{itu"}=.i[̰͗h?c1@$$M'Ret N"Z'%qNDtSx^u)2~9AOh/;yIIg'n@: (l qjU~^S^K}Wir۫{Ԡog@1H}/K9OqkRC}?! cy@Yv:': e=;Q'4$rgd'v'Y|{l,SBN 5ү+_quu愳κ5~N2g͠_:>qO~^US'jW-=A?lmVqZ| ST692Fߣ:P6u4m)'6峱/y}I<$扛b{vɍ}N+e'E')NsHWUۘV?mwܿG1(lWL>Ihr|[%/qWE/lڼ5YqA4~Бȶp߃ڨ-PI' ?d)ƛFm5M9>P[Yos''y;gE3Z'1_y,hoo$%OME'qEJscm;Zoζ$`,5~ILLqyۙؾ?h_'PWTp $׹mYOc7OxfP4+zy~}*Rwܴn-7oLk53m/!U~<5*>.mc<3ꌟ~E/;n1>V_˼c\`8>+=z,Vڨ}{o81q^8>N4|)}sOIQ'u-TfFjg O*>e#nc'!G-,qY&UW8Mi_޼cuFceRS#s޴6^޼i/qInmlT=)Z_v "ՙm6ߠ?ظhxߧ8Ai8|Dܿ;}WYל|gU?tu<'zui=nw9X@d5H۶NOs53a 5;^02:o:'vB2 XߐIEɒ:.N|ooL?s}Dif8-@͝<'0 ui"VI mLJIӍeZc}qOvSՏ?Ί*~M66Ӽ?S6O}tya{4r>(q+[׬{GԮӗ6*Q[X7i~"P8!7_n| N N<h}eʟ$dNOl_6Y6n|R'Csq'vjo[̟֗ba˧-K꿬GOUODm ʤ;5m묧5 %m>c\T=&k(c8ˎjRP6m">֣XΤ4}^1}jegfZ ߣ`S}ݞr:h78z?a ү.ڬ䵜s+~pjwڨ(mx"|St"67i}:? 7ԉKQq'ڦi>dDOPuv;nm7oܾFeW _Il0Wζ=#h,8N*TΙ$Wͽ<=NGm}j?fwLv`nrcd:~~d2U-0˂uFxZLQKQhf5(1+ާyu3R6{iƺd:~oSGw}z67yz?,cߋ=}6<ߖqR2ZWTezm~?4g\W]}e2a wlϥzRQjEpdpޝ!

y uj͂x|:!΋Qfh߼?~yqM+o>'1-SGNl"EI1TN:Lj=u-˕O>o7wlk?9[_/5M?g}탓ZUg,mtXm z?ůgGIc\R;ޟ1fcJ6]#-OOzDV9b1_&chъdXSj;JMB¼m'9;ALtKdNɂͼ]@ӋjE4MThm'#>Y;9d:1N<}NNƻh0FT:K(VO:˧HZ\4>v\O<_rgomvVUYǷ:Rh5 }wߣ83]jR㜚6ö557|;Sylu#X0uٳ7 oV>e! .^:?ܳ2Vyp;U;fo{oZ!|yv]| WZ(/Wuzd)h'&vay7[~ I*>MQy'~k{RKk֦I(jmkr0)[Gj*JT'^&'Ec`UgHUXƠ~g̺{:moam3]:~qkF3DžqaQj[~m=o=1^40o l{fy>j6MAމ-Y5mͺau_,e6C;vaNm Fڂx-g&o>b?HX>WtB0}7X7  oՁG .4|Uׅp}MS=E uǯrc{`<!==D"Czz@@!==DP~EQEQEQE56B@!==D"Czz@@!==Dmڼ5YqA4~Бȶp߃ڨ- ۣ'oWwl7 zGSo '& 'v|'©n ჳRFMd5H۶NOs53a 5;^tͮJ}\xVu5^;@?0Jmptu_}]~ŵYqY)k9@W;36*QQD+<20E77޼V ZcS;?&>ٮv~*e4z'=uU W1 M?tXQj;JMf=i1_&chъdXSj;Jm?p@:?gox|B{]ZzuV)m;ge2)>{wv^-Ws=+r첷cAze|=Sa-[P->kښuKx ֿh_Ytmh/!wW'zR*ƍ{ ZNϻ\綬xz=|=z,\pѥChVW.LMO{{xn-~>1[ izV*ЫO@cM:uꅵy/ BgA߮+tpo`$ }W\vUXp=kܚZCD7%zR*qWVmz@ަA8mUܐ8\kٶǁ^wQgw@郷%zR*qW]_\s!@PO8+[쵬W\6mޚ?[H1,+ԙzQ[/yz=B8S޴ek`ҲmV~T)+0miJmvv 5Fm nsoR6&t+B}Yd\Wۛڨ-Kzzo[6{ Czz@@!==D"Czz@@)(((jPmp"Czz@@!==D"Czz@ڴykXni#m7*Q[GO2zo> G˟է6j;jgNLNOS/gB7!Kz]mokm+ jjg&kɋw=?]9╭k#wL~jaKڨ-*JkRr;@ρ;végmTjVyeBa@onyK,5}Ǧv~:LM|]Th jsѽ!?Oȩl-~PquU W1 M?tXQj;JMT@5_b1_&chъdXSj;Jm+ǁ^ZXL]goOY~KW*<c'L:U:gaՎ٫ ΢@Z!|y>d@~Lu}>mX  nْj^֬XƫXXEB_ mC{<c':V_ `o!^nmmY *q4z`i8zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^e v^;`$KBqW\vUXp=kܚZCD7%zR*qPVVmz@ަՑ N/nOv8,Z˶>#:C}N-V+ [ SaOK|`Tt |{-믪7q׆MRz cg&J:ufneW[u뻅麁^*wJ*Ћk ћl UZmW;ʯ?*y~`>?m"5MڎcԮ󓁽fڨ-mM&߄nW/+~2~Б,j{R`)@Ѳ[y+~ "Czz@@!==D"Czz@@u'NQEQEQEQTj+"Czz@@!==D"CzЦ[Úu0Lc' l=Q=}"yu{p}W0?zX>QQ;pbpb]}"z>8/` XJ>j{[m]Q?WS>0YO^ξs0@DžlUW_>c SO_ڨFm GW^WW\zt@߱ӯN_>8nR O, ]zs[j@e;6ajj^FmW |>ϻ3 :*Z+}Ǧ\:(^ds:?p`v%ٻ ?2CCV$zRQj{^ S O]goOY~KW*<c'L:U,01{E\ZܮBmw}v>Ы_-S' nْj^֬XƫXXEB_ mC{<c':V0n-[rzgmmY *z`8zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^f v^[X7 }U0o@>+.*UY 5nM!@Q"w􁛒aNm80+T+\[u~@o `6kêknȂ}l[@;;a wےaNm80뫿®Mn'` y-Z_Uo+ 6o͟-~ }Lޕ udIfq^6z=*07m8Xl@v_`/U 0}dDkR]'{iʹQ[ܛM ʯP_V"Czz@@!==D"Czz@?yu(((T\"Czz@@!==D"6m֬۸a@?yHxdDJma̫;ޛcѣgڎٷ;>K|)MR2^Wm['0ɚ~py/vfW>.xey/Su~~F6j `8 ү.ڬ䵜s+~pjwڨ(mx"|`P"ЛoR?z- t@߱SlW;?2m\toOf豦}dl]}e2a wlϥzRQj赼8kf[4`\,Fz c?Z uJmGz8 Kׁ={ƛoS@߻իJi;C=+aNmٻCXcj]Wxow?||vvg C2uBxU ={'†[dZ|״56*֭:ƳB^6CN:U,xmB<-k>^*Ыϲcႋ. Gruajz:ssTO~.o 8\WMcR^}l]]z@;om2שS/{)fW= v^ۇ{m? W&a^`U~`Ttˮ +>Ug~7'q_zcܔ uJmānomz@!ܦՑ ZnOv8,Z˶>#:C}N-V]+ TIWȋnײz_qmشykl!Kǰ;vfdXSg&?LEAnfW\*wJ VqFA!zӖaJ˶ j[R0 GMd)QQ;{l_u~2L0ͽIЭ ecX>O:s]moRj,Ec1Zv}|n==D"Czz@@!==DO:6{p{Ra{tDX}-ꎭs@`~Y}jvyĎD8ҭ!|p6_ )t}iɻ0v}&L=~a8}` sٕA /^٪f} 08}vQ˯6+=.+y- c_=ڿ>|pF6j;Ja%'X&ƛO^]#wlj'O彌F@6c`t{-Õy cӏ.kڎR@o=8p.?u8 _|@!wlG+aN(5 F=uhk9`\tٳ7 oV>e! .^:?ܳ2VUªW-4+]!|y>@~L_m~`wb*leKzM[ncibak< ~. e?|dXSj[ł. -k9=֭-k>^(aX=.4+W]==<7M@pK-z}4=Vo+է u@߱S&z:ڼbEW-Uzn-c(tUI@o&ⲫŠ_ـ_%}N)VBµl{^7 6@/@t}|}a57d>Zq_0;vmɰ^ԶJ TzmFIWȋnײz_qmشykl!Kǰ;vfdXSg&?LEAnfW*p;%譽mG]S޴ek`ҲmV~T)+0miJmvv 5Fm nsoR6&t+B}Yd\Wۛڨ-KzV|@@!==D"Czz@@!==D:^((((Azz@@!==D"Czz@@!= haͺ GMިFm`>V_˼c\`8>+=z,Vڨ}{o81q^8>Ntk͗0j ݄l,%cuAڶuz(̟] S_'/g9khv|P‹WY @1]/mTj@++JJA^= Ow;O￿x.пr{8}̻Jmv6<J>OL(Lw͍7o՟VzE } S;?&>ٮv~*e4z'Dywpx>]z^EpeP;[]= za[?om2שS/{)V-Uzn-c(UI8뫞[zⲫŠ_ـ_jWO)Vòi >\ !M#zn?HB>0*>>`k-8/`y oK:U@ k}~P7ԫoc\b+[쵬W\6mޚ?[H1,w+Hu[?3yW2ש3&%5Bb?PUq=ƙB-[kmNJa^:6FmG}aj^Z3m`6&ooB+ԗ?a8ЫiБ,j{R`)@Ѳ[~+fzz@@!==D"Czz@@!==y<:EQEQEQEQ Czz@@!==D"Cxh40Zo 7wz@@w'.\xJ=D6m֬۸a ,/FQQ;pbpb]}"z>8/` XJ 5H۶NOs53a 5;^tͮJ}Mkߡcu~~F6j `8 ү.ڬs+j}+SׇμۨFmGi!˄txZ)kYkOO{^m.7' 3;Cs|&0v;qկS^l* ,NL lB@ikm, U,[ug/Ӆl@߱_ uJmXYaڂx-gn۞۲U\车_G \tiЇ?Օ Szzsy _8ƁOnj+ `:Bة}kaNzamK1BYз+ Z½Q諂y]~X,6ⲫŠ_ـjD7%zR*qWVmz@ަA8mUܐ8\kٶǁ^wQgw@郷%zR*qW]_\s!@P &`t |{-믪7q׆MRz cg&J:ufā^V{&W*`@NIzMm(A!zӖaJ˶ j[R0 GMd)QQ;{l_u~2L0ͽIЭ ecX>O:s]moRj,Ec1Zv뽿}[`!==D"Czz@@!==DP~EQEQEQE56B@ >>wFe@{' Oλ2pyW`T v{@,ց3y`Zzq<vwSNfئ[Úu0L@?N& l=Q=}"yu{p}W0?zX>QQ;pbpb]}"z>N`(t}iɻ0v}&L=~a8}` sٕA /^٪f} 08}vQ˯6+=.+y- c_=ڿ>|pF6j;Ja%'X&ƛO^]#wlj'O彌F@xG6ϖ}uU W1 M?tXQj;JM!b/L;6ɰ^v^mJ:?gox|B{]ZzuV)m;ge2)+䟽{jn{+ v+t]Eײcλ2u>W0;16ܲ% Y4Wn5~L}N>u2)b_W`m!^Yp MùRu?:zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^zpς]W8(UIO@g}eW* ׳Sq_zcܔ uJmāޮUx{^7++ S?LƉnOv8,Z˶>#:C}N-V+ BA0t |{-믪7q׆MRz cg&J:ufā^VKjv^~0Ptz=0.7m8Xl@v_`/U 0}dDkR]'{iʹQ[ܛM ʯP_VV_˼c\`8>+=z,Vڨ}{o81q^8>Ntk͗0j ݄l,%cuAڶuz(̟] S_'/g9khv|P‹WY @1]/mTj@++JJA^=BWnyQڎ҆'BX .-Sײ@05vSy/&kl:*Z+}Ǧ\:(^[ [@ŗ }Ǧ"ڎ +:O\ׁ={ƛoS@߻իJi;C=+aNmhU;on!|y}WZnz}_yTp˖,Tf0^º/Zx2]h;ɰ^Զoqpm_9co۞۲Eԇ0`:zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^zpς]W8(UNO}H쪰Wez6׼5BD7%zR*qvܞ !Mk F_y|pXu Yõm{}uGuv0 >x[2)z}Wz=s-em+GB^t>uW՛kæ[g _=@߱3w%z:3YA`@/ v%5Bb?P=M[*-6ЫnK¼B?0 c6FmG}aj^Z3m`6&ooB+ԗ?a@?yHuI}h٭v[=x"Czz@@!==D"Czz@@)(((jPmp"Czz@@!==D"Czz@ڴykXni#m7*Q[GO2zo> G˟է6j;jgNLNOS/gB7!Kz]mokm+ jjg&kɋw=?]9╭k#wL~jaKڨ-*JkRr;@ρ;végmTjVyeBa@onyK,5}Ǧv~:LM|]Th {C8~2]}e2a wlϥzRQj:n'_#e=@߱H:&+_Pkv/]goOY~KW*<c'L:UtuwjGf+B޶{]u F%`wb*leKzM[ncibak< ~. e?|dXSj[łQ`o!^yw ܖpO/C `:zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^zpς]W(UN"/tˮ +>Ug~ŭ5}1J>pS2)zjk nmZͷR{?t}|}a57d>Zq_0;vmɰ^ԶJ_aW^U =ƕ]ObeU&.ڰiB@a!w]ɰ^L8Ћz@ `@.BR޴ek`ҲmV~T)+0miJmvv 5Fm nsoR6&t+B}Yd\Wۛڨ-Kzzo[{ Czz@@!==D"Czz@@)(((jPmp"Czz@@!==D"Czz@ڴykXni#m7*Q[GO2zo> G˟է6j;jgNLNOS/gB7!Kz]mokm+ jjg&kɋw=?]9╭k#wL~jaKڨ-*JkRr;@ρ;végmTjVyeBa@onyK,5}Ǧv~:LM|]Th4 OcͳǚAׁޮW2\0;6aF(5 {C8~2b/L;6ɰ^vzn@_uٳ7 oV>e! .^:?ܳ2V-;U;fo{WmB޶ܭ v;}Y`G nْj^֬XƫXXEB_ mC{<c':Vpm-\[rz|ޝB=e-PjG .4|Uׅp}MS=/ҧ?F7{cs^_5MJzuu!wԾɰ^N6菱BgA_=(Uai7/`\v]V|,\[S"c;}dXSj[%PnWy@(з z#F_y|pXu Yõm{}uGuv0 >x[2)z}Wz=Tof0 ]!/^M\~ŵa/BؙɻaN, 0qq]s |ߞ_aJ!zӖaJ˶ j[R0 GMd)QQ;{l_u~2L0ͽIЭ ecX>O:s]moRj,Ec1ZvZoezz@@!==D"Czz@@!==y<:EQEQEQEQ Czz@@!==D"Czz@@`@6o km\P0}OL(Lw͍7o՟FOO{6λ3l31uU W1 M?tXQj;JMgG#e=@߱H:4kyG]goOY~KW*<c'L:Utu\WW˻o!ݮ߶;>;{C8~rvW|<0n;16ܲ% Y4Wn5~Lچy}N>u2)b_)W`owmmY [S\G .4|Uׅp}MS=/ҧ?F7{cs^_5MJzuu!wԾɰ^N6菱]W,ss-λp}U( KmJqW\vUXp= nMm(@Q"w􁛒aNm8ەr n諮_rOmUܐ8\kٶǁ^wQgw@郷%zR*qW]_\s!@UUmQ+[쵬W\6mޚ?[H1,+ԙzQ[uz=.k M[*-6ЫnK¼B?0 c6FmG}aj^Z3m`6&ooB+ԗ?a@?yHuI}h٭v[>`<!==D"Czz@@!==DP~EQEQEQE56B@!==Dy8״j.UB-M$.\R h F!2RQAFh4aPD&%1]MZyo;s3y݄3wjH@ z!A5$=ԐXӶ{k7ұ'{Xkɉ4{GZ?0zԽXkmĊtԓy*nJZW-YNF>c_KYۯCm1`gKJg5iU;R=V׼eg.tb3+u=M`TZqks&B>"+64sygwϞ&ayJ{DtA΁AB}Mϴ2Sz`J>ҭ;`Uy< +6}Xbb0-e>">b+'`A=#+6Xfb0-%#N/\l/za9466֜2saCiw38)m>5=G\c9ߚvGAwOv#}աܶ{Nrx_ Fb?~}|Ziw3YD}q]|ksз?I\M75u3z]zǛScz:>bΝŇy~x.WY=kU}f.m/nf,,GϻŸ"5bMw_M͸jv%&A_+oz7k;iy} A$_ FQoyzڼfطu\?QO ^WL=kq=CD}\m!vځ/;bw{~ִwx⾂A;y4ֻ C}Gz١!1SwQqFUD]M\oǺ8W60#NL4?W&Kqfei/:cL|/(Dbk7aL|3\}>Ÿ1{"=ԐPCjH@ z!A5$=ԐPCc1c1z!A5$=ԐPCjH@ z!A5$=? |O^HG?>ĚX 0hON7vc?) }֣ŚX;ls_L'Vۧ,SiwS7׺ nr2AZ~Nj8\j]U:_Is\m7wΗ*j͆-;[woYϭibM[/6q]PA_ٟ;ӽ4&ӮS#"wtu&+8ljL<ߜ~u5oIi?? NI\cgA>byw1vz 6_VJ+O+`A=#+6Xfb0ŀq ˯yX*5L?~}\PL)=AJO{ρq~>v}Sz|Š5qOǽ^F_iͨŠسhwc=?k4^5U7#+vJc.Gc9|m_;qP|sXnVKy~c x|M~-1({`<]rk XG&b>a؉*ay)5ά, E5X|̽)E叨_l<2A?yF3c5zX};A5$=ԐPCjH@ z!A5$=Ԑ<'c1c1=L?@ z!A5$=ԐPCjH@ z!A5$hmǞi~ڍtD:ɞ&ZA;~r"Oa0F>#̧nu/ab=">daJ3?ҽUEtlmRkvP[?ƙRҙbMj.@xsTqUk65mٺ ,X}4}n]Okb-0U}ǽoڜ~t@<t0\ޙ5vv}1]EgosEǵP5A_gSc3 GANJPU}}$3 M}<ֻX;L}\~+?H`A=#+6Xfb0ePU.6c_k={UkN~Fݹ4ֻXIuS?2|y> ߾|JOYq߸f_(}CۍkfTbsvY4;ɱ5/G*kNrxw#s;_Ϗ9۟/>|c`y^X.=yu1=s1ZqX揼v?k<G,Szb3z73]w#‹r]bg̀Q)qxm=/}ySZʦf\~5#~/A0 ^y4ֻXI{ w^Ľy#)ѷO|=mf3:}{lj⟨'`}f+n&vy7z~FI/_v>-bƭi֣}="+vwPiw3w' k!ڿIo;87b.*`TED=0._ĵ}{e10ADkzkbݼgVX ,M>"G/6~AvĚX =Õy0=ԐPCjH@ z!A5$=ԐPCj'RQ^IENDB`llama13b_autotrans_1to16_train_modelarts_inputs.png000066400000000000000000000517361471400134000366500ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_CkptPNG  IHDRNsRGBgAMA a pHYsodSsIDATx^o׽~CHZ2KA%lZГ$]J!RI7Mˏ1c @1s5"R!}Ϛ53kf3 g/-Ϛ˺z?kfH@"0q#D`$ GPGm?BAA;V.#iZZ;i߽AAAPH2V1s!`AM#N  adG  J#<0AA,Gy`!  (Y0%ʹ7/ϗ8o6ҿozS`!  (Y06umHoSOӃ-rk/ySG  JcZ^ֱ1\MyҍChRot  Uq-k L6:@l֚aF31)qȬ^(+n[RS_6A\֫~5@}~J}o=$^漳y~W #AA%mA3LU^W?w! #Wsf*—o8W|W|K}w:/ofށ-~=חz8BAAPJ7g G1ld3YK\Vnmy?+Y,'_! k$W??;f_3LE`˴ABr͑k횔Vv'9T>`]82lA6tà T1~bK3ΌdeYN_l#xw%~5Q\0evr@PIn$=\sԵrMJ+\u*l*83c *Alco0`4FYo}\U=F_ }c:b m~i^ffs#@)>kWA}4:h273Q0NV8:VW,!8;6(2A5N,@0{ŵxP>`S~Q8nJC %ϟ9xp!(y rg-?Hr^=^>TĂ( AY\u6 Ʊ>x7rqu3oGU,4:ԖJ8.o{ꈏ<.TĂ( AY\u6ќ!t͚83̤Q_ KSI#E(p6[ȿp9E:P 0Lgq*9(ؔ_Xg+G:P=3<ǦQ;l/<~O}kK9h7djtOSjX4`2(>kWA}4jĐg}\y\32̠2omcS͡<~jt3eLJ|Nszr#nF+4rw`JStjx:i6uEj붬 dP}׮ArMiEqoS֍K1^1>>=HBM˴rk/݁i|lƉU.Pem[!]K'ol+A'ǓoVŨuOF{uOq@V>Y0ھlsJҬn{4N%4:jdrUv U^(KCۖ*IcM[1 ܚ:6.qZUm2]B꺖p|h|Ukmo@UNlJ/`Cz?}u3cEFSu\$T*_h/huo+Zia*<}CYqasNdԁ7@c84Nܬ=S^Z6}5wS@~e2FD3=7JwWҢb\\1!Pg&}[n:j/^4vr]DӬsiOW־ t>E[(o[G8}ﻥ7rN=Δ{-U8&f/cMC,j1tơ}!ud,#yZ_e8C14Y|H)sm09-_y60Ɖ%z{{JmBzbf=1o!fju{^v6=?-M-Qq=Bz/ cc?Ӗ_f?=X:^Xg?zTܜz^㪧9wx<ا8 iݾ}ޡ/s5fC)쟽BeK}yM>˯ߍ7__INfs~\hHg9GMԌU@ zuw:;k S"7߀{DΗJ$U$qŴcK]O )M3?Ħ90@ǵUi+-2l9[6ki9KsF^ehNw|/-{*|xN=Ɣ{-ڕ􏀑Z!}/ϣwdJ0)DU$l+$>Ɍ`Bɟ6Ó2~8Pn)Mq5M5N܁ϼ ,IVtQ _q֍~C#>4s-ko=85ͪԭe f9z|;3yǘ}IӰi3ws酭#lx]:I7_+җNf_~ǯPޱYQIi7uR{f4nڱb^վ;N4QΝaϙOy{sVA2Y.bL0mz둁i6N S *UmrMi#i*}b]>x(Yu}e7/-qK!`p8FQAԧXHsKxTULo8{c7-@/gdK_cGBƱ6/(l@2Rrs a֯]ef>\}MЮJ5´մm2s5kUF-S 6MiS|ZIOnuM:o;V|pE=Gi?reC^;i3MW|Zq7 XGq6Ae)>[ʵ0O6Y*΃ٝEAB00i`5xF/'hM97P 澂8oeU tRsCfj9G[?Xe;Y1/W\QdqkcT~By}0@ordŴ|1xNer_ ii?C69۔1Qs?לS[9G%uNj}l#FcZ'&r`rHhc GPTZi61v2h]BZe ǵ/ƱmC=x&&,8&;P:z/I(,%=ٗ_Q Hq˂1U=0]_v#7 =6 f=˦Krm wW'#uP.GPmfo'-{^!4yv,q͓eb;~XG :E¾4Gl#$m>~z\G]@!PWGW=#hIwvT&|:8_]L+?TyFn#U SYl׮Ԉ ^rm+>:.|{pG&\]';,">m;)^'X/[u>e ]._P0mܬMiڲ7/y$Ls8Gp̥6Te]F1V;Uuô/0A;]c:' ;7^hrJ&$3fxk:g6OҾ4ܚ^>ng~}i!*h^:ju5yްֿV:+}4MGٌNӳ Ƀz1n HexV\GcćϩP}kWiCi`Q}i7'NVMd]ii1dνǃ-miA*y/fzJ1t msdwUrMi#iqbUBFsі~>YC_^9Ojظ2OΥ'~"ۺT}z_ Nsvfz nZ._O嫪*/y:Ҽyhӥ8f.cP 'i~_Lēq˂fMu^_d F d&5pl Xq4Y4łeN'^;cXqٝbB ۏѴ6-(y ΃WJ;Ń+>G:k.o8MAQdn?~Q לAn[HwHku*m}"\8QxKY'yy&異zZfTV(>[ܵ+%߼;jV<$~ޗs{nۯr 'x#7A `͡Η\ lJ/`AMS ;e_0BR8nYwU_ʬOSݢ-70`#4k6gGjϸ+ꫪ4c&AH|}`=Z\GǸX2f7S^kװvB۹]lfsO rn9HMfq~fYJŏW?BΏoA6`V/̧E۹Ncɨ@,J|(/ c=N:W33%yq D#:8IGS4O1_ێ;s*+{-UAsm?kU}z#}Xm7W gY}Gu*:_=BhKQ 3x#zȽlJ/`AMSqЧyV is;uOyf5}Q,S1! 0iY>ʽsJi< Y ͌NNASm"l u4n;DxmbmOQƷ?[6`)Mq5M5N,*f ~LM&Ä`S~jjX4 sW|>rQ5)p*9D dG(+:w񲺑H zk\{lפµ8ΡJ$B8FNAAA#Gy`!  (Y0 8BAAP`Aq  d8AAAɂq  2#AA% dG  J#<0AA,GyZZ;i  K2f.#iד_  rʅq5M,2#8@20 8 #<0$2#8@20 8 #<0$2#8X;\A=#<0 f `&0 80A#d q 8X;8BU_0dGy`kG l2cAT}8M`Aq`!/G #<0#AP@6qƱvq `&0 8qA۷M] QtκU+^JwoS[u49*M`&0 8Ec=hV@C ݵl+Eien=ZC5B'm[kFS]{-OrZ4a;|=Gw6ݛ]%"ۇ.CZJW瘡#dڡ(x`κ&n_ m(qR̅qz*ݞ^Wv/T07|c雋$]mk ^7h̲OeTbJ>Dc`2c8{;h'ԺP7OQ[):wZqzܠ]Բqs qN0cW7;KsNtqW[xփgfllq9ekm?J{tjkuj`=pӗxec4`3KT*w7x6/Sk0㉊9רuZR涾k4>e_ ǵqҮ7q]8췋.pw]q쵯3]38FWO*~;>NJ|G6֦b2tznS߰ؠs[YM2ogNY{ !:Y^=_p>$3^|N#/pQw'Mo>[St,iWyi~l>Z-|玻qI_Pmyua^s1]lZn!$)FlUvkm P$i0Isǃfe`(WiL dݻ)GCyeM(& 5p8J-3tK̐gm <6qII6~mDJ=l6kM_|m^#TGΰ1mȚ2,#w9Sl&mm3:lהּk2!3ͤ2u(C-& l2OqhRZeڱl5u4mzXKi!G8ނyӦo^]2F٤ܩ4T"m_DFcӢoγScHFjXƾ2H~FB͈S×y9qEȚş _4#ԟ`Sc;.u!aIȿ}Ͱ 3I[Tr~M}CA]{Y"3R}~p1O1{nRimn[D#E'w|I<I|=Ef\Lߧ8c-4AG1xTf4h"<&rMle8;wӞw3/]L6nLǤ4Ã}A<ÝvV?):/C||Tu,L9ϟ*s`qNO9hI#־ְ|i\rgy -v3:S@ǁlPa*eƱPߍɟ}kUo?VqttRK(u~;zO*hfRlqۍ/ۿX8M`A)8ʣ,u&/<;g /d1̜˒g$4hf~*FҲQpWQAn84PtEg =*cﳙAO;!N:>#ä4_Gx߱r]c51jJowq;~V{b(_Idp߅s>lɣ{ꝴ;q#ױ}agpm ZFȔ~|P{c;}/PQ>蔓;wLwK10 ^(ć'Lu8ݤI|ggq ySUm"}(F~,s{;ϦO}4ctJ>,%W/\/e9ׂeNCf9dnk'J=9w=7v*uwy[\h#[?0#kk+:m.qf)%_,sR2ˑTkYFi1R J!픗G2g1۠`&0 kIæ7&13q1ՠob5<,ؖ)ܫ:vc)8B/1j d^L3J>skj\eTM]i8M`AIo]3ċq͜ո9rfˌ |LQTq¡YGUJ+Ȕ2CQʇte);OFlU2j2CT%Huz&kZRy}zqr :9wRE l2O3iF*ll_P o7gM8`5q3B?Όc: '/atMQU}Č#A%1Ol۠$8M`A)8x&&Ic T eĈ C8ݸCǥ}TՏ׌,+܆9d9u1<0A#dgbq 21M5D}ZL6X OЬ9xY0hj[Q\,Ȍ;mq `&0 g f1Tz1S&Ѳ5QC3 -ά-͸ywc۬洂8BU_0dGyJqSG l2cAT}8M`Aq`!/G #<0 f  =`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq Gy`Hd9   JV!`$ G8H@"0q#D`$ GPGm?BAA;V.#iZZ;i߽AAAPH2V1s!`AM#N  adG  J#<0AA,Gy`!  (Y0 8BAAP`Aq  d8AAAɂq  2#AA% dG(+:w2m?<AУ\sc&] uU$q4rN5ɍd@kXIikWqA%M0 8BYQ >TrﳸvlJ/`AMS 0Lgq*9(ؔ_8'MEa ʽU`S~jjX4`2(>kWA}448 h* dP}׮ArMi#iqbATɠ,]Ń: GPT*_9Z=sn+QߡߡmI*8N6H `U_@`hRZMPKiT=k^I wN|6*}DD9*{ʵkp'Rm|6:u6m:6Ѵ5y+@ϭir8F-O4:R׵FCF8^%n{~}>`S~jjX+G/OqXMf6'شB [i;.`S~jJXJemx\6\,u?k՟hxǍLs~gT‡L?[K_r9C{^ͧu|(,g}QM%Fmh z/׽HMGquaFeex*o=,\K;oJB/hEcV'#FOx53*e 790f:un9l]͕kiiY9ml٬U95`nڜ6mM i'=561r,\Q-~j&n߲ R%u{/E4Q^G0i4c$ }.F^D@ } J\ؐhu4A,ɶg،uܩkuzj%,6YW'@j 1gvW+4H*&Y1ʛ γۍryܶ)Dq|po,yN=>%6AϘe~T{-Ug6}l2`[s)>d LLx?XW,ֹc0\qg$/՘QҒ󋗽qH:_qwA}44>r l0:7vK_qL.6gJs im%q~i1|b!+4m#84FXI|tlBcci)z&{̧=(*~ 7ޠ385/6qM zޙr nz0fUC9 <0#h|aP?ΠG(>[+ ޑA_y0 }H_iWO"2Lb{iAQql3g=%_F|1p[UOL{W)ؔ_8'V;>(wh:ïT=q(fO1m_~lki6Sz:uǹD/kwcxf {_c.K qRVGYn~W1d?n61oh-͎{18/mUq12$[$\:zݯ닌AT`?-}5͛w7_9ּY'Χ5{he;#q[c66yٝbrsl,K 62O-w$6Rc>x0@[Uo5Rtoyf'm M[zqqueu>Pbk aQӱGlP^r} ~dȭoغ9'U9e~T{-ڕ@(1%UA=<m@٭Aqwr_|<^+A|UrMi#iqb;gB+y>L[ܯU;ԶEiݏ]=9ыn^ZXO9>s߮'G|UuΓu멿1^}iIvr9L9w%,qryp]UxGg2ONӡI=HNX$xy0HxYOu/zzdffWU7sERX}UffɗR_r5`WUg #kv]E[]>YV#'XJ.N'm M[qh!frlvMY^!>Tk嚔VvT2#xYHC= 5G=kRZUPP%Mw!`AM#'   #<0AA,Gy`!  (Y0 8BAAP`Aq  d8AAAɂq  2#AA% dG  J#<-4|}z@AAP%ce34/AAAAcB8&I@qd`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq ߂ dA3AUQ0dGy`kG l2cAT}8M`Aq`!/G #<0#AP@6qƱvq `&0 80A#d q 8X;8BU_0dGy`kbyjn+U;y$z\TCG/nz\M8OUo3qnڶ)G #<0CQƱs `E5H+:HS:pڋo~5tKԕԚ ۶HtϺJ_K?F<ݴmC).f٧"*1_U%y8dƱԱ!]؄R_P<5գ]mnPo{ƶ2qJm>4NW/^Wq0 8{46pvR+%wCVuӱuٶ~Rh@xZX]=?qtq.{i\+״ yj'm}ǩrɶtGW:=6j3QAN.=p( .']38FWOPe' g`;1pZx%-.Ҙs<?HG\S]h9-_`J=<083oa(sb;eS@}޿Ljlukx$ߜWctbT%/x܄cs%m#K-ˠJs  k1vo@!Ž1w[\=UhwEK?/{@ݒq:'aUN?sz A!L3h9B3{^a'&t>[cDrWa)@dMQ8/o4d(+1v6{3Ž$  櫓bbtyǙ̧Yi4y"<6  ^yf5Q̌gra11yò͞_h㞽gKG?&cnҟOy3r 2M?*x%z.$=_'W|8ׂ}#}7ϐ,(qx@6qh71Z#FT9f3O"'Iٿ80bU^O! anM8ո p(8(Ի\ί7?Y/s`6UzLH.}p6VaEכyySit\Љ[֔I We[8ff}=9@_'/|\|Xy ШKoTfONVxYW][vG7v9^)ӯnk9C!3ns{̇[_w=i[Gr kE:#I}(e-8Η.|nˣ^=NzRGn}EɖDžI#dj8ڍcXbtQѠtn5 :8p E`փΌ{՛q ن Kj;c8/og)JÑ/H@Njf)L'٦×uى;w9XQUn>"c|1U0 Wd4#(CcoGRFg>aO/Q8&Kpl=;IPR=wjt\ԗ69wԬdR\8d|O=F3;N\ϓ_Iu-oq;yAKC~'sX7=FB/w6;/k~FE87my[ >ݣ{%Pw)%;qq]uYIݾ4DWG]8M`A)zQ̐,lc<3pǀ1yeQl&l0 HINRɦ8b]:xf,ڏz#|{\]>t>뢘ʝg]8M`A)8MoMbf; bcA#jntIkmyX-)S]IPm.5*U8>=:D #dbgӸT<t >L93o:#qXJkf@5~puX+N^<`0xql?)GWP8M`A)8x&&Ic T eĈ C8ݸCǥ}TՏ׌,+܆9d9u1<0A#dgbq 21M5D}ZL6X OЬ9xY0hj[Q\,Ȍ;mq `&0 g f1Tz1S&Ѳ5QC3 -ά-͸ywc۬洂8BU_0dGyJqSG l2cAT}8M`Aq`!/G #<0 f  =`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq Gy`Hd9   JV!`$ G8H@D?/y[UIENDB`llama13b_autotrans_1to16_train_modelarts_log1.png000066400000000000000000005561631471400134000361740ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_CkptPNG  IHDRsRGBgAMA a pHYsodIDATx^ϫ%GOp߷oK}oMf$h@<3"V_}g>#¿WzW? c>5b <>O7|+owqt?:SL=v8O|{G?}8im'jg~msy/ºگ=cmsvk Z msUӳG㬂~o5tӯԼzirϦWf?>ٯQ8a9_ulA=p!6q/BOX(.k1W/je{BkX .f~?>2xE 2ӻlp]DqUߵ;*x""D= _R]Ԛ_QsӳmƳ Nc7~'h;Lԥ>Ŷr7l,cl)&>5pfAߊ>}e7||# yoWXf6N }g0A_^}kg֗@Ee?$( `f2Z^WId?/>佳 %,XY(Fx^`{Vp$#A/]З5L?ga|TjḢ,gў$/ꬻ3;(8l~iFg{3y̬.cV#]c[/srbA?Jd]u ZO-BwoL|y~]>ܶ>| Q{~}{5s{)R~zp:ا8OrFa*A9"^ֵ"VO+~+D3+q f3Ue Ao{B^@\ιȬ}&B/}d֜oח_Ϟ[q0 wqwsܙ;CؖojQ '焂~|#yk Q Qha'|W\･0($\B<ƌm尷s}A{^ B9,p)H pչ֊vZ:o>޳QcoW7y]Ϗ޷*+~S/v> =z>tz=\kH7/5ao+l1~tb ʷڧB]Z?*$GŻr_ s"A?p˛˵(BOX>vXI=4n}Sl; c}3ʸ6k쑽.臿˳͂>CHz#8'V¡ΏaNwjsx(&Vٵ|Ww.^&!bϧ`F7x?FO/?O_6@+[1׎)7|C]Ouq}vϞ 1+lm^8 QkUqA*u 9"W#my$hߢVKD=_H='0j^~㬄^P=cȢ/|淜;bf}h\ُiPSWϏ/G^gF96T:G}!F-{ƭpHD<ݿpG ";߿_OA/x{\G IĵTbk=G a]{}IԏX70[(aDevm+'Nؼ zu M^mԽv 1_P (s^w?݈b 挄,g_[э[d¾}-ozv^ |%ۿ~F& QԷv/F9 U4<"<15)t g#vKn9gx{C{>ثЂo_wq7E`o]-Z~~`\-ys,?:>#nXebķ,_ᨾ~8=PV9=wF}GGoEV/^~NA9}p|7G"^3Ou¡5UV?׹.m!Ayz7yx^[~x~tԌ@|f7ϟ3u<~+uq/<.O~+hQ;ao%@"'ή5_܊yHԂGPA.*?ٷ`XD^>ھ7^ ޞ5SC"{5GeTasW].?۾rw[o?pzos>RO}Y{c3Db{~{Ѯz#(g!?#_랠ZS67j\p<~A[N kD~_bE"P' =A? ZhxUxC>+9Fz k)PyyȽ<9imdV[?q.R/c#{RGQ}-:w{KsoO>q7+ԕ`oӑb{c|3|=hgl3vÊ݈Ձת<%[+R5fR_="s?} A_ľ.sg(QE}SV?-d<+ܦ9'ɍߋXm\"}#[Q3Zcp63TKF/a~kq>tTbۯYؿUMz /bmz_ \GGu EDurY拨ϏI$@Eǂ~K_-"9zo=okkcT\{pZA=lwin۟=}@3 V EhX+[EeoWX?X +}g0A_WkG=rޗ=aOCD JB8 ?YBϼI_oET~Pr,qB^{;D“3?>2}YӾj=\gy(}5xxFg3r6oz(y[5s~-q %'!ߪAd۷>DD߼A{~q3A@VD] t 킾IC#JoGys6A s4rssx.u9{]w `KdR=m9$ii2~GZ>l8^%."oTXO׭\"nQ|B3ܛY-tO̷07<`S^ iϒg/C5(O ( pȨyP3?] W|-"O+~+D3+a+w0_'諿/$Y5EV}jܳ_7BP4ٔuZ[s s-X^uŶJ6=6oc?/"9 *X=jH|ckO%u|6@A_&Ǻ1a39SHs eK}j9Hgu}n~7{3>wяq?K_*?4gs9yA-g儂~ZڈQl Qha' Ʀ+RO[+wF.!ocƶ%OkǶ+菮:$ak@ԛF룄W/s]N״Rtm뛨zg+\*ʌmʊro}qAa޻9\l +g ˾@[p`79L^罾OP kV~yH@~rۃ~f$B8 YF,"ءT~Cc0o/OTC1Xb>74[z+αblc2') "h7$j~41W"gdt}# :wG}3n9r HЏb#\;bs9vm XWsov1MBujw~ `A أD8OlsF#H55G< •[]leXXA_D.臿˳͂>C=Eeg\gPĆ㜂hQwd+ƹGջv?-c*n^^D~B?O#kF"ybU|̊tV9`sb=4_eAnF }?cAMPC_EciPҙ֫G\%/}1כkz[X?Ӥ9J=Ooo'8yqA?]AيVH#vA`V7>a{yf{k߆ӳ1[~Y7?^+ɦEcs%'+OYBQa0/Bn,ڷh'3+A?y9*觿_o8+TB?U=biFus>(.>L'DĎf1EnQs^VvK_-qʘq-'R -ƇԺjNw$Q?bhGJeP;vSl݈ۂ{m'lk-uڶ>/k/! JS"\"5|:o~1#g1+4{+|,Nط/AZ]?(W_џǷ B݋zaM[}H\o\-o8xCVY`ΦWW\oz7%Ͻ_;CXr [l^n|<p> 97p/+k.-a+W}, ĮCD!~?`6QbUpݵo;5{P~x 5輍,C.`lf!nbی';ύ9ql@/'\iROl+_ϵ>31/~z3\#8A'\+k" $YynzlBЯR=ϣľ("y8|~/mZ\?g]c-ǽ_)5>> (NJ7 p \!ӻĻ.of2Lw=L؉ܼ~<87}Sm~~voUF~\[2ods:{霜UG0Χ~ՋOwH~62ky~5vAN,9^ :]!8g ?>,sY8w%p>Ãy {]jq >$Vr/NX(n/n\1U?[>Z_v1H +r\yUe[D2w Trְf{g\+gg'7WqǟO A=(g!?#_랠ZS67jeb E;x_]v;Yl_%G Ft,~)ЯoPWNGt^op ? +w#V_BÖBn ~H[I}q藛o{t>`j,9y+z>=QIzWֽy~zh^}}G~pڋ91+s6iE`g>{"g9om~lwfe/(96;dzыw2f̳ۜ,5LWK~sٶԾu<*g+5(y[acA|٘\ͽk=#ЋkFr!z{Cȁw؞Y| 1?򼸟-3Rٖoƙ}+چ"4wD{_M`-m"Hϲ+,n3s_/﫵 #I9o˞vA' !1"Y(7kp suIC;A˱9> p쵈1y78#A/]З5L?_RB\{&3C?tG{b˾YWWz{Hvb%xnJ^zq7tFɏ={w|q;>~)Fg;JsKo^u;c?]zx}kms9{ 9'oJ BΔouNgۺ?Lk#z&^Z\aoE^bZ*~}[?2g/ AUB"&A5p݊E/([9ӼuApBG|\| #a{à =a ,ybK9T΍ O񮵂@okMFN j?5|e=a뢌 ܇=A_|!Zey XQ(.>ybG_sl\=a+\済H\ BQeڮ^,`z欲1ݓHsFzz\-Cb;ڑ7[2?{ 9m4tcoOv3m=@tT wK<rH|D%^xB+4׻n{qkkVDu2wr7yY?\VBNc{ܳ1S3eGƫ>{!{5ٳg6mymݳ1:>tl5~tmo"hδ~P]q<`m4v[{B{^f5lz$s7[}Z+37CriOz''N  ve+[!Վ)Ywcߠtb~O uq}vOne~xh?':VMtuB\qi^ߔPViG4܀7 7' (V ~|9 0j^~lRϒp{j(N5")1Iͷ}Zy~zڜY+?~/%^yS cYk٭sؿhW|9ccL17o|+6!]t}VUc0=yWz䞽m'Q,իώ1c 覆O)ga펺ڛow}Qr`sbc"~fp$G IĵTk=Ga]{}IԏXzR-"۲{7`^ CZ ao!Dϼ7KN7!7xc\iXTp-FnH7H4- ?DJ?n} zX}/{xnb 6^`m2>kq=uވ8{TJ#KTroXFzs7NޞxK 8^'"mEᝏp9G<JXAz@rWC~1.DO ٷ/~CIx\_ȀAx|-,櫟􃠭szKTܭ]$|#_~c? \!W>p \!W>nz0=Ƿr{?=v7='o7vbz=y 77ϟcpcMTn_u_'{l#\妎^:'gtߧ_y0/tu=篟M_6}޳_]CNW8'NdH=Yë菽|և*]q4I-:;|H hA_k%X Q^$.bA5|6bG#;/|W> 8˶eAaθ*Vjm[ϔgOoz9U?@zQxB~FWT=A;`m&qoԜ ńv"66y#T/>wؾJS<>Gx8 J7Hks]n1dg3caEn@kUwzWȭ ?/ 9j⾄/7;FM߮oErM:sX $ASk/dEmSŇl{㷍B@lyw?ؾ?|LtE(Ȕsz8bR賲~w꣛=W}eݛwuNj=}-=go)V vW/[)q;j~\vg[cs<x~)c<qb\t{%7Ǟm˺9N{^[ǣrB/]6tMɵ!=ܻc|>f$OYǭl/ܼN~؍O=b*~yW[gm3-3 V EhX+[EeoWX?X +}g0A_WkG=rޗ=aOC.bDP`ov{ 榻9 AbŇwE4csd}|bkc|o\uqF/^&/kڷ~CnzujM--f~L6pc[_7}Ꝺϑ:wxJ\ܔn蜍{lmy<'w|*Rv<߼w2d;HXs{zEarN*F}dN)ζu/F~O/M~mmjm;ֽĴ]o磝sRsRT 'Ă~zIP c=;\"r "tG9 Vt4opof]?ܶs>W0HaO{u؂+=KCռs%^xw<@okMFNj?5|e2p޶}7$/k恀G.>yaȥ.=A(l=\X: l*a=4gZ׮2k-yc>.C^\9fOC:fڝd7óV4^}?յ׾IܴRG879ziHI̍9[}Q^||yuۯ8;I݇{|2xښ[NN D~s\ olkE~+$0nŇ} >@tT wK<rH|D%^xB+4:7٭@xx"nuͳqccm ȱNPFNκ&/6 7J;"UWil{6fx(xgz5ڹ=?zF֐9:{lUƾ-!;{6FgԇƏm(_k=}moǃFc+efPѾ^G1^M~mg\/9ss`!N5OǺNN6%A` VBS ǾA?@6\6잞 Z^N6u.B/S'HsӼ)Ҏ i~o/n,7OPT":Ar6aVA?z}s^U-bZPZUak+EIo Dmb\h=ybi{y8cV9x9L#Á=A=b>\\]oW_P湾= [LYW?7o{*A[S9'[IG|9c? \!W>p \!W>nz0=Ƿr{?=v7='o7vbz=y 77ϟcpʹW鳏{g?ն9a] lw19[ed;ǵ~m-3H{1Wc;Y}|:?j^4|~9~]g+ߟMWc"StI%Rs*c2?s~W\{*80.̋w7-x k!5ۋEL>φ](R2}deׇA`^wѺ];ս5WJ-{r MB:G'H~QxB~FWT=A;`m&qoԜ 8h'b3o=A î~G?Xap&Šy̩_fsDz_Y]y]b{O5i/(lK[ʧA#-{WceS vf$OYǭl/ܼN~؍Oε^19_Z[>5YΕ`,[6|#bmBooA:~]a`wx<m~A_~_Hqx{_ r<6?Lp fIB\!( Ys.Dߊ?/>佳}-"zYor?U\g$:}e }? YrzB\{B?tG{'Bƿo_U3^c9RX+^ QcϞm<_?Oe_νܒzXl19`~Zya^8BIۨ/Â遐3[ٶɯ>W[}~Ǻvcs8:!ŭc3c}㵲; AUB"&A5p݊E/([9ӼuApBG|\| #a{à =a ,ybK9T΍N7 wOqE{^K7񧚟97C:-ԼMna,Ue Ao{B^@HJq@Խ3P8έϨ^Kƫ ֈ4;^-zU~=&.(yebG_sl\=a+\8\jSRwG4}pmWš^/xY0esV^įIg9zv=Ɩ_kurUDJI6{:Ա7'zMrr}p>xIUKC=doo=׵Oenٲw˫K~q5߱UO>㓡ןC~co/q"A?pa7۵}`]͍"y{k7 þm ouG߁%Ew9$P7x !8ƹ#[ĴR8ë֠WhD85MO}9gkq}#V~^Ke`sb2uv%F=UlS͛9J li׾9-mijULl>gkɃf' z|ظ¸SYX{Hܼ-ҹisk74k~ZOpp`G IĵTk=Ga]{}IԏXzR-"۲{7`^ CZ ao!Dϼ7KѭD"ě-rS.̍jt#7ҭ@4tW&XzEE+?DJ?5kp*"rx+r6giWC~1.DO ٷ/~CI}{ >A'~1>T&9G*g۟xķ\Xp~}+A AB}+Ayz7yx^[~x~tԌ[Ɠ;1<ӛO18'p7|L?%N!ߦ_}i|0}Ɵ~^5ѳ2ONfZ+|Zy鳟ju~Ϙ|ۜ2Bֶ}{$Ա9`O>Gͫ~/ۯ?>lzej삜Xdrʽ9n18 Cq^E}X>TqK~]ƥb?6'n#9ZZ A,Bja7 +}P- a/@Qd9*J-uw{9kX3Z[33+8՚}9 $H?(g!?#_랠ZS67jeAbxPDAz}aW#Ur0z#K>+ <>xC"A] FVxvA_!~F;c=V&ѯUyaK\!!CjO Ga^XB 7 !~nv憹ܠ}E=7M~Fzɮ_bǃ_w%4;)^n9oea` x6;~l>p&JmJ9u=zbRY;>+C>jk^̉Q\ٖɷO+;K٫:^廧yN[r=۝Yl Fl]h>l6'-sMhz$핸{-8ym =t *JV{nbl51G6&׆=|zqH.dV[?r^yV^ocr|kZ0̂mg;/&D0xqgxwXx(83ѯ~a'﫵?0푔YuH7EH"@ AY8bFn{MbFoElپX=d7}`k{*3?>2}YӾjp, 9=!pǽZF:M=u_q7/ޙJ{s׎WMK/(g6}s/zǧ_}/l^ank|g,G˘[OctmT׬~cD~s$nxc]ǾE~+$0nŇ} 岩A~z܇ 2nt+B\qy~zڜY+?~/%^ySֲqP9 1|:u *_o6}zl)͜o%T6k_Ζor٪jv 6Q[ܳ׵A3꓅szuq>fl\a\]V),=n~HͭЬ-k?? -'R -ƇԺjNtaCeP;v[ۋ-{m'lk-u+o瞨׶>iP7Qx;eYn7…q^nF4tW&XzEjmi"{7o~y|/DQڽNx[/s'}55NF S1%UUtpvy*z,ŵ^>n/X[o`cȾk3XNٲn'翱߳=VjlA3]lǜ9^g>n} zX}/{xnb 6^`m2>kq=uވ8[lEZ?'y5w{Ğ߭~u &q7NJxRol~h=X;蘇3Vt[ ׵iu8|O}p!zbpu;|!BmLN/,kp&>Ŵ?|S~u<508?/sniG|9̵A AB}+A ABwӛ?᷻w]LwO8e<~{oy0yyͮzQ]=39?y WNO_m>՗vWjU=+c~ol—_L}ޛ>YojN`ɷ*# ?-kmkٷGߋMۨ FtӯԼzirϦWf?>ٯ.ȉE!+K?Uއe~>CUpa\>;"q?|wтJ8 bRIp]D\jnlk}̏"'Gv^&>q}UQzm5PYÚqUڲ)Ϟ ^!ĩٟO FD= _R]Ԛ_Qs.B{ㄢͼ%z^.A,c+ϥϭ/5~A9 J7bHLk. 2z@3ݰ"|5~C[ R5fR_="r6?݉H 7oq}=minM7Qdc+&Z! -zN~ݕXnӜj(>dە}5m㺘/M?/A7EzN]G( }V/N}t{J{.м.`xsbWl-ӊw}βW)q;j~\vg[cs<x~)c<qb\t{%7Ǟm˺9N{^[ǣrB/]6tMɵ!=y =>^\3 ٧֏s}n^'?BƧW09_Z[>5s`gh>}6!qWķŋ?ޮ~;NJCAƙ~ 2}YӾjp, 9=!pǽZzK:M=׍e俬zg+=Ds];^W<7%/:gǞ=[y; 8~~u{%%7:.cn=r<>ҵ9޽^Qq}ʷQ_aٿ!gʷ:m&˵eg_[[/g}c-Z[u/1`SBlFN,WNc=;\"rⅡp^Q(>G!ߊsc[8ܶ>| `z\=a ,yc ==q)A?xW'L#x({y#XFNj?5|v/~{3~YƂ~FDe<pvt(:=ҥgF{*P uL)5>:s3**/C5b'ΡWK^p;__23sG13=Gej^}︅VM;ks~ڜ4yL{ٿz:߉mkam֯˯K +{U9ɽU䰚44YCbnﶶ<~N({kۊQ@ Qha' Ʀ+RO[+wFr6fl@A_{lkZ3/󎜅7Սn% x+4Bq ;йQzg+Ɗ&VLo}mG f} '+~S/v>aȥ.=%uwLGmvZbG[6g0W~F3Zkjlю~Z!~/WUNTDiSH{xi^Gk$Gmn~)#GߛD_k4ÉAMs]XƜ-{}/IkW[=>T3읃>aM~ث9\Gv0z;;noeg =clDgaCqluG߁%a?‡JWi}<;~uͳqccm ;oed olC Tԫɿ#RuJ;gc&@USvnk=5duΞ=[յow@ȣ?ock{[=HƗAZw}_[mm yd靈׳=@́Wl[luk-0"K\o%tzp&dž w[q`?'Xob;>bf5}~=mǹm==cy㵿CAV7=!G Uov!Fz_ >nhߢ&ΛAr6aVA?z}s^U-bZPZZa+E4R"o&a>y~zڜY+?~/%^ySֲqP9 1|:u *_o6}zl)͜o%T6k_Ζor٪jv 6Q[ܳ׵A3꓅szuq>fl\a\]V),=nkM[Y[֪}6s8ZO"\9Z̍u՜$Q?bdnw 8[voMs+'Nؼ ZVތ=Qm{!03-}AӠnt+!&w~0To sZ?ȍt+x"M&xoFI7of~%rDaB=-Qd??o([orn$&f{ةd2D߀NZ?SPZ'},s גvwܩ5R#9[7{gQ49hr}&ؽ혳b30k'@ӭaC eqo;{=MLFޫl]q]Zc-vwH^۞=OnXv78 8)ᡂKAApND(BGfb휣cXEDCo%^NX\צL= p~jo#5@o19 yd@8?~ϋ<އ A\j9Rq?4|#~r}p \!W>p \!ӻĻ.of2Lw=L؉ܼ~<cMTn_u_'{l#\妎mgY}|:?j^4|~9~]g+ߟMWc"StI%Rs*c2?s~W\{*80.̋ w7-x k!5ۋEĕ>φ](R2}deׇA`^wѺ];ս5WJ-{r MBj;n@޳A/uOqN)XI52sxC"A] FVxvA_!~F;c=V&ѯUyaK\!!CjO Ga^XB 7;M1߮o57Ex}E=H7Z YlsxrT{G!ۮ^n9o@lyw?ؾ?|Lt(ڔsz8BQ賲~w꣛=}W}eݛwuNj=}-=go)V vWϏuwO3Qַ6?z;2ŻK}وmN[H+q%9l[qj:Ez~TؠkblL Y{⚑\>e~䞳s:b7>59ů5o]=6pfAߊk"w~K|[  c;px<m^A0ZRHqx{_:|}"F$ V\p,d1#|V7F}+bCƊ!sd}I ^ˍ<Wq]I.˚moP/gY s=՚[Z hrmƶn=.&e];s_5v!#u񊕸)y9%?ڦyN#\T+y<-9yיevs vͱ+Tr+=,9S9m0_l/^>z93>kqw{i['ŭc3c}㵒Sw6`#'ЫE oo[P8(BwoE9 -n[H[s>W0Bv`=.˞ZWO{<ױPŞn2Mye/4o{FNj?5|v/~{3~YƂ~FDe<pvt(:=ҥgF{*P uL)5>:s3**/C5b'ΡWK^p;__23sG13=Gej^}︅VM;ks~ڜ4yL{ٿz:߉mkam֯˯K +{U9ɽU䰚4gmcvk~sBA?[VtvߴX"m5G >I56]qzz޲_{7&WϷ1c c[➹_})w,.77-F<e!A8oWxܨwE޳QcoW&澶CԾA?t);Zc [0ǵPģ 8: l*a=4gZ׮2k-yc>.C^\9fOC:fڝd7óV4^}?յ׾IܴRG879ziHI̍9[}Q^||yuۯ8;I݇{|2xɈ~cD~s$nxc]ǾE~+$0nŇ} 岩A~z܇ 2nt+B\qYH}?׫WcƵUo%RC6Isҹis\[*v--'R -ƇԺjNtaCeP;v[ۋ-{m'lk-u+o瞨׶>iP7Qx;eYn7…q^nF<&n7 #ѤKϷr39Zh瞖(W_џǷ BKP䏷R97}w][=[o2ES"\[uLo@{ g'rQ\bmվU 9ky;nUTޚ^-yw=3kf4[>vYs5}z ǰ&`#Վ ..-WShϻP$msƐk=[Vl2'xCऄ .=[:z;ᚉsy8cA9x9 bq]F\34AЇ!a'W[A!$‚=gA [LYW?eo{*A[SsNJKTϾ? _o?vA \!W>p \!W>nz0=Ƿr{?=v7='o7vbz=y 77ϟcpʹW鳏{g?ն9a] lw19[ed;ǵ~m-3H{1Wc٣sVAh;N}W/> "_N_!}ʬg=9>{sr?xcpv 5g} N#K8YnO7Gr/NX(n/n\W?[>Z_v1H +r\yUe[D2w Trְf{g\+gg'7Wq5r'H~QxB~FWT=A;`m&qoԜ 8h'b3o=A î~G?XaxC"A] FVxvA_!~F;c=V&ѯUyaK\!!CjO Ga^XB 7;M1߮oE"x"[g ؛bw#=dWk/dEmSŇl{m\"}# +Q)9pĤgeRG7{'Hʺ7R{ݯN{1'Fqe[z&R>P,qgyS vf$OYǭl/ܼN~؍O/G?ǘ/~-,{{Jgh>}6!qWķŋ?ޮ~;NJCAƙ~ Ím}{\M˺wkCt?Gܵ+qsSҋs6J~ٳMGW_K1:۹Wx[r|Z32##]kc=?W9|VzX;=r|s:a\^=|6rg|> {/XzulfwV`#ߜ9^%."ox{;p݊EyE|+:gdaoЏopB):qYԺW-~ڳ乎-/ĥ'LsS^K77C=>ӢO+~+潱bA_^_'諿/$Y5?NtG >H:{>:>SzJN.x?doIsҩW%);{WChⲌ z lkYvWg;nU?ڜ6'M6m/4wk[;znG۲#w>^UNr/_9f3M>;nnk;焂~ZiEj}klz-Ae"|lo$M !ocƶ%OkǶ+=s-R>YP|\V7=ey+7;7]CleXĪmm&o+@"n>bG_sl\=a+\8\jSRwG4}pmWš^/xY0esVsIg9zv=Ɩ_ke_rUDJI6{:Ա7'zMrr}p>xIUKC=doo=׵Oenٲw˫K~q5߱UO>a?+?`{U`?'׎Vog'u>,[!8}v,>{h`8.M;p{>LGu[`Winėk΍t~um$6k72wr7yY?VXc{ܳ1S3eGƫ>k>ܞBz\#[k?[O={kcߖGf=?CVGzh/ʵ؍k\23hMgg{#]&ٶ3Z\a|EЗJY{Ank;~XPlG;o}:v1'8׾ gclB?o~_s]=ꆹ'HsqnWoΟMyHO~>nhߢ&ΛAr6aVA?z}s^U-bZPZUak+E4R"o&a>y~zڜY+?~/%^ySֲqsOrb2uv%F=UlS͛9J li׾9-mijULl>gkɃf' z|ظ¸SYX{H5Y(6{C淬Ul6pD\K%sP9g;H~ BApnm/V\O쵝y?Я֭{^B`gZ̯lۃAV7؋or72M榼q^nFnwĤtW f[d_-QksOK٫Dxw}!%(uz>ﻮv7):7뽇T9c(zv~16j*C]y7*w*oMHΖu<;홵zTc3tܭ| vb;9L>I=tcB{^vSj[h{\X+_Fb{(Y;Kno5N>I \ z u"Bw>5kp*=amI4: >\ {|=1fR 6&'5 8Abڟb~yxPA?:?pRTM29G*g۟xķ\Xp~}+A AB}+Ayz7yx^[~x~tԌ[Ɠ;1<ӛO18'p7|L?%N!ߦ_}i|0}Ɵ~^5ѳ2ONfZ+|Zy鳟ju~Ϙ|ۜ2Bֶ}{$Ա9`O>Gͫ~/ۯ?>lzej삜Xdrʽ9n18 Cq^E}X>TqK~]ƥރy솇 w7-x k!5ۋEĕ>φ](R2}deׇA`^wѺ];ս5WJ-{r MBj;n@޳A/uOqN)XI52 t,qg,{g9om~lwfe/(96;dzыw2f̳ۜ,5LWK~sٶԾu<*g+5(y[acA|٘\5#}Z=n=g{u#n|{k09_Z[>5s`gh>}6!qWķŋ?ޮ~;NJCAƙ~ t#}]쉾!~^|{gZDHcE9>ߤe.Ht$oeM67,jM--f~46pc[_7}Ꝺϑ:wxJ\ܔn蜍{lmy<'w|*Rv<߼w2d;HXs{zEarN*F}dN)ζu/F~O/M~mmjm;ֽĴNj{+PK<,޲9lzL3Ҝj^WcЯ؎vֺ{rr$BN= zD؛}FghwFƓL[Z=xT^&9jsK>8i<$_!N j'27l;E{MեGo^㸚ت'ugw* QukGlsغvmVvX>vXIt=4n{˦Zw_p&H#|؎ѭ OqőF7HCnW<[7?f~PFNκ&/6 7BJ;"UWkl{6fx(xgԧCZkdk Y{;'y{+gVumgctG}jV-Ҷ~P]q<`m4v[{B{^f5lz$s7[}Z+37[ 644yﶶ<~N +([ v|N9j#8Ч#lo{y?ls۰{z6v/+k55߃ѭn{B+47A7 HOp! ~7[ēys?=_A?y9*觿_o8zԫEL+S+=<\+ zFJSm߄8ԧ>AOs{7"k'e+o/{Z1N 9g!&]gaZbԣa_Ko{X1ܼmE 9>훜xͻb'um;yЌd!\^]|vW~W|J9 k۠& aG:7mnfoYq{т~q-Zbn|@攟ͯ +@'&0Tv ſ#lز{o[q=vP@BXf|zm Ai1mJu[E7[z#\Fn[i2{0M|,7þfh瞖(W_џǷ BKP䏷R97}w][=[o2ES"\[uLo@{ g'rQ\bmվU 9ky;nUTޚ^-yw=3kf4[>vYs5}z ǰ&`#Վ ..-WShϻP$95w{Ğ߭~u &q7NJxRol~h=X;蘇3Vt[ ׵iu8|O}p!zbpu;|!BmLN/,kp&>Ŵ?|S~u<50؟9Rq?4|#~r}p \!W>p \!ӻĻ.of2Lw=L؉ܼ~<87}Sm~~voUF~\[2ods:Q=:gtߧ_y0/tu=篟M_6}޳_]CNW8'NdH=Yë菽|և*]qXA?_;4Rx+9ZZ A,Bja7 +}P- a/@Qd9*J-uw{9kX3Z[33+8՚}9 $,gDKuSk fF͹ "^Lx9h'b3o=A î~G?8A>ooNVw}+oW7+ވ#1 5b{c|3|=hgl3vÊr3*Z;=l+>dpH[I}{b˾YWWz{Hvb%xnJ^zq7tFɏ={w|q;>~)Fg;JsKo^u;c?]zx}kms9{ 9'oJ BΔouNgۺ?Lk#z&^Z\aoE^bO:ѷ9)9lsrbA?Jl]Dv / "tG9 VtȜޠ ᶅ9Sp-tlu[pgs[_K &/f]k-87Лs=>"O+~+D@1~Yp*~Oغ(cA?m{OW#_H"Vk8;ktOju=}|twigTq%U^~kNCN*1N wֿ Beg]缿Pcf`{\X2ն:Tq sڜ6'M6m/4wk[;znG۲#w>^UNr/_9f s͑!˿&dk\g Ahڶ}wi9ZخI뽷žݳ <72x>PЗbG_sl\=a+\8\jSvõ]U^,`z欲1ݓHsFzz\-Cb;ڑ7[2?{ 9m4tcoOv3m=lGAOs{7"k'e+o/{ZQ oUBL&_nĨGþʗacy3[ 9>훜xͻb'um;yЌd!\^]|vW~W|J9 k-I \ z u"6 ._<=GcaHDXムo1ϙg1_c}mO۟͟_rn'mSL@x|}+A AB}+Ayz7yx^[~x~tԌ[Ɠ;1<ӛO18'p7|L޽~ 8O|{G?}8im'jg~msy/ºگ=cmsvk ZZfbrS6G笂>v>n5^|ƿE>nCYϦ{ rb}) 9kxa㿟P9+.=t)KCyc{Alsv.[тJ8 bRIp]D\jnlk}̏"'Gv^&>q}UQzm5PYÚqUڲ)Ϟ ^!ĩٟO A=(g!?#_랠ZS67jeb E;y_]v;Yl_%|Co#\9xC"A] FVxvA_!~F;c=V&ѯUyaK\!!CjO Ga^XB bt'"ك| 7ދ߸knM7Qdc+nLoeEmSl{㷍B@lyw?ؾ?|LtE(Ȕsz8!YY;J+C>jk^̉Q\ٖɷO+;K٫|85?i}kSg3-s}Aȱy9^ m1gde Md]c϶e=QT9[痮AEjM &ڐ]1>^\3 ٧֏sgBLŏ2}YӾjEX8^ FƔÍm}{\M˺wkCt?Gܵ+qsSҋs6J~ٳMGW_K1:۹Wx[r|Z32##]kc=?W9|VzX;=r|s:a\^=|6rg|> {/X^6}[?_2]8!'ЫE oo[P8(BwoE9 -n[H[s>W0Bv`=.˞ZWO{<ױPŞn+`ko;7Лs=>ӂO+~+D@1~Yp*~Oغ(cA?m{OW#_H"Vk8;#]B}|-Ry@Խ/RoN.x?doIsҩW%);{WChⲌ z lkYvWg;nU~.6XfõNtm\cG h[~^~]zG=WX٫I",؟Vyl 1~_5l{͉3N  zom[}> c;-lW$At[b{~EH\B<ƌm(K6zmMW^{Z~|ޑp;D7-f 0(4#"^ʨⷉU+ ? mG 4} '+~S/v>aȥ.=A(l=\@ۈxY0esVsIg9zv=Ɩ_kur刊zi_:fڝd7óV4^}?յ׾IܴRG879ziHI̍9[}Q^||yuۯ8;I݇{|YVx= 9L5zN$G19̭Nb7۵}[YBr;ocqa&Y|q\.jw~ } !a;JQ'H#xhѨsonG<1ʸ6kJX'(#{'g]ge^M+4Wz=3E@[AȻF[Ƣ}D.!By 9/W[gcTϕzU}i~CqjWALIjHԪCs札Z{Y/ʛ^EDԟ{[׮[-1Ѱ%fsط=ƘbnVm`CNli&'f`z8`={]N4>YH}?׫WcƵUo%R#|t퓍qqD\K%sP9g;H~ BApnm/V\O쵝y?Я֭{^B`gZ̯lۃAn|Jfx14DFPyt#7 {Ov_a$tcYJh^۟{Z^ |%ۿ~F& QԷv/A?KI{uMlͶSo5eL NUo1ſ]]=$~ާʡ vGqףO9VV'X%Z̻1VSykzFrolϬգilsnL{1gV,]_g>n} zX}/{xnb 6^`m2>kq=uވ8[lu՝sd|3UǨ^o~N>I \ z u"6 ._Lst:xˉgqd!8?p57B}B 7ژ75 8}AbڟbxPA?:?ϡ?H O7p \!W>p <>O7|+owqxx20zc'דraz;]zf/rᑉ߻Wy^#O7\?pzos>RO}Y{c3Db,sY8w%S IΓZuv.[тJ8 bRIp]D\jnlk}̏"'Gv^&>q}UQzm5PYÚqUڲ)Ϟ ^!ĩٟO A=(g!?#_랠ZS67jeb E;y_]v;Yl_%}}? >)ЯoPWGb^kvfzff;gkUvzWȭ}Z0y#',Xs? ӼY'bL9Fd}ow豀HF7_bǃ_w%4; Uv"9oe"}#芞L=#$>+z>=QIzWֽy~zh^}}G~pڋ91+s6iE`g>{"g9om~lwfe/(96;dzыw2f̳ۜ,5LWK~sٶԾu<*g+5(y[acA|٘\ͽk=#ЋkFr!z{#k_=b*~yW;}mfqfAߊk"w~K|[  c;px<m^A0ZRHqx{_:|}"F$ fwman.!)}^|{gPr,q|bkc|o\uqF/^&/kڷ~CnzujM--f~K6pc[_7}Ꝺϑ:wxJ\ܔn蜍{lmy<'w|*Rv<߼w2d;HXs{zEarN*F}dN)ζu/F~O/M~mmjm;ֽĴ~kLh圔꜔9U 9^%."ox{;p݊EyE|+:gdaoЏopB):qYԺW-~ڳ乎-/ĥt\]k}߹Oל!a ~j^[&#\U{E Ao{B^@@ktOju=}|twigTq%U^~kNCN*1N wֿ Beg]缿Pcf`{\X2ն:Tq sڜ6'M6m/4wk[;znG۲#w>^UNr/_]'Cl b{3asmN9pBN({kۊQ@ Qha' Ʀ+RO[+wFr6fl@A_{lkZ3/󎜅}n{+47M-75GdqE޳QcoW&7~FϏAhRNקxC_k}'l'K]zjPTn{+Ӌ%uLoٜU\={Rih^]eZlG;}zk]\9b^!͞N="u>?;#ng{i~k}i\4q|or}Ґn'5[ubs&#_q\wlՓ$[5!ɷ#49_É(:µ#6Ilv}+;VHna,Nc;$:X=eS;/$d>lGtW'H#xV inG<ʸ6kJX'(#{'g]ge^M+4Wz=3En} zX}/{xnb 6^`m2>kq=uވ8?usl/q^PʽbBcs 8)ᡂKAApN:<zx#\3v1g"!/'Viu8 Ր߰G k jcr<׼2@}xir?mC hxj?#w+@? oH`>ムp \!W>p \!ӻĻ.of2Lw=L؉ܼ~<87}Sm~~voUF~\[2ods:Q=:gtߧ_y0/tu=篟M_6}޳_]CNW8'NdH=Yë菽|և*]qHA?wfH-:;|H hA_k%X Q^$."A5|6bG#;/|W> (˶eAaθ*Vjm[ϔgOozTkO' @޳A/uOqN)XI52x1儢ͼ ~^.A,>s7+ԕ`o葘j]n1dg3caE9kZrk28k-{D&ȉ%m? "Ow"=ȍp[콈1mFd}E=7G7آ]6ͩBU]ojuY` x6;~l>p&۱(Ȕsz8!YY;J+C>jk^̉Q\ٖɷO+;K٫|85?i}kSg3-s}Aȱy9^ m1gde Md]c϶e=QT9[痮AEjM &ڐh]1>^\3 ٧֏s;HN[E_oɁ>#uhmmYзm,BsGڄ]/"n,{X+}g&W/}F=rޗ=}IB][,ft]1BR4csd}>|bkc|o\uqF/^&/kڷ~CnzujM--,~0W5lnlƿo_U3^c9RX+^ QcϞm<_?Oe_νܒzXl19`~Zya^8BIۨ/Â遐3[ٶɯ>W[}~Ǻ6 qth圔9)srbA?Jl]Dv / "tG9 VtȜޠ ᶅ9Sp-tlu[pgs[_K &/<ƻ Zqn75&~#zz}XEWV9C/ pU_'諿/$Y5?Nt He RΞXTzJWY:s3**/C5b'ΡWK^;__23sG13=Gej^}︅VMtbmO&{6Z{X;ѵ-s=7m{u;\ae*'⯊Y5˵͑Ab_v۞ksrBA?[VtvߴX"m5G >I56]qzz޲_{7&WϷ1c c[➹_})w,V7һ7-f 0(4s9zVFM:_PIh=?:I[8Yÿ_N_ }96sD.w BQeڮP2Fģ΂-P ~F3Zkjlю~Z!~/WUNTDiSH{xi^Gk$Gmn~)#GߛD_k4ÉAMs]XƜ-{}/IkW[=>ijӚ[NN D~s$nxc]ǾE~+$0nŇ} 岩A~z܇ 2D%^xB+4:7٭@x1̭y2n,~Z9 YfA@pYWG 9vpLϔ[+xhZCI^~ٳU]|<6Q:?գ^|Txn>XzОAMFn:{=ۣ z5ͶVq +^Vq|[:9ЗpJN +([ v|N9j#8Ч#lo{y?ls۰{z6v/+k55߃n{B+47A*BM4܀7 7' (v JdtA?y9*觿_o8zԫEL+S+=*l zgJLRm߄8FVE}6l-.oDOzW_,"ުLv:kQ}/7þa=6sfηlrڵ}NgK79lU5;wŨOvB^8360z+r\N~~FɁ͉-'R -ƇԺjNtaCeP;v[ۋ-{m'lk-u+o瞨׶>if!wo7 +.s^w?ȍx{i2W&XzEA9Zh瞖(W_џǷ BKP䏷R97}w][=[o2ESS[uLoW{ g'rQ\X[ϊs5l#Zrޮμc;Wj$g˺y~Z=v:MVwsf5}z ǰ&`#Վ ..-WShϻC]9Gϗ8C/\joPޠx 8^'"mEᝏp9G<JXAz@rWC~1.DO ٷ/~CIx\_ȀAx|-,櫟􃠭szKTܭ]$|#oNL@x|}+A AB}+Ayz7yx^[~x~tԌ[Ɠ;1<ӛO18'p7|L޽~ 8O|{G?}8im'jg~msy/ºگ=cmsvk ZZfۘ>X"LY}|:?j^4|~9~]g+ߟMWc"S 9%Rs*c2?s~W\{*8/^;2Rx+9ZZAX"qa7ы+5 Jkwf~tBn}Л0̫."Zk~5;Xeo=S?Wb0}π' @޳A/uOqN)XI52 t1儢ͼ ~^.A,cd"A9 J7bHsWhvA_!~F;c=V&ѯUyaK\!zͿٟI/F_w63 8oEY ;kE0^r[YvJA`ۉl& j-ԇ={ܬ$r>Y(7kpo⛛$=FHJ~Pr,q‡/\,{-bL.Ht$oeM67TX4^ FƔÍm}{\M˺wkCt?Gܵ+qsSҋs6J~ٳMGW_K1:۹Wx[r|Z32##]kc=?W9|VzX;=r|s:a\^=|6rg|> {/X:H++DG;/IÉ *uyi'pVd-B>Q|3-n[H[s>9X:qYԺW- ͞Y/lE &J7޵Vs97C:-ԼMc;G2p޶}7$/k恀G.>yE]zzwBX{7FBȁۘ}ZﱭYC\q?cfLOto[~u`\QiŃ-7ӵ}G=[5V6|e~C'm&>DXÿ_N_ }96s$MpmW0"UK<,޲9lzL3Ҝj^WcЯ؎vֺ{rD^BN= zD؛}FghwFƓL[Z=xT^&9jsK>8i<$_!N j'27l;E{MեGo^㸚ت'uIZuiMag&;|8\Gv0\7ݮ X>vXI=4n}S5(Aa(G#EMv+47#[]leXXB%r.臿˳͂ R&Hr+=");ʷ0^{vnk=5duΞ=[յow@ȣ?ock{[=@ŗAZw}_[mm yd靈׳=@́Wl[luk-0"K\{4N5O}6%A`튣VBSdNo/7ُ|~=mǹm==cy /'fۣÇNn{B+47~#yA_nfoMM?z ~rzUO/^q6F\WgI=5VzxUJϔ۾ qD;nj+kJ>{fӭowl؜H߫WK{|9II$t'ʓ$F$BZN< 'k?iFkdLgO~)w}wwwy^(w<pjEXZ#c7~X ,RYn3zH{\;ADynv_!d/??_;!އ2R]>GK(?/1ꗯS6Bu?q+JuODyimBcZ }`W`F+aNT_1g `ÃdgUrpz~|=i6v=!ЯQc<۬2p }V~ ύmC}hC~#:.rƷ˟`>p~\ }.>@ D"z^_zq3ݽZ]tLt~s7ݼ&tn}܇=}yiæ55{qf{׋7ӕ#ړ٧?4w1ObW}?־Y]w!Ҿ?.ˮʏ+۵Cvx1Z=z.muu> _I} 1}Un' `z?MCO?jzQy'??z>}ϧo/ݣC= )m SE>CW}"syN蹹8ڹLJ@߅" C1^B\w \P\N۾6Mza(lZcGAyk*70km|1еyj?f[xX !B}Jl_N?dCqp/;v.lm4zڰi蟤싴Geuث0@u=m;^z@kg(_f cVgh07zX/?j^Qs{译+E ̅57ӵ W-V cvKW!dk#y: }Q0qc~߮ܗTv/8F[~N%Nz7@c0 ,w\GWp+Trkםw:*8m9+%Yն8:,.{O/GVOޞʝ-e QFuϱhwQO3p,ͩH\+e|SSܲ?B1fv9(qaUvNJu?}2p[}x~XzrnGZsz-\kz,m}HDž]k\dk8סF k]`< r!́t6"bEpQ7|L vwHT>@c3]/]C//Sp 9AGI҆Tw*(1؈R_1|RƤ_0}W1_s8}q=C0_ pq_[2A^>Q^Urj_X~w^vu%uwcW/ <.~WF}鹍1-F;Fr}v{} k;.IrL1}u yq_+uȚmlC^:'eE?v/ۉmN~Akm<1 :SRZZ"0XC0q%~z8| AD}n66}})0}p߶}7/NNF0IT8G)lr6aQ^Sb7jT[^nߣ~Ÿt-rquhͥ1_E'Ikݾ=N/Ov9zr>f2ʲE]V)ҕi~1aPwܪ5vǪr䱾Y.{^ZXgPW1&i-_9l΂sHq}~MHa11}Zom|>512.< q[o rCUJa^x1/@ߍʹR4 }B}mΰ>V}5ė px Ks=n3B3[e%b_n$ ~"YF >hnNG{ʑڎW+ۻcC.Qw^S+z :9 rĵ֖l'פlgP]q\C9ֺ;+#Tl͗ӪN1A޵-wrz>GrecSs}=F۫1 mRWs G9ٺ^e0fK~]mmseKK|Uc5)6)0o1&՘ 1v CkF쏵olm)۟*wmBUs=GP0f!/ GCF/u@R}X!n*+1b:)B18 N2jTeְ6f9<制:uǶu+kĝ{bny|o^ĵb[m=}[e:4ʪ,{|_ZbfGur~]FUn]YgNzZeֈ@W5lWV|}7fXI/ 뱜WcLVK{!k3a!s|{9z7m:a~+n뺲cm藛Y=n@߷^݇v%D_[AT___ H/6'_//mV( ?Zm9?@?`)iy` Zz~BBy=vvִF2ܹ'oM.=9D[[1fOj6]E5mmc.;+vofڒ1*j {z܅m2f8Hu~ּ6\3/_X[Mq[3y]n>9<zL=& caiy sC*+?_n c&Prܾ#ʮ~?Je`Zֲ#=sAzZZdYN}|!;|o!U\鼛}B$}uzI/K miB+yn}"]P_{<cI{9[WX[e߮avs=}^]n_h B1-L}Hmc^k]fV8w[eFQUeuØtuf@:9?ʊ5mkUv'_Fy eֿ>Fu/F(=x|o~ӽquӯ1Vim]Mkmn7] y/ ֆ]8wt>oS}m\G<1C5&:U:4zb @F\n#X)?7~ }-Sz eDIm`>p~\ }.>@ D"z^_zq3ݽZ]tLt~s7ݼ&tn}܇=}yiæ55{qf{׋7ӕ#ړ٧?4w1ObW}?־Y]w!Ҿ?.ˮʏ+۵Cvx1Z=z.muu> _I} 1}Un' `z?MCO?jzQy'??z>}ϧo/ݣC= )m SE>CW}"軹u7G]; Z>X!q`7K_BC r}_O|< 7B|>(oYeT~sm86bPmQlo+ !=@S@/^V'L(.e΅ͼ ~Z[]]6w} );:.@8=/7^ѽ͠Օ>QmSZ\;C92-eWjS=;[R7c^/y^gV|=&KYjSVBys?eӹebZsP6\ê쪏9'~dηBsk[KZps~ݎZUkˏsf>q>n7ʾJ:'uh?vZEqs >`X\v u?l{m]5P،m pF nk0C}n~A8QAR!՝A 6ԗp;?vpԵ1xk׽LU mNAuq#kK~=Ы'k*[nW k+μ9.rAuǥJ㣯==\w:ƷhhSn/wmc_{r|Ϝ_ɹc,3u֊臮IcR.aX~D1˖CWrj|Zcֿ>Fus˶vy}ZוGll@hKcǰ|Lvh@"l]B^v v ~[>WIovbux:re{~i[5k 絔LDxa06yN&  =y?O >q\nh PTپU+غW@m{+#߉!r(z kO:>f0%הW{9V()h迩>FvGrq[rКKc0N ׺}{>_Zr8|:e#eeu5)qR+kucRjUY55,;jUc}=,=e]m]ΠbLZ*r̜ɇc> &^sюZuli~lVo|!Q*{%0B/vXm}`i\o)[_>׶\gX_{/5e `P\kdی 9VYۭؗ*+IuHjGDۃAjzor䶶c ǐKԝԊ(^eN2q-Ǭ(+5)Thq t_5@~{ cv BpDZ_//+-Ye%f_V:E1^yo|u \UR56,g6XgV;qzeso^s: WUlmq[v޵[FYU;eZKAlnWn۠˨ʭ[=2IU^7j֟QA|nޖNyɍoO(v} GuXbgc_d? n({} 㺮Xe:2_Vh[/2h]/Vb#՗>**mBVp_p[0"f0)ՖXc^M /ϯ8|Xg1kN 0ꬕ0I׍G/*ChggMl/z_ɝ~Ɲ_s~VMn9mQozE[|>[ފ1m/-YbΎְ]a&km['j͋m1e̟zkSn}rcy {Lo#W+憠UW6ܾA @W=Lf}G]uףeGz &sɲ|X睔;~.Bw2<|1.4,B*-y7ӕ"^a/җ^[hն` B+yn}"]P_{<cI{9[WX[e߮VXvs=}^]n_h bW?Ε|z2R[Z׼bĸUkl9o>Vٳ|fU:np?stݹ0f:,]jY5iΏb ;es5k]ɗ9z^ByQ|rʮa{M__y^ݸ:_ׂ6|_zs!i >A0`erڰ ·u}m蘇3\yADҳ*\ǁF98=}\v?a>b~Wܨ1om8>p~r?eJoϡ?w@러W6xhw#p@\ }.>@8gt޿֋ngj?慵og5yw˧>M|6f/˵3 ݋^G}.מL>yG}U~wYNNiS|r/b}l}XI8>C@pޞck}^P2$0z qKpBqu|t;Aoڠ7]h1د`CC -jm}m`${ + 9D2c1ÅBڹOkK|끾kæN/q6}C >pxCޠ.{Fh07VWGMq_hq L]~̪ CkVAs=jbnO\5~pٿafv_߹/gwKW1r>p -߷:E<;UK&Qߠ|d 7?eF~i!c^S·g:/5 Z)uݾkh>/A-먮 N[}zJecV-A}9Kk<>oޓǧc㓷rgKYjBfs2Z]%*#d)Kms54JwT;ol{:P@]r y܆kX]"Dݏt2Vhnmk_^_37|ܯۑ֜^N?kY}eB[]k$+*o8סF k]`< r!́t6"bEpQ7|L vwHT>@c3]/]C/mKp 9AG҆Tw*߇1؈R_1|RƤ_0}W1_s8}q=0Yf8Ǎ8ǯ-fBb(nl]/qWol;X;躻zʱ MV+k#s~{Mr޵Jc|Z>s>R|9~$Z<ΨZ+^'IѶ޺bWc 1m+t.[5˶^^>skY[--"=iĚ'4&rL1}u yq_+uȚmlC^:'eE?v/ۉmN~Akm<1 :SRZZ w^X<q%~z8| D}n6}})0VuV/FC\Q^@5i󓎗!:Αk+潜gvXה_MkT[^nߣ~Ÿt-rquhͥ1_E'Ikݾ=N/Ov9zr>f2ʲE]V)ҕi~1aPwܪ5vǪr䱾Y.{^ZXgPW1&i-_:g~E&=FkBz5Wq zk[!@!-vQOofzK=>`7B߭W#kǏն~nVQU<,smu nyr__/ e/exKY̐Co}ݪrT_7dv4# }WӋxCߕ#ۯVw8\VEE.uTHX?ZPkKYQ?VkR3|y_UYJ{]١M|ko*ˏiUҘ{UO;scI=V٣S9mn걩K޹>՘GkΜl]kMrln.ʶ綹%o{ŚpKV\gk5-1eW0Ƥ8N~sVܶ:صC-1esW }@jǁH| A0 %/ GC]$۬/v_s"Jݍ}7?:?.ANU` kc3Qh_3YH}lNY׸_Fܹ>s: WUlmq[v޵[FYU;e%МKAlnWn۠˨ʭ[=2IU^7j֟QA6ɍoOXcb]XuPl ֛GAme[w{\וk ]UF q]F>+% BpҼR@[@  E[0"fBrՖXc^M /ϯ8|XgKN 0ꬕ0I׍G/*ChggMl/z_ɝ~Ɲ_sCDckӵ[a{[[^6=OalnU7eIDATbL6kK*c5qzkXɼe "yZbp}t~bm61:orɹe1c1/5gttC\KcdVq^qLr1t_0RQvSQ*Km:\qo?Оkǟ7Ӛ(ϭׂ'm<`wR!Ÿ|ӰB|NLWx$] M}i-X_oMZs+!;g}'_ϗsN.D5l̷8 ȄON*vk'?S5Pn/cQo1~ԇ<5o;1nZ)1[Λs*Ul4žYUv]w.;KZgV=q cXN\ ZeW}e}Pv^jc/_bm~㮵JHuODimBcZ }`W`F+ua;:ᷩ6crj!HϪp=q1{ m{B_sĿy.Yd@拟?އ>Ж)?^2Fu]$|o?}@\ }.>@ D"t{w7ݽf{s==`wMkڻ{g[d@kOgǼ?}__}'WZfu8Ӈ_Kޛ>,>f+??le7=SlWvdk\L}.$<9&)Uy4 _?>'ϯg}|Vb]9_'[,7w~?~7{19X{O7Gptk `U!^F/!;~ ]c.B؂?wC(n'm_~=0T v5fQU̵u>~<|AEB !׻*)O\v+qNˌ?>=}yB_fz3;m3ė=/7^ѽ͠Օ>QmSZ\;C92-eWjv*8NZc-f"pVĵn{@UFGg1.+`fzy_cz{Իog/ozZҁ|1WaV5t9:tNx~^ۜY(K2׭yXcWu@?.D'8+2ê*0UmFe*AO-B*Uv.PkVV/FC\Q^@OHB^u=m̶ځ~^ysqX _~z-o~/k9֑L;Ǯ}kW]_o0f@[:t6ߴr?` lq zrx+7C[\(n^;~ >0wr-}߯aPk[3ma{3Zp`GpEYNE;:[e%b_nWV;\dո }WӋxCߕ#/,jפZ~[9^,{UndhsZ@_omk IU@7^֜ 6e}"ƟP)a\a? \W]Vv[E?>l޵aCW8o>@a<r}ul*+1b:od8}ncqhYYqo@ߗ+;+??W7rcRu>̉+XuPl ֛GAme[w{\וk ]UF q]F>+;{k[Ba7?tFUb!Z>0/m:Fb"Ǽ>[ǿ__qR~-5'* ^_tʲW.܎ױ(Ɋc^?/Sl{e6V~b_59gb̙̜ZNp0~x>lx0caiy sC*+?_n c&Prܾ#ʮ~?Je`Zֲ#=sAzZZdYNJg"PZ|!,A^B鼛 :J/l2uf_Zm rL;Bo^|3uB}]ka3{ u}t}}A>^Î\ǩYQ~u@YG_-cSy ϟ~ƽ雿v'_~+m I9^Q~O=s6uk,ך}}yq}pS?>@ D"p@=nW^LwovWw;]?StIdzO?]}g?zyO?~2}oV=o=}Ï˲cƾovPv35z̆zeGvhϵd[|OcRoL_I޼``z?MCO?jzQy'??z>}ϧo/^kekt%; [}1o}^P2$0z qKpBqu|t;Aoڠ7]h1د`CC -jm}m`qh{ + 9D2c17Znhk@uث0@u=m;^z@kg(_f cVgh07zX/?j^Qs{译+E ̽{3]?wj~S}x+8o {.X{}5=[}\-2@"! z߷:E<;}^B=qj@'o8e kc|­G')[e[2o[=Vki,m+MU7}Q W-}^Tāmg47nXQ!ngW+ˎgrXm͵+@F ]WѼ"X.r!o~PLkmKA 1ӁBl9 6S>L[ܗqpNi%K[E]/`֧b>wop|chk6CYukou.żH|pB\o86c:^ӻU?˲|c(Wio}u yq_+uȚmlC^:'eE?v/ۉmN~Akm<1 :SRZZ" !٦@ˬ NdI9| D}n62E޿moo;1_U,ѭ7Ӎ_ao[.X^nϪs84s;|#O_9v }jCgM `'x3m0!V+zj[_}7N+zK(*:Xne32!x>v{AUV"vJR}Гe$uh^|~!_^Ga_?a:zEmԧ2DUg}J2ƭ89neo7,_OYNnGN~sVܶ:صC-1esW }@jǁH| A0cmVYչΚ@_put8}eqk5uםe-{ߏȼO%FY@z@Mvu3&\}6QX2Ƿ7ُ|ۦ#^+?vN~ }>5}(`W c]1l{>3*|seϝPl[(NkEy5}6Џ/.{\)~|o|Ƿ779eٗ*om~o"`_g΅S}O )O?9 p6G=4=g_xGmї{!p?V.YdbceF>@_[ѲE_>tx {o~ZO7w˧ƾwtsw7ݽo^4.tes 7s?suJsc_LHyϮ[+(v7@?-@~qsC'v~f;w2п乾'{pj;9 !>ovzwVh9gs!/@8}+Do' ~ΟϤsuY?.пzLǵUY{y|OAhOGÍ&Q}$}c,yܼrTs nL\eB~\t9e>WTwhu^{˼cEte\?{pj{1kPqٯQ9/p^'׸@7V췫@?PjX;mסǻ9fhk1q,Ĺzc[{9?/Ǻ1s:kƚt~-Gﺐe[DžmiѾSCq",C1m=o~zu븭B\؞yK#p: sr :|;9!ף4C޸dpj,ԘPTY},04]=*@]qm?+͐wMj,\_/jxL.q^\?Nhno5~͹~8>v g:eiʐ1Aㆠ-}ׇs[}:܇_>h2`#]q5Ձu^Y y~zWc[𾌪mز]GUv!ݾ~Vw} }/ ݾ*$q~3roRy;ݼV~Xuٲ  Pu9]uK0j,~6]Gۭ貒.ѿc×rT۬9>뽵heoо{?A[S潹exZ@?}^'/BU-FF7/TGRn,b39el׽5L ~q]a+mF\C5n~w>l|n_?|:Ju"t?e}R=@Ї{·@A`o2ނIw>!`x'?sp>ĽCMp>ĽCMp>ĽCMp>?M/Wtzzj{vϿyaҸ7ӕq)\|8ϽpCh+u̡ $7gtknp臾/n~Nό}'N<}Nm@?!cw~電Fh9gs!/@8}+Do' ~ΟϤsuYu?!/eF} B{j88n6y !kc1ϓ浐H+Xpc(_>mq-弤CK}X7+u|,d}-OhS;ЏA ,^0~xHr8 }~oD[[A߮@IcٖB]:r瘡y;V߲\DZ1knYX<>(m\k>;Bm[<9GN~Qa O e(vZ ra{{/9(v@˶'u] 9-V\Wb!y\I_u6eTmS񐉒%+?r>x Ư9GڮL>,C"<C2ݱ>htCn^mas6~U wTye2 u 9,_Ecwn2cv7UمPv[G߹w/BDه"4ؿ2w|~ ߤߵvy% e+@#s|;Das9Xtrm$%[me%]{·/51kYss}{k5h)ޠ}Nm@惶`{s=] 4ہ~cE}*a #s#CI7y1fb{P2Pzeeư|[6\_F#.֡o@;ki>Zer>ZsO >ԎVtPqPoĽCXo7[oA;kp?0O9@ }8}h!&}8}h!&}8}h!&}8NoۗO}n{s==5=}y;߼?Ti\8.o>~8G!4:P~7{1=nm^8|!@}3|d N?/Z@g%>C&=z7fn,֛:t>>ssp}/?2_N?1M/MhRσtX R`!h#\;,ZFXeDɅ8g"vjhy?Fh\~9&@."/uXy>z}ͰZ:R܎op<Vgc$8κkΤ1YǾW}eRay5p2Σy=aS\]O ](x'CqLʐ1A_L}}>0DžIFWBk uAH^i6oWcaj|UT/v)q`ZhXbV0 _hBsXx"™r \JkuH@u?-k6臲,l+X6zZ3!tc VbzXYvnKnvqv#Ls"[C6Uq>[ۃ6 ~Xetmۣ֋=Џ|}khUѬ6msy9f@I^!oV CX4n:g0V;jncZe{j\Mu kW!Cޠ_@òX4,p!/j[8lW|qQ]eUy{i_=kvlx'ӿyOg HۂmCl\~YG 꾔!Qm0f[)O9 rȮu8ikYJc/>aXЮxncֳUZL\lH{^X:r_̳\߮mqeԿQm9oSsu}c51]}IOWcݸaU۫a]m祹=};!K__}UH>.6cߤߵvy% e+@#s|;Das9Xtrm$%[me%]{·/51kYss}{k5h)ޠ};wG1_?7_Yv!XE)N;ɡD/ikDkSؒۻzl} iRa.p6lol67{XZ:cm{^K:lL[N]I 뵑`Kw-UuY{ׂ#vDžsܵmc:qZX-\+׏*_w v>gE׌r}(azY3~}x6}uXnZvݽ yxnY 30kޛZU?ޏ?;}*a #s#CI7y1fb{P2Pzeeư|[6\_F#.֡o@;ki>Zer>ZsO ?>K[cџ׋Q~S.H" \Ǵhp1 S0W6h_+Lkplݞc{mXz-]?Vܘ2x'a4P^nC9L߬ubWp_}0ӛZcڿ=oA~z}sn2W=8=֩kf=}^?.с>*qjq?: }Fc޲ߓ{X_>[c`@M!̇"!úm+;6N%1.S)T_zю[~KJ8#zjksf 95?mg8fyI/s<}ggz01BK!/]/64@ks:?]ze!P7̭qR5TomrٗmL+hZPd\\[,+::NC mb8g=d%P-kYr:aa~~ \^#tN?Mۡxk5{'4ung}{G횾9rcA[몴RkU<^6^EY흷Ǵ跾F[b}cueOM<%"`x'?sw.$?Mo`oKTXƐb'0)!a޾mP DŽ2v/pN:v_ys:ݞ4 q]{zRHoIm]Q.ۯ76kh^zǬ1wﳱl߿/%+ kX۬U[ s]KZ/1JZ!ֽۏOG_o߳ǟA?4 6ZF}m 6:0 ot_{xYK9{k w?aU&,ꐲXϺ=q4ۗgK^/af8`je=&p-6u>k'>kܻ%'[٭CFZ?cE[\qi[LqI}7&>AMO޻_?+yYޞ QYjM1H1ڷ08IF!cO6l D2xY8R_¶-P #ơ;hNuܥqk[:^[cz~8ڬ\7_d6O vrӚLxoS{{CC(g (ւ5~._.3ucǥ?7l=1>e\GX?޽9ۉ@NqdjM\6촤-$'b2׎j]Vhi2MV[0l4hNxCVZkx_;f=BǞZBµ!b8=ޒ:v} ǦyMa1e쒲V{F1Hm!z-c"[k\vuord0^Ro,CN\}h{~'ۂV`'NR(O.Ȱ$'޾5pm+^P]joHt8OY'iKZk724~{ [Ŷ>}ׂ\߮/ش{c&h&v,y,so^R{Z=d0 fs9(Ʀ5:v֮Wsjn} KBM7?ӿWQT B:NږCX剠hXֶl \R9@ΪXq@k_kb&sc0S4yܖkɨ}Ӝn cbe \;\xϩ͏Q\߮/a8/6[q s'ǡ<7/e|}yjؖ݋}(^V[\}h:現Խ^0QK!ր-L3VRNZ}91VfF}W+hNuy*I^x=9R_ c=Sxr6yJ3{\֏Y+٦8ܷ_Ͻ6eH-ǡ:9/k%in]WZ#qC&c֡k\jC^v=٣,?M/Wtzzj{vϿyaҸ7ӕq)\|8Ͻ/A~1V>\ uR4qHLVn?7H匎 RׄMش(z0?~ ifgY;m9\7Vϣ9"kPPwL_$c/Y#Q&翿֯^~YW=Io^keR}}\fP52Z}'},&+ Rެy:hݓ[m'.с~ s:{}9q?_׾=՞]OVPo {~{/>[Ntwbds}O9?>v?l#³F ~0 u^8:~N!e bړB^Nk?4+v¿TtjVغcPid8Cs64_J8"i^u)%^ݧ{_Ͻ6[qpԺ4+;S{mXO=m˵aٲƵԎVbN\bzw$fί?=oVh9gs!/@ݐ5S)DGojyh eTd9tni}=´5!qt£Uǻ:n4i^/k7~Ucl[k(Դ'ўe c=_"?azn=VԶ}y{Κ~񿫖Y\x[!b|=op,&=J3~jev ?˟~//_4v1s1oq1sk;LZB_q>fx}'?/xVa~>ǜo>"F{1kPqٯQ9/p^'!hT7V췫@?PjX;mסǻ9fhk1q,Ĺzc[{9?/Ǻ1s:kƚt~-Gﺐe[DžmiѾB½GY;TY9=H0DžIFWBk uAH^i6oWcaj|UT>4q@>>4q@>>4q\tt~s7ݾ|j{\M7wwݛ驹n>慽JLW1py_?<>9:aPg1:2o_fz:ngӭ^  9!ss;]?3;_\ߓ8bzw$?oVh9gs!/@8}+Do' ~ΟϤsuY~F ^o-ӵUY{y|OAhOGÍ&Q}$}c,yܼrTs nL\eB~\t9e>WTwhu^{˼cEte\?{pj{1kPqٯQ9/p^'ɯ8}{n *][bpBPԏyhk+hUq5v,R蝶PW}]]34Ou[8\=fa}-KԽ˗LJc~5cG:Ֆcw]Ȳ6z4hS)!8ti` ESގCˁv]_sxС~ho%V\W! nƵ Wc,u=?c9N>З牺s9[VIho[@sNm7SC _S2d }P`ȇ޷m}` Cseۓ:ԅ+@+q1 ymߤ:2ǩxD9~~| `4êȖeXYχcV颮[uqTc ˵:_nEv;֘cǠjUfui!^u@,{8r5t-*lߗ?7CU-FF7/TGRn,b39el׽5L ~q]a+mF\C5n~w>l|n_?|:Ju"t?e}R=@Ї{·@A`o2ނIw>!`x'?sp>ĽCMp>ĽCMp>ĽCMp>?M/Wtzzj{vϿyaҸ7ӕq)\|8ϽpCh+u̡ $7gtknp臾/n~Nό}'N<}Nm@?!v~;}W{Fh9gs!/@8}+Do' ~ΟϤsuY-?4[t0}~?^_ShSpccT i_y<7>7F\GP~>nF]k.-%u^2oXѯCc!-]?oY:w8Gށ~ e`yZ!T\pN@2uOOԵ=c~# vj\M˶z:Ux!t; Sݹ|?2}-:8WY{tXuu/X7Fn_zXo]l븰;1͹~8>v s:#]xZE`(C0в뿙N@.wÃC|\G/yNu(͐7ٟn \ 5fa11U}Vq0DžIFWBk uAH^i6oWcaj|UTܼ>\mLAaBd:d0sXƘ:eTm ǖ*/?n> S*@sNm@_/EheU!&{ IktZK0a d2,VCձG@tQ-vørHtKlڢJrD _Vk1cP*o:4:kSAm9Lzihy;U-FF7/TGRn,b39el׽5L ~q]a+mF\C5n~w>l|n_?|:Ju"t?e}R=@Ї{·@A`o2ނIw>!`x'?sp>ĽCMp>ĽCMp>ĽCMp>?M/Wtzzj{vϿyaҸ7ӕq)\|8ϽpCh+u̡ $~3gɳ M|/}g _N /yIN~1CH[l_jƛq6@ {hS=зB&z"wYLz>WY3_a ezF} B{j88n6y !kc1ϓ浐H+Xpc(_>mq-弤CK}X7+u|,d}-OhS;ЏA ,^0~xHr89jg*;?n *][bpBPԏyhk+hUq5v,R蝶PW}]]34Ou[8\=fa}-KԽ˗LJc~5cG:Ֆcw]Ȳ6z4hS)!8ti` ESގC6sq{cB\؞yK#p: sr :|;9!ף4C޸dpj,ԘPTY},0DžIFWBk uAH^i6oWcaj|UT*qjq?: }| vM[8p>/o' A?LcY"Ԛ7{'2bW?77ߞ>Ze1c_ѷyev(o=fC#_+=כ&IOoEgF꤁B L/w(}~O>w_M/>*GϧԹVžKklqvgo]m2ËX.*?"'}m|crԺ?Әck NNF^ځ2@&n8ZT(Ko_,eu e.יY;ߺmFv\^ꛜtcF߽5ܾlk leOr .k\ }v({xmŵ!T]>'}?:՘7{y ZZ$ǩךm͘lY3qcR:_/՗[]?/A|qmǔD*E}oLR_1ihoOvz߇7ƏͿNphL~Bm9ܺ_cBدG"y:q5r͝I,;?E5x|$̏lXXZ~/˗cYOiQΙ׺eo^ ϫڭb[eu-s)jL/_ot\37~-:e&x5,{YעmjLʲVctKj?FgmR׀Ԏ0nr .Zϩy8aF yvxsf[m+ˎgrX 軾08K%|ɡ'mK{Y|m}_ta,KQ?/mHu /%O_hۡIrݗ ?9?>8dy Ժs,)_5-\٣{Jf^7,=?|ckԮǚrK1)&ko5?^K˗)(%}l_VCWgKj.o d y3^9k9?/rƀmoo1{ǿܰ_N~E"YSvb+Zp幱rPWnQkAFif7vT?c}|ǩP>esur(i:foUSx- kT2jL1\5~?q}|ݢOu|/OɪYB(>.Ez\ASv6Z3p:ehzc֣+~r>uۊ>v}+B%o}oסlu> i:s@4ΆP([B^ougG9c^kۮܗeNEE(]ɬ^My>q\nhR]]E/U? wb+ʫ81ʸc8r|52UwZ_:(bk0QcSN̾Xc~ˏYX\9✥m\ժۍN{vўЎb[8j>fك^ڥۅݚnݗeF9Z9OaM޲ǬwnYvٿ^D=uo;`>$ڸv?U11=&a{{|N٣Uc,۵K#~y+[͏יq,&v}~٪g8[N⺨6YˏoNƹ{Fǻcʵ׿XW~8^PֵzAl˲c2_BT:oڇ|8FE=AyRs.z#}|^ɇԣޏ_}@\Fpuu9W_p@|lf`Q}-Æ_\__VYۭؗ*+I_jGkסAjzor䶶űR;4gr]타Xk_~kbe+{kDٽ]zd/jםʙy}Zw*Ƥ[wX U|DF=E[ r<&N=fkC\~^[s?{(#k)b׬AٲOQ.eu<}[}+ۍi_3n.+]̹usS☧Zd8>~,ezrd\YkP О{37s)\a?bCܶ:T!H?>l޵aCi{\ւkC [Ef`Q}-D6;u=({YcveVٹct^q}Ȳ{J{G繴M(on4118ƺZcf./0/+/K~; BGlѦfUی@mk.1΢=sxmbœ3emh}Tv>l'5ʩa͹ƤVm8ɗ//h]FYv[7Zr1e[-ʫ=s_?fb>cs]fPre)`o@{:Awc_d? n({}ǿָ+?֣2@\_V[Zs.n)////ԫ/Eygw"<(V@_>/{_}m e5¿ՖXc^M /ϯJZ~cBX+ȌC&ǷV:k]:-׬'"N1,+uc2]#q\mz=YK?_sn9fkGMֶF%MG|yc*$Ǿ/|)/>±?wk}+t^}}r ZTDjgno'_ڒ11ו/ѷZcNk~K,˗r{"~s0v2 Īy9Xf_v9Fjz}}me M0n✧1Kfpvq[qCk!WvM_G:m׬ՖC,}-1s꺊TwXWjsn8jE\#C088&l~}:l{GQnwD1Oe͟o[ˎہ~c-IJsz-^]y9%~:,_W_鼛}qP_7O_znٗVۂ2lh)a>B^ |뿃F޿.wB=z<)s_@B cE 1eZpte١MѼ&0%xn:kZg^+O7uUw﹞r 빨wrS}cRnݡr]b̋XqJAGv:t|뚓k$$A} T&]lE{eq621Mu~Ҽә IJj 8:Zew={JٿbLzCilTGu037i:=fl,7Qt{a|/՗Z/_/]ʖ/˶60(s=.es{sܮFHD,w\'S.ǿxܯrk#>KòKy-z[sꃪ^QnwZ,eʶz}|T5|.W˥啕-ucC6o~O8\ww1u*{֝Uv5^|nП뼯}}uŘDŘ.ʼݵSE׿ y=Z \s([ύo Xm:볚Y5Xsz:m>>P p|XׇEP_l}cYoSt6x^XB(<. ĀsεpºGa:8dWgkk>K5&a[/αoxb ~ o Wo _TPݗ@߂/ƻf= wVdǿy.]bPz ЯVI5ͯ= .)r m@Jqẋm~"Ia~ UݶK #rYpt|xN_Oiyֿ$q/w>@ D"p@=nW^LwovWw;]?StItd@kOgǼ?}__}'WZfu8Ӈ_Kޛ>,>f+??le7=SlWvdkԾWzs=$b-Ͷy>!4$Z4>G<~:w_<*؇{ۗOmW/7ӕ<Gx(]@*Q˰4K%u/o [ nb!U}\_HzVߛv@cGAvk*7K}djӖ6 Q{}cۅ{}\gH.{ + 9D2c17Znhk-Ax(xC/{Fonu=m;K~ɖ+BH3k=[++כ?5wQm8 ܛۗW>?{ ž __MO1r}s22@"! = I}s_^cQ!qjw@'Kh!/7ख5 埤8n k[QomWX/n_l@k8_۽vs?Q[3}L_āg47nX7s6Ca߂o[\vBgrX _pwdʅqV@Wr)A 1ӡb Br >/mHu}/%O]x{6Gյ<$0>o}*}wN6O qk8a＆k3UQ;[uݷn15Q}-Sk#=}+<џONV1<`=. ٣, p6m@?eP2t6ߴr?oYm2.pz sUJ>~ @Skz0#ps?luxfR2Km`o@$Ze%b_n$W V;|`)m-@=wmm I^:rg WГ9ORkG'Dvjzm|coO_Ϗwၾ:f}Z 8}eqk5uםeUS]k^#G(DWT{bPv5&oG=|>_Bzmh8^Z}x(q3Xz?~dn?>;0C.[o}MG>̏k\וQvf/+ -zBs-c|S1.6= BsU`E- Fab=|pZ+}̫鳵~{u'ׂX3_OŚBC5 ZAvyݗZ'~cݝϺeŨ췌;.,w???OGZ# p\C8<# ×p-|]O<lBxffP}` h{W|y6Zk. w s&w{CǐV+†y1gogzs((66o-;nu+뵠y]ww)0+-!b7Ky7ӕ vtモ^HfTu ;,a `O>v_!d/??_;!$V&iiꯒ_]{=Rٶu?1wRy~ٲ_rգŨw S ?y|ӿ@"|(f|(6i|㣇>@߿9}[pQ0TvQ4o>v]g3kL古[.ܣA5PC{kžv}h<ǡ|ZvxCUzt~`S [RGp3)$_egWZ C!9ďn`iNpK mCi"6)\&/m7bJ5^_\lXguKcsU㺱]'o/g@?㇏#Z?6{^ :!c|غmn-uLv {byt5Xynu2סtlIwnVrZnݸ^\屲ܖ.'~즺F{n&.D-p?e*V;Ŀ /p?/ p/?3sje/ G|;Po[PcǹW7ÄhK澲n㰬ŖafzGmΎEkõ-׿y-})mrFbȿi=qs4]Ic@?_.+1ײ>ZkKou=FKÿ]B١=q8ݺlkf9oSsyxGI'~^ ˣ?;O|7v!hpzm3 cքpdžRB*\:x>ÿq:ok ͙]nk։}=5 >prOK>?'F{x#P[!C*f$N0/} w uV[J~Jo!9=/zokb4=e1iYZk&b[6}^Yor*ƾk9\};Hy֮95?Z%?})z/5ۤ uQV#k-B}1(ҿQ@Qn~>i4=dž^km^Hcoּ}Z먧:B4{w{Ҷu?օ5kmf;rٱy\ίtN:Zn ul)C'ۭ0gZpok[w默e-}}wο{cKFU"Zmсa;Q. ɡNpc*3&~;:`Jc_קqp$lhhFs n aк^ZY;ۇ+]6k5k_CRCkޜv{֯cwFߵRF -1juƦ>^u=YceRwyBҸ \`eo@"5 G£R/9w@l?.ԚX۟02̒!W1+cq۵pk<ס΋ݜCq[{]ḃy v(&pM6ug{s6"t{w7ݽf{s===ߤvW/^q5cp>ǟw,<lA@1d}جJbQ_p2~!Zӎ_r,R:q)۟Rm@kOgǼ?}__}'WZfu8Ӈ_Kޛ>,>f+??le7=SlWvdkԾWzs=$b-Ͷy>a}0⟦٧?}~5<=>SӷZp\: йdz nO?{OĿS\ޑ%@9@o;4v5Eg]ڟ=u: +vXڝK%n|Ri>؝3joٮ:+c%7֭i,sfIخ &nZ~UYk^e޺5T^"q\ӎcw\cMML-c}32o\}pԇ2, ǰw bKV#Xu|1}UwRQ}ʨR|y?>ڴe l0a^߾ߘva^?,p9!!)O\v+qNˌ?o_>5 /]A+O<?Zc 6{VN?LCR?\tkCu|[n|1 ZsۑU^oPoh<&;ưl)X}u:>-ZT)q5mߜ4k$?'[Kcz5iU(:ot]wٱr:.͍{䘄cӼ1k3Z*5~!_*)vebzTw|/0\;C92-eW2Zgh007zX/?!WV7Ikq\u3]7m\8wB;ބ P›c>OVMF~ /yLu"STv!qj@'o@?lw._umkԭ_W5~NFru҃Juv]n3!]=jK^}O*/cI=q4liS\~:~LQ֡F!C?LCÿl\o/lKt6Js1`)uՎ1j6Y~9?ۖcXm] m\k͏0K {!{oNJfz{!u[#54w-[ǹv>tS]a+|9&߮/a||/6[K%}NvAs#톼px<n3ao-eG*t/~&py~w}w!,Y uPgm[C./RY: 6S>aܗq`Q0n9ΰ0}W1_s8}q>Π=k]^?˿gZL_:Eµ',KY,_uef^iLZ׹${,##߻f~vLjqp~m(K~Cj(dF+<߶>H9:Tb_DN)ٶV9*+R uo-Ǧu]g|=1am9C,6 cZiϧۯw5ֶ0Wuc}UBPտt]kЁy~o2u!ckãwzqe6zBm-1 vm,K1]3fidZdk3pv}+B%o}oסlnp^AwCGm@4ΆP([B^ougG9)Ƽx*0iS_PU|U:\My2rE\a[V`wV/FC\Q^@@"t6zwց}]fv`[@_Sԝ%Qm,T=N}]{6_Vw|^:2ڟ?p@grz#c0!Mŗ_Zl3 Bh ,db lQ27u@>f[Kr\9^v}NkT<55^-Ǧv庵Nj0K 湏Wmk!k֞xF񽠽Z/~>o[kb{1V}2ֺ~%kn-+qa[\FWfoY|1irX{-zm.ێ~\ˠ~ aeli~-߲d]ԓǛz-:7w|H= l~٧ HaG-@^hu($B>3Ǟ%З#ϷJ>3uJfdv4@S[#@=wmm pci\_-No_olk&1I/ukxp ~<=ieYݯpN/e7=r `{#A}:Ycg5|`)#|iTk6֞ls4j5AzjԯiiІCG':(й}5u^/e=2NNs-s?i+FXgC}үSy}{m.N~C};mC됺>c i{\ւkCS!wc dWkf`6l~}Z 8}2_kXACX]ծplٮry/rDv C{,՘rAu1ot_o2 )8Ġ-)*VpyEB<+RJ6j\^8aC]>sG/`Xߞ.)IswiKq?Z#W粞Fx6,-=q8^]rwlLum[ŠuZ(y8{~fYz21ئ]C =xK1dAr&owtD8_v_!d/??_;!<(;.5YړcR׬=42asYicXcSRϰOY6}i0x(x;@~cxo` o. ud;\ߕU7}\=a>m*ToGNxg?_!;>E"p@\ }.>p~Ϯۻ7ݛ驵ݕN~L,f{_{2=履F>bW?77ߞ>Ze1c_ѷyev(o=fC#_+=כ&IOoinɝ4пno^m}0⟦٧?}~5<=>SӷZN, {@tTe?^M/.uj=DvrtRX!@]B.A;~ ~cj?wC7B0T{";j[cV_/'ӇV̵8s.,ߋf0'.Wbr8'epcNr pb{_o {t 7]غ6 %PNdKٕ !5 ̍ˏ~qҟ;@utL.sݕ.KoMW ^_гqp͕ [֛߷:E<;ȩCnݯ ;ϧO~C_owQo 8+Cg'˹-:OCowVd\yHCveo^D_˾Ysmm>r+[;KiLāg47nX7s6Ca߂o[\vBgrX _pwCй҅q:Ȳs ]\CжEPω.q_K8Ɲ/o?g ?9?>8d6|\%2:k֯"O{1EY|ݍvW1^iy;uWϝ5fMsωzꋸ Mte΁rEHMp:A 9ss1m^ۜe{+Tc ͹s<'Řw9|YAa#ʳMfy"TrE\.&q鿱~}@_|'˹ꁀ T`Tk[Uǚ5 |ݝua_:޵^ϣE/#>6|UsWm)i@?eP2t6ߴr?oYm2.pz sUJ>~ @Skz0#ps?CVUDeyf քq[+o/[UWR u&hIk}@=wmm+c?_^++!7XE[9Ok8#ϫ9Ic ȶ:p*\oW+?{J}?Mxv CpoF\%Q}n[*Xy$|S?wUP=Hzd\|vhG߅bUVbku3 CK>앷q&Wg?+0ڵkaYs :<_ƍc֯;eƬZ5پpc Cj|s]YǏMT1_/|l g`C =xK1dAr&owtD8_Bc۷nqldݪE=ŝ_>PM wo]X6bIVuXF-{|~-_I|+^n#\$}i|S#@\ }.>@ D"t{w7ݽf{s==ӗ8O7w͋u%ިSs_%ݿ:"cnw~w^aLN:{SzJdzO?]}g?zyO?~2}oV=o=}Ï˲cƾovPv35z̆zeGvVJ{~7#Ml^+E9F椁=|`z?MCO?jzQy'??z>}ϧo/X:Cеi݅.&_@?3ܕ*1fg f9}G}~J#]x-Sː8K%u/j [A?Mc>.m/~=+ Me v {u8=*{c_O6mk+[? "qh׭>/O2B}Jl_N?dCq̩\CoKޢoqϪٻz6j@?{߾+ BxU S l~]jƒkg(_fA?fxv s㡇5c2pezs}%t}y|oG0"5]HKWg1rveFsu vyLu"STv%Ǩv~o ?}>}~#zkY1z>=!tYmс~|ܵ"kG#e|Ķx}\uͲ}5׏3\Q{ ?uWS#UX|N|,ez>~u⩲1cۗڦr!ۮm?3&S[8ks[s'4uVEzz kiߠ_R=Qoاe~˺c&׈nsUk͕gpm5NvAs#\톼աlx<n3v-eG*t/~&@/] hOb9}9Y|m}__ K86S}/w~ OBZO[}@[!@ߝS?O&o3$G|Xeݕ:k֯"(J^1EYZr[rm~-#xcdxm==X~0*+}٦::qײ/֝f?Bsc̬=}y#ˊmmvt[Z:z[G(Waj\Wn.k٨:&;]_̽k5zMvx=yCY\G([#^#uYbǨ_i[c-I:O9AN@߿n]9wy!!|Au;7ǁe;i;4I1汾,6һEx3 _W_/ŭ/gUy2rEܗ"1VpV/FC\Q^@1|keKR׮mUkָq̮5'Tǹ6~ܘ5MylT\{]u;t1_+VDp6qN_ۯdQ<7(KNaoXØndkL;7kea̚`\kaP?v>k.gV-wØU eUWd^[e@?eP2t6ߴr?tq zx3\ax|^ɇԣޏ_)5s={a.vC,_X[_zo"|m|DV}㕤ք V;\d퐣e^|~EqlWZ!~u/w4xV5 q mڔNT@ֹ~<'e:fcTyC;\o=cƴfχhX|?/Ous1>c5Y {-U0)Ҩc0'U_cSźruw*KeUݡ>5cUt:f^N~HEQ}n[*Xy|S?wUڰq~~GFkЀbCۉ/֗ۿo*Kq݇!b[ v/v_sM!H[Ɓ?_o0l0c5۠5k8f7vk[' ^uPvm UװVZṁzd1[~r-u]ʗClhQǰ:֎ru걔|rZqm7eg/JLBL%(@ENR($\Ĥ%i.tD *8I ! pYŨ).AͪiTzֳ^6;v|f{gvNfnE݋E{ =ޜW휭ǎn;0C|G~9#c+XZڰNYS{u޷$޸o*[ozFc>.1(޸vmXC\7[osc mI6 "5@lh^~~}yGWhlm^A춲҈{ <'8^3yu]mmW}}rxmaהּط{YP E$ȼT ⽴֬ FZWSϵ_O[ȯu~nz/V9/5Nݶz7׊*s߼v>+\[wVf# 5r~Rk.с~ i,`ϑ!l0.;gP?njmǰh?6m~%'=,2é{oyh;|ukWK0h5zNƯ-q׸kʹ׳gsk:`-eyd!^YκOt}׿7eo^kSk{yki+96צkK__֬k~T8fzNcіy9^0Z̳Z +;|CU.8jo*NZ37Jk%P[; 4poQ@xs.!p7W.7 \bUˀoa!p@\ }.>@{ϯ׷+7W3y׾z~wŽpvzyl_W͞]J̹c֣]tt:j z_<E?>cqʎ뱿Okfe?"{k/sg?ڡg޺f]#{^>|RkuO$ǧJ67]3ߝ9i?w_}o6I~~_ώ9VÉ=#m3 ?{]W6Gׇ=ÿ_NaS}/?qz'stkX!^B.A; Vc^ ZvmR@{ӟ?LSanm޸}=N8/ŷ˧1mYkk[Dru텭|7RS@׿L2{|C?*n)Wx_Ʀ[ili YQ0n`n|ad^C_kZ/o_oH~y薿޽y5],~u}Z#yV0ި~\\^sl5n=+@_-@(~мxz=4ǺV#au^{iV;xPW{x^o9Vجz.x47r\_v}VZ_zɎE%vټu_Jnd5"k>Z_K }1sVw{S]*_gYmUMֶuHż}/\DzVێó}(w"Ǟ梬w]e\m}mVKMZRs2 g׶ˍٝ+tbUZP-O:6V,k^l[7k@?Fpm, kQym]*X|Ӷv(tvc~y==PЀAlެl /5,,Yo[o.o:UunVbŞkqS}h~&o>/ojRQoMk:-cQq㜃`!mwpQ_|o>T*U~? 85sj{Q<#3P^T}Gɍ#-yM~ݟp텾f~ט֣`qYUGn7cw..iiˉnhv}֫H#h^1*ίֽWp\4nǦnWb_uxva\o x77ُ}sLG=W~Uױ۵a#k~Bn'T_Zo&8|x8ӿaoo.6777bjosc mUߔQmor?ף~{e73֞kָ~e:<WuQD}]k_GkF]Sן+dMe͚9Zqm}Nxm}=7`#_^O7~,s_־ZZWۏ;s]jm7U}t;նqY؏z_C}[s-{m׮Xs$rqZ~W;;NfMu~_"s-ZskY/x~@rtCZ- sd> Ym~1t}DZwD1.Om__ Ϸہ~e/ĶkJ뽠zw>6qޛO˛q˲ ŕ7_ENM\jc 7}7 !{7E5@ nԺ{D/wv GS)TXe{tomہUV|5TzfŽ]EXV~}:RX5Uu^"0kլ^gٜsu-rNWY 8Ey5"'1Q6oXo߶8?!ͻ'7;Ezff\~u^/1;Γ5l=Uˢ_jjUıBk^ת\zc[הl~ mLGVԆuUyp }`Wox8:!ԃd$w/8r_ڇ. >4W5{N>?Ep }E"7e9H5{ 8zor0~=ٽl>Fb*>1@\ }.>@ D"t>>tz޵～yg/γulg@Wn/p{tıyogdi^GOwяy2Xz~tڱYvǏ'Hޚ~˼?uvh:YWȞנ4Zkݓ$齒M1wgN7D{ߟ^Nܯ??q?^$?W?ku?/kgN1K6C@LAoڳ/r3]YǰM{\3~XGl:)>;:wN}X,Cn/!~ V ֝1Z;6C y{x&B7Զzno>{mo֘? ":T>GVț{ @ߊkVo9Qnl xBͳ;՗o֖=l%a@[`{|C?*n)Wx_Ʀ[ili YQ0n`n|ad^C_kZ/wt[FC_{~콺 Qyo+o7ŵ[)T~\\^s7lKp Q:o?po >Y뽬#-Ok8]k#!]5޻׷Z᤻X4qw]U({mQ}^khb=w|_˼׫v߾,} K]jf5uGۨYsa\żϦ q|f:o<3iڃ9}s>w^|}gY^(kXsp7N?vAs%\málo}߂n;R{N39,_ I|ܽC YoaJ қq|8IcFRߩ}74p>vp |븘ٵOg15f'G/;r*2W`96>R͟Yq) 5fg^n߈k5){y5q:yTv}֜W k{^ mk[?_ڟ>7;-|FS#粟z,jN͚rƽ{%_[/جYۇyk_&qZ5\μ_Ks\.~ 9yMxNgڛֹ9 =Dubl.Pmg!C {سϫ wb\++>p"ZgMw'Y{kDe%il.+ü='tU-=uS1WZcއx66uP#]kxrxOv5ج4֫w{'ztq##cլ9n1.k=z%쿽Vmpn}횹}7dchZ{H6k@?yP2t6ivlg{o R{_uS#k0ګM>pnokb=z|_)o_3ޤ7[m%Xˆz%@QG취kW''{|Cߵ#;t^f+ pmcmZpTx6>;ڋƕ 焢}a]meckʱlAVgy.6sf~=VFMj}pԼg01[1TjfqqvYsrRE';^cIj7^/0pMއWWWmkV{^^sܭ6s 6otGueB`Rv}LڡٍaۇmSZCC?]'Zϕo77sƛV =gsZ\l |Hjow@M|8_o4j}zkZӹn[ pjdۡ>mwU71mw{z2nq Bzw͋y|nns}o^땵Aw]7?t=G1^ͺ.V,c9qMxnۮY{uзެus^:kc=/oӽva\o x77ُ}sLG=W~Uױ۵a#k~Bn'T_Zo\s͠x?S]qäkz>*(̹ȱ$Pr?ף~{e¤>*}E"0N*܅}U\ }.>@ D"wv}uefzzf=^OqtUowx}| ~xGOwяy2Xz~tڱYvǏ'Hޚ~˼?uvh:Y۞me9l{>O7>'U4 kq' 9ؾ?w_}o6I~~_ώᬲH7ӕux@po軀T>aiP?KIvXnb!H6Bďq<ԇlj빽qVθo·CAl[J~o}tX!儔CS@׿LBj]0nۮv۱uj-\_NK,KoL.(B{.K.$: ԭ#y:D7$7Kq]sy {8G[뼿a%7yBw=\=kS{.v=㺎goiXƮw׍'[VkXsywāc4W] sf ێ 8f|s= k}Pk@?tYZTd}b0߹Ű):[*zC"J.m e ԾcHs)Dak@::sjߟb?ko߉qjnGZykzmԏE 75KsCW.~^e-xioOӣ6l,<}M?h<'} w~';;>,WA}x`8 W2o- ~PF}[@ Bu_kl#:ߡTozTd\k9|uz|=z d`6R! zD[ǭ*땤FDFmnLߢ]@gӋ }׎|y(E\u'9:.q͆@ߩoW&%9&)Xk-|<~d>0Og ;]8}>pXǶ(W}>?8CP~0%|lƛCa~4bgM}S_' R| a;0rj2JQc+15>zEe @?wTg]g&cx@^_+|g_,0>ܽj^x#Ƈ }sLG=kgvۃm̾oVPs:1,cӿm W%.3_5"t3wP]VX_C<l^\B[Հmc},}gӯG>Ng=!P43 BmEl{y{'Cj}u\b~Ǖc㵱uk޹^+c}gwOvo^Gra Gߝn}<zϧ_ oa~ ܾ~7;9IV k-AblX@Bޖv&Z5:`%pbc=lOo[=:1 z=R}Z^ZߺVaxgWcǣ^zXBʆs_?Al@v[h;E-ݷ}^*j붍?kӞ(ֺ;Sցm\@e b,hoy\CTº~o:؂%0 Bo^|kcB}0^'TK=|ʶ{#`N;ӿ?=.~7f?s>8G5?4{٣?\މ tp/²FMSZp-kqsζͿܚ<mZj[cԡ)>\?>AJ2E?Zֱ|~A=>;|Cؕ7Տc|_L46"}c}hy{Ň?`PYoׯu@a~}lc t>إ^`ֲ}ֽǁ9]}bA-yQV-{<)o>/@>.F=a>8_Bd;2uVS};$?, ?qqkpoK) EB5awW#lѶ-a} {zFZ{!a~Cm G(Vir}]1X-ziM$Ѿ?PԶw>>"ܯOP_??%v^*C|[]!ySߴ`ǾAiXa k n1DkVcpnA5O,6RuH珆&baܦ}QYk'imqc߭1|-ui1ǚ^dZ55l}H'Co"DA?kkt/X=7 D u;o}[Y~7$;!h-Ko G=צUژV󰮫:XXv-j{"0׽ϯ s8cn}[^u]cG{~@3Zs'@@h.v;n3GѸl7xK߀C2+{6iúr2t[SzS֚Ik9Qx[cEu8߷1:'¾nC@#G<(O?Z{wݷAv=0S-dsA`-녦t {5 Feb=}'5) {ǵ~_7;0t+6)Tߛt~k?X~; ӣ=~dP>/CS!:&]s@o sn[KgZ:M֚Zq hSVCu[P5֏1^ힲROYlnÞsz@t3؛jn۞|=߼?T.tes o.I;?{KοYxu!{)Vߚt;~;S}!WYowOзP;5@2;~Lyc~I@֫PO}k X{^ZڷpZG!4\٫+CHD_OO6 V ,@~qsNίq@K_jkC`8)D{.$A]+uFBO-]g Vq9F+| !l\}aGeq\SeX}יּު_,קUv֫5nB?C{,*ַcx>?7ܱӮw0{k;kB}|d ӻ"냯?rz۪o[䐗Z =|cd! #;!zR.x}1ڥR`*]xi5|Zߴ/'됀XˬuZ~NZ/kZ.m6<Ҏ_LUyk@?!p^Vop~,Mzn+3kԟ|G>^q ?_S Tp5ssұ9:y9k\M41V輮1׷\wuk]̿wӵݵ8[wˣoM:w/fP[ j~~$L<^V5B2ԭK_sNGժk8jOZ:?&˼isdoYܐqFK5ѯ~oߧsҜ=}Ⱦz-w@X^!`:2 ׭dO>@8Q UP벆?Ϛc^'D5[ڼ\{Io)[@ȟ=~4]+̊eAZ/XLt=D,m o;ceP>کY@yK|0Cj@v_׹ANA}kmSaκgR=҇VbBy_[we)@T oe/)C2F?(} 9>0Džؓ2.ԅk }8'$/TCoү/jab|u>dqxuWxUtpP4!e#8r.ͮL>C,<-C2ݹ>ht:׿pn7/0uyWӼ|*eFػPu5u]އ yvr ,pז!o[87WǭGv&]wh@_/Ehq|0wNJ/7?rzVcJ `4mXYcV[}hHtK{2S 3!` ּkf=O7em0Z)n^ǩ4&]-YA^3yl5L yq]հ|[6\F%Zow@ki=^[ڮ?z:J}"t?ept{E|N8;] }@A\4Nn׃@l+ ]>p6ݣ?Gߚl>r _n3v@v p3Go1+B>zz/'c1F pVB.?{z9^J팅O>fzsO|{yW,B\W3s[j]ymk3z}V,u世5[㺎kUÆ_Xc3otmzdk=<3}N'R9cX,ΫW2< r<)@͂hOdx_?0wJ|]>j!K@m%^Ծu;ͱ{?Am_c~-` e>-i 2wAqS-8/ ottcԣ6.'} Sq^cVܗ|}d߁l[^w8}<<<<<<<<<?^|fC׋6z=]?7¾ys'r7#::^]sO_O{9GძfO:ca@ 7CO}l'L~ήpwK@9[!|oot~zn+3ofjB-uu==18&嘛ܦ6;>zh˲]Gk X0V_^9luq=`%>v:ȑ{~LFGV33t"c@?2z%.S@,~Q敎7"}pA?Cth/jK}y~oV{ol]vs}m:sEҷ^Zȍ@nv &Y,o RyvZTB(;CެGe\:^MϙH}qksہյvn1YcߪV&=}7+e/)RF?(PwG~+܍yV@___@A%З#ǚf<П5]>[#rIlLBIa@3 ݹ>=pZy= fm*qee!C/q5ec苲fε9ϊ#7ށcxځ;cF_k=c+5Z-ĸ qǫcka Q{82}v*c@2MZ"0[}8 {_NMAa|ʅ7fvG}g1Q>@ea)c/ʱu+LK}m>@-uϻ~с>s~}~}~}~}~}~}~}~}~M_N_>3t{{a_޼PzLWyps g9:>7G|1e1r>|p5)Bg@9f rzNO?F<CM_O/q\o}{o!_.B񳗯뵺a[|m~KmVC\W3s[j]ymk3z}V,u世5[㺎kUÆ_Xc3otmzdk=<3}N'R9cX,ΫW2< r<)@͂hzn^阯|?{y#BTc;Dׁb DK؟ڗm]֥>n>;Q?W~,}5x f@ k.hBP :ӻ3q*(ΟQfw`Yʸt3?l }kZ}cUdM{aoWBW._S2h ~PϿƱ#UV.ÁJ/CsG5 51.qx?k軾}8G>٘Tb- îL1,gais}[ [BGv+!!xfYʸj\sY zy fX*!٪smsγb-{c9ͱw/Bx؇"hA8^vXmcaoTr^R=c+5Z-ĸ qǫcka Q{82}v*c@2MZ"0[}8 [|=}E0zPT ΋yBr3]ǣ3]I 0~wi}Xf{m&6_w ۖ׺]?N@q?>pp?>pp?>pp?>pp?>pp?>pp?>pp?>pp?EϦn/f^Oύsp/o^}o+܉༎CdzWWC?Kg?Θst9>n3v@v p3ƱC1F pVB.?{z9^J}\<6!{֍yǹ-5q.<6wC+\:r\Ě-}q]ǵz*aïos,әG c6j=r^ߏ{1,a+`t9fA uEp=ʷOyc.k{\P5]ڋ%}/aj_۷w\_[GN\=~0r%5<I5 Ae蛂@GSAqE%0 <zTƥԗf[_?P] mƘ5vj%k"#p~CpY",AkhouLg;[%o182k(24wX]׌Ƹہ؇sk{D#)IH5)֒@9tR~fp;ׇՠ;/cGWN<}Me\\5,Dv=l,}Qlչ9Y^~;!RY/R;w\p}@B,~@Cm\uj!%whǘ 8^[{ a~#_Sߏ3Ԗh52p^)x>AqS-8/ ottcԣ6.'} Sq^cVܗ|}d߁l[^w8}<Ǐ`xw{wWO@`a"$nAw7vǟqaM0Xhy52"{~ٻM'>Ji|#\G Lls'ϟW :8İW2Yݰtr?u?$\>PK{GWj_#je?pl@jJF> oխ>ְS"p?5wo׏<~>u:OXɶ~Xcls{F5o}=ۭ}@k!n5@amOx(k- rW"r?2GߚãoOy6{G/=Tp3H{ߗ{e0ƫ ׽k 1lJ?q2}])̻<-{&J^jvA;~٣?\_=>OY7a@?O-6Ǎ[2Pcנ!vr:.%P7=}||އߩ}!g][椄qpk>o-=>W~8?}޹1/3/.`Md+O[KXxLQf@9|)HwB=bZ9kRS>Ѿmau^g.>p~Ǐp}z 'ѷ<=k>v됀tf ZTͪuk3gWόczq3F98@ؗ7/7ӕ|Dߚpޑ| sJƻ/5З7-q=Я}v\\>M/d]<>7?Qj-ۮi޵9_s1p;YιeczN/xhu ut58R@/w?fz? fnb9\M7x;@Gx K8Q N/v47CxR_rǥЯVeG9-0UCMc\ߙd?lse[r~sk!kz.6bVQzpv p3Gʓcnzz}z#'Зߞw`OM_}k}4~:GX~:ךv@ZI蟂?9-6GeyVx@ VB.?{z9^Jf6!{֍yǹ-5q.<6wC+\:r\Ě-}q]ǵz*aïos,әG c6j=r^#o+M.?+Xu}+ lmw).1ר>?:po#ۛGX+5C#g|.ށ~ eyJ!]6Y-p]\ow+,wJ|]>j!K@m%^ԾpasmMF=?:UcҘO v~%t 9EX*|:v(?J wcpe8>PP ehȱf!&%gq}w=GR|<>:V!i9ym-["$'7c s5П 92",-}w{A󨏧w>zzu@zjia@/˟ƕ=ǝ`/ʚ:6<+ֲ7ӫߏwGxG`?gh}8WV]Z-W)Y^{\_ uc sxvRPe Rk<ٸ5z}sZ<ѻ/d{"}X*V4EXj u[|m}n+m弤/5z847PWk]/k,Zq16*Wk>+~sAp/eTyw{Vy2j@C` k1l zڥ`,ۂ_^dz-cJsj(=˵};j4k`8\/l7~_=(nżr!Muу5WIDAT@uzDY@;؋r,y~6R_;mk1Wﾱo݇N0&^ޗ@n@~|oc{ƞ @?"c9=xo kXki/4@p8r?1GߚwN {vhUҶn3v@v p3;}9ɗ; g q7y~=q‡& A^^R;[Bӏw7:=W)B\W3s[j]ymk3z}V,u世5[㺎kUÆ_Xc3otmzdk=<3}N'R9cX,ΫW2< r<)@͂h>}7te !x f@{V[%O~cR }kɟ+wz؏FnvS5`f! }SzΣbz/'q=? e-*!pe!Xo֣2.d/dĸ5@_ZhV_7oU+Y>sՇ2a ZC{V.ÁJ/CsG5 51.qx?k軾}8G>٘Tb- îL1,gais}[ L[>bz˺zjia@/˟ƕ=ǝ`/ʚ:6<+ֲ7ӫߏ{"}X*V4EXj u=~u!_j?pho6׺^Xj;4c?lU?׆}VD_@?۩Goj4k`8\/l^RT= }ZǨGm\NA N(ǒ׭7k3/Ⱦٶ=!pZG΍x{x{x{x{x{x{x{x{x.:6]u;~87mz~n}yB=}3]NnGut58Rj@/s1]·y +nzz}z#8}+!Mv ~Ώ|_mvCӷ^mhSg>X7ĝs۔fgCpYq-ku˫8 ο1ṉNgr92tϏڨQ{xfzN~?xsǰXWdxx hS2uU}1=$ٻ8dW:^ވU3uXj-}[q~yWuuTKz=k #7^})_zd ZT)HCAL７wZTB(;CެGe\:^MϙH}qksہյvn1YcߪV&=}7+e/)RF?(@_#z(~n@::pn܏C@;܏C@;܏C@;܏C@;܏C@;܏C@;܏C@;܏C@;܏Cpсg1tnoos ˛y_J>w"ws?8nՕ:PO|[[s.W<"tNn y`{@yl'g q7y~=q‡& A^^R;XjKJ-uu==18&嘛ܦ6;>zh˲]Gk X0V_^9luq=`%>v:ȑ{~LFGV33t"c@?2z%.S@,~8wJ|]>j!K@m%^Ծ-k̯e,p ՇR:_/GՃZp^+T= }ZǨGm\NA N(ǒ׭7k3/Ⱦٶ=!pZG΍x{x{x{x K<|C_>yLޏ9[W~d0=5LP{Ϧѷ_<^  G4_a@~  O}ky? NQzzow޳ݩۛ:t:UX1íhzuy*{Rد~_QKWۃ^?F~ה=j~2滹7?ֱ'_LoYfO>fzi;[h{\o>6uOTbz=p2O\QX2ּN0{{)"gG\5{_= ,aCRV?_鋛lA\ tPg@}oC_YK/yl ~הj~]oMMݛn#!‰X!pq\v^@w~ 9s[]ؒ=x{@GS9f|~~xor7eb!8\.q&^JMJ?ߕ7S\5t~SSsy f{>6Tyj=*V},ڮEɵG־o(A_ڜּ1wm 񜑱~Bj/>;Q:0qL?e^kMu׮r׺seu:aY8yu6za\m~xx~km9{.Kc~Zq]|WVMӺZϕk}&[TիY6&ѵ^ȱg5WǺS[oM޾趎\&T|k C4w?ؿ/]~W'xWvq<̻2fٮ#_*꾴ݟ^Ͷ5Z}:_mb?Ʈ9f/k\[rPm{!`qyno[u/̎]ĵSSW9߆g.qoem1-✁y9jtdkt׭sLcgt,T3{5qf5!ֹԾ{<ܯ͵3Uk[waͭ99l=v{Azڞ|m'wcP W2QE RC!$ YAM_?7fBТ=nmM5)C|iK>Dަz:߹/yMxu±|5n7DxZޮu\k{}֤L yW[^U)DLs5cZ^PSw:Dztҵi~h-uo3KiܕP~u^xZcݗKǾuqV3?79C^ox<l^7CSog5YB,po¯ ,ƛ AHweT~n\珨\["e:בk9WO6zcpmݝW`FsYfev1l5Ȯ+uIeY-6ʱS3qڼm1A~:5s}zeǛmsͺ}km9o>#{_) kd.ýJӼ7"heoƻśy(ȸ1.%q Ezm_Xcj7}}׾;rת5^kt+\v;k0Xc ڢl :9N_o^G7 XY]U?텁+uu4O:k]QڲkcG;R>s}zبSsf珴\>[NX{Awu=\f[X6}T f> vx¢X( ׬Av7v䃡lrdp6^HUoa.z?VGh92T* lYTkrаv`^HŹmγ7ͫ?W76s^>Y>[ 1+/kuKx9cj&:|L؟'g|NV3vxow863mԧ;9ֿRy?획 h}nus(BYFyq??~FP'>ѧU'1_?yhaj-m;anZQ51Wǚnǘ[0vdy}xN^{^Ǜj+ t]Q*1/Gg{AؾfV=:cIe/li&yy~q~}97nz4$Y?'[}Q} lLWR:p=sXYi5N1^ck6ȷپ 5П-AN|C<Ȃf(vpD=*o*^yaZ77,x WAU=r:!er1 vD}4 n?GgW5u\?z;W~;dm#X_;sױ\ `oo:{niZ빙Q*dskߘkv~Ѯýܖqu3e\%<X~T1.\c)v3ߟ:7{teC-cZklaFn^b-ks--tk ڼ?VқWZg^p1>qV#dϕmcֵIo?^[w=s͵lUjkfkamw?ckjc8nեk>blo3VCƇ"u!h(СPy(AF~^<}Kp!֊Sc1C`lժvBm5é+珮T8fb\c]qsG->׬umxVb󶊚[c{?{M)k7vFZߒ/[?K7 ^3@zSĸD~& Ĝ\kT͚Ho^-&f%e휬fq-%_|r}2Źf\~m6{yEBw^+5q6;s~}=L_ItMtmX5_@\k^A^W(zyhY3YGmj} 7jskEmqVwb/ל^cGkkjk#kK1^'5 =П"uWo "5p DxSFoB8KCtV5`uՈNnpz4Z{+,UL_7vvΦy9ܖ{6Cg.K*'p z`L_ >,v(xJ!$opwR!a;@p 8 ;@e˜@{ _x?} `G{'N#8wсg1zu^M7ֱnYǰ^ ޭGx轺2ι[ݛakѳO{~XθQ_OO߶Oy:6{ϿXǪ-pϻstu>}Z^\ k^=pr^K>|3OsR&{O[~.mfݼwxl@GF{Q9K o_Ooo+ OPqlwUrlB=r7Λg7O臶O-W3I7w/.q`!Qy2D>f['gN çrl{ }\C!I n˽Ѷz}lA\tPg@}oC_YK/yl~c~הj~, A7=wP7ߑ eD{:g@%-soh6?Q{Ty^>Pk =зB.6D .|R;Mg |lm_ X䜲oZ(kvX0S0̏rS비VcQv=/(OC%W@'65/w̝[BF>y1^ZqNƞX/_d}g\k;_ǕUӴsZ'jֿ&~vjnEz!Ǟ\ o`ۿ&737oi C=ؿ/չ&c(yeü+c:eKyl{\Yoɵmy9V}%lljch;쿠؟s ,t׹~Is;fBҵ3㮛|۵kmϚYa7Ws|~ ^K*6i.F~LzYjNX}WY6]~ÜB,w?^נ5K_ML?G|~bNej`ks>׼+{鸼Fϣ>vͶQ[z}f~cz~C_ o\_Y}qqoP zҘ% mY "aCxk KҜq\yWqvצsg>Ss yq},^?Vk^05~xo3KiܕP~u^xZcݗKǾuqV3?79C^ox<l|_p}+8ɂdA txs!po93"!aeXQhS͝›K9=r=zR~#]=FsbmI@K+ǡڪͻ;"΋׉Z˼b>W5̟kƑ]W\˲>[#mc;f2רݎ9: =֦(NSכב{k>1F=#}=Vf:d}mc(Oz{aJzZq1$nغԴ\m:6gTܮ#ml;]_c9ipmc)4Qf=@wvބ635kP]xსlrdp6^HUoa.z?VGh92T* lYTkrаv`^HŹmγ7ͫ?W76s^>Y>[ 1+/kuKx9cj&:|L؟ vjf^QWv}%GX{#o?W{>kv[ʺ5Qfj{'j@9i% Hh۝o K^<yaZ77,x WAU=r:!er1 vD}4 n?GgW5u\?z;W~;dm#X_;sױ\ `oo:{niZ빙Q*dskߘkv~Ѯý;js[yΔRskٯuBjx %} cLV0enSf5μqHݰeLk9>cc1Hc)Zk[[5µy97ֵΦ5jc5}=ƛ2Q;G<ɞ+*Pko/ [jߎie_O1z4gk~ lHz^1r׮[p=ܪK9|,x=\ـkKp}0C. E\@1:,›F7Bn$mӷ1a,ax>9V3 V:`'F_3-]:ksbmzIxm=H 볌7/1vxC#{Fyk 薠tV׶Yy?t=<5꽗3;Gp}wlfkdz]U_o| ~MQ^/5kJq_ykjX}YR(~ob7j(b83(Yo7wh:7]/Yk:Y3+edl{],>չvEj z/.t9ּBgQۿ8k5њ8f88:Tko->N5n^Я9ƪYyiʏG3:bbOkkXTCTxº7BǣHx")#7j!%CRcI:NclM~}ܙۺ~r"0J "O6Gi*aEy{|nZkkY ̫\Fk=^Xjf.d6ۨq?c^wn}},zi콚YHs c}וkFǒ-r~wpPD:v56&zzԺȚ׻ȘV gj&:6~@C|*|ư'l[v>9y T/lYYv5iөY֎;s=;s^nemq}hS%߾9}j56.½ډ>]վ_X~oʚF4/'vzi^Ŝ9ul>[o_>utϻPu[efl>y9RSEY//[kyͶ_.իY:ľ ::eCX)CK~t/ǃ^@O) n^_R<={'v>w@sC{'R]oor-r7E+K 7N#8Gp4;@hw>x?} `G.:6]u;~8QWUët:v7W3vvya3ܻWW9wk{3ux-7zv9jvy>ֱtz7;Yfo1X7|1eޗvy׾w:[|磏XK!{_ckNqݻyׇow?q]|d϶iŞs,No}MSy_bxzz}{3]YvgڀΖ@^}XdG~הk>||^7w!o؟2m&Թ[>دg텓o^8])CX:e|֘_S$!$3:-FFr?[WÖssB}AsWK[B5娚b?o@B78vY+6 \/!7u΀tK[O9:jʑ5spJeN8‡p !ku[`H~S%D }0@\^ÂzMa7CLZ^0JF}r.5"@'65/w̝CxƾLa}:^.u\ZwCg=סa5z=5]F;k=^M_߼Yo8'ceui/^˾3뵝ʪiZWr}_j?z5kfa#Z/d{YձstvZFv oso[gzMutB2`w?ؿ/kO"7cbKOy/6vA?gcٿWu-_ٶBtcux>psǛ+V c#q>*UŷEK۲}YFzLqy8gྯGo^j;ֵa9pvw=ƚӸqU3Yn{]t^j=[ZsΎƻVaCYqk彠Af-ڙ={'j@?2xz%CLP/ q:?Bp(c!hAR6?%m?"oS=_ܗ&[xz"<-ڎyXoֺ^>kR{xk8ӛW+9}~U>k/|_kB4|Q#?Ju 5u}~,KM羫,]VaNZk;ˍOsQkkК%/릏Ukv>?1o5O~0zZk=\_t\^jQgf-z>[J>_}x<ֶ~3;\]oqo) _szeCoT4xoS=KfUe\KX$Ϗ˪8fkXwD}) Rs yq},^?Vk^05$cyk_#\ bn Y{ [?kǺ̳\X~]mY/Y\}\=/]:9KZn+,4k쳼v_Y?Sz] Y5}t_g5ss1TڏpKGß+jazZ>^wv ChA .ɂ.zCvzSvMC $̻ 2*PB@?OQ7w GP(aN\2jȵvkhќ'keR=q1jΫ0kkubc2owU gzDqd$ײϖB؎8\pm޶WV^? sUͶf˾׵Q`KqY^ZYCMmoqeiޛB4̲7ƛ<}dq8nO>kk,[cj7c8%_]l{}z%rknwmkS^{^PA[_')Ƚ5oX枑{Aؾ+?k˶1'0vRFIg 8v\[vM7wcl}g]jgOն\b3~snw^ _r }ͯ{GͶm^Oh# vx¢L$cHh/\%c-Ovt+/rrdp6^ކ0~۵L!j=#*yWms[X_ZhX^;0Z$z\׶k˛Ս9/]5ʺ~[y15w[>U3E>'qoxxx<^S;j6ӝW\}wzM)뼟vZ|u[S;߬KEz̲77kg!a /C1.1} ON9cfÚ3ZH5vG%em{kb l{5ݎ1|-+{a`7Vқ37B麢.Ub^|4}=Zͬzfu(+P]^vwM*p߯.-r nhIN'J}*{خtHk/{jϱ؇rakRYb>׬%]wZ^wv gK&[ݹ!aO[&Htύ{c6"Z/JC=[NYg#aТl[>&6wڎ请Ϛ@9OW#3썫y|? }{.ȟkm=ם׫{|TogǭGvdU31.q?ckϹ| ;Rs-:5qߖ;ިuҞs3>yU4֬1]{5ksl6e\LY e=W *a?|_ b\"'0dSVf?un\3 [ƴت}&&bhǟ׼Xzms6cB6O=gպٴ\FmL&gxUƲ}=ju?se[u텁{kR1O:ףlyk8)]Ɯzls-[}گ횙Zi]+X|N[u)皏5kS!wuX v4!H7ě(֛E3n$m3d[ku<>YYڳPdo1.цu!o&X8+sM5R5k"yza fIY;'YGK}sq6}mymsټ6"\} yٴY!LV3C#9'5 ӵa}^\~u_msryy]Zףqkף5gqgqqu5[|.ϭ= jݺT[a߉_s"{U۳ZOgXu,xŞ׬-LU =П"uWo "5p 7#͉ƌ&Ʒ%,IԘbhSADclM~}ܙۺ~r"0AJ "O6Gi*aEy{|nZkkY ̫\Fk=^Xjf.d6ۨq?c^wn}},zi콚YHs c}וkFǒ-r~wpPD:v5 kJ}uj]dlxL^x~ɯͲ7볟ǎb7˟׭7p<&dc-s+"=ۅ JPŐ%1 "޼9kβ&Ͷ{:5+5Ts~1s.vsskm-mBjEZͬcpFv,pkcף9/Gj=[Affs6綴ݻb=ZU)lPǖ(6ck\G[ Uu_j6Qi{Gc+z9W^򲵖=с{lk+=Y_e5&p}\`p~ 1?w>-+е9&z@п \y"'S俒^@_x?} `G{'N#8Gp4;@{ϯ׷+7W3y׾z~g/_%Ǿ^^}wgM i]M7Ϗg9d}S?Ko+Nv{ӿ.OVv?\/NZ;6+?~[~]?.mW=S5kc)~[kycsKu>!!oO'NVo?~v KM8UplׄS !k}jM}V xp}p2, u >]+ A:?ߛ\7gҼzojۅ빽qVڸy~gg~>Pլ"_y]Z+8i0 { @ߊkVo9xB  wqZW PC Lvث@жGMq[(qvm&[.t>Bj]0nց~Sŭzk)/1Kյ'待{3]oܽ~yk7t7]xjƞu< @ +"t܏zˋxNw4/a8G[뼁R#WKh!/o7?7?}Q~s?X 97зmǿ}}=k z+5ǖϋq>q׮[Y3y3Yж^'ϜZF~.v]_k4w\YgfYVz cnQ'u6}@X `{D>_mGKm+晄q}k-u8tYZ (7Yp_|EXyMq5^ϖkbM10Ŭr~J!kի|^v9^^}fmzχyZ\jrZ˶soW\e,e?p:0j@"t_Wc?5wyf C2os1z:rHb`;[Zu=1y}܁~|c޽/BF&<'7NXa@vN ?q:n16})a{6y}o[N kE{𠤀. TBEM.2^;6g\5pY׫oV7umַ;^Wמxdr]],Y'" hjGY?Y;߇?w>vrWsט諾Ni0v wY|oYd~@o>X68Bk`m]*km;~|Hc c#>o^Ԩ7F|48 fe0[m%X5JRf6jpcn8:Ϧ?\Eün}=nd y׏kLaYqn3 MR{x&xMYʼb#5 llصvzESg=\vn~ֺ}nugG5{'lST#$5fn*[ߜk|ky[c`z*p:kmҷxRZn1'u5\oķ:0:7{j+1bϵ1C:G}7pGKq?6~P皯%cC^+쿦tYMbf#;`M9vt]lCv1ϣk__g|Y߷I{=ʽ ?r}P=޲jopJNn@@Y:[*anM ro7tDact6/Mʘkoܸjsޛz3\oķfy?IEd"| [0"ڪ P%g\4s>~=ǿ__qR~/,u~0uʶ{Z>(>sSBryMq׳kN; c%/rCۺ&8;5[>mrZm[b.XV{Zk=d9Y:GveB-9sֺ9;~Vֻ{OvZc)>j^l.kXN8с~ q,tϑq0.;' ogvT}]6=ul9X5࠾^6Fho]R^ލRUX-u7ӕ{Xq_7uNѷPKmlƱ2YsBo^|kcB}0BS}ԼNvm^Mj?;Oۛޓ|S~w|9v5]u5z֤_s֚rn&i:DM_~͚Y{eYh[b!kkg^/nl߮j! :7i}b9ϊ\Zk[aGM5^-Zec^-I8eGme opЍa?Pxhx?}\~|@!!8w> ?$N#8Gp4;@hw>x?} `Gzz}{;ݾzq3~u==w{8ھ Xwb{kVw*N_>3dzu w3]Yi=j;;שǰIkYݻHu׃Ӿ>mgr1;0_SHwr /߹cV^ϗ rԩou^{ݻF'x >{~.gר^+ʻ5iZvKzV6v lSܫf'߼6/gUo=]㏟L?ǟ_>vlV8/ұ2oڮ{k֦VSI5FǮl]?T4m|rGhMߟ^Nܯ??q?^$?W?ku?/kgN?@lGЉ=o&ܡfCjj7ays9˛;tnu۶ew4dp O=|7G su?nsy^ k{jz|#Ͽ9M.t*?k~8g\gYewwo=П'vz?Ͻ}>|T췑J)kOhx2o_SN5G6>|?m_jstƷ]τohg [\m~wrghfB ! m9Pܚ֮b/(jQ]{Rfvg߈W@_#yƿU1 Qz~\\^sËsj;oӍIƭu@_-@(~pqƵgr?vzo{{g>[U/{S!Yl̷)ǻycc똓VF"ww~˺gn6/:նY_q[^m{_sYYg=fC554Vzsc˽l^d} GGCqFQ{Xl*ƿilz=ךjf^kMmI5{i5Y3yZ敿nTuyUϊj^ImBY]rzj5z sO؊6jEi{IE_ȽǝkC=_e&ueýcs?r\# 9u[Ⱥ\Y윬]Ƹ#s/CZwt}oYGGlKXeXX>Ըp]v5/۳5%OyRv1so}'_LoZ"#*qޙ۴k=sˎUPk/t{WҺɹZ=&_zַ~~Ԯ]3ۡ5K5YrXFvGdG{>z=:t=߼&=֦^/B9v{+D o#kjZY[EVjk_Ƙv.Ym=lfl%7uY}_clQk^.zk֟W⎛czh}y`=T}uʚ~;5d='USqNdZ1_zj'u6}@X `{D>_mGKm+晄q}k-u8tYZ (7Ypt1?.k:vVo=j55E_Zۏ1¯vK7ֺϜ^s=!] ~6UBR,:V\Yg[xk~,Q52ZjW|/>.ϋի|sy#4I=/]8ZU*t_k<:y_+f\[#^_S|}&ǖͻ%<.j^[[<'KgꩭU ˱W o{'~H5Ic5LךθZznauz{i>mq{P ׶㹿t~Gx j"3oZ)k&du_rz"ϫqƏ3oysߢ}סty>yumB>-9Wk/I2׮YQgױ5c5+jOJ5hnos 5j5kkUZȜG='v{\r=j/,ۓ3ߖ8߮h˷oz,m狵)QqY)ETj#ocv:J:}gb>;l=ykӵiZNkWWW̱[׸:+C_>5y A 3kApP|{X??m'_9C~Oڬ5̯:\@ޏK ㋗xnjFׄD{\unz_'~y _z3^ϫ wb\++>p"Kc]7rY?@epjmY/VfR3I1^mݙ6^if=en]Z <+5Y [s>D6~5Cw!=n{ͱUUs/lS%׎x1̚N 1⚍sZq;K{SHk̍C]gۄ)(Vڲ]9޼>X)׍Cèʼl}֮ٛ6Kqs5gϫyWnx^_eV>5z&_?}U]e {'~(ڭۼ'.jbա}Ȇ,jRs6|sqMڗ_ӾG:.3&_^26G|>뿭uz5Vmk5sU4C2jU >uk+kY6G|-+>wqkfc!5k^3[5sPZA+uRV~~v:p}گmvr,7F]]-Q墬[ZQxk:|1mwcyv_ _Sz{7/{}u95kk=u gE/jXc!ǯ}WZ[?&qXu@^EP7x+~W+_(/ {A`c? \e_E/mQ$m:Ϧ?\ݡZz®n k]3oP߇2 Y||^39ؽWGkS/=Qyy0ƚ]y \2:=sm39 F=eqv'?o^O1kf˯z8#Xkչ}]I7ܛƹ͚㚥:,k},^_:sT+қWX}ͱlS\/+1Ty:Y3ߺ6*.UCg_eA8͵~mPk~>Ae YCwn1:?Vzk\]kb;1CѮ߬k[o֨;džUԤ*j1XOծMj[g־7nnz^BzVVu]گm}ͨo:[Y ^Xפ׀W\fR_؍?_ZJi}ܞnzSe qm} OUqLfEMkffr=''_7>o Y̯gX[׏{>oᛱw`[{\G)Wd;\cԣXOjZٞ׺q}̫~q]e$kqoYѶ^oUף[껼6f_kǝݗ[\.SolST#$5fn*[ߜk|ky[c`z*p:kmҷxRZn1'u%^H {/sfzv/E=W}AϏZ\4Ao/v͇Á] }+' ugez?[l4k{0vos)k{\5) Ξ -٩}065=ͬ}<_Vj {\-a^53k9gq1s}m]_k9u/gVn7;6kf/go}yիrMe yyݿnO*ەcͫ?nqg[_+MQ-{ΐh[R{5kFmͫp==Y;'e~6X]s~'w.ŽuLe;*3-yޱFLgf͚ZU_k>wj󽫞Cڗ:ucZV=rVYQ٪YkSQfJ\Mt?s~p:,uϹsܼ+ktz{]שd_gb˱ce=/{USkF[|5]^{ϊyY}d1| ۋ{>8:ׁ>R skbo~9#Ep=?XзǧT}}oWduV^c?UċѭK_ҋlx.^P͹ȱ? A/@lh^~~}IzC@!-~H XjW {0toǢF c5|[kњXmy,L}gcQO{fMp?^ocʞu3ͽ#qlO;|M涾+k^ޒksg;C^cX)|j356'U[fug5^S{#{!+o+b~o[f6ݦ5-uWƼjZEoh\qV;rg?_b˪VOzܫ?gyXOIFrɷo-7N6&~Wofp}Tjڵ_++j^:{aVν98?%KgT{zZW_+1k3{tz7M~;orl^>Y_tm7V͋1x{q;Z]}6ϗkv-Oo弊yj5k|jlpDwkׁ|i^>ԮƯפ$x?/pbjz;Gw=e-y4~?^X'^jqc/֡3/Ǐ10aFϩ캝lw gmk}j6PoѷOym>Nekkz*o|0Џ!92 epW?Al@BߎʷѶ˶tu_0kc]j{Kڻ1^J^(fy^k~~ Pu. yF&g__? vOi>3d9Cuٯ e{8y?mg9?k{T}F93Yft5 K1 b1nyܵZF1|d{-ׯpO{zi,=5^6Xؚ⵩0zFYW?PcϯsoϦ{sRrn.}k%JoY3Gk꿢ؿF?{AEk7{(Um[9M;m}_7y1Z)}7׾[>n׊5Z}m m}Pl-ܱ|n1s[k&/;QFKEMRߝqzd׎~͊w~%um^뉱tY&5l ܛ3zV/;7CZǔ[z5o_z7t߲ubmjreUk:ɺ\M2]޻Xvli~}] m=N)tGP䉘 I:- Bx7pW+qzĶghu{XyؔZOz)Gv~kAe.y8ew _=%չȶx883^k%Ϯ{ r6wOe{اX>o= 7?(@V0?q;\xOaȓ,8͔xs H0b(gFwg[}w1_A< I>T矶7bI@ s sv`t@q3O]@yD;{&>Z [>mB*^8>,y!z9|z.>C:;{! Ti3Yq[x?qߴWVRh߳m`& A08;p>A08;p>A08;p>w-迾10۟n÷_/vko sxvϏ%Oo9OyfuyW= {9?g{cg~[?| !w?^>q'g]'[z|7ߪ{uO/5^\Hz\ϝg>~Μd3mi7[,4g'>l J4;U8;&M;,u=7_py덝ΙuJ@lzEUl}Y΅eOsoPk uD% zÞ W?^ 5ED⺦|U93׬E}Grڳ=.>;r߄,)R&J铞7CD}Aӭf{r_d7~Y۞OP|O~ltx"ߴoA {ݸn)n}5Ͷoi]kqr{9]D,4'a|yď2j[>!=:t&cʹfxLⓧk2gX`)1,%T=WѪ'=VcʈQOާ~H¾k7EXA0Gb.1D2gƷ4j]1:]?+k\-?'/lm\sA=Pc_Ί_X΍;c}j?_x73odK^_x־s֍l:6ֺzl+}I{(gHf :+|ȹR>WEt{tDH`볨Y_6OՍ}_ڷQqx?XYWtD+˜9o~l~ɸoseL # ֑-/59 Yk-u5<֣Bv#/Kٕ>YǶiEηk͙z"v܃X}鳼~]cD{gBܷ_Uzk2f1+]b=̯qD=Dάߴ m{9[/B-E,UԫBz]gY$٨b%V+nk鵪^pm1kZS2u5Q zsR|]}a|_W[l}ۮϊpN }߭K0k[e5gu6崌uLKvssAΒLm1o^X^˾$_ίERl<`N/>\8\ m`_|v$A"@":xQś0(Yq[H%!8Yh"K9#x=՛KZZYCG7V'ͦݏ ZlEq2fT +X4߼ZJoчerb[BXguϖzd ۑyn=Zi%rc gʗ.sݼ۽[:q;w]k-Uߦyr6O_Ÿ0BÂ8Цq)C# |\5lmmnAקJmA\Vumk#_ U}֨o;ä9|^PB[Aq?~B#qgx/0ףAmmQ/L_ vdIbw|#-]s:3S0u!}-s/͏)u\q6}Ե%}16>s6OYN }_bѣ`N&l'8<:$ &qHmN[DHIsci,C Z݌⬍Ts{}ԣ{8vD\Q-*o6l?\~zHon  Ͽ}#3G!osyQs~}$"c DΜ{#QxYc{(Vgl0`$a{s_}9%ߣql1Ql6 &#z,r{o֤w~?c sp剮9"WڳQ&rl|}T&NO$5?^,ڎe 3 ^_{|(Wbݺ%=z-xyaM^b{X O}ڵ\o} ztbc^kx#ߛ_]zy]K~qxG _36͟GBՔ?g/vq'sg{뇾k2gq#zoi$wr>z!ӽ?Fs_Sm~?g3}?xtPwX`~1=…8/3o9R9 "-?' #e?g;Bњ0b'l8y}#lkZ^0⒞gf 㢯z|l3"g{!_TtNt\yQ>^\&l9^\WƠz}3KHף3sr!\9KC 5Glc9GZzzgWzb TDg'^Nxź/o ω"v\-ԳQ(hR){d!1)xFDؖ&1y[]fEk^I)ӲȡۋZWe-X}ѫNJ֜9̩3VD=(}9c~-f>Yܸ>:(gZ]v1fm̉͟|FȱʖOeA'bkY]sƟw|j![I,x>B1W,bmaږ_%"}EBЌsmbIAL C77Y]~fbZڶ"Vl}Bzr6 ND;zpݸe_Ǖj何5{ٜu3ŜMq2hmeOmLLexoPczQ;⓾kHש^*o/G9[u&T}[5+:&ٿya{OagVd\7&bQF7}`/}p_HQ91mA=Vts>Z I6 xŠ8;p>~}\W`?pv| Ӣ~?eAϺп \~{!oN}@8 N}@8 N}@8 N] /ykg [{<'@z<]=q1[gr޽cF^kOx?_>B/ݏOIٷA~Vyͷjb iF>^:׮sş_x3)lL[l;}qz9< ÂN?&y=7_py덝ΙuJ@lzEUl}~?ͽyCApO)gu$B {"`W/\U@p{jR+!ŋ"qɎ[W] zZlY &6dL갭tՙIϛ !QvΉ> G[Aqς=V9/g2?SNY}scO} P-߃uKt+[Lo}OP_sr~g'^N=> ''IX^'񣌷VEHϟ,xe|BVq>&>a&إoJ,Jc UU* v,1*i]CW~M+Q̅^\4FoK(sf|~BW^ynq?ۂ5染Eki.Ϲàs/gůKN]Fj㱏>z}?ϙ7]brzNS//<_k9Fe^Nk]kgMϤkjm9ȹS_.x%{ $ޡz:` X9_śou_-sT\b3|v +j{w<{}36r͏-O7Wmvϴ[ỞsZ=+#[_jl;rumкş7nzTȎS}de}I6=RR9>v]{mײ9}\Q\_ٮ[bY^ڮΊ{1"U3yun}z|[ksVx׸"goZq&H1N-o"M)*UA!,uP#g>B;' -ZH,yx:qg60e/Fԣucm;OB|V/ut<1peWzM񭐽KlmXcYOAzu{^`G%_8m_h?[Ŷ"BCxw9 DjVkYݻi9-c>үw/cy̛9{ײ/ ׶k!E˾?*48g#zJ|v$A"@":H\ C̐JP"`*GΟ!3 ^55}tce}l8~hV0.#`FмOͫm}X/W(/(9~u^ZlG&۰#Y\"8J0 rF{|2wYͻݻگ^pW5r^Em(g<ު~ & }nC͇zqݯְqOx g[|onAקJmA\Vumk#U>rkԷaMH/p]>uO(:;j hyu{[[e $ &qFmN  4i~.s,6?~HA࢒B`4jnϜz}yg"Ю+EEmO)V͍?!co$b(mN56oucַ#9c|>5~B#q!r2޷];G:c nN~qs$q{ZY(>(6K=Asפw~?3?Ξ^_?+bxq=kum"zCk /mljM2?|= VeWS+1 n]gb{Ȟc{=z|a=:115OgeIF W:v^푴wՏႭ1qؼ5;_ SU)H#!4tElfqD==|; E\Fjy؄2+tX"lJA+6#ή7;lQuY7Np̽voɲ//-{.#}ksi(z!!}vs[?f Nzp0lx'jϫwm0`[7Nb{xe>! (կ5!Hy #X_3YBńs 'OrSsŵ 23[8<}㾸5bvw<禖Ql#lIGz!k(ZbS/]'7.,'l3-e{=9G\a^G0a{Tpoٿ2ۯ۽ZG۰F}9,g`?gn_3fl4hkk|Zu{pYƩQ|vlA_5I`"/ EACbT*ɮ6jq&{z=1f Wk򨅝V8%-֋ZqIGc}lF}ElY ȥ>x#ߛ_]zy]K~qxG _36͟GBՔ?g/vq'sg{뇾k2gq#zoi$wr>z!ӽ?Fs_Sm~?g[!gxל,ĴCv}P$(y"_̆7pke b[Tκ(lHlY9+}'I~3)2N^6`5Vĵ白ø 'Ħ^E?7ۃșƞz9Wks}Wz jWF1uv9>+qD~:wH'|+D}PavQw?([X/gN`ѨDFř|1l 6/XW%M9Q$nK~OQ(hR){d!1)xFDؖ&1y[]fEk^I)ӲȡۋZWe-X}ѫNJ֜9̩3VD=(}9c~-f>Yܸ>:(gZ]v1fm̉͟|FȱʖOeA'bkY]3.FC~5 sX|8oE`g@fmK_x܋$h JEdU4ZZ,*Wo/n,t},2mElSo,*?̋mZq/}[֣Rژ2fu/3nƶ).Bƶփճ̗⩭ jl]/avX|ҷ~Mx:+@(g ΄vX{C$?/l|~)a֊+=fW5l8 Â>/kච`A@ >hA/J_&Y$$oϋ| =AnWnu}_Acg'><-WkS7k !(_Iw&N~@8 N}|g'J˿s1ܱ/1A0xv?\b˿uO^:O5,g >!/1Qu{ r.K qz?gw\/ܢ.?{սq Ly?J\/xolr=oLdzOu/?"*ӡ#-Y~n `g67Vݴ[$,o`[E-SSb?{9MOЏ5^k?y^[ [  yS8}_C ykoio<nmgmv-}?˿?W^wc%GdMѯ9M cSж۰'kRL.}V *"zvEN*Q@Hkb!y(=Xk:_?}~&<^/Yѹ g@ϑ/sg~y7/~7B~~L֟3?6:O*O7xV\C{z!G Wx<ov<%MiKm_7'/Y"g9;o^]I>/Ƽ5sB'O._ *E3"UD*>=隍bXqtJ.X>*kգٸ??+j]Қ6={4~f\9Etj>\zcs][}a<A~c?m)r[޼?8=:OKC=* p5Nk)D⺦|'[W] zZJ㳓Q?/_'^=^UY(jb*lWEq6 O\/Dx}^ }tMjUOZXb0k3H&ޱ)z5Q7yI>yY~Q[zzrن[ͩ9k{Y3~{:&뿭&dL,Q$̈́t;QDA;c?,߳og0)g,}z?ɏ?O6-@~~-ҭܳo3}fksř9ۊNqX_p}?$­?9-4Yyb+ab^=D_=3cF$e=P`ۣE쮩"kk=j'VmT!tDV_ 9/f<qz^]k#d/k%_ԋh3+4;[>4N=> 0A:eն}BztĒ6g>a&إoJ Kc UU*c.ՂFJ}>CZW_S~(B/.S3N_V{u_G^gi}]ӱ-zYYj9m9yak_< >rVZ4unٯ6Wû}#%&D^:nd[ֱջfk]Q}L޻F9F5Xй^Eؐi8/t/Oyz"@$Y/|:eow3KXYWtD+˜9o~l~ɸoseL # ֑-/59 Yk-u5<֣Bv#/Kٕ>YǶiEηk͙z"v܃X}鳼~]cD{gBܷ_Uzk2f1+]b=̯qD=Dάߴ mXn-^!fa 1\p'1-D OG3'&𞁴E4'W؊=jhyڷR;m{=?eJnz]'soU\/v-l (^n=]{W}Mr2ۘΙ^!m9Lc1K%{}٣֓ -ep_.Z4Y YǫW϶HyB@ŎeE Iߚ%rPW[ ^]9MޯX*XrnO=aYYƛ^+5eN_/cq\z_A9({^QI{:'5Nu֧l=Pzt:뾚꿶UZsVnsZNXdk7.9m=,zvka^K¿Z,cqze/#ez8hWE`Ob.Ӌ?9ϺO""^ׂ^,ՀONBGDT.݂˟0 V^ƵtK(("E!QƩ~/m+SNXݮfq/%z/\׋Dt?Wx~Zs5ECٻ~G3vkN8˹@\hkQk;Εbkj.!r gBh D$.hA_C#d!$m"dyJgD -Wo.kMke }XY4v?N,hi=Qm44c|j)mE n= qb96?[6oGrkm}&Ο%*_bk]ֿwnngvM&W~~=<&Zb_ž% 0nCCD=h6 Op!lk8 D /w 2?S~{UE|]?gA*~Gu=g= Jh+:'o^Hs|d5 z4:-ٿ2 A8q+Z֮l9PozEkNG}F{*ۼ9}˜9e_1Yۻk^6æ/ƦGq$Z}U/"6Ɗ\bY!- bo O=!ļH$Hz]D,rmdI?ٶjn^h@}k,ս<[.\_3m_\zEU۸èW[,i폚6?^?X~7ӫF=U"5㾉}l_1>TN/a0'd{yMJvD p6'_ G o?9NB?EpQH!HxM5gN=ھ3hhNբfs6yᧇ+Ɵ[h}71zb6ϱzqۑymIe?G,G +2B̹7eo5_wޏbuܸ HL1ܧtlL,FX߬IA?c sp)ݏ1Ҟ5׌_6ck=_'/mljM2?|= VeWS+1 n]gb{Ȟc{=z|a=:115OgeIF W:v^푴wՏႭ1qؼ5;_mӎg' >?S˿J/Hq_/r$XeˎqU O}yl'"ļ!M97t|a}Us ǪeOk\zb/4JX.k^zzYSDK~Xq{G^/4Z?tux9dxlXgKב9?й~}{ԫYD>r$HE ]Q$ۦY&hXz_G![ˈU-o^fŕP[ͳ#Ri1h~>1f7.3!C5@=so][jKƁ{˞k}\7ʻ^E~G=;C9c~֏>&zӭ g'[4ɵ][p3+#Ϳol+ Ql_k.BA Faf  7O0nĵ 23[8+17[#f'͕97bfsM䔯= yO]lUFqz:qa9a{!o/a>⚵e|: ۣ{~=j?r߆4͹e?9s1c֧y&Fg^[{"ýX/_mc|vi/'HD.b$.5E$-<]zK}?9귾7lXz~ǴmD\7,ٷS!=GJ껨y>9G!rŚ$f0!DYQD"bJtHqw]#$m$L6W)ǡ ek*0zb0Q ;97~]qʊN^ZK:f7.gP@.Y|w;y_+3Zzs(=E(]~ٵil=9c~{=g;ۣD^?\9~{K[&'i#5 n1zn9ۆ5ICQpwX`~1=EIJ#X# 2+sе=!rVhOKf6Sne>m6׵krkc/qI3{3qANM~>yng3 =s*:':<^(t/}^I/+cP~=_ťsы}VTuL9sOV„횣w~Pr߱^Ϝ_ce{ޣQ^==[3׫be=16ⳓOk݋u uU񄮑԰bᲘ=#yU>X'`&Vզ'P+zoqyƞl^qAGb^M?/=ɘ)UoqW^k>F׫_57ϫkk|qd=Pce_to$_-LT5[1;NԼn^<=+ Jb]7Dl .0&L"{/<SMx;z,BT6_҈r$&bkb W+>i6~>cZ־90v{q@[s㲵/6zXQš3 9}fʞGž/g/}7Q{'C^}[.X5{޻99V?D?S}M>9kD3 -LUqa~OD|8oE'.cLl#|[Xc/[&trQgBg`Qcbc˟6_ Qk>_vkkEƕ~m+嬏ׂQI|v4hyy?",k$R#DD̠}abXWX$z"g%6Xb~ͤORsFb_z?#',1VWu"֟>k:1 $A{<=^ycUnkz{co&ްVt.<ʵԿڷ}Ge&&bKaARTzN5p[P0EkEM..𼰢1=|忾 #&ߛ+piHpbPٞ.*EG>_mgNJy֬Pc`Y =?s=!}^^heo/?ՓΠqV^.~x^/䘈CQTJ>Sfr#Qke$ʃWz͋lۼ8|<?=<AnWnu}_AcN˿^}HwuW!m2]9Ɖ_j39y>y3?Z&53O+sjZ5Y`}Jcd A"~=5&l_#L?_<9%ǷE}7Dπ~6ע􂳎Z#ž׳˿3Y)&rVH]A+<Dž&:ƶba"Rv|T[$x-<y[ox=ĺ/|$pu]γ?~|kgS2xę4uw∛-ry6}%*#C `G_.o9Sп@ -_v A_Пϔ3r&c6z{o_S϶#<-{PhaOꅫ n/\Sqa"xq]SD>ݓﭫrfYlnC=|-YaK{9Io"I Ja}fB~HsO?pbg{A &c3h8\xK?'|}(P=n\HrϾU#N_C=|-,邾'gp!? kIĶV۪v&$lӚo"B>a&إoJ Kc UUt* v,1*i]CW~M+Q̅^\4FoK(sf|~BW^ynq?ۂ5染Eki.Ϲàs/gůKN]Fj㱏>z}?ϙ7]brzNS//<_k9Fe^Nk]kgMϤk~@z.rƆOU{loC|o|0;Tϟ| Y/|<~i2G%;9gnqz|wmϳWЗ9cs(Wb$|qE&lO^L9GzA" #[_jl;rz@P˺ߚ_[ GQ!;vLɗ%JKsnc["u]Lq=FqewomAZbY^ڮΊ{1"U3yun}z|[ksVx׸"goZqg;N-o"MH*UA!,uPΌ>0 Тo͒PQ9-?iS]]kD=xX<+So0.yz7j"^Ϻ^йQ\W5=?DƷBJ/ɾ"bEg9J>QsJ{˚ezr9-sVKȇ4^cQ{(0g?ze)[N_}6]Aǵ|Y#>0e$}~=g_d=ɇw(~E}N!zO*$_z] !pTΚؐ_~bI/~qźW<~*dө+dA*[ioxD~;{Oخ}V֯ת{{JiMWXױ^׸GyFP䃰ƮzTIus|]m.[l>+9y{O,:}:.àNĺm֜սۜ23/͟Ka[ϩ~9K^0=<ǼyZcWz-m;9KX9^d+pًHsY.s'r3Qg'^N 5 !4"x~ȇ`ЌBHۊE%(eg'U⏜?C$ri1gzs ^kZ+kٴqbAKc>(a\FjyeWKi-_P_pQs̽ᵴRLa};36ϭG^+mD3q`,V]eww_7n'?3k2q弊3Q$3#ՏQVKӽ7FhXt8.sb~yK,\=,+<-z .Zl۵~*zUx5q0i{&_{V>uO(:;j hyu{[[e `?wbqbˬ!bqyvD*-X܏'F;F}eF4dgKM}8~yosy[KFy cȴg{(g/v?71gDplԃd;V{^k N~qsغwb]ã1n악hkxi{دxb9…'Ly[HZW-sj 6Iq_1;i̹e6kl&|^k~g2 dɍ ۃ +| |^h׬-CQ/L$ܛp avV6G'Qmeo/Kϙ׌۴>31>k6߻p*}1:=笏ַ{9[bM3K"("m@1%:Ta~]HvW3\ރDUe׫EXG-ܰu))h^ZK:f7.gZ-5r}V?Gq1Gޗ7 ^8JnJb!gvm?[)cΘ_㾥{O(}+dzk"GҖIZ}nBv={)澦~ d+ʑ$ 10[^\&l9^\WƠz}3KHף3sr!\9KC 5Glc9GZzzgWzb:GAΈ/XW%M9Q$nK~II3сI/<SMx;z,BT6_҈r$&bkb d7fS^Zڱ89.[bWu/9s SgvzQrb[z|}q}bQc̾:?c-qO3CxȜ͡sƟw >%*؂8DŽy1E-Y7Z W+bBfWlkMG rfjM_a23жfM_V׫X3/^pjmĽ&ڱ=녣E(:T6&בֿYL-lGk`,xjeb*s{[׋zؽ1_^GN Pyk}9ق331^1 5{ ;{ص"6{r6F,!>;rX#q aE{=187*}Qdp <-WkS7k !(_w&N~@8 N}@8 N}@8 N}ܵ˯߿v,oU^//1pT;o#???u<-ݛ9Zn^37[.}חϾKol/_So,x˟}t'nɛZ<|K.!=xqm#酫sz>\zcs][}a<A~c?m)r#/Oso{9?Ip}˶#<-ID^8WV)BN 5E=y޺*gȶV?`곈r|v$A$Y8S:&o$Iϛ !QvΉ> ϟ ~P{ٷ 3ϔ]'A(>{ 7-ly0&sO!.}SbX%KzUV=Vc߈QOާ~H¾k7EXA0Gb.1~[B3[:w#w3Ǵt=tq6/ZKsyupC}9+~}_^bv:7W}||쾑s"/z~ZY7-rZ]>mz&E]}u.й^ᾋ!>&x%{ $ޡzf{ X9/Y_6_u>~i2G%;9gnqz|wmϳWЗ9cs(Wb$|qE&l˘L9GzA" #[_jl;rVvO[yC1/k~yla/,G1G&_֗dC߳+5/%Ϲ}mӊo׵v-3qE(: ܽi]6gyj;+jLj7W9Kτ:չoտΫ5nebWd[f{_>zYi-t9@Fg'^PE?qa" $x^vI:O) X# -fC(۟?.{5eծuZqF,t>E Ywps!|7=_gp߳ăP>׫@l2u9bɊYկU,1ů0._XڞYl:5_~rlTDŒs+| og 5Z~o/_5)sRx: (Aك|zUJ[_9)qھ0>~->emgE8'oRCǥsԉXTךwrZ:}&_s9l9 g_ Ƕ7/_ sJe_m"g)6C0'Ջ}e.{qTi.˅q=oN${YHB ZyApÆ=8FBrE JBLq\%3D"sFzX7൦nOM'4يę6Wֱiyߢ 8^K-dַ#9cz6K>A ƒAho/1޵].߻y{Zupv3 &ZΫM?lD-﫚_Ÿ0BÂ8\>k5lbBfq@^d~ |}ʯ"vmʠ^Gnv\:Lڞ%i7`/Na>w{c{=~^֖_„ 8A kW@T(vǷa="5>=m^SN>a2r/rjkh]5^/paG][c8gz@g'^N/dHW0"ypI>r+L(‘ۜ|%A*$hҾ\X8m~ E%k7#8k#i6^9hD];WT%dRBouH Qۜjl>ǭoGr&}j_+Į3H2^Xy?[pv3+#{x3r~6cb[0cA~F=N2/9…ݏ1Ҟ\x(_ \8=xh;NlA/.xk|^>x*Ҟ2_^v:CLѻS3>5 zaaM~?%vokrvщy}x u?wo/K7g`?g3lS~l1:}li ˩*IbHMs8ŞB~8)FZ(6!̊+b*gGbЊ}bDog]fDCkρzxdٗ=4owq>;VL{vrbs}L[AF=8OhkqmgW0GЉ'V1=x#ߛ_]zy]K~qxG _36͟GBՔ?g/vq'sg{뇾k2gq#zoi$wr>z!ӽ?Fs_Sm~?g=;؟c^Q$Yi_( 8 ; b6{o]5X# 2+sе=!rVhOKf6Sne>m6׵krkc/qI3{3qANM~>yng3 =s*:':<^(t/}^I/+cP~=_ťsы}VTuL9sOV„횣w~Pr߱^Ϝ_ce{ޣQ^==[3׫be=1}{"远xĈ/o ω"v\ia$LCD_$yT?8ʧ"wYH2mL .5ILk1Y,'W`}l}6Ǵ}+rhka2=bUYK; ek_l걢5gsA s*̮==Qb#}_Θ_,7~KoO547>ʙV']l8jw[g's"r%.BSC~|hsV牉wUn?|Z[=91z1+·56k[^&@ P("KW((:&֚tmԚ`9T1z|qc5eg&fYm/bgz/~{/dW1(g^ls:~ڒצ}Bpzt"}WUVF֬ess6EVۣzzR<219AE=O֯ #]z{lAԙPnqkzdWBԚ==Zq忇lF9#wve'߯!EH\sX^@@ >hA/J_&Y$$o + z\oIAc)cG3`?L~<8ƤOBǧp@n'#:p3fyZ9|sp;N1T[o#_?8st-7݇kOxqFnoO/solo:=Ӌ/83o,~8}Oŏ :Nq|7ߪ[^Hk6]Γ_~9|31_6-M}lv/Z_#gQ*+mWmqJoxܷ(|ywpzΞb% ~=#y *l׽쑳z997(gߜHq?>Fy\3V(^'C䏷cz )HNRב9yu({Zd7+_GOR\&>dpO|[Jh~>$]OQܧ_Jzm6A=nYܿe*`2+_S.Ym.ӋN~lxߴmq [ݸ)n}֟so[Q\/^Sl\C}O" w~ɗD[mU;s:p/SA i/7R_ 0dib]ģK0P\[kz lǿ`'wҺ VL룘 h槿PՅj|]1:]?+k\-?'/lm\sA=P㼜X΍;c}j?;3odK^_x־s֍l:6ֺzl+yIѽk>6Ϭ\pEؐ|?]x!_/襷7@b[`߈Y}^ߦ[w/$P?K9*.q>s}=KKk{s}36r-O7Wmvϴ[ỞsI/⹕֑-/53r VvO[yM1/k~yla/,\Z ٱcL/Ɇ>gWj^Jܟs}^[ksVxǸ"goZq<ĐHq\x 6J_:?΂I^6cK,hQi[>z`W[1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRq͢4O[f<ׇxK֗%}2<_{B@ۣE~A<;IHES>S$_\D-ykNjL|~+1ֽsV!N͗_'URJ;Ͼ=aYYƛ>ڟ -WLk..8z_A9({^gգuNk/jOubYsb!>i6~XX?d+{3hh^YR6~>,+zg?:ssx-m~#mX.#Y\"8J< rF{|2wYjL/+L\k9"6{xY~#{n2 ^ӽV6/}b~{"T,b=, /w 2?>Wo{⢵ʶk2W[l ‡ua/Na>w{c{=~^֖_„ 8A kW@T(vǷa="5>=m^SN>aѿe5}kc{Qזt($ɦ8þ 9Y8z $^^D/v$DG{F\D›#NDbMڷ OcR"dfpem$X<ͦ3m_ÙH@k'jQ~dzwb%2 FfPBlTcV|vIG'&>Y:)޽,Sk^=N1>\5&a[cu($8L_XE9 6RCW ɶinJ3K"iWC1pF fVhVE<;VmG#ov~>2# 2^}.g7KM}z޲2X߷6.b}g#ӮbE9c~֏^&zӭ g'[4ɵ8[p3+#Ϳol_b[sw,a"u_kK($0T^~j8Gqy~p{#0}㾸5bv\sS(m6$LNKz!c(ZbS/]'7.,'l3-e{=9Ga^G0a{Tpoٿ2ۯ=ZG۰F}9,k`?gn_3fl4hkk|Xñ=ˋUbuz8$ɦ8#}-$a $< $S!xTHvW3\ރUeǫ 3S"YG-ܰu*)n^ZK:"f7.gPLւ(.;!]}^gW8<ң[DӅX]#j8?g/vqҿsO(}+dzk"GҖIZȽnBv={~)漦~fɹ!k=򽴽pB~߿bB!?GH5酿Lj :H+ʶ,tmbsG>ZRhQ%V#fs]l}>&׊6<7c; |'M=#r{qN=뜊Ή+׵>݋+z_ׄm5Gҋ+#Ժ_~Fqzbfcn8"?US;c"gbQc̾:?yc-qO5ŜC&\O2^#T]xnaʧ~jlAo=1_[1&6{E-Y7bA $R1 QB3i&]#9sXU/^3_Xd jw,2mElSo,*?̋mZ/ xl!zzt"}WUVF֬ess6EVۣzzRH219ޠvow'}ׄבS=TZ_r L l8jWl=tL+!j{OvmȸlF9#U[ \ή>- mຠ`+ Ꮘ`^O#ɂ&qxxZXV@q.ܮ\c}]qqng5sxҵsO??^u&p=p@nqq 7}Ɓ8OFu+g rtt7voO?ݝ^yc`^ޘÓG~<,1&޿q<.9Znջמr=|ߞ^||uz/_xc pzgXwp}t'-oU%9xqm9O'>rz+gc2lbO[lt_qFUWڮ"oˣ_$}BΗw7)+Vbz8 Bo/~kʥ988@/c%zŞ8?^8]qz 9K)^\G ʗ~6:=wd-/᣸4gyn~L``u-z]~pD!zώ>R O ~P[oٷ~>>}q|&c5墜e%_}b!iq>z({ q]S,-V#Ͷo[Q\3ߵn6>B=A>HQD$V۪v h [m5$b9&svH7%%_*`_ ca;E#L>"CW~MaVL룘 h槿PՅj|]1:]?+k\-?'/lm\sA=P㼜X΍;c}j?;3odK^_x־s֍l:6ֺzl+yIѽkmh[=tW"jlH~E"~A/zx;A` X9 xr9e"}X]ǥ%ݵ=7lCK'ᛌ+6a;_gZ]ϹZ!:vFζB)u?)eo/-셅KQ!;vLɗ%JKsnc["u]Lq=FqebwomAZbY^ڮΊ{1"U5yun}k2f1+]b=̏qD=Dάߴ m#arő~ =lXW smg$DY RȢh5KCG堎xMuzxAWFq^ +$ֶ 59(˫G!K)]Ǔ/kN>kk|c<|H5U{A/?ϛ s?g[Ƒb`1ٽUgڱp\:ר889ۆkt>/ŋX/ɽ}N~!hG,P>ϫXl2syG *ԘW/b{umB6/{`g*[i7?"'lW>+xgUs{JiMWXeЯqr=a QI{:'5Nu֧l9Pzt:뾚꿶UZsVnsZNXdg7.9m=,zvca^K?Y,cqze/#ez86d_GshBt AD$4h!nP^̿!"9n+ ĚUH8Yhr"K ;#x=F"KZZYCG7V'ͦݏ[lEq2bfT +X4߼ZJoчer~[BXguϖzd %9cz6K>AǒAho/1޵].ܼ۽[:q;w]k-UߦyrS?>_Ÿ0Âx1O/ʥ0`^_"C@>U6P!lkP -<-z٥=*"vlʠ^Gnv\:Lڞ^ Jt+:'oׅ:;j hyu{[[e GYf`b 0Ȃxin_5i/'y"! cE )6^w?~VJ{\3~]DĮ~MHXk~X'6Yˠfl1^>x*Ҟ2_^v:CL;S3>5 zaaM~?%8֮=z֣,~f^tn~yũgl5^/I{XM.1:}lS=a*I‹IMsPK^dI1pF fVhVE<;VmG#ov~>2# 2^}.g7KM}z޲2X߷6.b}g#ӮbE9c~֏^&zӭ g'[4ɵ8[p3+#Ϳolܗab[sw,o PH`~1g-\q+ 9z5rG$a}qk8e6kl&|%1VeWo-ȮWv\#Y[Ƈ0^=I7_i#mXNL>ۜ^533i}gb}쵵mw,*}1:=lבq)k& L%AaV H V H&5FR\-dserzTV'Ld^ȓsEzQk5.tOMߨϺ-kcQ-gs{˾KܳAҫ_G-B, ̮MgP5e3;Ǹo'sg{뇾c2gq#zoi$wr^z!=?Fs^Sm~?g#r#kx+ʑ$S m-y~(xbB!?GH5奿3o9R9#-?' ]9"g;of3V#fs]l}>&׊6<7c; |'M=#r{qN=뜊Ή+׵>݋+z_ׄm5Gҋ+#Ժ_~Fqzbfcn8"?US;c"gg曨=e ͍crl[,c=ɜklTpP})&Zl5>~NtFŒO~jlAop}A#Ce'4cZ-}q/~-Iu_%4fXkҵ=b3SkP5ōEրvM̲^+C6_j6^|^ϭ*?̋mZ/ xl!zzt"}WUVF֬ess6EVۣzzRH219ޠvow'}ׄבS=TZ_r L l8jWl=tL+!j{OvmȸlF9C~x*skU# \a[=1|ɂ^@x$Y0$oO + a ؅s+ L涿}qO.\~3ǂ[!o37}Ɓ8>p@ndWN޽r,o_+oNNw_zcg7vp9^8yrq/r<&&ɶNWk 8]}%* 57<[r;}x?vgOqX[}ő<q~|q6^S."zٞ97(gߜHq?>Fy\W쉀zP1إG ӿu|9tN^|nʞ;͊Q\3m&dpOl[B{$\K4?'(/%|rz6,߲o{0),RV&[7OP,EmP-߂uMt+[eOrP.᣸0gtγ/\wPq䳈~ Ds/x[mU;s"fH~B"9/7/_ 0dib]1QO4S\[XA؎SOR?uua_5$8#G1zqO]3[eW>ynq?ۂ5染Eki.Ϲàuq^Ί_KN]Fj㱏>z}߿ϙ7]brzNS//<_k9Fe^Nk]cgMפ5Y4W\#3kuW=s56$GoE^{ x[`O 0 ؜v/_^#g"}&ׂ%k,q\^"]+˜9o~h~ɸoseL KzAϫl~mveikyY[c {azTȎS}de}I69RR9>v]lײ9}\Q\_ٮ[{ֵo3}ׯ빳^}|~qtMc^[j>/ڭYl+WlcQ378g=څ?GE4"ě\Q"E>vL:O/Sg_@Ȣh5KC 5i| ma^|G|w &ۅ4O/&[^dY :wUjcz~o^}ek[r| գ%ɗ5aյ5v}>1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRql镼?_>_l_$i|<_{B@ۣE~A <;IHES>_$_\D-gykNjL|~+?۫k{?g|}Q!Kέ+nk鳪b~ϴR㺬5Q zqV=*io}\i@9T-]᜼='J[aP'bWS׶^kmNi̿%簭T%Y/c޼|,̱+}}I翶?x,T/_}Qq,^9Ay~j p Y@#F!DDABr6;]_ sɢH G%. A0Fs$ϠvFzX7൦nOM'4يę6Wֱiyߢ 8Μ^K-dַKrkm}&Ο%*_bk]߹y{Zupv3 &ZΫM?G'+VK=7FtX/ovWakظ'Bv϶=k=-"TqmA\Vumc#UzUx5q0i{&_{I/(ѭ |H븟lQ\B#qgx/0ףAmmQ/L_ vdIbw|#-]s:3S0u![^Ο)u\ql>ڒN9Q6=]9,RTl/ioQ?g}0шHZxs bMڷ*(54!- LnFrFl>s=Dv"7KΫw)V͍?!co$h(mN56ouǭocoM(9q>W]1"gι(}eڽ~3gW0G0k9>u[z-X ^ۗfM爅}+0?Ξ^_?+hxq=kum=GUlnG5qz"ambvd-^]KzW٪J{|%{!ۭL^BX\l3^`lG؂O$-5 J8o~[s|XNL|tS{{Yҹ^9Οx`{$bc4}`kLÞfQwim۶^.0qau$HE]$ۦY(OB'(2U&DYC,BaVZZO(vvaq\Po,@ee䱾om.]/mN9>vs/ͷ~0;n-9`<٢Nű՞W؂a\A'nX|c(g}hM1:D~ q; 0ƺ/5{x($0T^~j8Gqy~p{#ֈIseM-F\c09k/酼Vy*zkM@vܸ=xϐ·zq2>y„QM½ gnnOkoztb\}͘M<fmcp//6V8g=<q$l0YDJnjQI/ BkDZRq=HPY[v03+5zy"O _WAEո#`>i6~>z'gs{˾KܳAҫ_Gэb, ̮MgP5e3;Ǹo'sg{뇾c2gq#zoi$wr^z!=?Fs^Sm~?g^Q:9HEb[/P I\^w"_̆7#[1ZuőQ\ٖ2!rVhOKf6Sne>m6׵crkc/I3{3qѿz}l3"gCΩx]|н{MVs$22AgΑG/f6Y#S1 ̽CZ?[9&rs k߽A}zA_s ~{F&zl5?,\8g}r^q,~@m7%$%‚mNSPxYTE2mL/5IXk1Y,'W`}l}6Ǵ}#rhka2=bUYK; ek_l걢΅5guA s*̮==Qb#1Xn>3Dy,khn\{}3oOdbqyOD^#XeK\}M15b笮 )q 1.sܮs39Mp@nqq 7}n/NN㳼?=twz'/8L݇v-z5"A y˛zx8}xʙsC7t/?vqk M/~zγomx~hlf?c~i'ikۡ *:g}oO5rޯ5Qs]쭟K[Pq? 3 f?N_%??|O#b |ZHԈypteׄ#+ MV73U' IiSW`L~&ϊv(gQק_y{g\Pz6 m$G9EoA~!FM9]!%[5 ?V_E d=q;[6OYpp!¥W/>+{s_-{%$p}o }܃v6+[lԐkPO_c[ʭ^75oz|ͣZ{(]&އwoP4~Vdy)OB779|,EQb@_U sx{w*\9ouť_~7ow|{ /9V>>?z*&۞X޻zy=uBt[߮{_[ubӾ8.ՇU]9tmSl[׉k8Chܻްq:ڧx{E϶wʼn D@g]{ٶ (팙8+ٯb@䷉ZkZL`.~р,-EԢUscV_+a\Ă~-ϫuj eƒ*HdţZŬ:gq]g,OkU\숙s`g׊t;r=xݿI6p8gfl]Rrj|aٺV_zVZ5?E۶~ck^5->sk+i]:#{6n^=5ʗKjmy1fO*[}xgu~l!)qD_&"l[X$vZLn<MBtv(1f,o(kk>_x?DnQCgT\[bq>R%~1~yIO/_/ٚ|wD3s O:.0ؼ7!e;WE"qEuIO}E_2{'EXI@ 0j\:Ĺml9׎]S g\"*ΗWPk4}53mri>{8 UasG>i>o6*V=Ϲ~gsxZ$ga6wzq!V1.r}/ОDfG-sBH܏lOq+Eʁ -!SZ{?>&' /w-!YK%BKy!: VX=[6 6<_+h(ȏ]<=Ov7I:F>y/sܽ.|𚲠{X|x= H{wvњMƭZI L2km[{֚}]c]X\FmϫkO^=\!qTG0u4W|= 75vmL\_/NgՌU`71WuovOho;91g}y}))fŰ$Fhj8٪+h]{3O_O?As>vs>Dp뉘kJY`lx=cڏku냵m⼸ִ>{Ÿo+z^`/uhmUzoX2Cbgq?{-QDu#_O؞-Oc}?ʍݻ|?ɏu!Ֆ䗐_Q{^wD |OX>Ӛe( Fbpce[0X$u$~?Ίחz)I'sj`DW8Eqi{J'P##έ{G+b. )<ιx= _s¨I-Eĕ;9YAui~].uO7Ƨ,:^r׳Ճx._ʹl1f_l^G=D8oP9yZ'đ/Hۏ3p{*AW#X9:1 ķo]`;jK~^ JO1x9 uB%/Q^ sO>ފ_e?[c쭟z_"RX/ٛY_"}0s% gӟ_}" H#m_בMg_c6y{NO,]9Oxw围\3Y}i9 66ZJT+5g6=gZWZG}&3Wr7׷fKϯuZs_գVZ;5=Lt/(55nkF%WyGھ(#m?f7-|)8h/W@7I?}?% >@8qq 7}Ɓ8>ŗwoYޟ~;ćݗj|D}9S>ѹy9y^ N޽rëOGD7J=L=z3I{zÎ׉CsF~5s=8϶%w<3΅?g[YZۻ<r.SGŵ0Nqr{YλV߷*Tsy}|>.G_\Nqgs۳_9Vy?ս?}~[6|&u$[??OgAڶsyZ'vʷY[SkMԜ%f}1{RVùEJ禉3??KOoZQ>קTk?}ϑ}1v$/7MD_:.x;l3}9-꽏n}lyYyt,WyaPϛF\4Wm\w׹X>{|?Q/=0ߵ{s"v_=dWroUM6^v,maEs}S.Km]r鵲׼X{zn\رʛ}resʱ?S7!~rkxMuJI9y\o*Xʄ$Js4[U0_?sgEZAUӯJ3.q=ObPd6wwᣜK M#㦜ur׭{]zqX"|+$?ylMC,B?76ᰇ5ۗ?f8ӗ \k>̼<:0zv>'uin|JÏ?O_\|Q{½p{ϋRӋ9|%=8Isq={_vέO|0ֳϽ}V̵\# >KxM9*FMvg\saglVgQh9z5u{s_-{%$p}o }܃v6+[lԐkPO_c[ʭ^75oz|ͣZ{(t?f|S {Җ E..H,R9u,~3g衏zoysoZu|E{'߃,=[:|r62nk.]QɜmѸa[8gz,L4uQ\:&>/-Vcvbz >f[Q=MMzrubӾSLje<53ο>^\Zrc.{^\rZ`歨^нQmƽ moٚr>a>Ft66ksI߽K3[ԚR<UXu~hlهUupT 2o漤9b']'Kԟ[Hz-^cA/^NMt=S9靛y8_cy3σ6}/d|zy;Og|^ιRas|ΝŦ3BbcAzaxޫ^uz[sާu~ޱʚ~z۹99YmY؝A\Ys^]].]ʜD=D5ms|{O8|+1OC#:Fǣu^H=lNj{i2gn\g|1ߺoQ9r?OY:=Br&{^Ϙv^Ksk3~(.{@qJ~Z{y ׶udߍđ\l'\m=|5k &_goӷ#[3V_lzG*9R|$߃spsXq5US>7DÃUjϴm |yٯ=n=G98sw_+yθ$n~|KӟhϋI~jBP~tu {g'k ^4lߞ VL9W9Km8n&93y{]3Q[:&sfrOwAd{f=v1+BաQrkԤɶ9>k\Hykf+Q[Uիj7Esw=br?F ⪤::8%z<>ӶؠΫ7_#kks gK(kg75բnPy$2~aZ";br剮.htkE4cq|D_\ILC %rڴ>Z:zɇ/!˅]L=ٚ|nf~9XZnl|?6|Suzs."~g$o{>?"k=իgͤ߇Cd/D67oJ?asEoVAv.Zyi>g#TW]kg?3I[{1߹fqAZ?q7,d]lμ1/ Y1z _<'*G~Vﳺw+Cڇ~\9 $[Zk2kw#L~hmoׇY|}[͜|_^Wqώ^+rs)mxG}U>{5힓}MN< {7[ȆzD=Lql=QQk&G?vgُ5soq韣c~d]ûe9x}U_ҟgB:&Ytqr5>$?k{5jHep[k{~5󵟳㜜-ڪkf/g 'NԠrAS ۻw1]GVڻv~\ +G ;\rRk_rq_s~pۃzǢW?Kÿ=3ˏs.Y/6VHضqF5hz4֦)z[bi!g:pkeڄ#?i"q? η[ ο}]q[~)w$2;bo}F~d{$\)R}cۿd97KNӟ |R Z>f>t{eb?sS{9̝r#_Y6v^/š|q?_5@C :iX{ pZgd[regݜ}>vmn~}eex8^F%9[09Q^x=.ƶzx&86u^~ {wՖI{/7fi [\_6n;^GNÞ{5N>L=yݭ_ѹ_+q%z_ߏX{&κ =[ۯǓ3c۱7p|\]+l@냅g?8/P.~oogFƮI¸v%) _0 j͎ M~1'&.bI}/Eֻ 1B.4j8^Xۏլ%:۳;cq?~rS<?Ca}&ݰ{L”rS:ss禌k>g#>J 'v:V߽،g8f† >{fآ?7Es\9s*9jOX^[sK~[nΜI}N9>kuMC7~/T後֨>zX,[zAha[~usQks7k{Uߧ^ c>mϾ?Oc&O"_Aϻ&'Ko>`r rj0FƉh8/kк:=i~}Dܲ.||vf s]|ޥL3 ݜׂY\^xwձaj_:纱ke?㼜-kf/g|N2կ俻6[ʉ;u>~d7ߣ=ݛQ\]*-gQ=gczˏ5Y.ql_'gƶsgq&5mJ9"џF;s?{-QDu#_O؞-Oc}?ʍݻ|?ɏu!Ֆ䗐_Q{.l.tpNzaLM{+Ŧ]XuՍm7ƩFD„~z!z}X)}ݰLœM᱌{9LHlAPa$[-ٜJ-/5uoዺ>OuQZm8fԌN 14Aӟ?M>-~_=^\^oq'wiT_wqr|{6W~uYerAtL/^5Kcb#m7n|MϗFWDL.D?5Gԣ1>$l bsP{/ ק~#iϮSO>?rg[cQOɾ77^N~M7F~m$3}rMvkenzy/ z}6879[wvE\ҧv빿W^/_]scKvID Wdg7R4~_]`._ `Ĝ*btO|w$z[۫-z-+=;<僸 {G֖D=`z)=x+~ҽHŘ.B6 u}(%-^Ի7zkw3A^O/s>~mpRֺכаϜG7.7x/J?*66 ݣeOK 9 ]zהQnMlok&osSU{wxm̶zʽ!jAcW9탌LHjjDk{kgzx3 lmy3 ˰lMm΢^0kEM z!W+Y"o-cH)W<Z}&{kwfK'ZxKP:sJnz^+Q>[W^&^v/^z˵GB9GFn:g|I9]16gDrP:zDS*'¶I}%UUrپ׿s=7)U?h=[S֏. !/0ׂl]a q\n%չOȶ883^k%k^MS{}ډe__is]0S{M??4@+A1[=4_L=l!o9wLCO#[>l=njΟ凸$p]^B ੃~@|ދǾ'7ٱD/ozv#xD' }p gq~>8>y~ arz+gc2lbO[lt_qFUWڮ"eoڋ#^/XxE!˻Ӈolw+Eq_SW!7gc5咜\X8@/c%zŞ8?^8]qz 9K)^\G ʗ~>:=wd-/᣸$gy}_p-=aWχ <;K ܷ^q?-[e?>_k/im~>C@~lxߴmq [ݸ)n}f7-z(v]p=p5=A>77I$_>'!Vs&dʹf xL/c2g4.}SX%KzV- Qz lbRܥ~H¾k/HXq0Gb.12gƷ4j|]1:]?+k\-?'/lm\sA=P㼜X΍;c}j?;3odK^_x־s֍l:6ֺzl+yIѽkm$[3׌^\ )⫈E^{ Ex[`u| {}x?#>O/}Xwnqz|wms/sPRI&7M1+|s.Vl~mv^̢u?)eo/-셅KQ!;vLɗ%JKsnc["u]Lq=FqebwomAZbY^ڮΊ{1"U5yun}k2f1+]b=̏qD=Dάߴ m8#"p{.dd_:?΂I^6sK,hQi[>zTjϟ!Ż׈zyW~`B]HoEnuQ=s +jcz~o^}ek[r| գ%ɗ5aյ5v}>1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRqmu5 -]yK"/Kr%toYzIBB(TU,H6Z8YQ9kC '5&>^]9M/X* Xrngv]q_MU+}5eN_/cq\B=5rP ^/0ΪG%_8m_h?[Ŷ"BCxw9 DjVkYݻi9-c>ҟw/cy̛9{ϲ/ ׶g!E˾?*48gؕ}OD "BBFr6>KɢH G%. A0Fs$ ^Zaů\ך>>i6~XX?d+{3hh^YR6~>,+zg?:ssx-m~#mX.#Y\"8J< rF{|2wYjL/+L\k9"6{xy"GDUtMmzQ.B |\5lgg[!hx>x[))~{UE|]؈ՠ^Gnv\:Lڞ^ Jt+:'oׅ:;j hyu{[[e r+L4"ޜ|%GblL,FY/~&/Q2/>"': ^\iϚkFݯkȱsT&NO$5?^,ڎe 3 ^_{I/ <[Q\iOt/duK!{ [)`š&A\ gokמrvщy}x u?wo/K:7k`?g3lS~lc{c>s6Iyz,VQ|N'Hm-^d EQe7M0BKX.0bms?bQxYsA=soY[jK=4owq >vs/ͷ~0;n-9`<٢Nű՞W؂a\A'nX|c(gs6Ql}8 @Hc /&#b8>y"D#ýX/_͑Γ.Q)k& L%AaV H.t/ɮ6jq&{ x=af W$kEVP%-֋ZqIGc}lF}ElYɥ>x#w_]z^8JnMbQgvm?[)19}K=g;ۣD^?9~{K[&'i#5 n1n9!1}؟c[] e9 饹½#b6%Au~ˑYWŕm9Y /9"g;of3V#fs]l}>&׊6<7c; |'M=#r{qN=뜊Ή+׵>݋+z_ׄm5Gҋ+#Ժ_~Fqzbfcn8"?US;c"g\E_*cƤ_Qa[Al~v#r{'ͦܯg#xL7"n/.(vk}^cqr\F+\Xs\2Z(6Fo3MԞɲбWG9D-G1nldN5BUE~*z~>s^-~ꚘU3~S&vAWc }r<Ce'4cZ-}q/~Ӌ&"}EBЌsmbIAL C7Y:]>71z m|m>{{!muE9bF`Co^ݸe_Ǖj乕5{ٜu3ŜMq2hmeRmLLe7ucI5uT֗-:>-c[_~JZS8][+27&bQF7kn )0=%FB. z#b *}HIV9@ +Xq9nW9@Atm&t\f.g5.Cgnqq 7}Ɓ8ɈNw?=>{匁Y޼Vߜ>᧻+o tsxr׏%7Μes3:]ˍzS.~x\Ӌ>=ۛNϿ olN/ !9N|oĵtͷjb=/|"p8Gׇ_N/_yLMiKm#pJ@UDOM~˕$oQ۝=cJ@lz\GUد{M+g9/~s@{G1DPtbO o/.8@d=R/#I?]znʞ;͊Q앳l 7! { VJ/~<}jt=gGq~) eq}g2+_SvYK3j$scwEmP-߂uMt+[eO7۾o_GO}G8T,"p'b2j[Q=A i/7R_ 0dib]K0P\[kz lǿ`'wҺ VL룘 h槿PՅj|]1:]?+k\-?'/lm\sA=P㼜X΍;c}j?;3odK^_x־s֍l:6ֺzl+yIѽkokm9ȹ/䋐H c/t``,>/FoO/}X]ǥ%ݵ=ϹCK'ᛌ+6a;_gZ]ϹZJ9 힟Zb^-^XcT|Y_ }ήԼ?y:O+r]>۵lycW/vֶZշ>WYQ{/>FDWY&1}ۭu^O~Z,|E+^l1șsAA R1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRqi9ρk?~)?%}"<_{B@ۣE~A<;IHES>O$_\D-gykNjL|~+1ֽsV!N͗_'URJ;Ͼ=aYYƛ>ڟ -WLkʜU`jcq\B=5rP ^/0ΪG%_8m_h?[Ŷ"BCxw9 DjVkYݻi9-c>ҟw/cy̛9{ϲ/ ׶g!E˾?*48g#z/8=-B$у-mAa|E Yq[H%!8Yh"AH ;#x=՛KZZYCG7V'ͦݏ[lEq2bfT +X4߼ZJoчer~[BXguϖzd %9cz6K>AǒAho/1޵].ܼ۽[:q;w]k-Uߦyr6@1#{n2 Ey!Ef2|\5lbC:^d~ |}ʯEkm_v;6YeP #F};Q&mk/.‡ua/Na>w{c{=~^֖_„ 8A kW@T(vǷa="5>=m^SN>aѿe/_rjkh]5^/p飮-鋱Q8þ 9Y8z $^^D9y_H$|Wh-9J8TH$֤}̱p4!- LnFVFl>s=Dv"7KΫw)V͍?!co$h(mN56ouǭocoM( 6ӯeWd șsn$xv/kv-q;̑=<bkO?g=t<|lL,FY/͝zM_7x >b#b8{zycN~|ŕf_=Ck/mljM2?||(Wbݺ%=z-xyaM^b{X O3淵k@~;LJǼ>Kx<ٻ%53~^qy G)V?FӇ ı=ykvN90qau$HaB]$ۦY(OEᾈI1pF fVhVE<;VmG#ov~>bųnj>3ƛſ&˾_=~yosy[KFy i11|lbQƓ-Z[yu-8tb߉U7vyr֡PŶ58")X $D PH`~1g-\q+ 9z5ď$p߸/n|snj6)_{I/r?u̳U[Klƅ{lG4c֖!&ljM8WvvZc|֣Ө6粷 kƌmZm5{mm|8{yJ_N9KЄ5Hq_ 7I`/ "EbqCb_T]#*m$L6W*ǡ Aen*6zHQ<97~]J[V㒎Hdٲ6K}V?Gq1GO=/t-9q".Ģ`m4 US9c~s{kE\{Mzٛ1 ÞgDt/0ΩS9q6G{qEHzqed^Za(.#]^lG䧪cA{~·rL,^&l%{烲+ jMj$X^/qζ`sˡ2pW)sIė<$~9GT:_sGU|Qq'~iDMmoexVˉ+X4rn1-k߈ZLoحzUҎ9qzsa͙s]Ȝ>keOOԣ؈b?/g/}7Q{'C^}[.X5{޻99V?D_SyM>9kf9CՅ_6_֧oⅿ[$qx~QoKv6抠@fmK_x܋$$h MEp(U4ZZ,*/n,t|}nbZڶ"Vl}Bzr6 NB|¥E(:T6&ϭYL-lGk`,jeb*sĽAE=O֯ #z{lAԙPnqkzdWBԚ™=Zq?6{rXvm!Hhs+>z#b *}H`I+>s`nWϱ>s}ԟI!oIυ+>oBX^s9Mz恸8>p@nqqtûWkKo ޟ~;PNo1'W|xXbLy\;7sӵȫw=a{9==Ͻ_N/>Oϼ;>?68O\+O[<|K.o!sz!' st=:O..~}WΜdpĞ48j=|G]E 2K$oQ۝=cJ@lz\GUد{M(g}ٽq+!7缼?R϶x${"^8TLx{0v"둂dyR/ɋϭC3G"۾Yq> sF,߲o{0){,_}B[; ~"}(PoAw㺦X[G{P}4Q왳?C}O"-D.RH|NBHokjK=}״f>Ћh>6oiӗ^߻׻řcZ?Wtl VָZNN^<:zסy9+~+~/V;ww>~>~v>gvv9Nt|}ٖulx9uZWT6]{(gHf :+|ȹRW/䋐Է޼z@0,>/Fo yb~/[樸zg<|5V8.]/ye7?Xj? d\Ʒ ۹2?oz% A֑-/53rUvO[yM1/k~yla/,\Z ٱcL/Ɇ>gWj^Jܟsr u̫svW߼_117_ma~ !rf\}lq)р os&%x>vL:O\cYtѢf}(۟?C$e'wً(aXNTc%dՋ7{>zxAWFq^ +$ֶ 59(˫G!K)]Ǔ/kN>kk|c<|H5U{A/?ϛ s?g[Ƒb`1ٽUgڱp\:ר889D|qos}_~ID_s E{eG,= xv>էy}H6Z8YQ9k&*ԘW/b{umB6/{`O60bɹv}#{v~7}V?\[̯֔9~eq] (Aك|z8>sR|]}a|_W[l}ۮϊpNߞ y߭K0k[e5gu6崌uLKvssAΒLm1o^>X>˾$_ΟERl<`N/>\8\ m|qhBt AD$4h!nH? KȢH G%. A0Fs$ BZaů\ך>>i6~XX?d+{3hh^YR6~>,+zg?:ssx-m~#mX.#Y\"8J< rF{|2wYjL/+L\k9"6{x RMު~ {&#:,Ey!ыtb2|4z{"T,b=,A݂O/BEkm_v;6YeP #F};Q&mk/%i7^Hs|d5 z4:-ٿ2 A8q+Z֮l9PozEkNG}F{*ۼ9}˜93_!Yۻk^6aG][c8gce-:}_"EaN &h'H$|WhE$-9J8TH$֤}̱p4!- LnFrFl>s=Dv"7KΫw)V͍?!co$h(mN56ouǭocoM(9q>W]1"gι(}eڽ~3gW0G0k9>mdͧ3f`b 0Ȃxin_5i~y)~82VD0?Ξ^w?~VJ{\3~]DĮ~MHXk~X'6Yˠfl1^>x*Ҟ2_^v:CL;S3>5 zaaM~?%8֮=z֣,~f^tn~yũgl5^/I{XM.1:}l-8/ ZH^Hmm↥"R'(2U&DYC,ZBaVZZO(vvaˌP{ 7,-5Y{˞c}\7ʻ8^~]VLv[?f{ Nzp0lx'jϫwl0`[7Nb{xH3D~ q; 0ƺ/5%_LGp|D*/w 0 uőA\2GsWfoFN+snj6)_{I/r?u̳U[Klƅ{lG4c֖!&ljM8WvvZc|֣Ө6粷 kƌmZm5{mm|8{yJ_N9(F`Wp &J 0+H[$VL x%BkDZ*@8$-;^FOx#w_]z^8JnMbQgvm?[)19}K=g;ۣD^?9~{K[&'i#5 n1n9ۆ5I\sI(^k!@Cs,D,0 o9"FI/8F-G*g]`Wda'QY9+}'I~3)2N^6`1VĵФ白øANM >yng3 s!_TtNt\yQ>^\&l9^\WƠz}3KHף3sr!9KC 5Glc9=GZzzgWzbmQ_s,~@m7%IK&9GT:_sGU|QqS/~iDMmoexVˉ+X4rn1-k߈ZLoحzUҎ9qzsa͙s]Ȝ>keOOԣ؈b?/g/}7Q{'C^}[.X5{޻99V?D_SyM>9kD3 -L_֯UR[-Ez/=cb`W-}q/~7$QQQ\uNM51ș5rb~"k@W&fYm/bgz/~{/dW1(g^lszZbCo^ݸe_Ǖj乕5{ٜu3ŜMq2hmeRmLLe7ucI5uT֗-:>-c[_~JZS8][+27&bQ|-fWqR`zJ6p]P0oG@0/' zTdp8Nj <-+s`nWϱ>sܮs39Mp@nqq 7Γ_~z8}xy9?|8}鍝OwW7ꑯKo9~fuyᵧ\s'G'k8>{sox-zN/5^\DzpSů^ʙ /Ӗ&>6;G\3(v/^@ 7<[r;}x?vgOqX[}ő<q~|q6^Sșl8s^)geYlfz(Ya `&dpO[#^7@&Cxvn\}lzܲ˾U dW\~_Oȥ{{,[#7-r<=n[G/}nAw㺦X[GƻP}Q\>Pq䳈~D3b[mU;˂$lӚ_(ߺ䢷y"M,Kߔ0| U{K-@0.CZW_AŠ)?b}sWʜߢPϹky[9sOǶge䅭m~Zs0{j׻k|׹~gx죏^_osflyK wκmYdžZWuE?o5)wr6_Dй^ᾋ![|q-BB_ 1^ou``,>/oy&:_ڷQqx;%k,q\^"]+˜9o~h~ɸo72M KzA #[_jlgl[/Ak]-u5<.G1G&_֗dC+5/%Ϲ}mӊo׵v-3qE(:ݽi]6gyj;+jLj7W9Kׄ:չoտΫoޯZ˘Řȶwŋ0?}9~Z~.>s6\s1ph7`Œu |U\Hl; &y'DΌ~y@dEJߚ%УrPW[1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRq=U)3't9xK%/(~E>էy M/}."G8*gMxȟWV¸|cݫk{?g|}Q!Kέ+nk鳪b~ϴI+e,^׸GFP䃰Yq bS]v}VsX(un=t\:Au_M_*{9{9-eg2^?öSre`zxl;y0zY%v,rb8sR~WGő\x=OIg*@B$у-m{IQ^CxErV8(q)7UH8Yh"K ;#x=՛KZZYCG7V'ͦݏ[lEq2bfT +X4߼ZJoчer~[BXguϖzd %9cz6K>AǒAho/1޵].ܼ۽[:q;w]k-Uߦyr'ůɯ~_Ÿ0Âx1O/ʥ0`^o|n{u [KzK,b=, /w 2?yni{⢵ʶk2W[l ‡ua/Na>w{c{=~^֖_„ 8A kW@T(vǷa="5>=m^SN>aѿeUl5}kc{Qזt(YVi[8/dĊƜ@5L{A#+L4"ޜ|%q*$kҾ\X8m~ &k7#+k#i6^9hD];WT%dջ+Ɵ[h}74zb6eַKr&}j9_G⊲+2B97eo5_wޏbuܸ HwM1ܧ>_[z-X ^ۗfMO/ I"! #E) 6^w?~VJ{\3~]D2կ k{͏&kŒ|}>^ VeWS+1 n]gb{Ȟc{=z|71kDplԃd;V{^c N~qsغwb]ã1n악hci_3x /&#b8>y"+uőA\2GƑxŭʜZFl&ar^ yOlUFqz:qa9a{!o/a>☵e|: ۣ{~=j?r߆4͹e_9s1c֧y&Fg^[{"^^lqXl}8RM6Kì@"mX1%@TŃ\HvW3\řރUeǫظE"YG-ܰu*)n^ZK:"f7.gZ-5r}V?Gq1GO=/t-9q".Ģ`m4 US9c~s{Lb{9ӽ8buNEDǕ\k¶#ŕye jݯw|k?tt=z11J)a 1t{=T]sV; Skl/s45ѫgaqz5_'9s 8C%ddS?l/&$Н_D/O Q(RkȢ/J1ncJ|/-M V?^ϊ@`9Iv:>i6~>cZ־90v{q@[s㲵/6zXQš3 9}fʞG~^Θ_,7|KoO547>ʙV']l8jw[g's"r%.BSC|hsVt9T]xnaڧD~YW 5 ^܋1ߓz1E-Y7 Z, Ep(U4ZZ,*/n,t|}nbZڶ"Vl}Bzr6 N<zpi=qʾ+ժs+#kV9fl9"dln=X=|)zqoPczQ;⓾kHǩ^*o/G9[u&T}[5+:&ٿzapfVd\oM~^9cgWqR`zJ6p]P0oG@0/' zTdp8Nj <-+s`nWϱ>sܮs39Mp@nqq 7Γ_~z8}xy9?|8}鍝OwW7ꑯKo9~fuyᵧ\s'G'k8>{sox-zN/5^\DzpSů^ʙ /Ӗ&>6;G\3(vv/_%v|q- 9_ޝ><ܟxcX -_v qq?u),_ܤ{ʁs^$be sVEeM0ɢĞeoMBh~>$0]OQܧ_JqaAq-V_rcI3xߴmq [ݸ)n}^(TE=|쪾C}OWB'|2j[ά1RDB}Z3 <&T1D7%%_*`_ ca;E#L>K=}_`ʏX\Ec4?2gƷ.T{s_G^gi\ӱ-zYYj9m9yak_< ^Z4unٯ6W}#%&D^:nd[ֱ;fk]Q}tM]}u.й^ᾋ!t_eX/H c[{@0bs~x^ߦϡ_ڷQqx;%k,q\^"]+˜9o~h~ɸoseL KzA #[_jlgl[/dA힟Zb^-^XcT|Y_ }ήԼ?y:O+r]>۵lycW/vu-Lz|#"߫|\,]WV:ڧ˿yvk-ccn"/6ACMk WGE4ƛ\b:^*.u~mbʋwEB[>zTjϟATxQ2!LH5 i:^MZ~.t?ze)[N_}6Aǵ|Y#w qos}_d}_{/B>w{B@ۣE~A^{v>էy M/}./prքykNjL|~+1ֽsV!N͗_'U0QJ;Ͼ=aYYƛ>ڟ -WLkʜT2^.~{kA>{^`UJ[9)qھ0>~->emgE8'oωRCǥsԉXTךwrZ:}&?s9l9 g_ Ƕ7/ sJe_m"g)6C0'Ջ}e.{qTi.˅q9\}OD "B A 9wN}P/,.#"9%!8Yh"K ;#x=՛KZZYCG7V'ͦݏ[lEq2bfT +X4߼ZJoчer~[BXguϖzd %9cz6K>AǒAho/1޵].ܼ۽[:q;w]k-Uߦyr6"&{n2 ^=Wg / vh^5lbCf@^d~ |}ʯ"vlʠ^Gnv\:Lڞ^ Jt+:'oׅ:;j hyu{[[e s=Dv"7KΫw)V͍?!co$h(mN56ouǭocoM(9q>W]1"gι(}eڽ~3gW0G0k9>m$f`b 0Ȃxin_5"JGp?l~|ŕ"D׵MZO鉄NjEqb zav>k/_g2+)l3y aq=dtzc ^>E:CX$+Sm_>a=:111OgeIFx 猟W:v^푴wՏႭ1ql{~plޚGqζa*I‹IMsP^] E\Fjy؄3+tEX"lJ C+6#Ύ7;lQuj>3ƛſ&˾_=~yosy[KFy i11|lbQƓ-Z[yu-8tb߉U7vyri{(կ1!Ny!!&.b">"'Ry[HXW /sj=w%a}qk줹2禖Ql#lIB^+S<[Q\Ħ^ Nn\XNg_[zDsq8fm¼za&ބe`_k{ڏ?Ʒa=:1ls.{{Y~ܾfئiYֶޱȇc{pbvHq_ 7I` "EbŔ`^X]#*m$L6W*ǡ Aen*6zHQ<97~] P^ZK:"f7.gPL.Y|w;+?+pеQzth>kz$TM1[wxI?%oYox\[29I׭^H5uîtϏ5Ŝw6t kxW#Ie w-y~(xbBW{/;B/fÛkҋ~Y#o9R9 -?' ;~*6wY5?a,NLqF\[ɵ"`&=ErBl_sS?=xψ^`S:sumAW5a[͑ȼ2;>Q\:G٘g%OUǔ0ioY:Ͻ*Lخ9J~e+})5V9՚ӳH8s/_maW*/(!# A<'dM|ɟE"I?|g"AΣQ>\E_*cƔ_Qa[Al~v#r{'ͦܯg#xL7"n/.(vk}^cqr\F+\Xs\2Z(6Fo3MԞɲбWG9D-G1nldN5BUE~*z~>s^-~}=1Πm”O+e_Uc Ž'F~Qo~Ohpǘ·56k[^&!Ah(KW(D (:&֚tmԚ`9T1x|qc5esжfM_V׫X3/^pj=-fizpi=qʾ+ժs+#kV9fl9"dln=X=|)zqoPczQ;⓾kHǩ^*o/G9[u&T}[5+:&ٿzapfVd\oM~^>- mຠ`+ Ꮘ`^O#ɂ&qxxZXV@q.ܮ\c}]qqng5sxҵsO?aܺpqy 7}Ɓ8>p@nۯO ߾+cחcsԮgA\vD~ᱞ͝Q|sn^3㡜3y9?;sͧ/ >?cJ~g.s{G$Y3myqk.`In-=̼= ӿw!۲^o~_85Tk. Aכ܍%?a^fw}q^v}1wl@vd{W^Ӌo-CX&h֧~< ؋櫴=șF,~%W78/+>tnl}y$ I`jtWA(KWalH{5_o|ק_y{g\7$~`mܑIK/:_~{_+o̷9^~85_G|OE]w#k~7s{n=s̆3PjyΙKA<_ix_C{ypF?zq؎}AfM8yG>[˱9{j׳kz&xg!ysg'3קx( ~9>A g՟ޱ[%|y_>8N̻B?=(ǥȿ?uVqWJٓپ*妳|_4~bӋ"$ח\(][V oʿXѿ(I6bz1ۗ 6!AI 8;shmbG?/a~ ^1tahM7WN~yŷk/l<ƃO#gςj!+'V<ioQ)NBs=#oQI9sO~d[ϩyʿL˷+1C{N9sUvk[:\j62]gOwM99* FߤV}u=׍3ݽ^_/W׳@I{1nO?{~XY%XYcٳKol$_3z!$񎕜:M{L-)fMU>wx~оݝ{J?3PYu֍:{_s;b_<oze+A_4'ӯi]a^R8<ЏG9s|{=Xl{;c3,7c F0ꙏ3Q]wZ~9'_.U$q+}&rB9[scbo,zQ}"bZjk޿i6;Y:"s5'&]rp ~>pYԽ3=|R~X`C_)y"XY!<@v{~Vgz>W3ŲO}F{_7_em:f ҷb1>y{g~-3#s5EխA_BjozlyuyM'gU-u\f}Уgo" KY>/GfR+WWcc{8Iù;?ةR_xyجy=gt[Ww3AB.ŧf'5 5{= 'fB>" HL\NKd:@_&lXO{ubO87~N%ͷ21ي"p=VDs&S/|X/v.פ k=և:G-&@ z4M8r*@V!ʏb{盨?,Rm~=n{}P3L}&`_l1y<+=dVcOWd3)uc)m_|}RlJ6.>S!?ΙQjmWã7]sc=*A]FXu@paޙG\kۇ.n΂m*GOY#rץ8=_iwAδ˸>Cj.3uq<W|y9PƱ!GuBX&_So*yqyF页Ρ<`~P釄z0\i^^~N쓟SL9f9k)Υf4gwx~YHPoQK7ןŶ=_ ?/h7{]_ǂ^6V:ـv99.¯]-ָ_zMÿ6I?Gb蝄s<%wp{ֶx/aq|y$.əxg⌧lDZn\Jc^=x.Ϟ@{MU>r0g~|hʟgm]=gחGswP96c3I[+c=Yh;ߓӿ;H&,֟uf~1yA--gP=_g kv6͉&esdIW7Y%Gs~OcjL r6kbsB~Wy!~ 15ȡ/< ~j#lcF9z{e u[7g9&i}LNgcύO[Őz{-xy(t{a Zi8qyLrT<o_Cjgmoоbӿk1;MPH'@2K҅ey_s/ogy/j=q_q/d g&^|AmnVq n[0 _r{)TV9ἃiFqe1r'w'=Ü/VVŊ%v^Y[֦Gþ69H3{Z;{z{L ,=e[q/ '?eyF]_̦g}!7<3-K:!6aCaD}M:n{X9ȗgHvX.*unژڮU\"Q,Xh'>1a:Ǧ?ǜC95YOk;ǽ@͙?ߟ:nuB3ov ck9aވ䏟w7_*_ ;^`וNv;(6??[y0C~hW?z?6=`W/gZD~x'7Pc$TW%|QZ^Ir7//8~#|~1C|۾v:֨(g^X %zه03.7Utp+D'۝ sox7~ 24?~,;"#B /rN zXNkbmI`~1%s>81:pt콄_iV~D>C^sCGNhNǾdWŸoh՚g.ױcN>ɽ Xُ>;71~ c<5?'٘'_jm]N<>[:Dֵ4h;/F{|lM:vAFkyұq~M'2n9GWًZ%"1L] ?LJ;ֶ>#_e9K1cr6ӹ'䮌|̞k,z=t`vjʩ{VL`o3{wܷ_&ml-9% cg9 x2K-}599W{~O3,mi8eY~rּQ=6?[1z0g=X>\~,͎9qu.͟ '.k?Ⓞvjzl͙޿]<,{߿lǽ [W1o|Xx^e/!3I~偕?G *m(^A,3B}:'u_Ǹ?E7'Jk腯Ǜ7ל`Y&ɯ{oI {8]Hٚt!.E`j"Ρ4?wm9Z☿jےs/?; -M`}6ϰ{yN8ty:ҧlt}?=%{׿`}yv?_3_ Ѱ.eعރsKov>-|-aM}\sZ]{fk}_rd)?=LC<^)ש{ns<G+~1_HI}7wML&iݻWt9ci~g֫]Oڸ1zMơj[kԮPm~̳$bRB N!+m%`%񪭭_pbջ>l碜RצAFw0O77Ad:9{9Vqɽedg!;3òփzY6}ћÞ1q4YI糼W\"a~%MsLy<0j-?ٯ ~|/ևm[L>ˍrvpk.F@>sf~96Ψt}4g) K_H;zԚoN}zثkTiD\`d94ǻNr&smG AZ]~G5Gyz\zl͙wDWweAWOcLj!>l\q_+^.u?k;P {ڏ͍~`!:9ʋh]K$pJ\Af{K!|^j!޼npmI1kkΔ o4>&ٸ3B^p].\w|MDٛ`}8ߢ 4Mj|o:i97:"kp*p8b[YaR~f"{u$z+ܶi!}[僭~3{⾱sKy{\_5K_ !abmH}WkN~N^sؚ6.ݿ4lm!|X-khޫO$n^ Ja[s>13)aZt 5Self93:l=;YPt6fw=I؍uLzUtybqzwS/ʙ&ZYm|"*=I/~ς[+66+ WTa9yWqs|9 V%& =1c\z9}!quoI>}6^yg>?#}z֫Ǡ֦ϤQκ}Ʈm1#+wA\sgtm{~ډQ/5ۼ <&RZQ}F(ߋֆxjݜvk}/6guiGƷ= sXq?2!5h ]ϹЀyS% LBņ'/ؗB|a?qsS粏SOVo7үN'7Tkqqwn>h!6;>z%< ?]? գŌ|4'ACoJOn<J7[x?ԭ/ ?9?1y͎c #%~ I c^t {8>ft9@@nqq 7}9@u|lׯ~19?|8}pۼzpr|yw7xD^~zH׫7Μt^]z~5e}/Gݣ޼8}x3|=p ?'(ע??$t!߹/x3Zsl_:ĞNߩ n_z&_-9鏿bv?/gG$'wڸroZ5k'Ϩ`ᬟ_vO1߃cv?[z-稾W9;I׫c=:<;ۯoVp^De}A=ri,z9$jMJZj!KjQm^$xH5*E/v%JD(؂yoO/~8}> /Ͼic_~r?_N/_%Ɖ]R羉}|ܞfmi'gbsޥV~<=[tg]iq x΃П>?}oRyv-o>O__o|?c~fם3A]o.ڷاZ[Ưcd/ W߱ks^gG] \N/zu eĭ@}pFφ3j%^o3wV\wv?[sα?ӹs#=W&Q_e?g,y]E V'w&?ĥ&(-WnY󷎈 އӇb\0za(F.%ꏧ$dY]_E+]_1{QB`&m޻Zd^|l<c9;Xc/S g4Yjח\_{<2IB~x3PF-Rʅ,讂~YWQxC_jۯO*θ"oH&$`#*_t>v&qMMW@ߘo;+L\ږe\ ͍$_5 %AȚ,~}y=;y-'?>se?DcC\֫ͭsQ|zcvm򅭕q>۸0}gAfV`´vVܷ([l'؞7֨_RФ9'Fy<_^֕ڍp=̜Mѻ.{#Lי]9?{Svk!'[e>H_7kvzUC|޿z]yGTFfe'YOS4$ J@@,LK#O6peBJK9P\Ӥ ю,5("dhVA49YAYk}z=]k]3Y !d:"DIaTv/)rۧ9^"m_=`qv˅d!S3|ೡ:E{sN7[ܴ\uhdPs7e8Tc>(yQߣs(gٿ_Vf+}7G}z8f\3fSlUg.n<p=P#h]xzk[^2Y=vp({aYM<=C8dCݼw6x9{eiknz^Jes2/= E ob"אAqxOj7wﲸ/?p3eL`9|,VBڋz.*J&U#* r#)AMxw156c755N$, ]t} 72}"m"qߎ5w&hǿp+_8eWrG0˵&m>>uD9pM^UY`fmر=ykx,u0j%mطt+絼%Ea-j0*:v(%CqPWD"w}㘈nc1,؝ہ>aO{;3ū.r_x]~~7~t۷Ke<|Loclg| s[}e\ij1lm| ݘ炵>\WY\g|g9f9O-d<6-^1;=}[ fVusMcP_\i>x MO+xK]V.}݈8*Xي1K._3uh-g Kj,:ܾϺ\D;=^ub[<:Wϻ_|=\nΊ?TgXy]/3s1Xqx%wlru6W/&F%yR&W[;,̂YHKe6X[=7ʹME%EmfހC6v H5oo}~-0o,"`К1D7vOx'յ4?3; -%bV6Hq{LJT!BϢR65FgHC[%<H}} \]"NUMBt[29U oRFjopEG ksuܽCe2&F_+e[1b9jCשl/}@}L̿{mPevކ8E"nSʥԏ" <ɘ>iL3uNZ3I7>ƴ}ж`#ȳbkgAusTc.nm$~́g}{̅꼐x1 0c8v φywiN:Zw^#smjBgTf1ڿ^rrG&ɍu@N<>!D66%M>"t";.cOkcά}ysr}Luuߺ?S]}8D{hx}Nuk+~WhݗŞr#'ɶhk9oQ:L[>tr_'ımEjxAl(r!}vsHm^6)c0=vn\4f^GkEi$mkr5ǣ1>?<3B#Ώ%ponޗF>xrablM\v}6/7~&"UsM8&^ay%ö/-څlO/|0rQW" P\4'Zk^CS}wb+[9ʃq}{e7!p'6d ۛ>GYƣqh7\`},٣wVi|FpܙOw\9^4(^|V/_سLN2^?)~߂ZߪN/%wgµ7-~AM,ncP7q/ƛrs=dJՁ@ނ@O}(ۣ"%eZlWƔ|VcO!qgfsf "s8ao4wo݃gז_{{ozTb7Ҹq= "\W-\t#]q~^cSs% PA&c 37߻Z{Ľb&&`E{E|{l|1{ڐm4FfT W"ǥkq]̭O{n'ƺŌ\JH{lA}S1gv0X?p xF0zu{bi9gS9]s9w(>yy 1hckgTf\۾O8V~u= >2c>=7OQk_7Oh0\ܚ!]^Jw^;+ΝNMsfC慾ͣbVuȿx,b?]ϗ//nCD%~7A}6YX:[T&ٹ%V*W { <܈'6e{fZе%hZbtV&¸#QB|DF[>YBc 춼.<m,.M{^n S}tmlr]₷9|=wQEߗbrt@V؆_벭C.o^"|_KBg/=h[پIv> r}170GF۔Tkow ٗ]'9a-dYw)rŸ>Z=zq=c3ţ1>slǃu{.91׏}U-RgeOiިv}6=ص?kjorym;L6؛ElunXS_[ۑϸ|.N44vޑr؝U9=ʁ ꯽~f ۾:> V^~;g&m\gNW;tz9<[咮ϒkN~1 |ydl5aO޵o3տyzM}/9I~sm>эX?sep^bϚ@qnsdt3 6i-uGT}m So6^'GCjw&Kc斓 ] 3˅%ꮟe<7']X}#o8y_7J4[i~I"d1j> ] ˡ d~{mcˇzk!\[؞^AޡldEg藕tѹ6|)8e>rljBmn88(ZatnưE ӋgBVB>ƼEoӯ[i!b^q;_$>+xw:}K q Wk*S=?s,}(VW{Llwpqz_%ޭۙRIkd׶wvXfgeݗ>Rc]gXLC\;awg69LƗe=k0m~ڌ6c{+tL+!d_q~>u˜(n?#fOSʟxx u`T8593 P }'qqk> w~n˓:ϼXrV>}9r$.?'Wϣ|\y7ZnnkMp\EeM}~{ĩ>X}̇߶u^>>~K}ك+o+ڿRgo_gW>~iNwo>ֶM}^{v|n6cG99DH/o,O~˯opI-O'/_园U($ʇc}_<}p6q a| \T'X,JK8ɛMXg+ OB;^m+ʼ CxW2D;EF럽|qζ~`yo¦^ $QXq"I/~/~L\-?m*,y`^EK|Xq?>5 Ɯ #"jO-pn{^ TUdggaOߎK Nf]-?opV.sJ/0Ds8IDATcvWBl,:>w}HY/_3, ~Xe-[UIN~$T_֏&Bt'1C! yi@p)ewr.i^$H\ 7&]VHk|5n'21WfV mv5v,#q_D?rc t"aXz֞Fo_ښ2|F׬COgl2'd9Hy$pnDt/]جHXS_ C ;@o= #}}i Ks*ɖkZy5UE_%,J/3˜BA֍yQg''G}DYUX)hрK1nSk2-Ow_S?ZC_/n /1+LWΎ|]5>Ϣkf.$[ʸ{o:xt7weʧS ~V;pn۶s,VO M1}s)l8 =Bܼo ِ/8ERx׸&ۤkR<;okQ_p.\][bXɢ_3iqا(چK:7̴ߙm"5+kRg{Ps ]{G{,a29$V׊NA6R_wm:ya?߉~Ⱦq?;ʧG_!vx]X*bX`E}4M)%n¹HR_n>be$N q<[󿯯1΁_> MjţX>׼9eFH̊?Ʀv-QݞP݂ί#-־>Δ8Dy 'lQ#bh^AVDYTj}~A_cbwKܷ i~#v]s }JH_S'N XN`mK)+LD"fq 79m -f7vyIpWfj-$~П#GKɞXsʼnu̚:*G'o1V}}\MvQ>91[~]N9۾<0I~e+۞c1e־ '>'~U }p_Cqp }\8 >pA񧏗y,yӏ>9_-7/_,׏2.O_,/_\n_^/2p%WNW]ߴGYK>%\/Oea{gęD{r;|}Lf{ sw.?6_^[;)8gpxk?_ ˆLҳlL1>x~~vT')K[Q wqӹ'YUcʘ&i=rve29_-q na?rSY>|m[CQT;k'=X"x!jWK[{~:+Nwo.ֶM%.>#rT7)7sZ)U ?oűN$XwD?}8bmsKr@蚹٪r37߄9Ccụ/1`0K$Awy;ώ$Yak|>< "Á|wM݄n,GbxpT̓F?/X'9 3DqiGK}_Q_xdw~H-nj4Gٖ܏Ǩ_-(ag7~yH}9`IbQYj7A? J7a< M5Ǫ$a[w gHTtokݑ,j}в]BxH?HXH9cϋK<u{`XO'}Ʌ|ỗzhl>s@_`\`6l-||:pځ}Ag{)}5{2*?wyc~1=kR<14s}  wl28JLZbܷX Ag: EQwsp_;u|ڥ|su8P}N.c}ݮ:s>I.$q[gA9EE˃eݳuRg0…]|Ȟ|K{ 5~y"$샽,o0=m5յ4WVR8 誇?DJy}Q}#?؃l{}'{cƣ^dm~uGe}3,yLOdq@nnہN7'm]:m|[E8Z8x>جHV{bt[Df78ͧoyI> g:/0ܷS_qkV^a]/̋.s ἷ=4fLg6hݗiY*F՞D\Ǯ}}Wef?k8<,)[ڧ}/+H{I>X,i[b/k^.$ZkyJ:.7-6WehCxzd{9uDƕ|݈Lyn):Ν}ڍ~Ӽ5"׎جUE޴8"^ vcDZ:1㘖6-95 AGgsĞz>[?W(m3}{/E=g'♮?9Dύ_$۾Mݵ޼>AoO6sL碉kzy73k7օsaKYYg rxO516D.qsrtثO]G|>Qf9;־+?3+; |J@()"'lO4cɽ^oӾk;9c ?ׇG4m]͇ M?~-}Y='(ؾ ? .Jf$öZvH)uتn,o47gSƎuE[.Xf㏉H6Z[pn+L4v(",t#f^^cCw+=~Nqr/wi~SJu_N kR:Zsmsslf/|:>n1}o2qĹo"1mE*qvڪ}fư)tcޢ ng p9\9dm3U˷>W'Zyl^[K\=k,*rW+=껗_4_d稞?vZ˴֯^~+W־n62/"'뺶3uAm~&~UUlbq7B>#8gIīHvg#VqkU9l|t >HOR;1V?o3z^~CJgfR~y+|˼ b95 kdo{OyyN.A!oeZ0C:D!2^s7arnSmQl;:7;[)a,7PASћW;4U !m6~]_iI[Bt,pӸvl9K ٯLO LXJd"ޡ? q{4j7Ao[}yM V0d.3cKxSX&/sYK^x^ z" IN~$fT_s{{)_I4_h޼4bG=~Q~l'5^_ƴ֤P~߷(ۺ~x{Hw}WNLJL^6{D١ Gqj+EܦK=o+3EL/|y1f83PUZ3I7>lq_3k%e%ܷ_/#ǖT'8ڗF.z-vNncԷIk٭y!r@U(JжҜt߱=܎p <#4VDQ̯ ~|G^R_-?e 5: T58˶S?zhz~]utA 6 f' 2pA( }~k"z3[T=#XCGCoX4jkA<}s3ڰ \7h6Lޤujxe4;3_|lGDx-o¼baB]Gb U´yC]"]ְ[}\Dr mcmӋ36g>oA>WѶǒ}Y)$ۺ~G`|?3hODqc%ڊ8ԺX9Q B\m!<ؿ6|f*g,o^6)c0^Zƅ5%I|h7oKr}vojv\aj9"^lOmYïyqL}3WLeW_J݆ݻcbL |6Kwn|ܚK1փ x_ecFy1ovZk]\ Y؞RuBٺ h|';כY ;Cfv~.tvs>փۏ?M5Zo`K=4/tpRs=wX}}PDxP}BNDz*1CApqvVfec6Dj@V\CN<&U>9o2tpTsߝ/+ME^믽AiܸD@g$~~YNjX'?on;5W_Ih PA$"%}Zo_ʞWH=]mH6k)tX8b-iOĵX񽣗 Zߊk3:ΞW/_kRԵ+P}~E+i6pY'Җb46w#]ԶkGkVԷyv~NňH@ Sbר|,78Ե>ЏGЗģt}6ӱsVlemv?~|&2^[g~^O^9uj =okny}[%1\"!v@]3}_*nlmo ͹ӄ |6Koݗqk.9dd+)lcyQ8zh^dvVy8nw O(A.*MaRD_燵V\,J1@oу[.K{Z;}?Vɾ3rѾ%5aHEވ4| m 6q mrps=c~ӵ}k׺lKio:x3WK1<lD7A_R@D|M}68#{|jK4.!)m:ǰs;<k(~au0ygQ\\ NO|׹G ˻6R[G14vnh?{}n^>~6ڻ1~f?[#Cf!n"|@JV7}Q|K>|/z!1) ^őD`vF Ra-VTVBk=eE7˛/)Vv+o`fʻό]Wb \ȔMP8FVwݸerlf6Lw7A \mu&7J?%:RXM/v++,'}2yo}w/5 W/¼aώp?,V։ƽ{^{}K qWk*S=?s,}(VW{Llwzx:1sr .f҉E=Jڬˏ{#G+,n;zHeĶ~\.u^ݟzgqgsre^y{g@]U%]ʶ9և?} 6p^Μ g{ev/9@(j2+rkdG|N53؞}x4~KܫfYoƞȅ9N/'Jol40&n[>CwN spoF/IS\[x1q^4D._Byk9s!zɂ^(^xFu{Ww{} { rG~YonƷ ?Da >Lp>.3.0zW '%/oG,f1y].plO L}!23q@K?@ܿIҝݷSVw6g<BI'n@ ϛ/uyO.OL^}# k#n<_Gr?Kc~oyh#`}: .38% >p@.qp <? jG\/GQç//_.7O׎Gˋ/Qy\+UPݫvǮo#^,WQ%sv~{ˋ2=́3Q9AދS$N?Cp_JM } 6`{w l뻫O>Y#-<kamƣ~i{ՉsA=;at-p=o|k+}k>;l~u\/\y [qyg?}|1g[Yc|^}xL_ti>izy9@͎jɣU_r-DK DqycYÉrp4"B|m[CQT;k'=X"xڿRg߰eʮݴ{mS.MM3}lߜ^ث=w~Y:у2=ݪray6~q9P 9h%9 VAiyq -|q6rV\9"ٞQ2ʧ[߃ X/ Ҥ9{|]8\w[gTuPlN{xudѵKtOo[ʳ{ygSN=Ε,`n>,5{~|],J,n?Y~+K' FmmvdͧvbymvIqnSDp, )?P5RkI@`AxO"򯖷$<ťI:.I}G/Gy ,]Ҟ^!xM~ue[s?SD q?3r(#X,JKQ5ɛM\0禀ocU[W`;{$*^:7nPhE?{wp.!<^M$Y@rd}y/eC۶ B\{`{PsUØ=gF4.Nк" UsҞ>h?y}NqJBӫ(z-ezNsI~/~gӹi G܃}묜*.}^~|gֵKtOgE[YgΧ{G3ǑuLw.&q՗k:FW틿8fPA7ƛ7eifSrMm1&J +聮zھNo,G>_> A_Sþ影gQ}m6\2a>7dc&C"t>KewO0}:OkT KNjVc׾ُs}w~5L*BO^VNulr-}㱎Y%Ҷ_g \H62e8{[rm9u-\nZlʺg4vwm2f p1siXcIg漨ѹr_j׏^پ#s >=v3z3kJ7bFsnW^g\g/}N#S·\;3Cڨxy&1>g2lʼnxBm x.^<-~׬SQl8boMS,ئh~ion޻dI{Ҝҽ5cb=E/9D>C1>+eT_/0ae#goow x 7!)77t큿h-l.>`1[ۭcg>4%YJL>[s}4窚Y9uޞi \_~62k7g-rc׏3Q#>e$\D'Ϧ\jh3%1>W^z29ۉdmu["|7wztCkHǂ8P<'nwY˿}z`XG&} +|KE=?v~fEy LjA){}1֚K>>m@o>ehnk5KmSݞ+xl߅Pܟ΅akM||N6ra;GPY!cءs{xQsx߭8!;}*}Eګ~6u_N kRr@uoޯܞcc6{ݿޗ1as{qsv}lanKv/R3V33O\v;[܇?3Q>OgR'o}8u򩅌Uk>FtǶϲoc9,Oʳn׾i ˟+3v?|z#`b똽xn#͹E@aGYx(1v?%]^-Yl9g[E\7|ʟU2>c{T;A>3ؖ#q_brK2>;063I+.ó_o.&7x%^Oڇ}\]cG7'ӋG3l\?mky}# S;y+}ne!3O簿e {>aWE"`}0G9cZJ'O_mg^y+Zy Yǟv +eft<;3#9ͷ|_\hԽ9RdjbkgY0 Ɂzk炑Q9ȶ 0p`hfnxɠ o?hm~&z91M`חe>e'gھxkJmԐ4UڮH&yӹ'>DmD;"D,+ulsPcx!;t5[h4{`Wx̿e-{u{!*L'!:MPCYw:'|ES{ҀM[7WE!>TO`+}T\3)ZkRa/.o=Pu#6{Է&Ufm?l\T[aOmOml,6\M9x[/bzcuV:W:,g:sqd3PUZ3I7>ƴ}ж`#ȳbkgAusTc.nm$~́g}{̅꼐x1 0c8v φywiN:Zw^#smjBgTf1ڿ^rrG&ɍu@N<>!D66%M>"t";.cOkcά}ysr}Luuߺ?S]}8D{hx}Nuy*h36|ҽƋn|^wfl7:l%d)땷H"7a^ 0K{!#׉*jaڊ.G}kXTn].y}"ض1Rb}\3aIZ}*KH7kpOҺ/=/F~Om]?s|l׉u`|?3hODqc%ڊ8ԺX9Q B\m!?<3B#Ώ%ponޗF>xrablM\v}6/7~&"UsM8&^ay%ö/-څlO/|0rQW" P\4'Zk^CS}wb+[9ʃq}{e7!p'6d ۛ>GYƣqh7\`},٣wVi|FpܙOw\9^4(^|V/_سLN2^?)~߂ZߪN/%wgµ7-~AM,ncP7q/ƛrsͤ#!;tn n~`1Uf1h@s5mdy VmIzot1whAf|907B;7ƳkK^믽7=iܸD@g$~~YNjX'?on;5W_Ih PA$"%}B{%q楮x/D_֚{}gH9jC\s_;QN6_GVu1>麝3wr!_+[qyѵtK(;{_|mIQfSO{6{v>2&lj<]?mkN>9k\>k 9[ ]?v~Nمg`)haɧԵyPzNzԺiKޞhK;]jsdfZ4C\Gb+m=ٌZ5h|v`ldT>$>K]c6MsLTt吏ߡu`N&b=Q[^sm>XM!KםdF3gQǣtsmrktngty͟+ymD;wr;=v|6q̙CZ37f/rZ!~gu8D=R\_?t>_^ qŗ~9ږ8faqXnR= TdX\-B- _8r#ؔF^i }>? L>͢=XᲴ'5{f/%Z耬 e[P]M{E߼79D<ؿ/zᗄ^m{ ќh+3}/yPq9~.$^_BοNuq~Q}p=nƸKtO;FgmewjYΙ Ӯ=/^?~ qT.XjMjL/pOvyf1~}|8M0B>uW -nߣ~36s\<3Msy<Z~^sW2+~Vޙoaתm`3Ћ]s&[fczms[V熵;DsOc)i\ӓؠgo;_Qc:o:c:yfr8fuxL_ù^.,dKGgG{X]Av>3P+?C~hGFg?ߟ/:kAqn~/r@:_}Du~ּ0fu8xY_?'wkt;&an9u0SY\XYQ}sG 1 ^87uM&] Ra-VTVBk=eE7˛/)Vv+o`fʻό]Wb \ȤÌεqHq-G3eSjs3A:\muƕ4-Jxu^ LWWYOX1: {d:/K_jN+ ;=}_yf;z9PqзYFϭv z_=`ߗ@&m#u־?UzTvIYn;Pϭǯ/xlVgu\iچڪXrZsHc>7r8w6Iڞ^|9?3.V;Ǔ`-ۤ-^+~V}oS+5ZUP1~k}uD:@9:z8=x'7S=!ȁ^\̉3bv=Y/g_ \pMSϞ8t: uiP@z_8a.m:L _[ нU<\(BY={U~ H/7sa^gA8&F}Wxe3q@qP}W w}sqY{UL D}ԟi9= $!HR٤{S^LJݲ.qbqOO[^Hܕ8qp1`+8+?_ >N }|8/w }W. %q>*@xe ,]//^\^>gyrzy]#SEpà y?ϝ)oDecbq~co{ç/jYtpBN-,d>([ǵ,_i|]彯>}:cNuv[m*bMf;|/*}wYfP˓k\_g˓?IxfUvFRdqnO^9+{r8AK<\?Nx8ObYq7A? `L75 -ѕM$AX¶we^Xp &͂Bu.ٻˇlgB-m;y7ʭWHhHxI/~/~L\-?m*,Ιxq==@KԿW_W+=1񯹭Ǐcl3P}WZ~=};/5B!}B澭^ֶxqC}j}Cp䋳@Y0~I)؉F$IllN- vc3'|9O{'_t95jI 39wcos릢>(1uf9LY+'f r1&w9_Ϋ3\ū3J>uq3 [>|ϸ[a:r{;C{=ߐp]#A;z1 ;}}3&$o~, ,?ߍȢ wY)—BѨ:EB>g ]FDMP.@\VP"<%anjm~T_x;=q7?yq_EsQQrAƶԷ][QlaşXɳ&٧۞abr-=D*>cz/lL瑶O+RȏTq7y??oO~tHaמ3, ~Yle- D UNp#1H!M[? YgK*6mq<˸3ln/*9$|!DQt 'H|# -׸x^[FS"]?(]>+H>Ͻp.xQ Sds>gA[OÍgoe鍿'վ @^"Q]pG/-r^V /^no~]<(kdiI/\Yuj.S{m硤>]@{e9'sÉXފᶼm; {k} ZĮ'ڵ-;)x mQ9M`cYT$򉺡-pQM$"ZH$tzM8iM2Ű-gaB:ϖ/f~`bI|pEu(@wũoc'cmgF|ߓ&mNm* ZI}ױ"}SC'.״糩 YOvzJ\_by0l:.W[ݽykq:y8Edmu)ЦD;UbT; ٮ%TN 'Z⾍W4籸ȅ ߔproC' |I)Kapb.FbD"WMv\'4Ҳ-8YDJiI7 o|SkO${vl,X |RʳP]}\P˳w+; 9ȿf07u~xz-_pP2هX_٭C=ův]q>[ȳyv,gҦ"&[jٞ<'|,[]7 gK~NI 9%.w7'a/PIȕua#dO,Kq^"vL},Qaϝ= _Fq;4>NIaxkgj֊|A_z͛@ @/՛1r_VzKxMseį7=NxZ߳_QlqTvb鹿H ඀.@~󟸗 yO s@yǘ"^?Uh[]R<_&/ ZxOBF ϽQ@${`GBvts収];G,@x1}jk LbkO j^oAsns{⾦K>`~ˈ4fk~s>}e3e^G/gRCŔhm~඀reS?y (ߙW[m#A9zzq?חi_(vv}{ߜWܟhǛٟmXa )@ ,$B?e/_ ^/h5YLȧbN~yv'^bK~sS^ "G:gdGއVy^}>̷x՛^ O /i?_h 1! Im=ܗ8UhOvŌ4VŁC8\'o/'R_ޖ2F X@;7P/~F>!:D(徽"l=Ѹ-Fa~gD先rLlc#Vm_'[/'_`{Z~HcٲylcU Mqi ѯzGo孠T-xH0_BO݆/TDBVdWXqҟO+?,նՖͧVeD׿~bo}>\4 !Ljq^k`;CKp ƭv}oQDes/6Hz4BOcJA>Bc?['7.5u>ţ˗Wq,On_^/?b~d ^W7OLYYr%\^<}n̮/eW 71 {zE,4O>?i PVߑmP9몟>cq-[1.vmZb*$~'S_}<;'m뼽|(}*q-WZe+~Wy﫥t߾Θ|>v+g}mmwmSem,'grVq? ?^-<_.Ͽ?%}~<{|a7oWevɂ !ky1;fh=K ՋUqNqWT~>F_P= -7|o[+F8eq]c-<$yN]-3}݈K2σ|l)/Vqi8ObYq7A? `L75 -ѕM$AX¶we^Xp &͂Bu.ٻˇlgB-m;y7ʭWHܟ$E/o둸凢M_~_}L/CBw&䟿r?*;/Yf'g)j! [U.I"H(lJ/Xa}b<,`x\_xz?MbQYXq9H\./ :sy먱;,ބ6Qmu.A1#x&3Kx1_T _^O]4E$wݿ2`n懋r,[qvyV/{}ckۘWH3^eզI{oKYO:_Ӝ9=|B䴄'hkc?IOBoXoV?/}eev|IBČ@_ܟy@ d_Z!~?}+}5qt:v(" z!-q%'Fvb@wӋfLIߒRIY_Y(#]@SD+B[u.so_d6إ˪ Eu}eĦR?*uz7o~, >ߋ_sc\u#ظkG%* N}ICDܯ}Se޵;{'uÈP(&a9i ’*zE .l[-TQ/k^oV`[IXHO۹nF^K̈E:VF纭8f)ߴ^AEԜQgVxbk.kN%(OtDeX?o"\ILP?c^Pi#硸_ ƞawv{yDW!q͈Ĕ, =BJIDYRBĠnh)6LJ|bMۮ.}qE|~$SȂV^H>+zɵhPOd 1#[{NeO^۸iO']jY".[R{-0졈 I0_Vo} JRhܗXv\c_ȶZr}m+,uOo~\4PHD}hqn@^R8u ?.0˙$j~[MªHH[S]]?[/F~IX$ȶV?o)+_Xx1Kj]E!&XBʼn n(8I5(T,yQ+ԵB8x-Xwr>|<-ܿ4\0)\}Gd;i^Xmtc/lOЏc$OeX>Qx6)>' o/}$N,7} V =l[@>]{^"v>׮m|dߌuiLOijϱmr?(΂W ,0-PMb"pXnlA8i[)"0!sg~u{?Y\h ÜO*'$YTEs}ؼL;U'5ia WΧ;f^wgϸE5ZܶW;| . vއ`[}۽/ ;P?F.nw0oK]~>! P$* }.n+,X"EC*pc siٖD%UҮ{Z|!7Z>'I ?^9/X{}7\7e~Äc7Wo_c,J R.qX5^:v[ĸؒ{]{_Bw>ͣ]/^Kx0*zMר3G ˉIaxER~a }\8 >p@.q|'7/uy\?2x'ifeW_./> N U7f\O2ڿNr|~^׋\pN-,d>([ǵ,_i|]彯>}:cNuv[m*bMf;UOD<_.Ͽ?%}~<{|a7oWev"/s|!/Rw >p>bdIW/zd.~4\Gg[?{ڤ絏oi7;ˮx#"qR\$Q@y&Gϖ6g}Q3sq?ç7"'>iC^¸3./i({ʗ6ZSr_~]{#"ǿn?@bL,1x} ?mE2'/طNjcL*S[5`=]`x\_xzIH8W$ &[[)qC)lu˓\GMQvoc]A#x&3Kx1_TK~v@^n7=+d"!r,[qvyV/{c16}.6MR}^ʴjt9{5s7QGдM%z.lWn{l7sXbj+kbnQܷ3&ѹ!syK_w+L4v(" z!-q%'F~b@wӋfLIߒRIgcY % 4Eık#Eu)DD aōMJiEP.KA%:ԾV2cmbS{X,ϋJ/ڃ[$)ύ5S4r׍c#+<3;%EM?ښۋ)rF\ao,V#kgwc=Ǝ^}(DZ{~}Iǹl'Uxto2=3LwoY}vuc^(+MeMڮ.}qE|~$SȂV^Hzy.\A=E$#{F,ƌn9RPc =zm+>Ow]jH+Zԇ޾|c< >,׈cY<:Kuֶ޵1;|>}$'aU*I틮IlVuȭƩ/˂v  j`<[}O"Hg>~+Z7Qqcs1~^ 'HA(Ķ{A-d(vJxph}hu@<)~OkmPcEV/ޅk4r=a [/Xe妍&( hS 潛ԇ׸oKWp?aT}s]Dh4/|7ֺO9f ~S4~/Gbo ̱X=%D7mD{X$w/3F/H3cq>y#6$@ĈD*4xE}¹sumhd1/O7/ ]Fz(׃kqiGb}"KDwpn}a~3uA U] 潛[DoqgsAv_\5aȉi8657;ݖ&CMf9+[1ܖm_wo_wm:ya?ڵ}3":qv_[՞c+_>!̋!ڤkZP6(.'&4Ng7 -SFrj&}lbVn8+ "g O*'$Ikj˜e݉7* |S׮΄v[a_d;O[cum;~eslzo\~z.4/]sK^_)E[g,y~%`3Ͽ?w;wt5?ŸO{km .k{9 מYD(Vבmz1@Suk +F,J/}ٱ@N}؝<вᛒn].QD *O%!ڸϥrEw%lz;.4Ҳ-=q6$qFm=-IpWfj-$~o ΅՟_9>l^ڳK)ۨ&DZ#f܂y}K*kf1.n;G iqYRQڮO%|<kTӣt.I;]^RhcluU}.n }Xnw{e}kωIaap(Łrm( .O"=:qY~へ\8 >p@.qp pY<^^||~fy0N3/G\=x0,\-7ޣlИdiO_Խ)== %9fsǩ>X}̇߶u^>>~K}ك+o+ڿRgo_gW>~iNwo>ֶM}^{)yݶwyG˳;~z}D^1X/_^>?[|OO=^0m7[s"ʮXri"˃ O=_=Lӟq{ŭ'I,1R0Nb&gv&gىnC 7oa )Wy=`y(DZwd7-hE?{wpqm^h>-zw;]E?gL$Hq\k`@የ_ q<#qgE3 snx>qNwq?*q\>"ZF VXf ;S?> {v ^j`B}[ mG}SV@;Gyp8,OxzZ=&ӱ,cB}Ă|]Y]u.Or5vKQmuQMGM!gݗAe/llȜJVfDN/ӷv$*8KL߶hmYZ>j\^v iΛAԷi+s즾d\U833]g_m7:>+y,^~ױm}-v^z|ϩ[a:}C`7- ܾ%dH܏^ Nq_pzqߌ> [_*)>qk ?̦==R˃Й?|?VŃy]y.+sIu$6ԏCOϟeA^ܧ6W_~{ؘgJ[yVYa09啉̛RW˳ç7ܜӗ|XǾ}zĥݳ}4-yyU[<3g۴L#E3mg|pPȳx<#Ks_Է-agWw11Vbb}r{`-##mz.0c/o:\|=gY&ؚZbl*]]Q$ 'Q? s/}yObB#pشP/cϰ;n,zna=7}a]?(]>+H>ϽpvRpy%Nu]q^g0TW)QsL,ɗxzdWA3dG{>o>g⓾5E+ڎ~GEP}Lǫ19jG{;Z }=+ ݉yI;y]y& nLhJhؿ~ln߲ Et_M;* ;L~VR߾fUGjR _ j #q$,B{Ad ~+Z7Qqc >6}3z-ŗAx=\wlP>̖ m*U(5+d-;Mrܶ b ?[|ܧ~6ϋbdŎ@iCn]Lq}d޵-Dm _`zsK-v+m|Q:oOjx⼪l8iYV:91ʻ ĺ[ή~qN1q~6Utc;!}Al疗iZz>/ʱ}#S僯ߚ[xg]pq?P\W Yx+m/0bݤ/7Ɇ^ `ܽ ``$dne7sN!8CBBほ}XogaE}¹sum~ }7xZܿs_7JZ1Œ˥bvs%+ׅauA;.џY7^qC}u獜U>BɽKkњ>򙪷3ٱڵ?NvQc[R"?Mk;rhŔ[k1!ʎN Uv45(* ;~Sgpop[öݽcYߵ鈾=׮_;vj[T{lɯ8:{!uz $C@:?`wYlԷs-"ĩγYq?㼘yp̬xB$%^Y{K:pG6oα^ΎjNkLӘ/^ef-tgNjDy#o1Hw}'+x<ڹ򌯋SF.nw0oK]6i(PRvWH(K?n<."pc siٖā@^?? IpWfj-$~Og[F=X0>眱볶y9kUPDX^8K$v.DV%mÿ5>$2hM۔~q=O./6Kb]F L+OC\8 >p@.qp pY<^^||~fy0N3/Gpq<|͓rԼNW!W_./> {|UwEZZLs;m nڽbm}ֶ~ϋ}6e:].AXmIzor5_| ./t$ ׿y|տ*Nz<\</{NiýQޔT zrRqb!(-$&o~`7o"p(;rS?Qm}˼<wM"v;EF럽|q϶~`y/Ik=$o6W< br%I/~/~L\-?m*,=@YVܿKX?P~ _rĊT qp_+=1ñJ}ϋ~cۿ*ЂL,1x} m?opVp/sz}kQ_y8>>ܼ`B |z:HY/U_]_'/&qVՑ" ~(ZaX]}:'a|$޳"f[1⾁EkZs*/<7LBJ Zw`~d$G}DYUPeGI:d ?[|ܧ~6 4\D_ܷPb~od5ݾlk1%~m־ȼ* z hvd_4h܈}u ~V6:W|6^`)!I_?on;%ړ ǽ@"{A0z07vG/I ^j R\RKl!) V'M]u-%kK4D4F3iqD[b1aX ׼"tyE1bvhW!;ɞ9PrqmFܟ%ׁMa½myv@uV?e}צ#>׮_;}mK^m>;B"N$N"XQ @bQ+ҝgDEږr)4D-|L\Ŭ}}ޏqV8D8TNH/[c+2y 6'ɌZBW߹Zly׵w_y B4f|?^}Ҟ˔ORyӧϘW}}~-J8u@DWN)B,'zT; ٮҗ;[⾍W僽~#Zv7|Sr۝ +*nE' #'(f"1 !,4ľ"q 79mMdqd*Njs$]Bn>|N'o58VBm$a{u:{g;SƏl}D2_ZhR׻^PceB5Q40>D[ݿٯhڇ~nx1Xغ{ɉIax5FkA$Чk{_zn")f~KFHW~U8~ ?O~& }\8 >p@.\/_./_Te2'R׫J;pwZY>|m[CQT;k'=X*[}{_-eo-}u|}7^1>cmkT|žo2.mwwd0˜V8$X/_^>?[|OO=^0m7[2p5'$b ~}3+8ڗL#&y]/#}B}Oܿ\{x^9ObKYi7A? J78 Np¬x#KN C|W$b׺#YnlD.ٻˇlgB-mNVr%ڕCHT8Pt|I"| $gz$lhS}fukbq|ńOo.G' PY,r}o~q] Ze\Z>ԿW5{bt_sS¬f ;S?> {v ^j`B}[ mG}SV@;w_nKOh$~~p" ϯ_FudcE%:fpk+Ŭ"P:sy먱aݛ*nWq7_w_ge67O([Yn֙x r@D{F^{rmx/^զ2_*%Ԝ:6Ģ ̣]<دHv/x^/]~Y=/Fټ0ߎpzNzWʓ5U}ۼhܻ}'qܢogLsCD 5qtء3w+HdHܷ!0S^7cOyֺv7$xdDHt "? # ZfCE`rYK#E#Q3&Q] "=q7?yq_E{p{Pg.aJ٬EףumM`ص3b|1>ݎ 6>_&7r-#ѵɹgY&/sYK6mG= IN~(P~Y?6/<[Ŧ-`;?qg-\ q_ $" )+*DJ(ܮEDj7WbjAY_G7,omEpklH&5GíW).rrVԺ鬍h xb=avdҧ}ϑѼfc].2TW qwnFnC?;a\s.ފQ.>>suX$pnDt;Z߾fUGjR -ѯb2#}6ߊ lo/re,\#zײַc=3_fx n(tV ,HxLBoFȑ}h@6q_ }g˓ԏyBSև[*S2V #tuwu+l x#lCcקic2M[ƪeDy\N!}m|TWONP^-"?|>.:~{D~gS?y װڵR֗!AOj ³|EH ^mx9&}qhO6eܼw#_؏ P"IWHDHTkhp.\][#ENXC}[obMև[*c1,D]\ ޞ@ ilcRpd\~^XexεPq?9A}>c>0ۡr%Ko>>jWg .xD1k޻\yQMa½myv@uV?{;6Ivmlrdߌu/.NmK^ER~HHE!'*PhbDԷ(¹HR_n>B"T˖0!sg~u{?Y\9c|R9!) ˜CE9Ss2uz[ݮ=-v\]u_^Qq}} yBǶG4/kwL}uc]`yFvҟWi]01 z!U)ϣKuy(f_h-šWnѼ}sRq?%:RM/vNzd?aԋowDYvءmzB^q -)9[mg!MQ!=lPHo+znChRq 79m>r8#۶랖$3_?ωDms/}2nCijKf|yx_ʜ36Dlsm31ۯyb؜1>)c#&1JofV};۝&C닒ύ_<^?xEƯY[7ؽ{BE䭺Nsi.IA.>`cU\^qξG ωIa6]`Q4'k 4}p_CR}ˈKZ9}`_?N<8Jqp }\8 >pA񧏗y,yӏ>9_-7/_,׏2.O_,/_\n_^/2p%WNW]ߴGYK>%\/Oea{gęD{r;|}Lf{ sw.?6_^[;)8gpxk?K=w$l{+~W~-,;ya6Go^3g.~a>Nk|0g~"ye;o>;Irkx?xt]nj <[\Soy_Xʁ488{ĕI6r2"B4S89|/v?-_,PYlW[ٟd{ ھ߉]>GxGqrnj\~4f3e&E󞈇y/8}},d>([ǵ,_eo~ZF<Jy﫥Da ܕ]'iYk[ۦ9ށ]*փ=:>Y>7|~h؇|cH}a 'c9fq9P| ݵi2ѹ8 VJtylUoBƜ!̉~}z{{q0B;,/4S@|rhcgo>}g8Xwun7O|FNϡ~gƳrw|;{͗ڥ|su89yN1Z~gO9I.܁_c:p:0=_8T>C~mmvdͧ cŖkB&4wcinpWC$lNxj>'QZ=Z׵%=vbuIÿ#ӗ# xy_2YۑQ!vvSwr@0=VbXkW9|Fx㗇ާ$ Ei)&1yTƳM_#xjKv ,a;{$X^:7صnPhE?{wp.!<^M$YHm𿕛y0or{ w/|/8瀾3}k3FfpMZם|:pځ}Ag{)}5{2*?wyc~1=kR<)N^> !u8xO[?!h}ЙByg{}f<+'~>>x _.`3v4l%N/TX_ΜO Iܿ,#nl8]f{`Ml+?rL~y"(ϯ@|3{?]U5|˃R-` #]w-G^/ X}r09z"&xݙ{M0 s`F=ܓe6)隬#h|`m~;ЉfM}+ZG4To˶z]O٧Fd ]/ѩ1q7®-NְςٮK m?yA˱kő&MrxTn?GxZegn_и/^/7އkN]?q%_7?S޳0{sg_v_g4jȵ56uU7&W®]qlNs8Mgή` 4yPQ|8{/%^/<`+gXJ[9LCSsI)[qa+o#iN(`#}_Q7Գ8ZW=Ë~WK>ă(\kn\݊u~nnmLg&{t띕s ׶|2N7{f|Q~Eч? پi4 bмր_xLGsMݵ޼>AoO6sL碉kzy73k7օsaKYYg rxO516D.qs?bPC]χܰe"?uz+=vv6#[~(#9\6\I Xm ]፵RUYhnΦ늶\.%mطS+<絼%G"m m0>ءs{xQsx߭8RCu5&gk6?}x_*cK{u_N kR:Zsmsslf/|:>n1}o2qĹo"1mE*qvڪ}fư)tcޢ ng p9\sZW̟y6)|j!cy>o4.s꯱x^r']I|`^~ќWizi=t.ӎK[WzbGk[\Y{ȼ8.gvbFھ\֭w9 K@C 2w_M,lƒ𸵭>#g-G, tu鳝gc۷рn<<-Ѽ.2δyD<loc  /oǶywtgdʶӵ%%|cyv1шw2g:G>+u\9=|];3haHXoGO_ϫy4$l>}gQi;Q?yf/5GJl̻ yQsйHa<Ǩv8~C}/9!T/_A5/Tm-Pmg`\orT]<yb{zZu( 6,4Df%}nEMvH|~1`b>}nsQeQ BV.kF-pԖpbm*L'Q8zc1}o/%?-ڛTlsC^mޑZ%#pþNgiIѡ,&.o=Pu#6{m212Cy"G8VSSM)vS?zVf^.'cz~\_yk xk$\g}P*)mʖ(?Cs~d؎x[^P7{k_m|9g5{^DS&ůe߯[=exoD']?附Ov?߷z. l#zf7c9ـ~JžuY2w8Gd[vKvcę_pV}z+39Kp.ѧ>/">FqdʝvؕiO,lg.#w&Uck[R_ύ_qL߲i\f xBvV*W~!aM>ckj?^^\. Fm5Yb2WFPꄶu})vlG7/' f~xL s=jBL7`j[.;OFсP~_0.u T Riz~]utA 6 f' 2p@IW*@ϩE8iU3VᲧDyZ` akkݟo#^9BֺKFmb>ՆɛW`ƣugvywchEMB/ NMHuqq70mPh°].y}>GV?yA"kZ8qfw/:E=}_{A?'ɶhk9oQC ؇-e:?9猯?Q\X"` v6vN_Cz>W[f;9cݵ>y,'6I䳊` %5|1ـ~m~uEVVܡy sEh3GmIޙI Y,QyŸE=<<$SuDڞ9FB683y۾/ݼ_ٱ|3;Pg:_7dk)B }6ᯨ( wŘlrK5t]c)7F#Vǔ̕cߌ5YO1׺ǹ@4=܏"#u/(bm%MlO?v7wR C~]d}>{'`~Ш˳`6utG'1Q5|P2C2t <(m= 6>Ѿ/e[b@D u]UņhCs-Gu|a|5\0MUpnbwx%k73HPA"7"؁uޯ7X,]q~^cSs%aܽ  9q_)$/5P^?Bܟ=֚{8E{E|{lov}W|;t v0:#mp:q}?(>.Xbn}u;q-G7-f|BVA>YFlm/_kRԵ+P}~E+i6pY'Җb46w#]ԶkGkVԷyv~NňH@ܗѰ|88#[o|s4S׺l@?A_I2WwC>Ńǖv9w+Yj砯׃__ [{l!-X+3Aq?zg5yO757v-}Js}.Y}oZ_; LNƮ>/_7L7لZsiq>ٷ5S<稿hQ\;w/SV觎=f>t}rT-ן͍`^?O?f>3Pݛ'u'4vnvn['O~#<}1_Nu,w0/}qdvLK24ހJ~KόT!xV L"M ̈7dW079_ϊ>s<9b>ݸع1DE0y#mx#6h6L۷%%I˵}ϭc{+WvXe679CKHkc}s=x mac4cO9]{%f^(})!֪/GGm.=o/潱!%/E ϝ9oVgd%O_r\HX녜{MNnƸKt@y},cd蔏p)~u{q5qƫ̣'i޸s^6ymo$]ys|gH0$T'|PU}Ųږw4%z~Y-g#0ϚkANV[ooEv#}zqmه[_֏˟H}72yuAog?}q'QTG5utdYu.؆ԧmþ[ρ;׏;</X ]/ P)/j2;}_TkKڔx)$z!G~Ibd9P@Z*&b7X_f6)'\7 !p Jhr寕ࡳ_٭ C-O-m3 \pJ׵l7QΔ<ԆnpQ6hatnf_IcgqޢDW]G q~! +Fg!yoLc[_GfjN+={`_yf;z9!y_m3znekǏችߗ &m#u־?UzTvIYn;Pϭǯ/xlѶPxH6VlZ3lzm Dp-2r v1? 걱sf\v9kf~|<-yV!^f>:GX'f `z^.DA?LI"VS}1W.qRu1t>d|Kי̷3c:/Gzcow~aClGi^(cӜzw˽+xt}'[cm>+c~'ra/Ӌtxuɦ3! 8'Cnӫ\.۸K)^TwWg}OVqk 0K QecfxFu{Ww{} { rg_"#=p3M\}f"ņ{/8-@H!}{{=Ξ[_<ʿH:~vϿ7yDσo5^r/<:!}pp9ɥ;p?eugs{^K* .t-R~|:P}RʺcqO_^1ܕx9Nlei/~oyh#`}: .38% >p@.qp <? jG\/GQç//_.7O׎Gˋ/Qy\+UPݫvǮo#^,WQ%sv~{ˋ2=́3Q9AދS$N?Cp_JM } 6`{w l뻫O>Y#-<kamƣ~i{ՉsA=;at-p=o|k+}k>;l~u\/\y [qyg?}|1g[Yc|^}xL_ti>izy9@͎jɣU_r-DK DqycYÉrp4"B<Ջ t_'-[GX>ؔ񯶲?-Ȕ}|T9Ը4./3i̊IgFۭw=_-ooe-L=eFo;b=9̶iq~< ˣϿ|۶ˇowײO>{|EwkxKy﫥Da ܕ]'iYk[ۦ9ށ]*փ=:>Y>7|~ޥ7*s엥=X8!Cڭ K+Wlce}38kt`,ZJ=(y4=%Ҙ FS|s+G>>jCFt{0?<>5pq~q3A4=b]}묜*͉58xOCy㱟n,vtmKyִ79ws~:{iѹE q͇QEfoY܏ˠӢ]Eɠb\͢'o~$ȶuծl4َ>^,/n01mʜ&Q< A=MBs݃u][Kc.^$1?2}9R=-ϔ8L>|vxr~ݲ#=L౷xC|0a Z^!jNU:)\\ |zќAY>s=Li.oEoq~:7ͽ=>{}soS>?",ݺvtlu>4s\yfq8ΓYc!!o8]7brm=[(u?Z}vq':"zxF~y",lʭ/W?] +f59ʃR/xA%FBp@]Q+6m q?YaD@a@6iyGISy{E7&1GxZegn_$+~c2ٟΫQ>{yX9EKʵi:fH{wa]3r!4~Ȕ}fLk)Ce|qX`6}檬;\}Fcw&cր) ǡ:p>U>>Dyh΋+GX?Uz5[82c1:1˞gZ>vدt)Q9`46~~zJ|V@02|;= ;]>l23~&SVXߎgy.ـ;z?A|XZ\yͪ=ņ#Fn $?my!JK')+K[3v+SsA34:ORFEaf^>|F:<~盀p}ҟr|s}COr=/R[Z/㱵z~SsqQ*1xOHVW-bgx{wLW?t,ţ~R-|Q/ud9grʗ^s}SiWg6QY8(PI jK"1 ZA*? #V,ϵ/#m@o>e"?QvPٌk4_l}%+Z~pVڤn(xt K=[>̬M;6׵19o F-"$ 6xjgHxg;C{{}3]q!|GjGX1'ש59q(K\nAZ溍uɧ2vWƞ?˾-ׇGll?+ϺKs\1/H]<}և V崙Q<إή\J+־nDZ[}l鍘%c˺`4}!g⡔zƐɇV0;w}xUgvֳ[(n5sWng]."[;i:{vfiOv+;lϝ8.jc$ܞ6[{(V}ŽSЏCgrb[v|}-ľKv6$ ~16xU?iquxƞO/<9ss Y6ϏOmg=l{ς.7g*3.ΗѹS|j8D^;XDt6}q:OīGkS[D-nd{zZ&>=|!)2 Y =q{Ac V@??qUOHۋTG/̼t5%@+KC0c,[4rM.2F\)Go=:k\CK~2̬ߙUu{NwOeefdDdTT9n]Ӿ#{1Z݃Q!D0aX}tR?>H:L8 osSpN:`IQw-/kW}UR.E}[z?MeR.ŗmԳS_^جuݖZ%mE#m4ˢcQLx{0xI綎c !acwXYڟ?6r6nokgyz#maiTnAˊ tanu2FqyPύ:= Ts"c2N7M@&kV}=F^9e?8 w~v`N^k XU~5^a|ihkpFmq&ZGtk: u6:!T>%}I܀vo9Tlh?d{4ʾ+^ȘQ1>',oKMQnu@lʇܱ/ u [ܫi&lY Ztk ~c8 Ά~֨&nwq-\Hgd Oۈtmrly``L3MO?*jO*,MƱdtc'wV6+U6"1uI#%ع61epvgtoҺؒ/ý|l>)̫N1>,X0?^>Hq/?X /cRF_ژ&es;h37.n@i 96wGq\utݡn(w38(1KPIG'x sdbA+iv(b~PׇO7"[[yXBS?V}\G=q: &8t2c3,䋎J g=:j~nlQr Eϋkaz@ :*q+´, vYf׭9aqob,柿ϙyi׀wzE+1Իk sĔ]/F:&Ѿj+1G2=z|d=Ĉz~Og`akG`ŶKgNZ P|`H sdC_R1[ 0_l;mtQ}|őٽ?u SF/n>wz{\WطOmi*? Cnv_6wvuWj8h?CQYgAc'l93\5Pg-=XJ>?$4W{J݉AM&\\dx~h<6c\\at|al[d-/T:їbPekXns #Mb:[Kۿhk?``-/X:!'LYq[PgX95v`Hgmj7amٗX:CQ*H_91Gљ [惿qW(1'Лx sIWdD+jQ~1V⾲Y4^}>_.H}/_\n^g\bg{(m]^PDlcS|xڸdžó'ݱpB6u08ts` nYAw0k:F,2 bvPwMV\c:'ｭ=_%+e^{q (Um^V /+X,+d뢻]D} 3t? ̃Epe°Pϙ+6%`l]Wܗ@b &OF^Xa?u }8#s].^ 9=Vp\n ~~ˎ.u޿tޏ\3cFf߳/G^e_=Ixڎ(}lTf~Fxp9 dsxG8 TGx͑?T%v>|㜋 C 6'߆3aᚅص~^=̠SIܘ7n\FƗJ"˦1rnѹ^-֨ϥN#RD6-5{}y5t6:MbrrίŰǦ1 P5[vt|ʺ|Ho/:ٺvdScF:696:A[M{ +m~9u&y\lc,c} [ S}|k^yFDbwb?>6p$g}(_s}0#{i]Zelv 1\?x/9QjQ"yIg0Vqj^>󈍅c/s )cȱ9W*lz9`(qh *kfRI8Xw}m`(:A.cm&OehXNdC;vkcy!/WЇ1>X +igM-$,9^g?#[ uQ;E>Y Mu.~L1'ҧe?{s /PYPg&(v̑?z(t\*JV؊/!p/Əo~26ƾKXA2_*× }/|u9Cϗb?"7#zUkc_(Z^Ffn1`{ݟ_m)JG~{Dsi?ڶw,ԱWt۪ϬBg?zN#Zp6/HrLElJOSl]BUtt,eF[a FkC-TYw\8[B8Џ[F_HDZ<=؇zG}!gxu4b%g-Fĵ.8(W5Tfs zi.sؖ ^e^ӓ0X39\/^-HH{[|c~⺩τ[[߸ϔh/#E5u?rJe.q_8ҏ,8@&;[u&y c/5)zS}:?˼׆\0$y#VP:}Ml<.ڲAɯc=?<Y{#;$ӿ[q*S_=%a7fiӍE^]>3|zlt:ñXk#K9>pj_%M/փO(rPn-T_ `~͢p-VF .dkuQcXkgZ0Ln>)|mMzz[[d؇xA`Cڑ8 n"QL$L6/!*{ُmk/QSq!_Vtjb'QF-d18^:0v-dc5-Dnf)w僭Vɹ/8} \7[5ٷe??z :.HBFmƾg{_'t~t.|cs9cR{,Pٖ>Зͥdlk6!\cc*487_5vϊ(=,KB6*!WK_{%3ku=x 2|_[hFkC~\|_±xm.ѿaţ7ڿ&ۛ~`8?[mS+~r_96{Y;څ6lۣabsoqT;ݼ,&-c\k4&rxLeY_< `6z:ScOft쥍B8ܑΤ?;> D6Ö=h X6?>X~ϸto}.,]ͭdݛ+iMaB-Y8V΍ C~2uØ תWS-|k)|b͕2|rM,{oӣEϋqtj:ѼNT6D81uRS}9s-x 8?թO}O0| 9=T<fO)_ßlK$97x$O1J"u*IsrRP*!y= \%s$-c\_7ǽ:(~s+0'˳ "Jo9I/ Y?ToK~&yid>ojq%IX27/YO WoPP|Y=jg%Os$yPqSR؟ix5}1@q%q )/M/d~4_~[<_}$yXUzqzS-IkGȎ}X}?=6c"7OͶl>En/A>|0K o牕WDEho.%޹8>pyOW^~`7,cO//-ON?YΧ?O_m:{jg)g{e}v#(}w9dwa{V=ɗ9}q~s񭬿Q lyc\U}9[t 4vNSL*Je΋̳_C8L;D~~uzO;|zGϧ'Vyע߭ӟ~<=oINO>~ځ2kc>noWV;=b6% }]I;k^@/;esJ4!jbOOT~+yϣoHG{Yoǧ/zƃ 95׼gv&v^V: 25h{bo7Oӻkx]?/I;;A|`%?ɍ`\kiς#lgd %}GJCn|a;%H3@n)K+eOs\]q;xY.cg=&׵co=kF6  Yfq2p>c7q+㻥x_]+n9sL~ ,ۏ)ycsq߾NnHv!qe |I1&~ ɮҀ/FR!U?U^hϧFb5F{uz&悂y1CrbA(a^|.cyUR~s1ncuAuѶonNW2f_m4D/Q=] }) `I$PT%T,Jj)Z] pZZ/suYJq7(E_ǧ>ɍE_zhxrq͢M:U7[ b{*Kn.nK1w=h]ZlȮ [y[m֜ ʜß. )7w O;y=f?$l!qdgO*^9f(5^&>{Y5]KҋO=b@is==g ,n4֗ɶ@D̉W؟$c;-Q3B +XTvˮC[{ڸd;]dִ{j`kۂQo[EߊXцP}-M}0ΑxS1)Wו.;X̽|^|]?xMj1|r LBgm=.ОD֦z 9:|wR U0}v=Z_zV0[ɵpBv<+켅{8_q~P]3c N)}i7-k:ۿ&.Ru-5=供ٛagz*1_98Xϕ,JyLy>|?;\V&C{x>@r@=oEBk̮)~+`eH7+9܋s"G^Njo+Ykif\2:}czu Xc|][.\?^c~&6~6#zXZvkfQ|pK$cS0ew~V(_*[ܛB D1cBM__`z26) o`XgV:ȭl~T Qs\%{.XoI5\ɠyM纀3$Dlҟܧ>RʿqaY%8 q"09CX!C~cXEohC갘 l=UNnrxeH\\Y˽Zp&۷} tpA\~XpmCTs㽤/usp.8Xz)yVx]`pcKAVg,E9c#aqkczmڱ>DzeukB=- ⇵?g+XHd-y,co8wqMcu|[~sG,u1mn28]Z}v3|YYܦK~5L^i>Qs*7üך\el'RU>tqݗ@ #YokE;yLuvec8& uYƢv[CO'tyG!m_dYuP3ikvy-¶{^}+ʱg^< s1YFc۟yquɳw֣Uk'uRƒ:^ !?h\[ՙ#jhȱ/xa|H@ku€mc _{s"X3 <30s,yMf=/KQdy@v?(rA:|1\'=y/ʼsrn\s=pTgL,G:ñHy->]YGO=cEaEA&b>3[O\q-\Hgeз֏ue5v39~4l^dsbۏ)tu5)TR#{XMhl3'M/QhO-vL= M{Nj-owMt$ا(X ź)I^D:Q !*C |Kx_tNSaY3\A6ͳx% &lBC[os>ic.<~& }n+䃋r6!17<8Z[,)خEVmNhਈR.E} ={YrC l(0Jfp_ep.)OksbCC%uk^fL%sSM&t0V40cEX,6S:;u&2c2Y@JVwhn+WȜd{?hWy(ym,{/e>w4xN`rMGqlW_:]tPgw5Xyxc4B qLe(vX٘87Az@[;{g+ف V:z`_󭛟Ծ=dq}),!εPE?rGA]BsӾ/8״,EYGC勍ΰ~f_v`3D}a3lm_:3lC$cbزWk\0I=[3&@kT I]\5Y@_C|ejlJyp,Y|>oF0dPϷXg}l2%c 7{g/{)/!K KXGqe] mf I>{V; pRnA`a-a*/b_f]kme1_~&܃ W׃zWgF[w[k6m ׋X: J>?%`f%|g |\W;Jr:j~nBLj_ hߊ ZU.Q\JAiF4 8Q,v}olKJW,x۲iA/PܟtQuJGg 1Eqxx{ΌwӉ m)|a,r3L4~ jo.ȵ,6Scqߙ?.Ѯ-(9gnj_" s%Z®\b^mK-;'-`b5~Gs"Y? ue3eu1`xρ\E2k'nw䣥9e\;x~;ze}7dQ/^Ů:Uc{-긮8zX?`X ]@Jּڔ܆,^ers-Q~c~agj]d.;y`nIb/k&;Ɩ~´$L\7sy+yg[g6~?ʑT1o"cҹ<:ܐ3uѼ~C o/XY[ EceO+lcaYMm#.4Y ?&:W 6,LԷkQ۵I3t4ڍy|r> |g_Wl90"(rz+xz}LF$s'骉!@cIE".xB[+Q>6(gOz}x相\:ᤂjt P4bUtѽ>2 eLA{2Z}/syp:(*ѹTl7{l[J9,gz z 0a ܇`6J* ql+'ｭFåk7oK3ˊ3f,EwX'NP݀yH@,7e% ÷(뺌9?jx ĽgfRn3 ♓h{{9XH&ʹ`0ֱQ`ί:iukz,)Z16.ܻuEA+9IYyXk#صUv?xu:ׁ> rCM.k"sLGs@U`fs|\83]` whR y+gFxZS:s3&GNs%AI#RtGxu9=M2>@; >cS_P~1~nFg*y>T?[].2&M uѱ{k^7$锭::3*k cwblmtbb-kQڇtA~̍kk}Eo/ݣuuB_YgjlWHgM龱틭Ak}2s5Fl:<˽}r{nʷ&LçE_@z;.̥Vo/ *ȹ&%^b]w JpRɄp9t~%*`^yPqz:>C%+ZkCF@돇t}}~ խCJCH+ 9$Uwplȱ9eFO}D/76믻׹b 2{e?ۖ sfw\b@. ;E2x W?2Ɖ8aƞm z"` sX|ñX􅭵}@yd 9%=i35#[+{6WYo[`ϊnP#CY7JnĹ󨼨ҐG*GKxs}P:$ԇA_88>Hgwu:ĺB[:H^k dzץpOѷzsұj_c>Sx0zw?WcI/u" Z(W8}Pnqa]+p(9_, Vmyw,Ը=3*gb?(o?Z="}_9+gD󅘍c/P_අ{!<\ɂsppF:?#sMwہ=L/XWOԣ9s7r9(s.UA0Q4DDM|:z/$6d$^*l@P ABzP `V_3XV,[6/jxhL N*زx}I{P±AtsK 6)8c(؇kmA Mڑ8}(/ of`kUmM9kq^ŠZT-_ =}k ѵuLY:ցk!i)px)w僲~+ _пw@X_om=_6p/&g39UzQs˸rvX/aSr5>u$wom޳+c1m},9& r|gDR?PɱGy%s sĹ}^2qK/vW͑L!ڞc`D1ٗ|D:{ {!#NǯغuJll[\`>'\|N9K[C^;v-*6yEv&c*lAM59;瘠ٛA-" Y+uu!oV_S;9cI푱 @/Kmmڇ9őX|1>Bhh_߲ !$9~.9scUy r$1A5Gـ.WYdyLDϯx~Jgr[; A^ھ;;gۇmæ=tO`|93wܺ/$U>S?skc?ff5:Ǵ4FPG: ʅjc~2u/c\uLQiS1u*$M({oӣ%{Ee }Q\=w?}$ r\$IryΚp mpr|nȗכ"Irۮu q r>Ϭ^kt#,o/H|n=n[|N,sM枋o)sPĶ,^ZKҐ Xrڷ!Ѓ&2S1Ir^ʞ_WTmnIʯ_܋o_aY_ ?sʇ}!{c֭~nڒVo)sP.T_dӃ?c=ؚ_%[|1I k!c"V*B\y"?b8[/wK?n_yg琧6[r٭N"g}~$WOIDATDƜ7 $I$I$I$I$I$I$I$9'YO$I$I$I$I$I$I$I+'I$I$I$I$I$I$I$Irdq?I$I$I$I$I$I$I$I,'I$I$I$I$I$I$I$ɕ$I$I$I$I$I$I$I$r.PNC'i'W移ׯN/>ڒyͫׯOw_o8Ֆ$Ey5#d>,>^|1ꛧfEx znry k>y]%h,;o_|z琻s֋XxwIJ//u:>pyW^~`?p6óGEu?s-nyP~sr9̹r.p긕sV{$&{=5r{$3畛ޓNFmzīYlH7ͻON?@pJF=t^sV eS*nln|A=ӌƦFgr2lTӠ\j{)g(>;uzO;|zGϧ'Vyע߭ӟ~<=oINO>~ځ2kc>noWV;=b6% }]I;k^@/;es:]:NpǧO*7߽}OH lwNM5u]BlL.ڞ#=[?u_ӝ+?/mӚF1 8Hq}k9Gl}Fvlyȡdn7Zٰd vtV^L++3_(\vpߊ|Ӻv{ygn<.y׳'^;Yl7ʑ.Ņ,{uq}:}'Y/ElM' _>9@^Xq/U?Up@-}>5\WW|^ԹM1Crba(a^|.cyUR~s1':&Y'4_q{Ymfte-c FcK.OIH;y8AhJ kaw-kke)] A}(til(b{Grc~4'\,T_uؾ(ϖY[_X-|.\ei;y0Ŗz=9WX0TWD$&ȩ:;! Wp`e4k݊N |Ol_K|1 l/%cIɽ/kC"-)s=Ӌ.XĎ%;ygUyV{Iѷ}]o oEi]V=q_bqt+_`MsK}؟_;٣ģ{,{y uQLgNlevgOȗ\mo/sxb1FMHw_̃{~uzZ`Dhd-ZWV~Uѿ/%8F\kvKֺo&5je[mI}a|Xa/M1?\7|}X"$eG]$4MEYH_K)$Ke|CfFu5+ދqTnsw _cGil[" A\[ {_mGoOa(E{ZwleL!cgg'^^`ϽE'˞m149ޞ3D7xTd{bd[qx"Vx+cU|!beÕ`e,*eWK=m\@j.r elk[νuoY9Uo{ߊXцP}-mKq &"|E-(,ߣzoߝ^,󕱖˚^9՛jo}W=V?"sDZV Qq]|lAȥ+wyo^B{P~e_7?0(T}⦡~yS"{Yg plR80u`rq;Ω>Ś`^:ζ5[ɿ(N3`O;8IxؕZ,y QD̈]Fq(zڒ؁i]@y%_|h+H{ Au|Eu&|ot_do rP'}z Bn)?k{CG7KE&?}-{ ږ>"Rט=>>6N`~ cp[s(xm1eϢz&o긽_Ǯ/ȹž_b~m S7%,Q.~ 94eBNC܏t|%u'Nzhu/;wK^FNn׭Ac`Et{l1Rʿq&fA zi}Pb /zpР`ZEkyʞ_>C\*~Aq_3tM,k0+kw] dr6Q/"NCnZ/} -a v@}0$f4)*rnvn5R(:Ud?-,L0Yd7s Ftq  Eۛ^LN?9a0ܿ٣/\w|ص|^eAu{s;`\TX]gRK;r9BU?";};CxشCq򬁶cӇEx2Adg||`=UPuD-uqla ];f:, /YW}.gOJ|汒Y9 E`lx˿ց|vTXju2ѱC~@h-3ʲUtesK\rt^ȱMnon>ߍ5ϘV{k_%=XC}}VKqb/Z'm㕁"^QL|3ӯ~G޵+eݤ+h`[ 4N~h{Wr|L1O$ 1Gِ sy!^k!W[wG= ލOه/;F'ؓ_=g/@u{`4ÿW@+m?OmtYí|rUqa `w` {pPCrȪ A*+!mNH+:끳7w=`<&yV5D؄_hh}Cu؇5vb`)L~& }n|fr6!171hl `YZۼbU]1/0]¥}ួ~)A l(J_`#~Y[ĺne.ŊF/_϶-//ql]32gȱ,yHl,Tl/ot0V40cEX,6S:;u&2c2Y@JVwhn+WȜdyCˡ]Q9Jif!Y>M^|L{9%[cOsP~V:2 YaNsݫ0䍥*f./'Z_dnW-c;6n1R\ $7ܻ1XI;z`xg =y.>`,U(,.aucc~^K.+ޅ}#S![ѽXb?]. !焅_c![|V({b[)yC֮ A٫- l~Dd5? #͢uk kMX!/E Q#۪v C?z h!3sstؐ{fl_ѽe>Ož{v˪&:،_h:E¹ l`:Xј'çd.g3bѶ06osXn;F},Cѥ\+]2X~rw&s>~Vq/sg˔tpfj8i6Xpr96ͅ㰍 e]wn(|'6>G:]CTh|~?ϣ~bK;ՅrQ}{Pazp]١5j(o}x6ñ-zK^)c^XI@Ge{%HNI;1[>`Qb&RJ<-BVKuTWERxaB;a2/~cXeCݲ5Zo|soxbvt-sf{L%v~||6cn[󅱲FCO%?XK#췠\˂i35c[2JbMzNp%:<_\-:%F\-E~3 #q;8>&;} E6KzqQbkwe[%8ddO/s=~sU}5Ժ2kt!t:sGrgn<(AQ=<Xc9YƢg&A%653@V ` ֽy t<3 {@׍xKؽbʣLx<"dP2\!Ƴss~-ӵˢ׍m 2j[0l-e@F?;`?,O-X|[ze P¿ {6| /,{ Ly/]cu}ú}=D:#>a'\^YmÚg_mu _/U@F?w>e;{q~65|Sq2=,{FcKe݈d[ |g2{ZLP+*{^G^ŧx'?X\:6p:1><}<|lp= `M\w0*{FJ(](k=md8 ,E{])Od(T1mؗEhfhu7Nйyz{l7{l[J9,gz ЇkJ}N $@?w;HNxmIy1"QxօKr@Na,EwX'Ng~ |,[-[_X}C#rxo#c3f3{9XH&ʹ`CF΂9DG륲hy Fڸp f~1u?ƹ:9A?]1E3ܛA_@:YSQ=ÿ=7>#{bGTG u zI?[ڕ?]zvP❞] `)<3.yPu\Ǭ3>?aؗ){7ı+ލbr#[Jr/RŮ;\$6Xjc)=>(h sC-3hW:#A]ЗA_G:ES}ƙ9^ƶܤԻ1ei= C/`*ׄy8~鬌kܑyvF{ك<܆_Zy^}|{c `퟊;lﯸK:s}ϊ9ZQtʯ/fea\__R;6_bdU2(O?M.ۅ́c/s)c}y؋AN6ڌ%IvƃO29gqQVlwʃE<(;Ex=v!`:ޡt~YCqVΈ|?!{EmΡc@Ĵラ%JrlI20!~:rlNY,ȧ>" X^ݽe|e\=_w Sc2mT9A~.˂28_ _ iD};4/sDོ#cf0>E/lpl:xkQ^rƒǴ[@N=zܫ,շ-}0gER7(ۑϛ'Jn%UyT^{iHRGe|y|ǫoH{^?؞D3܇Rg.XFvFm#{~4F c ]_m=?U}2&[{hdӃ

cyHJٛ<PT,g+yL)m~ 1(ECD65͗ܨEߓs9w[#cW?Ϧ>ׁ 3{Gb.j_}~~10Zn}nkdo9gh + ȷcG2cv;DZirgZϖ,Sk@tG:>GgflXgjl.͇]L]>\dl˶+2Mxn>go܏ʴ-s]׾Q.>f`9ޯ vWero/p`E_,w07-W7nLg!wUq;_(pm :~Ifƅ!UظQkM1~dhͺRJ᠕CʓWe,P_lټzhec1i+8c%f1_-u6<Ct?$؇=.I iGZ0(K!HNL6޶[ƾ6?$x9kq^ŠZX-_ =}ߨ,FB1gXƮlwlmї0/|A ^q\7oc1G2Ǝk4D@x& z7\/jn7j&be4\MUAwl.>f'ˑܽͶz:S*2YsMƥTH)kcJhL1AWdz"f*?E/z~⸹.nYst\ڃt9󍁏gm=Y@7oݯ3Mˍ(boQ#&Ц|kE>nF&s\Nmz+Yf}dE;[tbe'[ĕUoc庨=y@u#Fkl!HO%"C_?i[ƄR6s*DX!\|?^ rI!g~Sg;@A*LƐ=.ˈ\mu.K{B3f{S?i)/ Mz?/AdI$I. <`}Owoܪ#&pgci߾D6~8tn`ɦ oj tyBBx3$IBY,^,Ag&^[B|7*31M{'2S1Ir; 6ؿg/糸os|7_}$ɛ93$I>.T_z'l $yc`*Iʍ|M QNݦS!ؒ-bz}3G Vdlsb9⤮/A$L$ /'ߍI$I$I$I$I$I$I$I,'I$I$I$I$I$I$I$ɕ$I$I$I$I$I$I$I$r$I$I$I$I$I$I$I$WN$I$I$I$I$I$I$I~$I$I$I$I$I$I$I\9(gՓo{4~+XWYmIr<ק/^^}jKyzzkW=^#^ߝ[m}KQ_>٦@6AH>?+Q׋9y3ɗw˜%쵷wO_W^/ng<:}g~s-' _ӷm׵SQfA]]x?_&u!V˾%>5|OEh﷛Α+doR５lY|移oKtNĸ}9[_"+4˒i~&%'_:U !$,?O?@[_w>qm7wߟ~<=og.<k}n]iSؼϿuFYE+y5?GΞu څ/Lzs*ĭp<~I7<:wXO~| i_m<4dyz_q{&hkgralWo)/SjoO.&N,("U~_{mwqp%x(`^XD/8/,k'GB{WQ,Sv؊]+O7k޿_~ 0gbXy4?,q޶s.9֋)zvP:'Y_~qxPo9GwW3^L6t\.{߼rQ{}Z׮|lA?iÛUbY|uC$0S&"ON?l3ID\&U?Up@-}>5^Q /hWW.ׇ܃ px1:!h)`xhZuˀ;=KFj*ZN)ҼKQ{b|yl\2cЍXWdEcQGץ9ҙ`?M>J@[̞ŤVZX-䵠_ ka_ X#p.K)ǧ<8ɍE_zhxrq͢M:U7x}a6}q,V*sr?J^nb!_p.\^0ʉ^}ܐN`W35ۃM*?e~!2Cn-7ۦQm71Yb_w3^ܿOgX ?+{X9aYsӫb 'x\7Ϝ;%ث/PǻM˵?k׊{s}W8\_k>/˞}]/p&2a/߷힭ϖȅ/p//F|8wZhn]vP ݢjPt١/?}ywPY嗅 \?{}͋6|F1v$FI| 9^\> g`6pC4Lx_:_N\>t_$0̌mVN1̽,-yl-ccqil*dsPg(U򞦧e K_Kj;ێ ¾bT[gN/e1A9J=rx$sc!ڟ/S9ho*=}>,BY%ZM2g{P|l{ J"_(2Zՙ3źI_yڋ47ْMy_}tW}ԋ/]A׎g#SF_0wYSGa P]*Ƚ ۶ i۴t2􃾎v#k(R6yq6;d|~!gx@km sv-avBE:~޶,016!CrLg\'E\l%17\ ײ~{?xA3w?\*zxEp鋹xv⳱=3f=GEC>!>UO+u\F~ |&!ޚZt#: }_T'^-[,=FN;[)ֵϸͶ9(,ïںBglZe*L>#5][z;rI>ܮ>^0} -; ֶ$f!kS2Ux/tK~K/"# /sE??(2daUennB5ckc}1gئW ,:+|3x%xPN&h׸㕊92Ƅ1G-b_2Y0i]#^c1kB\Ǘqba r.J;.+יCZڧ#)c( 5 {&ھ!O ɃN#|^50/%ܿefw)Q@ؑ}%}txv)~/ֽ2ݪ=lHuC>|_=k[x5<pA⽟ӗZlV4{(7V8V:ctVqas^ZB_c܉!{΄N:ڣe?O}I691Y)Rųcrw7\0-' _gz7k8GRs|lt d1˜l0r_u򱥎lrBXD\n9^gTRSKq?鬭ߣHGͶ1uW׹ .|9\܇Έ=e{Ό=|[k?41uƖ>,ttKZXgu0ùuZ|ɱ3ii~g"~uU=Wl萯c8B|!V(1&1Q_1h;{qFH I^Ky·ޏm-kHLhҌڱO%UvaM j|pw;4( P}txcH;HŃYU<ɜ )cd{B7nP0[ Cc}yBQ:^93@z7N ^]c؁ꣃ w~y K3 Y)u,Z<_6؍u1ǨٰR.\ ={.$/%>_|i/0U gm.*/%8DZXɞ%x{0Hj2i;c CbHs];ֱ0ɤ`lkm kXoq./>vNq@6/v['ctT$%){OJ~0D\lFiƶȟY9ٵbga(w[ ac Fs[O\|<}7l>~޶gƔ9<;aP,?z@[xu/-J ~Y,$?i3k+6G:teЅygQt LlDl7ldm?;ώM!9I>+1}>A"r YM"c|A`r ;PбJP5c66#Drzrier:+cr>5c{rOl t69F_ sYeRX7xk`-yayާ|Ao/h6kdWF1e uVǶird kg X_h 8׭:4,=TB_؂,ӿ|_r%? !$-1:{qҾ˿L >2&dž|8T.@:ЍO n t~H?Z>?VcO!qcZ}YpYtc}]!5ޯ3!/gZ~؇ `-ĴA;{j~ncDy~-BRpUѸi[ tLЎcw}olKأ=mп `%H>6G_FsvǗq::#}l>b̖v?~Dʵ{Wj8[h_Wg n4ZqWs=AKk];U 3_wUdׅyFc2=6=Rg[עC:3A/:پ7ϫgg@gd)?#ώ-)s3H.`be, {E3Lyj#Qؽ!D.2&gPY@ 9ZKR[uaml 1uY[e }Yc8EȽ6@gcb]LZu@ÿWF?zhjc/V׼u uAIBYiz._ Ï@u>rp}XFR\(=woJg0vRtnz;>&zKŧ%?OWM 1aIBID'FC[+14M>6$?<{ғO#Q6u0v(jxuYA)kv;zkWm_ʘ6wpÊp,^CgV8P9=/E:sv CUv <58v ޱ*ZO$ʼx`QF(<㽬^I> n,u.V?26Z@`[}o-.[~̻wKqP4ۓ$$͊s[@W7s].N@dtO0¾_߲gkr/1# }ѽ"z>_ޤ_|QrK'I]1_l]>[m?/ܫ}ooǗ}+䲉k5Ku6 mH?օrqt\ E$-Ѱ}{7,{y*ůk)6KjQ͓u"ز:s!3;~-o=t;%{뵇`$%1}/uge}/' |ʩghy3-kj0&r1l<n9Kkv1Q۹vסο}\V3:Pg@iy:ue]rRVD=6lg33 ~Qc[mM">8}9}(\3?ƺZ2V ޻/Aw_|} }a ]ꓵ~X??Ix{{#|{K%pJNK rgig/bp~ऀrLVʂU ^~^ŸQyjQZ˙A}l#}]/0ֆcYDrhCtk9r?[_E-c %}0[ bYcK N M-?{qAW).~p/6=X_, s_4pYWc{Ej 05F(ضL֯5ߟҏ2ucp(V d粕88:c 㛾27y,`<<{ ஛%?~qq>3XB¯767s$5^d{#TR Fٞtg1"h-A)oVܑ]pmG/y:eq<>y]m?_$Ց]SwgbܢcaXw~vPyU 3_;ϊe^I]m[ľLϳj"?csLgJ'̇Xg#M(W 7r{lc^%1;r% @<,u`L#kFfi󳚳;y GieL*T[GDξ{]\Y=rcXcsw}K\n3nv..SߣDNG3i.9 2|=flMf#Йg‡- LO5Y' ݏ#_+Nt@g G}ݮ=c807/7qLdqN.S_ҵ&῍DxMjuWj(J:zoI;m>=vyI/,J9DN[79kѣ(\b v!,bJ[Yal_F m2:sY}^;H&=K>\Y%p}@ڑÃL h"[ƾv:wr8/a]~-Ȗ/~>[oA~k!y똈uccYKfkZp~nQa /5fcnĸ9t\*D#}Nx|>6@e[@_6=5oGP庲2s[ :?fdi;EgEj|gFGۓߑܤM{g!̭MOev5&^ sOĚ `z]@]C?_$b+7}P6UaHY"mӅ}@-sh|c] Yynץ_9JDGhNt&$L΅&}aTr}pF}?>tuu3ƴƌ&U< sRp>{u܈L,a<;&wJ$9ȹsY|_4S(?[y@.>vsaJ"y-c%u@.~Ϝ Zg{A6/G˚X=15_#i?sc~pֵ_g@akˑMQƷYdG: ~1#+Bvѧ+g2ޟbwBgK}ljeLKja'Gl"d1@Eu֞UuSmh98|}0.v$*I,'̄/`%Ir^@`!O 0`cn0z,=Ƴ$Ϩ 7~rd-vr'}?e`k$ycԫ:MƑx iPy{E}$: p5͠/?c/?)xSLN{x}iu|V[$Ӌ_cB^>7gjf@<_/z7O6 hvF9AEȹp07Kekܞ!N3^?<}!-|ӷkg秗0|@>vߘt>-V9_2\6GMvO0z'Gbp֘~ sߺG峲=/w/}^_^yZv{̣ӷ_}v5-ON?@'y兊zӕjުUBi<|^W9(mlj$\.VRܯÉ8"BEg uzO;|zGϧ'Vyע}ᝏ\1<~]Oٸ O>~ځG}WT.6/vQuVQziE^ ?>WGߐ~k׏O?!m_6್ ?mmk3A[; cr/}+H|]=GT{{r1udA߳Ao:zl 9qeL kZXZ8,/\XFS/# 9ro;;(W>qC#xEdLޘ)Q}(^< #S9"7\o\8qq=-j}[\6gzs؝?;q.k}s2ڵĝ[o\y{e.`1<WbQr_1{{ujtشR˵XT)/qdU?Up@-}>5(dž>>^^n y sA x19!h)`xhZuˀ;=KFj*ZN)VKQ{!\2cͨ $6YQtuuh;|zt&$,6$eP,&T,j)&Z] pZg/suYJaw} =hx_?>Eї6 ~Hn,zoCÓ o}o*ޯQsl/ g^9+ +yɃц7p KM)9ɼ9liߐ-XUn\8Y lH:0 =2rH }ϯH_M1?\HI/˭`}kHY|dp2!3j"Sns }gz!`X\cGil[ ԙ1?e)8|җ=үζ"!@`,3Dqb[i3b.όAgN/e1A1n{H ?UC?_3mtҾqTF~)M#:3ֶ=۹n<}Ž[˺gYL1vp ; }\G![o9noC P}n?&/=_27Zd\n>WlB>BYokѺbE+|tW _T}[y{Ŕ?*(q}M\ڤ,:󨶒YC>%$<_ߘW>2~ܔb 8<߿r^s~dpм:\ ~ĕ>e>RʿqYjrXm$Q=$ͨ% Vȥ+V ?CT< ,.>ae?"@^? k׆2 qZk:0ptmlw }]k}J>ZÚ`C5Ioٽ3M012UxL98K<,|+. ːW"YcX?06" Ռxŀcvz_~0@W謬8@|L(oq~<{.A%^{kJc˜#T/z{,lc w5˸} A۰9PUYuL!m-m[ɂ1wtTjޱ5GIG yJOvWj-mB>kL/,]X26cR|VhI6\ZO #{=|^׽1t<[:r?掬Aۗ9#{ֶa0y"ȡ;c(fs±U;ogr.l9rmQ>uG: {@۶w~}\c˚8IƷv|/ ?syU.sh 㙊:T=e_M 97fO: LNغ+rg./Z c_WaWc4}"{~&}xk4 , x~YM u&*< ;9Թ7c_sA`dic^BM/M63yVdU1/,7_| _Eŧau_VFy}ψޟg׹Xr {Ep;O%Uyv(;6Wvۘ h8Ic)cpЎ},)(V9u4 lKv`@1ȸA*ۊ <ɜ )cgaW8yB_\s/O>J+tC念>ԉbAkd꣓D>/?/Ed9x՘p Kk_ۼb7^dd":K!sb6e}/߾ o60Jf?,W.\z;;a[ %1.*x{0xI綎c !acwXYڟ?6r6nokgyz#maiTnAˊ tanu2FHe~lΕʾTs"c2N7M@&kV}=F^9e?8 w~v`N^k XU~5^a|ihkpFmq&ZGtk: u6:!T>%}I܀vo9Tlh?d{4ʾ+^ȘQ1>',oKMQnu@lʇܱ/ u [ܫi&lY Ztk ~c8 Ά~֨&nwq-\Hgd Oۈtmrly``L3MO?*jO*,MƱdtc'wV6+U6"1uI#%ع61epvgtoҺؒ/ý|l>)̫N1>,X0?^>Hq/?X /cRF_ژ&es;h37.n@i 96wGq\utݡn(w38(1KP@IG'0_ۿֵx+f_0->Qt5L7֗k /F5ޯ3!x%g4k> XL? n5㲽>FY;1%[:F}QG)Z^Wץ<*q-Ҷof0~ ʗlG{,۲iAy̵)lX:n%g{1=ILbcƱs(cGCA8tV ?抾 ]l[|?DP>wY@f9tW3M͗\hCm砍αc>?o8r#^cJ=Q>6]p[3sσ -\a>ns`[?΁\{.j[ph? =`3xq7GtslأŀKIGۛܘjXi;1ɤb 9t8?^flvK# n/,m*Pg3RW,?l mcaIP|]sk)} {m 157]eKgC?S>䕛)kv7y,b+lL-PT-csLvM?q}# J}06ʩ913>lNF gì+&#:5 k9({ Aߧd?1oܸb%/0mD.2fMcfܢs.:6[Q]sK#b'F6=l>e[*k,Pg3|kWlmt::_aMcVAj~ knuM^uuY+dtmrlzMuփ1/\ZhWr=6L}XƺkȧL %o9:Jr(~|n;mHPaFҺϗNƝ`Ab$>_roբDç϶1`Բ|^ ^RSƐc/NrU6r Pz@<U jա,[kUqřA}l#}] J cm8^OT (潿8Dg#_"0a3,ɱ%ā% 8us ؜N!jyȧ>" V{̦b9~e|e\-B/Gc2_[/|jH?$iێ9 N,EBhRU\dV^}y {A7~|165XB&>W$<{ ஛%?~qq>3XBRϯ767s$kdOgto{7BQ(q홏]|#؛hPǶ_Ao>~c 9bh}kdhC u2}+?fLru V9dcбm=&yqPgNlss?>lo>Sr|?tA@pnl)Ί~Oz9E:H?暐\\{lՙ277`њOx :`,J#_"sMBT<䍘GZC9 7e }<Һpk%u_fuF;hhOzo} TL}a}ޘM7{wh b,`oA| 4@Z>$X Ѯ? `Ot-4VF 0eVGeq_w`Xڹά{T>Ί$Axm%3>\  ҎXpIb"q`y U~|lc_{J;eZ.dH?qj-7j uLY:ց,ƥc5-8|R^0_oyA ^q_ `zqE,eϭRt\*#}Nx|>6|.V2M&V0{l,b+Ɔqx?SaLkի)>5MTP>k JEj>&=ـҷQʢE͸]Mplboh^|*_\}:)>u9<^NJT٧'>K*ǔgO 6@ˇJ <[%:$9})_ g~|f@9q1/^ś]xv9ɕ@YGg7@Ɯ$ID,*%aʹoez7n$o,YOn _C,' qw\~w,ss'Jo9I\)t?4<ؚ>I8Ir&p2?H/`Q-zw۾gUb*AZiy[*_=n|[>\>$Ր1'Y\,~) $I$I$I$I$I$I$I$9'YO$I$I$I$I$I$I$I+'I$I$I$I$I$I$I$Irdq?I$I$I$I$I$I$I$I,'I$I$I$I$I$I$I$ɕ$I$I$I$I$I$I$I$r.PNC'i'W移ׯN/>ڒyͫׯOw_o8Ֆ$ύ{"Y;!F;=ڮ<ϗbK=|}zSM:l>x>p!|~)V^Gճswf/9/K ko>\twspɱw`Xυ`y>ύ\rˀLrrmG{LceOgy]gHg~gwp O%//KRk; $\BWp^XVǥNUj{"BE?4}9ӓ?O~M{~<=u}>#Fޗ[yINO>~ځG}WT66/jgǿ7:G#^dl_7ΤldeϱyÂzPG'Ja~>'~|`wNM5u]BlL.ڞ#-Ѹk]g|]PπGTw{/f"}Yq Ƶ@۟9a3+V%Yoּ͡aΔDh~XBmӛ/(;>9p_ pIBK hg0;hgAoTq}?}cs/tPo9GwW3^L6t\.{߼rQ{}Z׮|lA?iÛUbY|utCⶼ`@RL',v"698Uqj]s"{AZH楇x`K(>)Hb,EI-V,mI% 3w5ZQ;@ZЇ/~|zK=%IҴ`\kw*ܿk-F\]l8+kA}(til(R{Grcz\p_~{S~ /:Okg]χCY3Bľ)&z F2 %m }2EU7}O`N`W35ۃM*?e~!2Cn-7ۦQm7gvɱw`g0N=o򊓇80\?F9k!{Ru9CBVy&.^vxQ~:޽o_|wi]V ՗:,XY|>^.뼈ޓ9.O~.s{_>[^nDȅ/=b |b1t},ﾤ:3?{' "x05еEbZšeцߑEo NjX y_ :XvYc{b g`6pC4Lx_:_N\>t_$0 Qt5Q1̽_C⾞G9[q{ilɠ:3G/ 8-}c/,;Zצb[e~r?E!sEy^xߣ,&"GVG}dcx0^8D3E33oVsn*Ї{Mpb.[߲rcCb]ۚlKIΣE(m="B{ |8+Y+uBd^<EaK.? uV?relk_5 Ty(tPm%sU{yq;vֹ'|a 2a[R 닁yM*皴[Q%?LX_|J*qf&f\P68 8{}4>Br=`bV*p.R\Pǰ{[a3[ORgTlcΰ@*sϮK~q?l^W>ܮ>^0} -; ֶ$f!kS2Ux/tK~K/"# /3~^ڽU~ ",8069>@_-;ZsYL/59~~6. E'zzVϯQ{.A:_W*92~W׳#du]{ŏe8Ǭ q_=_چ/ȁ(ԮR\gbikiFhsJ|#s+kB8[^:{7'j;y`D0pbͧ},ߥDGcGk^8~ۥ rgZ+Xw~ "u&׉_ Y}m}cSL)Ή~N_i [\\r򹟏؟.!__G,iX-u4&c׵΅+|^DZ:w.{Jԙ{^J~vIf|ՙ#Js[s8rYĺϛQN>Q]_H- ;_Cj{3#u{\6:7ڣe/kqplrߙok&Μ҇`_,A~8NKVߜo39~&|]zw!͏bsL/G\7C-?;ÖulБG/ E^?:1K>mg/cI&HZ2_% I WxrJjؘB;鲰ʎ2XIdt5>8˻y::2X0 t>A9x՘ Z[*)EWeumx]]B-0SFzKQ+&={Yߋ~_/%>YRQ/ 0l+SPR.%j9/ks Ⱥs~YDg$/+~xoOIYM&xg?u }8cH c`:6&?-bYxm `mrK-;څ%Gک0_&ndLToy)98{7O"_su.64}c[owĬv_ Z|d0Kױc~euߊr]i> uMub?On~|o[W3[cvlNA p& ?ٹǐ83vfb][t6iQ.YT6T%g ɥM({{B&jY{k;gcko;:0+|&{Skl~Ɛy >@K GRk&r1y fZyV~r ñ]u"A9=TNj9xCX9Q6:[u׹2)CAt5B0FS [^5YV+#_h:+kcv9i g~53qQA/DŽ|`VK*/lA_/v\&2cI%MB.6$ =qd)e]wƧƆ7cNPk:PP(ʢuM{%x+Ǭ,klYpd0yqN)n;Y?snop:#ZzXN J%St 01Hq^1xg4{[O->r2"8LqxpX$JtL}8V|sce{TI0ⰞY%+Q/XvmE;{k,Be:Wߗ4gaS1[ m砍αcF5 %>`%H>6G_FsvǗq::#}l>b̖v?~Dʵ{Wj,P[*AQiEeг :v\2dy={k5TV:a3ѺJs-hQR8|vؾm[bɘ{{^c%g֏־8èC6IJL XXKÞen>s 7sSfHT>vo1TyCΠ-ǒTz'flXCt̾jE֮}ƫklB_Cth:ro10,XӱrC$U=f0#F~=5o]CE}8PglyZ#uh'\$WocG]ҙ]ƁTx@ζڒ}iIO$5yI$I8Y ㇧oD]CؐIO:H|ڡT(Fa8h $xkC'Hn˘6bk6_#w A2=8XQ6 x/+_a;FnX dl;VПI`,/3F_0x} ̱̽~1 3,! ^cЮ$a$Ib1UM=v(0oo嶰ٚ\'kKL5C=ft|^k=9/-yX9k$mbAv-9^&? qL _]}qȇ}ک,3r<ϧzf9>ߢ-m"6cpd!~}mk}pWeU: u6/o- ƜgYܱXZZk%'e+Idcqx<#?S5e/,蓌ӷΊ5c]a{qqک!{lՙr}%̧}@p>YDL4e{#@{K%p0_NS$,΍(_;5I ȃT`h`mtZlW(-q`,#u.sJV 6II{:+c륲 U{r^o(%5a b+ZAL? rl)g?c>ʱ9eE>Yc6/׹ 2{ҧe>{oSA?3.;HXXq@*y.[#c+zӾ<?J,!ca|{sþ|I_K?a=㪍/^-/-z_`ss_a^B#+o7BQ(q홯~uLm$GrKb [3/~=wdcFK~i/&qjlܸ.Ϧ~^!l[O1AulWic]Y?"}`-P8Qqg ^L-gX8}54v ץt<y#|uM߀J-AgϏwЅ}~^mP|L; S+N!)ZN*{Q:rn~޻|~4e_Í`\'_ܓ1hCz=sȑ\^^!r eL*T[ m}|H{ȱǰlK"~ۻx.7ҙ\]]~c"(o;6LGKNȷg`[S0ta~?SutAItBNJ}.:(rD~7CQ+md<>*srت3u͋'/p\ߤ+SgYӇƶiג7HuX&t3ͧ./<.V)9mqop< [ +T{Ђw=uWk1Cz(Khy[D9lS -u*=36~PgZ6lU kA؇k k>H;҂~x390ټMdckUC@NCU%VBG_ZH:&{/XlM N.K"~+ VocHئ^}?xjDxX|!:.H"Fmƾg{_'r~t.|cs dcX-}/Kٚ#(r]m @CRP-{Oe8޻CH)>+V+h,=3?ڞl&mrk?` `nox(6jH}" Rb Bg<&[dMֿOۀG߿i;h=]< gT.f fs+^ }ylYb#u?SrzZ!5Mkw=mDK5FA?LX9=V<13>HgX9f}dE;X޷ beLSN |Mm@׽ ru [-H_?MU>=(Wڳjbʸ>R6o{_P"I.=&3I{$8PNo spvKR$v)ws{3 [ G7ܕ{U]`ח`pow~KJn?.Y?/Top9I-|'x|\YOn _C,'y}-,cT/L7`ϛz{ګ~ @/^'yFuFd-vr'}`k$yc,N$n06B%CZ@3IOL^s3(T?-/ykPۣa/s19vaO< Z(ṯG$\$I$I$I$I$I$I$I$9~$I$I$I$I$I$I$I\9YO$I$I$I$I$I$I$I+'I$I$I$I$I$I$I$Irdq?I$I$I$I$I$I$I$I,'I$I$I$I$I$I$I$ɕs~v*_=9ָM>5??ݽ~uzՖ$o^^~}n/NO$I߿x|n 1v4ll^_~No]o|uzwGD٦3n>QIٸuv&CcW0[<:ő'G'Ja~>'~|1byz_q{&hkgralWo)/SjoKD~6dX)c݂,Vn}~Y:®Cڽ' ˈk jayl^V젴r_y=i;Q3{cNprD_xc0p^8N _/n,pqqŝ@x]~n]s.ۜawxǹ=S-ɬkJwo-?s9}΂Ĕ>P^E}987F΢{.שa.g`J/bscQnġIcTٖ|.W({uz%`50etdMN@UZ:5J/4)&üPl8:\(b50Y>_0P:C}w=`*1\uqu}`AJq {2(1iBk-E]-³[c,?/~qzW@كvP}`P"/=4<׿8f|^){)2D$dOobqf<m/u`YiIoRuOƍ b%c-;E9o]aν;w7K{׮ 5X|u: |+/\?Fc| ;2Q8{[q?z!8E1\ ~.s{_>[ Ѐ:Y7,xk \>}I_fU)d-zs:t`a$( B8}y*w;-1|xaGkY>kDۮ3klP\'tL> M$/&{HҀICK$$IfVG]$ڬMEYH_v2s8qߐ!F8*Fj;[Ȃ0gX%Fk)g/kY+6zVrY7rc3<IpDgx&T޷g;u61ǃ/^}>xY,)2Ʈ4xcOSG ea< m|!}(vה5ge+l=Ed d u d `-ǭ~MsO:d7{KTq@^1YG(k-z-ZW⃛b%{/Ju9oXSsRGe3/PIK7@gV2:sqڧ䚄JڇBo?TQh6{NP}n.`Ml9٫Bv{??ӇX_N۫{.Xo WPtcU#ҟܧS*W73TM͂$ `/ۃp39X#1y85~Qa"eq knϺ>(6T9麡҃FL؋=ԟmlXeu/g-DcZSiާBm?IzmiB"{e2FXbaPu[tYxKW {/{a|bEdc#aqk#۲5u rM=dq_>Sks=-A_W*92~W׳#dds]{ŏe8Ǭl_=_چ/ȁ(ԮR\gbikiFhsJ|#s+<5qS?umOvWj-mB>kL/,]X26cR|VhI6\ZO #{=|^׽1t<[:r?掬Aۗ9#{ֶa0y"ȡ;c(fs±U;ogr.l9rmQ>uG: {@۶w~}\c˚8IƷv|/ ?syU.sh 㙊:T=e_M 97fO: LNغ+rg./Z c_WaWc4}"{~&}xk4 , x~YM u&*< ;9Թ7c_sA`dic^BM/M63yVdU1/,7_| _Eŧau_VFy}ψޟg׹Xr {Ep;O%Uyv(;6Wvۘ h8Ic)cpЎ},)(V9u4 lKv`@1ȸ^1Zvt^U[Em Unѓ _<us:Y:kn8fkY٫=悳A*cɌG'%1 }3_~^sD r6!17$@-`їaע+ynk)#0] ѥួp}ۗM,z)_(ŗmԳK~I{ΦK#c?kf$"beVEӍ瀋'I tY~nHq0Ɛ=vul>|`lkm kXoq./>vNq@6/v['c>k{=-O*Qo $5\b3J7E v_ehoX=]gM ^{rY| ^~(^@91M/wfn\܀rlhP)C7PfqPFcZPCA(reQܺF&<ĕc_1Uܗ22e5;rBLgi]:ù\dh]/R> XL? n5㲽>FY;1%[:F}QG)Z^WK,ƭ@ pieљ]}olKJW,h۲EsY?sq]o\]>O~ (Rb#>oyyeSvDq=6ʀa{#=U񏮹eV/Cw0a;ik&@!/]YcLp9cj:f[dD`9hs옏m+A{|W8ƘROT6rM}q==¾e~jK+Wmrs p˸VE>gڏB*X? &{pg6l=鎅}&;ơk[)Ư_f]8h5 ]h>Er_ƴaX|/iՙF}:Ʈ:X#Їk"B}~`4w[0qޢ⽷$vk7jK3ˊW3fƎEwX'Ne,/3F_0x} ̱̽~1`(uJqLoOF^Xa?u }8#s].^ 9=Vp\n ~~ˎ.u޿tޏ\3cFf߳y Ңxyw#u$ik;sf$Qc>֚?Nj}Zƾ+욒Y}/ R5G'LPsSE9`lSs.cf3 }0؈~FYWL>/Gtkc=rPz3O~&qcj߸qIJ_*a;\d.K6ȹEz]tlX>F:=OlH{|ʶTXf֠NP t696u:Ǭ2 .Cnl)!mhd+VL m5c,:_I'v#{lԙru%>n$OJstzM*_Qމv 5 ~Ìu1/k~藍;kHr}~?ߞGqE%O?mcXũe{#6̥6.ܧ!^ \mhơx"(@q &* fYpcKKplȱ9eC@O}D/ŭMr, 2{ҧe?{oŅXozqpn%s(VJ>QT {<űi_C^Ѝdl} %0σeU/ A,^&}s/~\Dn\F6ǾP({q [ Qq#We{ߞ(7iڏ ul{3g9ЙS-gܿM8 R.S`T)7[o?C;Kٮciњg ~A]3} W00{@=:gxcp93u9qn}_Y91oݣ*-XY)n,{C"q6gʿ@6z&,Bg3Fڃ\l/&!.c}G:{gG&w$V`<b-׋c =}$^;>_:7n,3ÖV73%'3~Hgd d}fdGWd3#! =h yu;V o^,,sX ~c AT924!2$D?Iވy:<@ {S 7Cl/ lPXGp~o^g?Vܧ BfOIYڪtcѱ{ό6p,Rơė@IA `+aE~Z𮇰vYddXa,iRn䔲CN7;\ԩXhAi|BIf}>@.&?iA&Dd"[ƾ5wr* Zp-_ =}ߨ,FB1gX~AbkZpv^ѷ[a^WczLQNe  $!#Hc߳:_i9%OR~_+V磘$vc/ ٨,_.}NM1Էum=Kzc0_s~ &㵹D6F`hnov;P lNcOѾ}0kGrDg|kڤleY_Q<2v~"V rӈgE1MNJg-nZ6עgWDCL=ak {б6 rG:`0O[3`xc5?*cнyPt97y`gdvoXn4kd HgX976cuɘB c\^ULѶi"Yc5WU.:VU7MR=/"lmcx NG:]S ,ZIUN x1:VPoT>=^j\LP ?=?}zm`/\>T`ْ<%($I-KB}V80 7k7r̉8q}J(MI76|œ,:7<+2$I&/dq?bWL OKH5??ݽ~uzՖ$o^^~}n/NO$I߿x|n 1vu~þ7|yyXq^{{p}jW^~`rS>wmo[uT|⯟\;{'{5ƳKE/u!V˾%>5|OEh﷛Α+doR５lY|移oKtN\_,ۗ5y&eIjm約MpB҂2E:.uPCH*YO~M,;x;Hvz3r|z;Wv{.ƖmoM&5swzڟWw><Ǯ K  qCOOT~+yϣoHG{Yoǧ/zƃ;95׼gv&v^V: 25h{J?BZ V^h1ӌkL 1&GTw{/f"}Yq Ƶ@۟V~5 ir+V%Yoּ͡aΔDh~XBmӛ/渾bXo#^L;z&vIٻON?<@d_y|7Ϝ;٫/ʎw&}:.zٽo^=v>?k׊{s}Wu􀠟y*1,}:}mIh! g fߓAaI*ܧ*ҟ*/ϴSe 긋C!4` px(JR 0L pKNUcB{Gs{'fk ƃiGPǒExXƽZ r" ~^/zvAL0 &%҃-fOb bRDŢ4-bZݵ Zxv b8W'"/N({.~|zţ/lޑX'\,T_uË8'#m/Z= eݵ#{ v8NA,7bYK<m|K/+WK+'L{`?]p󾂜l WaD, ro6=/l~-@lC[\!ȹ(4{/Kgbq cb  Ys|{z9E9;xIAuZqwζ /T__`gRܿJyM(d~.s{_>[5 'ra=rE bY>}Iuf`O Fh,P 6oяNP1|@~y8 ɶ g`6pC4Lx_:_N\>t_$0 Qt5Q1̽_C⾞G)[t;ZOcؒAuf̏r_ @plyOS)[^UYvM#n+.؏eM!BQFU_ch=!ܝ^}u=b*r1n{H ?UC?_3s6:%T>{|X8>Kڵje, |A\3DPd/:ֵ3igW5or}HS<7=oJ{}͋]S/v];iDLv~eO{+dCewث .߃|n6x;mә^:}9={k?C|/gGtMWep￶N̽e `[X{v|Skq[m=s:)2yʹuG6Wh Wעu:~[ceNh:(l)ץGq!=gRnm f[D>pjO61n:/qsJ|N| |-\fKRKrJt_|J*qf_Vr1>8C mcgSLI?S<5uq9g]=EN46}wz E~gaG"z;.E'vMۊr|Kͅpx)0>X~Xv>ӇMTs㽤/usp.m,|+. yi Vcow/0YL6s`;* [ igؖk&{ڨ?.;[|x8^iVq~%xPN&h׸㕊92Ƅ1G-b_2Y0i]#^c1kB\Ǘqba r.J;.+יCZڧ#)c(=خɽXk9Z#_ ɃN#|^50/%ܿefw)Q@ؑ}%}txv)~/ֽ2ݪ=lHuC>|_=k[x5`g}/9דv<#9ŦB+H\ѩI!'5}i@O߅~$:BƱ#yy8Q2ـdzubXOёsb-lb3I_suϼx%i9y@\uF)#wfZ3g6a#Xu:kyP'Ӓ7LI;_ǝwH\?񋭫оbOΰ@|t B׏)D6yAًARIk< DK"5iJOS s_tYXYeGsC\wXto^ZCTqYY^)/xñnm^x_QXܯ_"l>7kiJQֹ/m,}XcRã>0>7',x`@&d5:J ky]ۼb7^EW3`pK!b2ܳh~RE! "x,_R)ȺQ_ut~u8.]<",_$e51U`!1$9z.|dT0edᵉ6u ,}[8k֗Ci Okr8| ZVd t1Qqۖ u{JkfolC]/1٘,rױ~:V{o4ڿ9[t\5dzwÞۼ.Sm*yfkLÎ#9AknN}IO[g~. s+t =q^@-@RB/MI$+@g/ <6ձck1ރpHgcB{}or{}B;LAR5w<{3Hj-"D.2&,sP 5ʯT!_c8kc<N$('֑I\V-o2/S7`16w 4@gck:ڙ^&e(uWOВƑ}ackѺF: >@J}e SPgeM|l[.Q~8L@?|x8ޚ/*Й}cIC%-:.W_A3uҗĎ&lw7$ާE96ǡrRun|jlw3fܸIZe[x\.ǯe Ѻu=Qq;^e^! )壵}x):ZAL? lA3Ṋ9F`pJ<-Z^b!kGEpU4n^E^.56',{c+_V`ߖm -ˢzy ^~1uZc_4q?b<\&1eWcƱs(cͧN*G\}m+җء;ϰ\g. ,1W܇JoHD`9hs옯uMnv|5|/үMQܪes;{~@|mM*{:?4sSikcnfym8Wq(s3H.`be, {E3Lyj#Qؽ!D.2&gPY@ 9ZKR[uaml 1uY[e }Yc8EȽ6@gcb]LZu@ÿWF?zhjc/V׼u uAIBYiz._ Ï@u>rp}XFR\(=woJg0vRtnz;>&zKŧ%Y?]M U& %I LY㇧owKó'=>"eM\wk7;*QXfc2 ;䵾`AñɽkRe^ښz-l]Z{c}=Vby^N@ԟS,7'g2[aM"cRyLݢs,б<߯3bsC1Jgt&=Ә t6;6K[x䤬z9#{l?Ϛggvƶ%ښ\CE}qsv!YQ>6f+;7u1Yu>;N[;?d`:w_.rﾄ'k1~X? +U=^>^S\ϚK7$_;5I 䐴^+lQmдMXgq͢ 8D&O-x]^6Vu_Xл->uȱ4_ߒ0RH L9t1t؜N"ns ,_[1\/ܽ_o8B/Gcm9}* G[o]st4bJiSϖugBǥ٫l%,Nn'cclk(M|/% &A,^&}s/~\Dn\F6ǾP({ .ُ剃i˾Fٞtg1"h-A)oVܑ]pmG/y:eq<>y]m?_$Ց]Sw=;|oygiS7BF,kA`D-Ha؀QKfD_@V  m< - 4UsaZ5fwys""'~ͼuĉf>%7kZ9u `[Nm=;1ʴ}֋6>U(}hsPlYotEgm:ީK%>1mלno=:~ ضWWb/@p__N|_P}#{}ڪK4.!)mǰ}@6~~u۶፟[ٖÜm}r=~;tbkSF@msTP[y~̎ ;d\x7 Uİ7~n50qVEŸ'r>b<:`G+:>S8 eط=< mvc4'O/JY{V{98d6mj8.C5 TʺoEj UZ;:6^7?\DMxImD /NZcA5|_gWw#Wf=>XqIo6m7m:-21DHgbV<ԂY Dl-6\ޖu;r(Lh?%*'btTԵڗ.է VO¼aOuv ~^j+͓raֳ)}gޫ_n*S>?}Lu(P-]Cm1_b~ۢl偏P񘑶ej5:aŸcm:b}qf_m}7^K$FZ^3S=EexOby S{Qfwx%<1}~Adۋ0}7>k+n[#٢'Ec>L3rP<ߗg LYΓ0b A|ytȶ[u3)U }@,ԭweٹ]AlZ<7ظzPYsr,yEKF֜:z4gDي)By^ygql~!{{K컔m}L"̏r;λ/ğge9Oa%hCwbXZ>+C!oMԯ}FOz>>zV1YcgPƣ3E{>kƙXKsh,ە;Ÿ2uӴb)-bF{2c53=u(+(&B.tڄzX[J6t=K0F)A4pxBc 818/$n'(n cpFI=Q֔z>6Y J>2wOy߽gn`<6EP?%^ѵhpFu:nyf1$. }0 ` -#܌O b3q^€@, />;\g1/o8D:}=ʅ>]p,xC>?3߭?۹͉彉#~锱uk_=OpF#Zqr9ɥA)35C* ΐMR2?F,?o6suˁ<'g'&oސ8Y kŚ?=^/4bAM34}l_[?J>r:={8 _.G'3>p@q8s |vo?:>_M/OOxi7?^x1]?_:<dnӓo^z0ǺG]={1=|M+`8i?k?Мۿ7(֚GǛ3T ֩Ns{ӧ?|4=}K;1ݛu|sT콹3 @ge߾\p/s[v^㏹w fz~л߃~}t_?=?^mket$6w=N sѶ~^y$qteu=7:jqqn'J5iPRrʇn%Y#Dž]Éq4"Bo/yϿMw_;އ&;/.×ӫjWg~zniׯFc*إzߟA/RvXwcx#3,\C6Bߝ|Ȼ[4Ѥ+`}ePgߛ֋O'_T .Jܯ']đ UnX=n˾ob렱陯 }qc=a=w\i;闶8sw 5OcM9y/ _x6,9{bq߿.&, 0g&7M a] 2;Ѷ?^82;">۱B &^ƱEwCQyJbn="D:E[B b B>_5M~a7e'wsNwu/s⁥G7`!l8.c3V8gƧ{SmW[ xH듽];*"fhvlC]Ď9EOsD=}MWm6s@8&n,1#ˤ%V?ڰa8tIQ+p}"!o#v/]v4֝] [b~g>##ۈ:_x}3 xLo=ȈEia8.{:?+|_w<ɼƅp}H>,,ʣW`+WVV6'VWW 7nouaoi˱-e]FrK9~TFtc/M/f⟮sgz6m$?A_\.7޵a8qM}TQ;$z ֻ̽JL$珯sT CN(VcYQ_su7׫~U &S~%!Jڧ}/3{_$˽$]eLi[b/LaX68^#-oQ^_8=Rezq߬QĦƪ۝mzm7&cIsۡ<p?}q(!sxz=t̄- /}u>pf<]Cun_WֳxSݶ_YYZzЃO=`.sq]/Uk]c J.^ ?my%T/C6{,#ЧpLeM۵51D}>C->Ki_OcՍaGc[߶gnBSo/S;K뀽~nV{y-yկZ\Ve}"XU6ss|ǧգl{x3ŵxE9ǂqϟP?n?Oﱸ?Ly(IFydqrbA ӏ]\hP%;4Qzn$iPޜLŸ`H|( 9wӯ/㼰ЪӾ4_Y^XDOף|NܷK[юZ[eNR1Fχe]oBi>,GrC3kSĶyfL[mQ /`ož-Z53T/x~{GP dž u3MqC^7^[5K&.c_*VƂ[󵰮:{—Qj尾 IdןnXs\slf-]b<l.1u/2Y6~n"(0%;!]3V3ӆk;>hǂu1vn=<=Zs} uD#jȱ+q~/ՙ=m۱߲ncQ"cַY]fR纩 W~x$UfzNyVB\Wj0u-ƹzr=>3v|-`ƒ6[Ӛ6s >`Z y@>H<3-o;>3m볊`gng(®T{vg6Ekn{vLwun q\[=Ke}[Ooe[ne9tN/vQYn#$׽zVzW'y)oߪn9>m{X7hCg8tm[ݖ/]:Kwꋉ nc_'6E^cGsuqڛQu7Zr/CRrib&䁣|!֒ H\&ۢ"6xk\CyvHj\Ӎ!ӶaQu*_{+\Ko_ps.DvzPiKٷyX]5cYD<eźAz8mkC\W<jm?hv:[k]3U}£여m}i=IŧkƺA{įdM_( ik/fտ]Ăn},0Bk~|BAɱ2R3-Q[Fk| 86ӞbN.kGg-Dݢc涤(Ȼ=ܟCȬ-}d}J{?/Wn":VhU,lJ_ld}Jܶ%3Igᙜ0WNRՆ,;܉s p=.ڻ^!P4E[` =簔Z@dGdSV!<)KԫkK^.ر=.|&Azh¶ay r3-fe-ndj4oD_ih_[mk^0/^+y-$'q0ߊnEguݫ[ku ]¶o[)[1gq?]O_|A@^L-VB,u;i wZwź]kQ/5rMezu jOGaulH')7.XbrN{ٱ}" y1Х wl&9/o#-ci pm+^+HmО޶?oXez{+]ؽW&DvzP/wBPo2>u;>֖ݵT{|HsSSuM+ռfJ'%5K9p_r=vf}ӵcݡ>Xcl Xvw}6⯢D̫_?6nz>į{}5֗0l~iUox -n1TW&Is֯ZtʡcY6#cHwQN]X -|bcAW9߹7#>ByC{W?_ m[Q5{iOlh{kzX>t.+܋1f܎x6}??>=Κ4Ed EAԛ릉r`qhʉ#'WQ.`^;PcsݎQދB-.px㡵i3 e=_Hu/h17"2FFCxpqܒsoM؅Ю__ #b;3Ubx {+jDŽRt8?@P_vX.^f^00} }^\_\ h2B5;m4׊FAQW7Z\}Úˉk?c]cXxPqP{0Z{ߵ}b+^+sRԵړ͖G$Qq dW69|Q *ΟoǦX蔕>k11@ѶAXdF#~,ً5S=)a[8;9(~ULE;G=r j 3<YC.zk\_"o!ufʺ53Avy/Y7V/h5ỵ+VڶbSفc#c nlx_Xglﺶ޸{>S1]+bC)ǽOկc2 Cu7lmlZQa?8uAr#}[Go~->W5a?buCضa.c%Ţ}ü2QnbVƿZlݖAmgnVh`ք~7Q'}eZUgp/۬іATmP79CMHic]="~K_ٷV/X9ZeV$tD6sUܗ Zʹ.5o[_kcCXKD]ߝ}Ћtky1/DߢK?a7uc\ߎ@^_Ok;/y^g$]˟hƫO1GziOj;Fcn'3n3o[Aw٦skƣ?uƤ;bTm PmGZDk[7_a;D3N難|}686ܴkc>(Ϸx otvFfIߝLܗb ]/'fH@Kֽ+R{etN`Dvt'E|^(%zѬ-Dۈ8,ә場S>VXwuzM_+n`8՝= PY.[ӌ}4Vحmv6ofFZ? $|x"G3iQjq3򶜮ۙ׵Z1΄6l8oQ«#b(g~"+F!ymLK]}AR}on~쳪fOITS34Ol^S}`OR@lļk3/7tz})]s>A[uy:(.϶/nmQVg]93Ҷ UmZ3l|~, SBo=NDsG޿uĎ]AgItg.ۤ-e%=WqV9ǚ?|]5z r]zo ߄]t,qTQbe+[ԗ2Szԃ˶op F3p/$(>ua3@=Jfʱ`(=_u ҫu+(^PbCz)y>ŎA{,68Tq[ }p j_p@(s %/vBp+؅n8+n#G?Up#BQ{n $3$<.6"^$~Klf}x6>>p@q8s gO?~8}Mr6~Gg/ Ӌ/G~Kǃ'oL4 rz ^gWNtmߴFsXg/iG"3-`P1Z;byˮ znk s~ӳ{ӧ̟Oo\8i[3=?vn0-I k ?|4=}I|¾:rg=:CRΆ>0}(JkUHg|5{g>;bXty8y_njN77>7غ]I隴Ou?m ]iCl>XAt2"Bi|68|ʼsYfG_ND/LyU~u.僚?ޙ{҉`Yv!/R >OO>ޔzzu ϛcx]Do [1;P7rp塡*)EnRߕ];]{Yk[ݦ4.x6 mtُxczQ틾m [f G&[vxVw#eq6q1ިغGb+%FN [Ul}b~ 18iצVn]oPŋ6^7}{/}?a{NIL>x rLmϮyr2-B_Q׳W;Q.oeL>8<;9yQħw\6# ggxpƚK,܀_cڸ͛p0 ݰ ?D}4}+M*o&?6g \c}k|z{ X8>$bOwEv>d6#z]6. W=-)5S C䗭m~uƫE7/rK9dcIbQYq7Ka< UՃʶgA87 \wHtnsޞ,¯}!Qqqc,4 eeޢE{uYX3Lux3yȃZw~9p1? c ͷ^=7{ij9Ә~3C^ɽsox1Jy퇈XF8+9TkS;׹Ai<\[ uvqX1)$F9zF{ y͝(w eng \g޵szۼ /T_YϻBv?=\׵ՆdՂ A)7E=a;Uڶ;& BzNسUMu(ۚ坺VfgAOHtc`D4tM4ް2flDypҢ#ߒnn*,˶z+Oj؇j%%&U5vm5u0Wm1/)XgNlWI~*PcACui~ :|ւ=hO:EWxKafdk8s^"mk_=`) &]>Ls)4?ezy-r{$IYMU;?P?zH+2#- qn Tk}_9wTf=2Ć عnSmT.gTqxB-szy9>=zB?yf(gX?sc=x}+ OVkġqQ>!~c}f9S)[.ZutTj>5F8^z#s\8*d?k;x}M9zq^_kc~8彗w13t /i:QM}H+~BZ[/b,F~uôvS_9{:R?*/ohK7s%Ùgm TXdgOۅ{i2X 6vkiXĘjB+?n+jĘ@,qs{&MGn*︁\~'Qfڤ"_au+}XIvxi4S0 BH&A~+٬ޯ*>ug5j尾 ݠ?_ ^k]ck۬vk_ϒ›֥?ZW1ˆ8o_Ʋd"qv:۪}fڰcm'Xv0"3y/ٹl#)TC]{hAR=W(<RLbWJݫ__dX(v%m9g|ߢkX^{H\l?Ő׹|cKL"~LV- -B[|rL>׹s۩b;u^S#9߭طYi}!OQN>Ь;}'6en->HPqgmܪrl#q/0iO vo`5w Ky{On~bu6U}}|inh;\s~Mb]§nv&f|4"5P_ÞRsb1{ظ \9ʸ;u@pOޢ̫cgd~&=;#)ZV9$ZLΏ6˸u5ĒMgkw/^ʧJ,j>m׳\vŜ[/ӲL:;oW3kה1O쳻!NҙYxOk㋼RN~%j'H8h 0SV|>~|)}nY8:2,ʶliN{-ڹ9G˵p#S}筎egʼdk}lݒ'e>FD9QdwcT%AjaS ^joA=vdCد)*cvx4}ba;kv>*mmkiP*o+OP;nc0]RC_׶Rˠ obWgߒ/*5cgu2ֽZiӎ'ϛAQ!Sv6ʲ)RMyHkkzuk=xe\GݗŚrƯ\d٢o}9el+6g.ۯ,Ҩ>G>?VӞ3vvm^ tiBm!m?ƨJm6xs#Mq@{k9.4^,I] 5XKuyv^eFۿ֑ 4ˆUZiYUe$Fqul\o9VBz8c>H|g mr^kK]O sįN w8q.}P}nYw%l_n ={fҫ¿[|A:h6:>Pu=f%[{& x{PWU^D=kOLsH>Kԩc-d=Co[eүc8l۪}Su~{:5E)/yu\E"ؽzl1&:>ŭ;]7R>%\WlAv dybCou }O?=?7qc(%ag6~vӨ6q|ozm ͦ6߾'1jIDAT/o>وW/J|{*Q"jA] ` -pkEݎQދ/X{|O>k{xcߢZ6^-1o2sTsݍY3ME^kϽA7][GvDTΫ%yM һ})*n.;WAɋ<-*x/Ih>aImu/kux֢6'&6iZ߷5P-iov)6h͵b)4ƱXo|Xu9qGsk Z6}wMSrVVH3k.Kzzn5lMe)oy^KWbȷО6smxJ"1-8;qDb;zbx]kbD$#NJѨ9+ڔy'9W_gfPyUݝv˺7)"Sw\] ;[|3T}ߎ:yG]϶ٝ y?{='(d)'m7k.QuP5>Yk3㔓cW>,_j|d}] ftmQuJAvJx?٫kgE,w0ExdWg-?E=1WnqW38hl8R=^4zRe1X3~4M~۾~D1^͆}XXé>Z|ڸ}{w#6=^1|{nRbsj7dyhS 4e@Ɂ)QC #`/(W+`{LMn Ŝ&d0^+6p;XP; [ӵn:Vl{mTgO XoIucRQbo!2Ղi8i9wnV1ں5ͥʴX^~)Y%+r+ڴqa?hXrVX _\0u:/۰;>S ux)ע\bqTB9eYP^M}}+ym}k7~湱7BT-ʊ,߾vs?|(Z8ƥX=ZDƘ$kvHfc/)|'*hp܍)ryQo7/n 1'>pR?jk)+US[/0c/5I!}T]$y}<3$bDUQyq0<3:cg5jqMt|hǖ-ǚυj~+|gݶ|^5 x9?zxc8`41mhٸⳢZS:ؼ{cgWr{Z}]yNv8P}#{}}BҶ} n?>@ؐK7[m{m8gz/C'6SHPMy|u ;9OZdBjoߍ&Ί~|BSD^,[';>֣}h]g G{cܷǟGtxں~$[7kzz=Gܟ7KZz/rn*eKްϤʺC|ok x'692VTNxkwЇqq[,11o{5ˍ^_eyGGלO~]ʾe~y=&sKo[<653Ҷ UmZ3l|~, SBoٹOB=R%v ?#Xl/P`@Ι)Sߞ"2rAp x>5I-lQ" 1"J}s~~(o+Vw={Ѫr[~WEro%zͦ3+C!oMԯ}FOz>>zV1YcgPƣ3E{>kƙXKsh,ە;Ÿ2uӴb)-bF{2c53=u(+(&B.tڄzX[J6t=K0F)4pxBc 818/$n'(n cpFI=)t2V [b'9e/k_ps9ΨN=W::,<Zr/7&9!e)Cl83 Kf XxF!͉e9//OrES }oh~]3 {gq;]"gT;Ar$qK/SV7gpkT!`rc?F,/9oDفE !pXY7׆18e83!q_MD].Ļ~> z=K>r:={8 _.G'3>p@q8s |vo?:>_M/OOxi7?^x1]?_:<dnӓo^z0ǺG]={1=|M+`8i?k?Мۿ7(֚GǛ3T ֩Ns{ӧ?|4=}K;1ݛu|sT콹3 @ge߾\p/s[v^㏹w fz~л߃~}t_?=?^mket$6w=N sѶ~^y$qteu=7:jqqn'J5iPRrʇn%Y#Dž]Éq4"Bo/y̏_U>5ӫ\OMwU3>|_B_ڥ|򙟞/|6)t'$[7dP'l>tcx#3,\C6Bߝ|ȻϹ7$W{qhöHgQu94iggQ0(؇93YThVJOpuٮ,7َ>Oϗn^00-'ʃT";OPYem^{UP샪S C䗭 U_jэ#›y6MBDeQZ AL^D(./_(scFAIKܯ:6յaq58x_wWlE>CF^v\i;:}9G:GuS03g@ܿ8z8NץÜq\,tn#Y+?{!V8yMٓy |XYG(j堔}{Bw}mw>6$UD8s鬕|O*/z-r}֙I_9ZP׭}unWif=1%Ҷ_®Zl2Eq^9srw潢q߬QĦƪ۝mzm7&cIsۡ<p?}q(!sxz=t̄- /}u>pf<]Cun_WֳxSݶ_YYZzЃO=`.sq]/Uk]c J.^ ?my%T/C6{,#ЧpLeM۵51D}>C->Ki_OcՍaGc[߶gnBSo/S;K뀽~nV{y-yկZ\Ve}"XU6ss|ǧգl{x3ŵxE9ǂqϟP?n?Oﱸ?Ly(IFydqrbA ӏ]\hP%;4Qzn$iPޜLŸ! jП?a9p-b16b'\r_=eJ['ڹF|x]Zrнm<-_AyJ|aݣZ*2u@lpxãM(͇eU[nxcfmض9݌~k-1%ط]+9&szU/:VmWdwE1MDܟ}`QmOmo:Z+2yɥv_ k^BAy/^5Ǯ15mR~ٵ/gclSB}-+e#/r cY}Z;zm>3mرhyv,X [XGÙJ=O5ؕظ ̏ͶoYp(1,Y3vsU+m?b<}և O3Hm-]sv7벭72:,l^s=+reoUUض={hǃ!}f3q|[:v|u6-nˌفc.Y?ƥk/Wa"9ǧ581rԱ}tuYT6Ommgޖmlk8ښ&6沆_AoqHQv6m):A>uf3+֒HX7(]bmHja[.Qgkk[Oxu/W'tX7hG+E ɶ۱K/ƺh?Av>E=Lgu q!m ڬ=ݶX|֍FwsٯZ>29VFvo3JuCu/fٞo9#PɅXu[X]sܖeyA8'~xEQ EMB WO3Qd7xw#< lv4=ĵ{OVm_.pEyeo>ukprЏq*(73bVƿZK֫JFkϻexޙef+ hmN)_+FPRL%W>Y6FkNWh:/uk~" +rފbGŠ^=WV4n{g&bM9\d٢o}9elyg.ۯ,Ҩ>G>?VӞ3vvm^ tiBm!m?ƨ푚ssIъ 87d'ϛ"V*v3A㽹q~nǣҿ>2epeCr,J=ބcc@Fqnu׹rs}yo\ݵz_tXwG+[Cz%]h?>,׏ۇa~%-_Ztk<[3Cm~{c>gE U }\rƘ|VȘ,ҝv}kS7!(e XUhw.oLjϪPՏ?F[4Vix ^Z!Z{ڣ7{?}&K!1zug7c72^_ϟ0~OO M>_sѩ'31©jbQP7=&-i"}X\jozr_ V>DՂ9THK ]Gu;~Gz/xphce/_ >QMhO|kxp܈o .ƁڳsK^kϽ7=_lbB~}5#툨WK@.Yw)buSU~K\v?P;,/3xb[,T^yqV/uOI,o(tS֝m6kcӨ׫_r\jc-aM5ͱ1V,ki<ڸw(aonn#23xIfShORo?7[voXGQ볊ǁ8]>Wژ}D]+l:5bSVUoČEaet!ng/LI֒{LŔwrv?Uɀ胷V ݰ3JukEٶOx%k{0qmX3 sh#c/cܛ?LþPlzO(C-m+7AƅVƫ~lD<_Ux6bϿՃ/o韦N!6;1Nv`P]!mzc&=UL\W {*s &nm"& T9qB<( y #ArjlOО -l Ҧz*cGak֭ JƿaA 2^-n #FŻaxYjqu[:9ZyGm[DEiU1½RKoJ,gW/WPiC~#bѴ~m@``;u^aw(*}9ݵ+2sUܗ Zʹ.5o[_kcCXKD]ߝ}Ћtky1/DߢK?a7uc\ߎ@^_Ok;ssIҌWb%ҞvtO:?fb >i1f߶N3M;֌GIw<6łFڎpTkolîw YW1vg'>W!97y`tŖ^\#/gP^W'φdy@> 3uÿc7cyXy^3s38\W[+z֍'{i!Zlzj]~Ɩlg#c=ms{Ӷ[PR3N =]wHvvULb`ϟQܺmymƏ=>;^@~cmzug831byVU39Vk-=J2il[{m wx_?~23MuN?ze:>^=n{?<)Zžޯ| mpli ?6=^|6 Po&x8%řWӿ;/Ďh5'q^N̐-'{;Vʺ9R彉lOyÉPJY[H2VfvA׳ub![ C>KJvv)ץؐyHgޫ_n*S>?}Lu(P-]Cm1_b~ۢxsr?/XqݢԖ)֣\53|0𝱇{ )WDy]X;?3eeAU7;oPE8k^ݭCXYvG׌u:yY8_g/z]%ڭ˙RIsdײv:vXfgi]L6] 5&o5JuxȺ4 vns?V2at|F,~ϟ:8 uȹy}Z|Zۅ͝qfF&J# kƫ7Z>k_+zt9~SdbM)ǢNOkJ}~D>Hh?%J1m{gBGkK.cLKkmAtr?b]sr_)\ww_kF @kzƣXψהg t&75'ؓKtKaa,.vګ`O e8Uy2|0xAAgݻoL{'YEЗ{pC~I 2f٥+Z Kf P>G]X,axqs]i.6pY5p9O3 7gpk q8!pI шC\_%T,br{͈~LeCrp\}{\g0s{?G'3>p@q8s p^9/F֔[I+/^=I619'>nNQwl;ӻ"洏>;6S7D^OiNo}.os9muW1>kcm۔|?NݛNXҶ,l\7G~8ƁtO?~8}FzӔڽ[s7}ٽW3U6m~4=}-gݾ{[`'AϿ~EsD^;n͟S+'+h#q_k~^ļ-~U'T,JK4E巁ˮeM Ŀ1f%F". ]|U$R=YG.ӻs}CϫY}p׹{ҿ.wq=$¿7}J¼⺸%o?A/P]~6_^>~EݔrB?ӝ;w]ޟoӽmܶ}.nWxoޖkn{\-1(w iיXSwA祆 X[9mԧ {}c6{ƭ;]ڬy$?mIr+>^ßuIv|yiOWӗv )Pdv=}yper.mNqnB~/[MOk@VhS_/Ix-M^/9g 9XA%DT_0kݳi^mj1\ل"ƿW"Wf̄7a?r2NzЋAЧ =_l1܆錵u1ު'<#"wfA~uљ}M9idKӯ5_φ;x0q_;3nd+u }+o+>}|O[a:UmWdu,V31&{{ŷm^O? ?lg0qY8";ŚM$'Av|%,qLBLSwyu]62Yڶvۺ;1c 6mIy|!|U:MM6V{1Fc-za/ٖgn{nyv ?՚}ZXVGWqi}v=곃̌ Ǹx$:u&|.xuMge?g6{'MCo4s5{B:Q/?#, ~]V|(OY<Btvk39xD쿵9~!ƈjl DnPclH?Xƙz1~<ԦC(x(Cr砹9VZk\NȋDlMڵNIP_'a^t= Z̏ŋ\N$`͔G6]^ۥEb@ mhCϸ} &| 7 2.41Ƌafmv:_Womxuxrk1fey󼏎" XmC͔@RO1bG`w?LW?'<W2}{+eS=Hhs]Kq~(}`ͭʵpY\;Ex9f{u%DZU+d5;*MaQ!ݛ)$?ox!o)_˼ow :_u9g8,]8|(I']ڭlnlChv. XY(Ҏ!Vo/%UwYY' L_mX+VbA۱=GKus'6k10[pNj%=~&5Tvhiwx_,ox\6}?&=!;mўZW$K80 :#._Y/ׂ}Qo)aa\2}X-veA KN:Ro9}"Q!pP8\s .,4|mk^] /~_J[ݛ>}=:Dg?> D⥸_EvY"og0A(EoEgk[ҋx1\P6mR̒[! z wM(*^ 5kw-2Zio{^N 4PGg0^-L6#}[gL}N\wf\\|JY$3 ԗ,4 ʷN/r[}UYh~Ugpop–E%#oiӆDžB|kCu۝BYѿm[Bi[| Y{xqr +$" Q=AX~m ܾHB]Cl?/,,ÂZ$+qfϜ/G}Lmkcc7aEeCuA+ΟRѶtbx%^ ^g,qB?m}5\`aqZ~́O=_1zvp#3.mђz_:uogo{nx}hQ-1cKܵ mS?4ƫ,@=|5AΞsBևP> {+5 _b~+|_u'To-菇KA>uI~uz/)](ZhH5 ʩ 㓀yPB17~}<}ۅ/-(3|@_m*Tm>8Kҗ w1x9H_BI_N^o2_5X7& !<,+ t߶C{iO«M@ O!)5'nbOOB1}=^R\}Y~*ώ}IY qm>>}OWy(ZDB4~hk-6?O/Q>v 8!< {Y'WRށ~BLį/ʏ/nDyG@_o؟A+o16#:K>3+pQ?d_HOzB|+|ջ\8/?";5z}3$ʿ`aQz˷}f䵿]Yq?>m @paП_)gX A|nS~oRwˈOPk=zS}~}}^ _GODd):c"^~z_oq_sӋ??pI@xdzٕ>ʣ7Ouy>=y`8\ճ3:ӓo^L׏ kӕvf3}Ƹ qz*nz= bt^<1 p5]9վdTog}Oދ3{M9/}|l~%J&ιeG}|Dw/器_~o5೑_;=Gw߱yL_{>ZLYwwߞz=:];]{nS{u8`ފ3/;O&F9oQ8{W_gtzkpʔ_N?կҎICIbb$w6_~}9ݣv#oC܄w7~2Vw9y"LjW?ΏG'Uo"Ύ;߸&D}Ntn5VhF;tbݯAgu3"ƝO*_ ★2%r6ctW8x;X:\՘1%zLvB*;kVj{$ZstNi?=y/v޿{Ƕgc)ۡXy^zӝϷkn7˿+'%Bdx9}~ 1cDT,/ `)[{Q/E2?p *+D\YEЏ"REdpMtu1Bk6_mo+R`WM"x۳C]t7w>nݯub["=D_~+dlQoDcq;9G;d!.\t}a?'W؎aE* tAW퀋K]l}%M5vÁRNgg)$Tx :,H5%ˡl,Jh樱]wE7ٹ tqs:-vv/X3v) qIzkTn=0~՗3zmKl oUaťXBq`v8WϋZYlu6xKs?$z/? ٽ/=1g_K -泰Җ߆ ~ _+J7nKf!57=. Udg]gbMg?]]1붂{SĽo{B}Q7{oݶMqx#~zЃ : ⃤?H(?V4S>en:yR:$Q̣_c:[CE_!͗s9g2N/ 9r|FDߏD~6(XjV|XXH<7f\_biISdakc{ʁQ3 | iyqMsvDo{M1qV̿Z g3sIafMR8B'nhie;2:Z㳝LPf"[c"8I} 8Oprgg<ҵ`Xk ~)=c){V"ģWh@s>i|Zkwmlgmk8ksl"P̏wqzo!y;eSŝ\lzmjֲvFgDP/5_%vֳV,[bNz2&5^i%-{ S|-ܯ-/[y»g3W=ֈkjßp9o/~SGsۮ#tDf#\&`WO^ /El˺O'ۗ)WҟonEz!5=oway {!烕@R_B*xyik9ItL?i}*S~ߋGs:ņ7ǒ8 era8SwȲ=7ת>4Ӳ&Ҍ3b0#TٜY:u8#kv [q)B[_Oi/E_Y4 1;u2 9^a_I9:m+l\-97vzo+Fś{9!)Αڡ4sucmK]Գw!o>⳪ϼ~57[4לkfC D{2qDqE_!//jb7_k YBt{=1b8}qks'1C" yi@M]ܯӨ[y8SbH\c82ò> 2|wVZ˛q9+6:0ui>ж;駥-_{?(/eE}G#YBm72]n[[Fh*w-Y!6/ue_+/Q<@Պ5>v8 4q0r֔tZ;.Qi^ѾPB8Nu33ΩN*oAy-˓Y@kkώg:v\5MؒcTa}(8|㐮uL%^ݔ;*Gh1Y&] ij;us^0)r}1mA!o#"Ş_sb\.X;V1xo[r|Scr<{'Qb璏6~JӦvE_àNJ۵]YϺ pǭ~h_e<{tz/j_j]rY\xei8¹5#ĺ_-V"j' :+׾/z~A"v}6ߊoؽ8^?V&Cx:b0rM jPV9TqZeyHvJ%R{#/`A[KG?z/y cjC}n6>(ĉw*p-KqT]m~ub_95Q/sl3O!%6x7}g)ޕ=*]%ǘG+&q ; Qr Aj٩[-26n_|vޚ&l1*K1k<uђהA7;q+׃>u/Fuovz6OoڡJmX9.uD|͋)qáG 1OFu9}{&c-Cyϓ,eGO^ڱZƢ%bܿgh؜h{>+ӥE_vWʶ?-l}o<Ҿ?~Pڦ)VwcZW}=˺޹r{ pj jz>~|Qo)y|kh [C=ٰ~xA{Rv8KT^((!=ykA|.݃Awآyu%ܾ}-+_aU}?pX?s_Wqe嵵uÝ;%AGo 47s6?fcw,VViց ~/ooMpZPUHiU8yl|#mݡbn_k|v8X)l9h376#i33`n ty?{a}cWf:u{T .tڎb`h͚bx=Ƚ-I&cY(O'VvMkԇιn}l7͉iwgoH0W%14կJC`ݏҷ񰄴_u{f~~wϷg@'Q #ƙޚ"aq=Wev^%N{m^lw1Y,mݯ11G3x9-9F7qFế7dÏ(]*Ы뤵U;(<6;W[ݩƦmEt6ы=͊5grc<δ<(DI1SIث{&kOhPVn~}g )RCaK^\.cW3;7V8 x _P7=ӯ*_nZ>_kNoL:uy_c.^D#oIcͱͪ_sbUuXxu{p^y1Gr .h cRrM#1۾3*W=ܥ|Kݖuykv[mJyn;ucc(|>ӊ [OG/k'?ӣ_]W_pi~vDR| ?! " jY6OO>s|6IirzU^׹dQ2 FW?˽s?'nپt9pӕvl|V e5ݔs]wQsHއ54av 1n]Ƚ ]}tˢuQ|]DYʿQvY?oL| HbsW^U7 9onk>Qu⻠Ƴn[?RZ7oRXD_~+dlQoD#1 ŕsG`%tDĔT[a&%[ Qq#[(8kqp{).y~,s0e[y;]{K|elz?}hܴm)v:ka#glV溯E.璎7379D\ pvׁF[#>v]17M/(.#, ~x V(O4EtEDLy>-6'/<$͡:m!Ɔ,ccM]7ukd~'SRVWp,؟b,iltjR=68-чLJ 5 sSP/ȹcgr*[ڡ}ڧ#Rl{V/u8ICLuXTBNw{w?q?[/(?"Zhp_# sV\ۊiNX!/ kF'XbM?S8nsQ+Əw_dX9\)YaŽ1Ϗ))VLn*tYȏ}lh?m<*U;k_'AyR#&`+W}V?Dܗ|zqmSi,}~ x> $ W|( ROUyHvJ%/,:jvT`qF[=OG}G^sp⡶,+[iʸ1QƸp߬{͟94M;՛Gjc5kHM4)|6#IzB}5pcmkUmݻFz*4ADǢߋNF{woC9b8\KecXn@V@%Kj'|(>!"FeӿUូk;Ҏaͱ9qc1D^K/"R~ j; [Rj_ -To döeA K> T>cL !k*B=JM^uͫ+kQЪow>~8,/qcOpycD? =F(! >Iwn5$}ڧ/3eacn{_D1&M!W k˗Bf)*8ZzX elߋNFgw~x1U0̛s,hxj^i[OY7+նח#UqԠl|SaGJk*t-cUrYܯ n[زWK~ wƛ6m{\د~mXr>PdE7p8N(BTw ,["i[G)۴pÀyN_6[ N^oƣ^a_l*GPދYwn6=aaϪc^Cu|,}#R!E{kKh q+ a>L'D_C }/mӺhp_T}FΩ?8MK<7@J[**])pp\S8[neJt~Ap_㹝`ij9ĺX+k5o2ӵ٪99ӌDu<¼qI ߽.]("[جHa7 ʩ}s1/1um1bY\Z5@-I0M랩~4Vve]AIiBȏrBe;1zߛ?9mъD#;6en9::GǟxqL2/1LTбfuAn3; ^&H_``J⦛~~s_bFċ@ }p!qN/!)͓`pkS ^,nC}.~ a`' g}́93>p@L_^om2nLu,-lcrTq?L_|O8OGLٿZ*(q01䁺/OOe~%=._wUO˗YrORY`j|A? ӋJQyo&ߘ^l{#]uPnsޞ,Z|֣f]ݹ>45{>GS 'Q?AW}*{ߊ2[gX q81q[նc>eY~Nue;6UiיXSwAG/k&n+r>o{[wƷYmA$~M1 r<.ϮԟySKaK9 U2OJg폄g%ݝ"}O b^HmƣQi] 8 Թ=6͌_Q0}wS_ S ƃh;:3p7k>M>'0*ܶ+,[d#Tkj ,m~['V|@/qlo}/XfBBQHɂW][>)KE+-(R EbCj;"VbZKC哨D"^NIK/FA\ܧ2W(4KOv-b !fblxyZ+!*0f~dٶ23a>|-ʘ->kXGkLK_Sd3B}ݏX7>8*{߃|<8Ϣ7V(0 au!:~!jl~F݂cCٶc-Bh'"L!I*;Rt&+E 幜D{"_[EJ8eMۮ蘭">'/qE VDOmSxG2F)GS#DϷÙYw/wUhOޚ3P8\xVAZxV>dϚ81GY5}fDX E ڪ{|Oku+}7$Ěl M[FSx)_vF"/,k=^⾵Q/m+`*kΚ(7Ěuxp=}uu\3_Uoo{M_RڠhSYwfWq?%68K}ky@/.ϒ#K >p@q8s g}́yŋų+?}GӋoLT?|zpd?~wO?g)_Lwmkw:7xYSn='{c$瘷wRܟ|)_;=Gw߱yL_{>ZLYwwߞz=:];]{nSjwų^f619q ӏN߾/c6_3׈__ϼ7}_7}K>q.?7xzR͏Ų|zA/<I4qe_o僚?rW?~~ԅ H}~ǚ1oh>vI,B* RX b"GvV)"GY .[ L Ŀ1f%F". ]xU$=YG.ӻs׋dӻg q}t>ҖUqAx!/),K]\":M0o.q}IgqτDT׷嗗Q7l뾽\P~Ϧ7t'B{݇DCתt/g  m߰K d/l%%F[;ƣ~َ@_Avzu&]y`W \S ^طqO"z΃NcDS>/)I_-6]!z$A) EP"#T]{];ݶN s[<_~~=}:}F/nz@ga-xј?`?:ǰqKeީ[ޫ距?my#Vk1U\Zx83c;1.5Nt ߇Kl>B_D(rZMVL΢|gGt_/MTObDҀ~ji{U3qMQCqQ&\As?^s(M׸<|na9kUO¼zwP".ShI8T)Emb׭&*%Kmpn,S9i}qyLJL/?I o?e\i ctڤlu(3ۯxcb^cˮ^y*NE6ۆ8)a灤 ӟb<=,؇#,>\/]hH[^S+׾/΃UV e _Kob~E{oކO\󫄅vq:H\+s\לCaWWBeM_@Q$ڽ?Ob{SV=b5K|oz[w}Vb|}ѥCBq? ^ۥʖ^ Љv<6lЏ!2)b՝O6_N:iC,g&6^ ug}+|Mm ֞%˹Xl-ڒDaq};4i ;D?!~K7-!8,򪼾X][C[R7Ɇm/  _(v.|l/u!Bb=/4{ 6fWWע,R8V}{AgO'ϲO2?x)"z 5C}?+CG#JQE['Y.Vbx%^ lf;Tdd+Vȿ`8iCf9+ƣt]tƺxٲ',4}_ϿYD/KyY3I`W\9_܏l ٙT]7&^nŠ*XVƝ? ^ۥPmJŵ>Y L> ~Bk>~'g8~1׷Qß?zc*B^bGg]۞%tC,=[b6kڦP!hWY{j.7<=c(u)Ed`u)D9[D?=bA՛m}˶5qߎWK}MܷvW|bh7ōV{OQÁDJ,=]a ZIEۂj_j9hWeyw,'Er+'p݊nېD~-ǶDzʣʟPܗUđH 8)ƺVHOmbW)4bX -mv?~/u>S By/{"q0m];cE4.㨱VMuH!vƺ[_̝~3ѩh\>#r:m~sϖ^}==QqڬǽCg㌈mZ;DW9/ц,O8>Lu'ְɾl')QwN8}IaBK>8K w1x9H_Bpi@ O!q8s g}́93>Ń'/ϮQ]O/y2S ؝Xܟ|b~Ϡ7mXx=3q&<7mSqYE}^j>9}YqPWkyl9^l9zv>sgzW?UÜwfʺ;()ue>?m.ڡ3gmmuWH?#8`Vn˿1&ӛqh:*C dL_|O8OGLٿZ*현ð/rAÜ|6M×ӫnzdo!G_Nț?w7!ݤ8}]Ͽcg~1Qaq4ׄȽwIczߟ!]h۞ѫCmtz_v%bMw>~)DOsҧޙvWqv֍mueدtnѮ ޽iǙ_̹)׳"Pl7_AN]5֫~.]}M@̢Q]WʿQ \v@n!H˂sW_U79onkvߘޝ^'ߞ}8]>K[t;c_* yK&%}Fᓨ q>=qoE3 s,nKVvb]:PH A4qF+wӛRe=} }%ιM5vÁROg'?xeR㚒o>ÿ ZWMs߮p,fV}8[ZZ)Mko|df|ƫm8;nBo͉:݊fݽ6F/(}ȟg8zq-γۧqFkݓԿ/~.W[b4 կh<* T%;C=:k.>+y붂{S)'ط>=Tܧ~/nZ>O6zi'O%>| 2z@M_ t(JEGFbQ>eaP:yR:$Q̣l$F Cc7^-7Og?u2_ޛTiN}3z",mݎ^e8bvXlاGq 7Zc5l䷡a}ZEە9wuha|:71)pʶ6 ?~l^Ʈ6 [n>f*1U/cQޣ5Z`ʁ5h{pPܷ/&ѹ"{"M>gܭ0*ܶ+".i5>Y>@/suoMzծ%N+LJ CRC( l`[~`R۩~>(1-xP*Aik9tHtdL?i}*S~ߋG#PiSlȱs,[6C&8S{=qW[U9:niYCyѦB4e8 5po*18~D{[Sg%#s8$L+n1;Juo3%;x]H-|a,<՘.@ CovoO[9{~*ƶX=dMYχW rDz:uw]w' 6BbX@m5[{9n9&iěN7kL>,i2ydׯ;-= ߅Qt|f?gao. }]&ǯ w=ku[8z?-]umݢ_6~9?`L%|Ϯܓ{pcN12/$(h*E`e7n#(O4EtE(DKy/^7omN$_xHD1/ >:/vGX-Xr|〶B<@C}P-VsQלt}x幜||na=`!?kUY=q_D倣2DpB(1<;7v;]K!H m٬zG0ו}GvA=,[8AP-KaG+)m/ooMq8\pJMb(E$gq+dgj33o۝SUނ߼~QLǎ [r5i|%|}c~57DXt=}5`q&IioBV)`Zl'_1l aN ^Ыgͺ{1e P ; Y=S9~WLPL=kSS|ai%ֽhWO˦ϊvXWۘ+ۄ=M{c&ވX0^e#NXKm j1X΋W_9Vzʴ:Ike{N~}shB $t~"xU)rv(ntv+ n_{ mi׉V̯HRdW.r5b>-C0`JDX~OG}G^sx:X˃.s:=k΍ÏAM`i/!K=~2žrlX*9pFl<0=<[Sr+'{TKIh|k'!h-bCֱ+gn xИvVY35MؒcTc|)ibG^Si|Xx'1 ox Me31Nb<==wrۼϨŶay\݉fݽXe8L;TOM.pbë,f5_kt/bkuZͥ,|(Z~o޼=lǽKegq*3VbMoav_w/=1xM6=.G_6ֿ"Q؝^06Ea^i x8(ztNux恻80j}芕C0 sW~u9 3cmz8lz# nR;7v:Ci<61=My_F6sžb x}Dw0v; C;:,1<M_?^'a*۴Z-q||hkPg9-s טϘ#Gu]–r q՛8窼'E 4\g."~xܱZ Rz{]{ݵ/+tq֫ feڪKu~ 1 ={5u\j[MMmqTo.2?ܜzl钞 {َڽ71n5K]pK?s:cQuϵ $~^7z@C)ֽxn kesk_x]U1N+gU܏"E:hËS~ -bt7PwS]mWஉvZ/,[+I1Q񃽖-/uBA:C>H (SԷNszv0h%վl A*[ϻ/ApWgjN zO1"JZqvq6LC"^.[m/4aHVnUL*c)oyAs0.!^+`m֡)⌐V_SJĎq&=1)^w1iݹoI;V<ݯϘ,c"CS}(lkeYZqik~'S]6D{vęD,8{>Ǥ:}03rou{M1iǺWw+fuwb! Z^ܿuN+l7ꭎK֚cɏpn졺k<\'CxJkBD{To:龘6|׀xA%5pr륯sۏ5z8XgPk⠼g}6ɾ붯'{n7k{ý4kKU2骵ҮǽKd/O!y0r;ph bzy l=g|ܸ8ہ Ye3p`vnk)/qa1cn3]g }p!pRPVZMz܄Sܿ\Tfx U9 aXSo\Z+WNYg3>ęJU,p@q8s g}΋O/^L/]<^|d]ӓ&!y5]1}Kۉ|8l4 <|aoqӃ/~1}yH*9N?͔uwx QSګ[<}~\N۵CUgX6%߯9xG\qݖczMwH_7G [OG/k'?ӣ_]W_pi~vDR| Jj/fޟo/dnJO>#?rU#^JW?./WJOO{&ϝKo痜WgӛKZDϿi6z"GwnZ۾Dgn}бpa.1"w?q׾ll/{0"#h__sxT-1(5POĚ :/53UmԧRPOo+B}{i{Oed^>_Xuˤrnw}.c%$h1E I -[}*E+,.+vp9d<[^C/V 4ʟ˛5[ęG⭱VUC/3ZI6m'GxϧC}37 0:,|64zs1~RwǾ/^ZP^] }vP /s#<bltu6^.N([\=v&3V`n]{_4v|$nenp?6'8+?"QH=tB -;ӊ2 HlCؐNLcZW(Oش)nuOIK/FA\ܧ2W(,k`ܦqVTW\^s }j"E2}4>T23Rɥ_ϲm6O^8ds+JWc03Qv+>"@[. 2|}F| O b,qQh`<9܎G3:GPObѯ.#>p;fuP`grf_ e^~/ju[᭑qC=sUAeMT>=e1be|6 uJ OK"(2KEY%FL' QG 2F%Ğ1'ǺvU:J漯oT1ޯusOMeJ[_"h sdn.bk}$#^٭61I8`o.{k3,-{~0{E/i}^3˶rq$q\#ڇkZAmAx)s+$HKxlc'Ga%cevD\vcxX}^7%U}˳3E*Wn3(3St}׉qSΕ`wMU۞>s[{}s_x-cSxYܯ n[ز_7mP_~mXrľދ\GawzI_iz )ׁ>#K1r<|--<NyX-uYa&ЋZ?wa{RG/nB]6Z>!k&7;7ܑafʸbJsT3chS~s+[_]S}vX ǯ1Dml4gez} ? o|SsO{tߝn[УWQ5.ο!rzx uv äϨYw6Î~oyn70~3n"-R=>kZ6DR,Vٙ1]v)M,:rNxcΫ #s.f UzOXVX+@՟~9DlĶ'GBzSư?3f+m5+榵M;07ktǣ wM5бr==&Ϋc]p"ir1ÄoO\bxAd>|vr]aT;LH Rf//pCė{gµ/Β=}8N/|:>-Wuj/n1Oqe!>gg!3>p@q8s g}΋O/^L/]<^|d]ӓ&٥t}9t好<~Ii׏teu\.'8}] _p-~wz/"cܙ>xU0}釵!ʿ=zJ{uzs]w?xivjYk[ݦ>':UoLnU"f'ӣ5_}Nѯu?.+S/}8VJ;2J$ }Ly4n_{Ih]1}V>*X*RupEЏ"RE«\v(k!H˂sW_U79onk>Q>B%[ڢ%+ˇs% LoAO~T\Oe2(7"ϱxqrc*em%}Gc/Fdukk e;6UiיXSwA祆 X[j~^ອT mEO~/zߋ˺ ӓ5W,/ˁ$%ߋy鏟K"F֦w5x0P43E3ݵN}-L1\b΄V$:W|O]G_vEvWA#Mq&z"xl:MI=k/aO-BQHɂW][>)KE+-(R EbCj;"VbZKC哨D"^NIK/FA\ܧ2W(4KOv-b !fblxyZ+!*0f~dٶ23a>|-ʘ->kXGkLK_Sd3B}ݏX7>8*{߃|<8EߘV`Ȣ|Qt9xD쿵9~!ƈjl⾏]~{vk1Ջ!oY8}3%FlÍ 6 X܁mIF_H88јI~`:rG̤bB v{ժUw9]]UVZUw}c .SFR Ž'MICBuD2 l-{߿owDAn+쌋qEVDOUەm69^QyӱmyygaW]#KX=k_['ޞ3`{X |L|]u h|ڴp+\3;^U}\΄D~"x:Ri Zή [qf{-_{b=,q\ι8{9k1q &0:.I?UV FR m!$BloJo5VD&v-iG/h~v/m澳#9}\۬}sNBɾed:eۏ̈\$岞}Mǫb}ڗY6<=g5c1Eܷ±fߩŬ;-xXbjC4&mknfp aL }#]Xମ-VOG׏ %5Vm=$yK|yEMBPZ) I"i]3$K$ˮYXmVѴ9}q?=o71w4椈kdd:e|.\VKR$i_o3ibZo_ukbQ&V'Ŭ;-B%CǩΟ½uy ݶCu|MN=.`\> QP>/v{QV9_b@Z5pbcn a&$B->D"s,ҷ`Obf@u^L?ۗ}\dyESffmϘ9Q.J8DS0E|̄oyxꎉP5Nշ+S׀mUg-yl,}H#ϖ?jt| 1fm;FT܏"&]:R vYmpSDohEmſlG!lM&jJlRN6ǝ2\sؒ:NL!^jJ.L R$QQ(TR-0JDzm { A^O}Gݽ)Y}:b.U\=/ض|~Aΰp,uv#x2?9jtYw-ZKei|,׍y]ͳ_־ԙv93r2^=쯛N6~s`S];ZaU8a3'O Y޾G>[my \?p),6¾qqA8s}3qA8s} |W_.Gugj|Z\MѳWK~r*mBm=$=]=bOO_~n닳keL8xXX"?96'qqtjlVs {΃?N_E>X׹?}*q/+RշjeJ>~OhTvovY[_|~m`o67S>W:+(#ڷ|jɹ'ӕ%>_5bzY^ϟ=~Tm5YَdBsճGva ݝv߲p.PBcWD'fs~O;RS&v'E!9=`l3 P) g)7b ڒy0mھ׷]޲>l*;X^`jg7 ӳ?]>+|Ra!({G4^l ūl;{ۋֵ(^\qyp l׎,cj>n嵋w 'Q?}![?6 f@X-wr/~0\ bF;,HoL2;,D*Cra׹3'> 7֊_Kks}6н^YQ|\_ lvpfuMule?74 v6kltj(kO+{r~\c5Z 5y#"Ÿ"W._7{˅"_^o[udG\:IX)Ja6:uid;{>$uz\'wB?7lw6oyqxsm6}\)o|?9&羴Y68 Kkh!5b,m7 H߲|Yv8 [Y8Я:ry/a|ᚬP٨ u,4׃/ 1'Lph^AgU̖1El}@ڟ ,8|>1Y\ع\=hЇm56H %2^v\qՕq-k/ٖ>-X먺]{rw(g.+_ͶNpowGѷov3#8ψeMSrg>‰k4_OlɡMUy[yΆۛqQ³D8nxug6E|& JLz~`ksmYM3G:fW̅Nl! xxI!k_:9ٙ04چϳd^CXP2.Q}U~!I?<qqߵ)^ȹ2ZM(s)̋YmI!Kʾ*JTb[6<'p9O]>_(rӎqȿy|!ZHRc}J1],1~Gq$ǿmev僘1[(q]׽/"WbRvγ\k%;cCيg\JeWc\ܪ5mvvcPP9˅#s!ڬge"OqI=b!,W)8r^=9'kʽNœ'}OwνQux:~L• _{YWs\{}6IA[/uM{Dv#L5({WU_Vu[mp bvx{mGENt ?¿G?4 ˬ:֢|7>m&/c;~>"ġuk>̓Aa&&d{NDouXlg03=Q~}/-q_(mွЇ:ET}rп#Ey7&[v=j`M3fVO;R2(%yQ'xYJ1-/_6h̬}=s_BXqv&Tl#](c"cl0˸mfg@-XqiƽF>ۻu*2/oֹ;ڶ8ګ}vZm$AxmAH;gy(@7ڛažF{GܯD?,ac`6Bo{\/s}3f8ߋνvoDŁyԵ8.偤?(BloJ22)!Q?4q_LO?}gG^salgμ9V1~>B2:6sKtU_;l@%1b][˄GՆ1^k⺴ qزmvΥ rkOG3'Bg=one\:tu_*ƹy=\0nĭKK5M^{_[?%icnG;#Uc}C~}W .kE.}-_ u,\s۶f{ධ]C5g_7{:m/fu)}!p*Vkox(3o!du<nvͲ0bh9$ϖɰn~F1_2y3v~nk"Ő/j`Mk85[K:X_B3BSsc|OUo^0Uuk-a@%<~9wc)msYg{kGg@-X~Xs~Jvθz+W8K0뻌qnG^oχ:6V翼vm_i\{elռvIw_uiֽue>nbm}Zߗ1_a]%bJ:{m;Ş7~Uux[d,o7}:_a|uj~=fѦE&CVokuxk:aHxH[?\t+!/ǹ뼘~/a#mB~gjC:٪|ϽM3f1q >?ׇs@!Ȫ%Ң >"_0{#0 2жL :R:fiS{R l/u5v(\V8Yso\\̡F_%"D\>K|_okR}snK͑@b19QΏocFkcܻZ[ޫmLA>QYL0.Ԥӟúߋ!sZM}~bogI!Sg[ֱ]M&u :q_  ݅ſl@$}ok]]Q].:u[#2w^ +A,.i^ -b7U Ĭ M\ju&-.}p|Wj3m}1C~ jӹwhݓ=Oh[ܿ{Sq7q|ֽ?K;(ѹ->{Hrc>a>W+["rܯ=.2u9ן:Hٵm*u'{{ptD.KBd.ٷD ̽`C,Io#3Ŏ16>}3qA8s}3qx|9zzz.u.ryϿv\LyzzYg>Nݻ\LW1ٻ.&bdO$WӅUVc1[|ט)t )y0=~ȧ:OE/eo?YʾV@iMM~.y닏|ܯ lW>cF}jɿ'΅& oa MLW"kKN=.ʖ(`m3vAGg+{}[]يYs\n>s:qb6_;ߵ29م?^?k;UGy$P;iUzP@Iޯe,Oc6\${>huz\qzC~/s&yzq6 k>_yS .||KAͳ J*`CYrX D_/f=@Õoؔy=H}kGSs3;.NgUmWvy0̢>b,}|r͓}VUeΗ%_T\Ը֬#13NuS91Ai`wQׂD犀k~Ix~_8m=JGdvm;b 3zWOܷ#}%q,uSӉ{\߮u8@u}m ::s<HAuU^痷AdwD>ϟ7''Ae\;IPZ"P"c!iwt^-g[mwr@}>%gsߋY;Vo5xYd{q?hsjo>hW%_?lcq< ee9Pc[ޫg~?Z'a]̏h奎ّj̎35|l>1t1 nq?jC(X桸hCr㠹Zk<z7wlg==Q~-do'a^"b~@xpqm*'p-`fP*]vMTJX}~99YI?܏oop[ŸeGrC {nc尻}luh>q{^mGޮ>w-Ulsu',OE>es9=@9$enx:)ְX6$s>\SB~"xm:RYDdYkoLJ ֛qf{1mo^)goY﷕g2fZ?BBvq:H\+q\ԏ׌Cae+!Xe@ڨQ$3?bg)o˺߮1{i1a1R桫EL1\?3Vߥߙ/-|ms|8f;C1 mRb;?L!qt3Wp+G3י+br[AFK۶˵XKjNX-|ʵ~=O\S۱y搾vOC}A!s|G.) (k!im#jPqq׌5:T,bL?"9zG"8(CpYA%x<٪nvͲ0bhX)p7}gv_";]HB >{ 5C?Lj^?s{?q+oR̒vWP[ <7ʆ;ΧsWG+5u-ҙ29Ruk5ZR߶T8Ggaޥݼ~[.|:2oGƶ*LQ=A?} ̱H߂!J|/5ǿE?GqyS^|[_$g|cq?"{_^-uc~mMhQSNe<,:q?d"E&hQC.a ZI:ձ| C=k[)^_${\ |~¶H".ǾDyFq_WG,I1Eε<]]c96eqeq[l"AZazȵeȟetsySo5Ң~It ~;s]QS9j2Y{F1;9"VkÑUKCwqy08fLk}M\(tNPЈ }8K%;{7|ඁaஂp 9g>p 9v(O׿\NξtXE-'Wp2mϣSl{={uc^*;33n_={dֹ1Bm^7\Ma߳_F97*sv͸M^'f[<?Uӏuӧ,e_}`zVvDIemuG]~P1k}>sߓEYWBBfƥ~oz[8Y=LOs~|/Ogy?dQO?ge{UZrbw{fydk!u[;M71Q9mzQz})^fn6OS@NFgyڌmǞ&{v۹PF#߯GSF|G>6*T/JKa5Yug8 (mKѶ݆}}Ӹx1*N`_@b֣+{-oLZ>泗g =.lC pKΗ 'Q?}![?6 f7bnj; ·rPq5GN02vAy8t syyvxG,=%({!T̏<HYĶ}5or#VKI-YY[SݸBuMWqme5?¯[^Fn6OSRy"s@mAw{qXջ }>~$|uݣqNog73^-1:öle?7?&3iL]}̊x^؟5ǥ3l\ؾ'i_=)[>\t6j?wVܥJY'{.Eö#طtCoO~dWjM b_]ri.~w! ghw難C~lYLNfȋJJZyJMQiǸo{OQyVZ/gpء(!!#}1P#3518Zc}Q9|9{Lg..Cys?Ka۔eϯ˃^ĭCy'-Mz~* =Řfvgm`{zpkv6~c}(}6"s1I%Ů1K뱌o9bQՌw_OۜT^Y'R_}>+y8i>TFn7*$.)Sϓ4aQzFq_ +/&A]W}"{!rGێ4h=D'ۂ8:G=ߎ{>.WW/Ʀ7omw?x8?ƒoy!z,Kc u\C>_ƣ\T*/?.6[]ߍ0\+\ܐs}sI:yX9Ѷrn<ʯ2Pm+z{<$rZuק;4!:_Q~;da_I0XU۾X5'bОc֦g[s|,} +{e/}&ukه{YdoDz\p]Dub"z!:ʥpu{-2;>x:V 䋸?{܈s6f:qwʋewܺAom5a:)kƏs۪vO[e;;-{߿owDAn+F0Sm72ܼ깐E& 20Ɔ@ߵCZENųirbg@0ޓr^9AqH ۱)Q,%̭K!. !n3gsOG6TvV`9ElkUvɳdL{~*bO+r{ϵٱ=6]\?W&[֦~qrՙ|G#4Iӧ^9Y?~1 >+M5G[+Q`7r5mۖo-̇cIUc/nw-8k_E6}l';^8W}X ̶ fuȝ)ތ,7ՃUqf{[V߶m7ʳ9o78Cqk8+~0wzkf8~`{'![l[U+<$?|Bh1)ߕ뿓=*%֚0Gar(X.+gvV`ken$7U1q?k怵 _u12bޛz"Cg9^+DЇOӟ5yu3jdOqt߅rܿel{k2FN^sa\/ξ+G/bVYX~\y>_}:oV\-lA9ShWp|kjޮ#ͺz엡7+#ogu "!붜Ղ=K=m=}aicǸ뷷hBYurGm09_CogYBG-!VH5{;a,!W:Z]l`֧1ʃӥNKdߛ"ּ͗Ǘ-HFe 1Ob>κ?^}X[){%_\)s"Ժ\==w+W/fb}l eShGkyuO^CʪqCۮ7r|ѱ3Yn_3H~[p½uy ݶC|o~MN=.D}/_W MDziǥ1YXEqwm惛x0P 9 !̟[LXoVcpD|مqy18*_?o_C<5.opQa (5p"LUœjW9b~5R{ǘ7Ks_ayfŦ< q^d=Wez]{Kyɣ^߭}g;}XQԝRq̳Sh w˒r y[<ӢqLwBl7Ԁ3"|sWYڧ4){͞ѵٟ<=]{z:k2>:VNC;{L{\형l|L}I>ЇFM}+krG'*ٌ*J7ic}kg>D :jn6ɄӼs*euЫ -FGC(2 '؎wa?}"uMrm-Wra16)'^uu ˵kzq~0!A1H Mao~Ǽ:x?܍~E߂XjCN8m{=[? kr1YV;p طyɱx qOx{Scr\Z7ּy͒Y.9 @Qar=!sUv:Y7KJ[1XSIѳ} k`>˵к=~p 9v(O׿\NξtX0/͋WOjmI38nB~^={dcZ"~ܟ>叿d)['4}N*7}mn/>r6]Y)$/R:47@ȧus|-*_zg`z뿦|_r_LO?'ӏ}2f?+ۑ_s;tG_IAkU3Xܿ?}?u=Okw`_ovMh ky4]rCLv ,r~~1֏Zw3aWPW~ ȵҽOLᗟu<{yvZ𦰩DU!zQZAtE(¬?Q5DaoBU73Ҹx1*N`_u-f= }K>(j>{y0u|ʡۆ%  "|N%chmF:;s@AAoսDq]`.E ̢Jϋ;_|{mzkgC~&.i#ͱ]?s8zXVY^0O3>o*}^PyvbQD[bt7ŝqaGpG$w~;m5 :"YWW׋>s|tϪ WWm /ȷ[x x//GUGylĂmBiCY'{.z8ePb՛NDY"Z10?A\tO5 3ήC/V2Wi&ʶ3$˷^W~ J)H54ka:rxx:k݉Gu|߷FP9>/YʗΆ&s6)Zoa>}/,X+kTak)R}\^'-q_DA\wmʷ2}hr5`\yVɅT̋朧ԾW>>hhS)_j*Y ٯkr7]ާNKs+WHDAo?}\sR 5)!> q+Fg=0ߏG5y6M}s/1kƸܪ >XrS wsx|mnɇv:t{CZC \O|X{v%4yh+r#~>%eɹp=k_]%溯om5cy}\ª#ƿ=ѷkSz Y3^?kזBoZ~}츏|9{*\t<~Ͼ~.N\z]맑{lw/y!URΤL@!w:yvZfs%ڇkJ5l[pVGY?,B{M,7;b-b+| otoi~mYIe\·;|zapQK]lCYC(kay/[l[U+0跈kbzo;;ZE:V捶[iYy7sR ^e%%!Ŭ6'\{r ou\^sU$J&:rv*][ܗN2rz띰(?a-/Zn_q0V,ko=ySjj]_?mլ$.E$ʦ5|zwJ>XkH̵ܟݿe> >k#P{%~ټ˹׮_ɳ7 !?+<3f9-ˁq{ۘi(kD1a˅˛%ڏ`~7klGe7kr9.QJw:ޓ7\o#`[Da_WLZ]{9/su\:92oʵh]˪m˘<{#y~TzyvRf\pp]B[o,6|:_v|~9_)|:vҧ5~u\'Zdyn hP {Ϯ|`:XPk:d-cv ? ⾯bqT܏~޾c|+H|7Q_/E\9B{+g͵ٸ~&V̪sڋC@XF̥k9C1VZ@Q uE:z3&n}Cc4/mrXn!qbkb^ފ{r$ƽ`K䰽~h>Vj¾'>\3"ms|hyhϮ!~9ZƝ_Sf쵹`vC?|XQDdp^GA:˟ϯ -FGC(2 '؎wa?}&RB]+0ϔyY/:D~q[= 0S)͟SRg&C_ݯ[PKͷ@~\@8mu%/y?(AvwǻsV.(~sH}B?!ǖs#lg*?͢3vb1c[vc]o1b9ge{ AoD),mqP!"It`Hw磘 s]\Ys yn=k3Bi7_<|kq _~9طk: }M~Twr8X?Qo5?k,+/G9s,kt>`[gIza`a#Gby}Gx;>0}Ä?đ7YF܇qL1,?>9p 9g>p 9v(O׿\NξtX0/o[n^LW׵p5]Xeo qwqIeWO6k<.1ƺ ΗwLq*ǺSQ{YO}0U+;P{GӇ爐{Ӈ:}#.kUx똵ѾiZo@,+sI3z |Z7®~!O>߿_ן=~j_Lv&7JS@}[_wLܯ=$ŷǻ*'1fDU!zQZAtE(¬?Q5DaoBU73Ҹx1*N`_u-f= }K>(j>{y0u|ʡۆ%0]0x'"|N%chmF:{1ǞyrsqT%>ogDUJ,nIwDžٲ}m&.>fE<_]^/zVϚuӉOb?(']MmW.#Po,^={wˋP:>p.V$V:uid;{D偆,MyZ|>2n~e0_>:ՕvTVRky#⭫#!׶Y.7׭(SBŽXٱt{Ѱ}cWQul{.S1WT1sש)k;: (kAvsE5q_$Y6EB#2;i zWOܷ#}%q,uSӎ{9ȍP:',R׺6 |RT94ZK)ϳdߋXQ4ei,kD :-Q}͂Xok:iLx" kSE-?A+懲+Z` 9p V!̧Y' R}^6݌$.׼j},1}'d/ѳ+F6G7c(Wo}d扶D{?klœc6 :1G,{a΃}-"?(";"z"z!:59mG9ӱZ ¿G ݲ*|c< q? /BS)#)\aGšH!:d{Nd?Qo[fEplF·};?v8El+ֶFlu/HѼ6DNo k>fVAZ|w`ie>bmZ8֌}/+>\SB~f"x:RYDdYko|`oOm[cަ㿷SHo+J*:v>`㢏S^k`$&B)T([VcJdlGܒ6j~T跈kbzo;;Z7$[ M_F۬Sx):̌Eb[X.T~*[Ǭ}[џeSs^3]}k.kZ̺ˆ%Fya;IcҶ6k ~\ ˷ormsk ôSlCUgwyxXC5 qz$DK *Bi5CHDe+ak1,nR<g]IDAT\M{ӓaqF}SpGcNfJF۬Sx.eU4.E~:/Xo Zb.ovo7O{>`{\؏^\;>z? [oFa:{-Bh%i!•k3Q)T=ζ̱H߂b8U߾O9r>t_U䱱p-#<[11&Pz>z؇>1fSq?ȚLH17]VgD?hEfwaQ.'[o}=_Jֶ+~%fcr"/Y{\v]/ Y)K-\WӅA$*~E߂XjfQ(P-Ap|!7C7e<+ϕTG̥ʓgۖou2/4|n#OBp 9g>p^<^~=~yajzȺ{^MU9@B?zjɯ^{[]dSy^?;U=Ͳc}q6oӃw!az@[gO93ku>ZĹ=|Zߟ<|{#y>n;.N j.^=߱`/y0=~ȧ:OE/eo?YʾV@iMM~.y닏|ܯ lW>cF}jH̎(?˧fq~(ӕ%>_5bzY^ϟ=~Tm5YَdBsճGva ݝv߲p.PBcWD'fs~O;RS&v'E!9=`l3 P) g)7b ڒy֮mnU7'VI=M}' ؋R0,Gv^]YhlMxmW~ߛFyF/6wU8V.>g'3VL}ţ?4~y$ kAO~‰?B\Lm(Qg' :>: ŚP׊"-qOc~=пu9mgH x8ۦbqU/(ec"T[FI+|;/s'Bv ;Kƒ8?D>R|d\`>"e.wx9zXrY)##WW_Ƶd[`jҿwɍKgߡ5{W;QdYEYk͌<#W6O˝p 'گ|=Ij볅_'6EO^N pFqxEH;9UCߦ[ܣ Gێ4}=D'` N@X9bU2޹CRwHEF|0gqg}pvf> -C;1:}: xT_Ahoxy-q_DA\wmʷ2}hrLy\ :mVEj⾹n)kA뒲J륍?$.=l>_(rӎ/}'!/]WZ6bPl7:bYbrI˴1cP·'B$J8;ZLNy axg}([לKɾj+[T^an̷y8gp|dt.DlSI#.iٸB,:GKռ'd}Wx}D/n|^bй;UǏ)Wrko>"K*|k0`&b={C°|k{Xl&vh*Y`o,}Q1./w_|]ެ vkS<<$ 2EA]\^/dcf\ ġuk>̓Aa&&d{NDouXlg03=Q~}/-q_(mွЇ:ET}rп#Ey7&[v=j`M3fVOu-n%yeLAJZ4 ~2OP5_re[kێG0ǻPDƸ7..`Υ9q۶`$ ȶ+6_#^?_o\ Mv(KuX)c8f>fXg\[=& nWړu4vLrjߗ\'6_SEfCn昳8q{?sgw>6øq6ZͲ-Y}9e+l.vDp폹k߶㬎~d}ShP]=~Xm+p-_:}l}ù85ֵ R<Z-[ QfV>E9$6j~T"]:g[kuW]{\cb 1۾}mOul vj\A쵅`\nfyB)}!p*Fm/S7(Ǻ@<[ZʃHyf,[ s,XCR?lۜ g%LЙ7h&R ߏ?::شމc렼-Xe& kKNRu!ƙ)֕"cL 0]k |/ٟ艇tk^[|ޯ.zmc؇J}"c pl,W}[[s} ߬otqa&vƵG?њ@ݧq'a!Yy97Άu !mpY9L#5⾯bqT܏~޾c|y y , /s[g~/3B?:6͘WFzL ˰ĵ;$]/az[cTo5]_bcc*~Buh[ tE*E0)ƽqXqZ}zf~l; _+^q|XŹ7N.tPGov.Xv|X%/ѷ5)9H m(Ƿ1l1]{r-W[V@uNt6 (Ƭbb}ja|9ŦR_?wBA}oY#zY{*6׼uÃ]V|D?hEfwaQ.'[oS讉zmW?JL+W2<|\ZjNycugCLUsjw5]E}kp&aiѯ[PKͷ|(j l@8BoS1q~mwwq|.p9IP_E%=܅%"nbQ69ʸew yp2[j}"´ܷwK@v\}KhPq(|8_U5'-}1e?,cWӃ,7D{Yҳ]Ӱ:#b7U Ĭ qz]ͲDe9ߥO2N\/ո\-}涍:"fZs}(pBflt]ZdS}!ewu*jbtnwˬRܘDϕkOm65d+[zm6z|{ΕHww K7̅=62Hl>%{}82~6qS։7XF>p 9g>p /W_O_^<^r9=;WcU3w.?)_O=27ש{w*7{wdX)wjjl=fKOx;e<.>7;8}c](Ž'KW>ު(mw=wRٽose>o}oTuhߴO\竰K{Z*(qv5}O|~߼Ͽ5+}Rk?K7?2kH}>^‹oreݛgo rMo|kO7]Ύ7S H}}\O,Zbq?>6*T/Ja5Yu$1- goBVuo/6w:Xe Kݞ^PŬGWZ/XΝz\곏٪"n !Х./"< 5o/^j7BD9> ۮm?O /8ؽ?;=w}od<pl~Fx:[]qqGp_ v6kltj1+޶׮/V|̇e zd._29م?^?k;UGy$P;iUzP@Iޯe,Oc6\${>huz\qzC~/s&yzq6 k>_yS .||KAͳ J*`CYrX D_/f=@Õoؔy=H}kGSs3;.NgUmWvy0̢>b,}|r͓}VUeΗ%_T\Ը֬#13NuS91Ai`wQsx[U)߶"ǿ p {hۑǙWG/""Wl;/@s6! Cx)$Ctlj@L C;Q ePI,B2,Vyy7R_w""UT>ߜx}W=Dp$Ai|NB̏y-sy`o9\Зq~/fXSG/fy@aY~D_|mǡ0ŗ@myrWkm{q>zv%2?}:fGރ1;:9.5xr ;KE_!"?n] N] DqoqOEnON,^717d]o'{m\RyEjC(X桸hCr㠹Zk<z7wlg==Q~-do'a^"b~@xpqm*'p-`fP*]vMTJX}~99YI?܏oop[ŸeGrC {nc尻}luh>q{^mGޮ>w-Ulsu',OE>es9=@9$enx:)ְX6${Va70[8Ց"w"[{3> 3֛視_=H)`bo>9+QѺ6#^ƹ 5.v@4b</kf>LQ=A?} ̱H߂!J|/5ǿE?GqyS^|[_$g|cq?"{_^-uc~mMhQduu ˵kB XE jpZ4h'@(TR- mx~Er-'[S "N}9_E'\9ײ?~w)vO#;wgڲƕm]̲!k{NW η#'?"yVMuHq$1 ua;G/N=ѱ]m4ʘ9y^=[2g=OnXGW.=ލYd1a76?PpD60 xk7/4"mqΒ=>qz ofo;o _}m & _;ƟjG 9g>p 9g>yrz]>ӫ/#뺳y5]>V{Q\L{D[ht*۞G^oWjΌ=LWungMg!Fg{v؟1u]d{✏rw8;R6ߔ7$;;8}c](Ž'KW>ު(mw=wRٽose>o}oTuhߴO\竰K{ d=__5bzY^ϟ=~Tm5Yٞaz>0\KNl @9af?q;?LBڻ?L]?18nBɃכjZG7;O_D~9޸'D~QN?\yF6K5.y&_/qYD͕~Uge+R޽=fvmV/;}m#7\9z|# q!z^T>K1BnkIvx/xTwRx+ ΂~gQ՟E(DWv !m]/}ogq=bsUc¯8_X;;-qur`.ޤPwk "|N%chmF:{q9;4·!""u@W8BO&ln^@)݁a5w2ןgKWsdb㞒^ay,yoFmTndq wɳsB\\_M> ~YO+}?ݻw{#;\B!*Dpgز͐iqV^3ST믖 G!0凴-a)lV]ĴgC̷ُθl>}h,Dz4PǵO?e<(շÀ%0(菋M_~ws:农gK`(A?s:3ƆyUڝFw0c]w+z{\y8Zs=Kn|?q.yֹ99_c"< @x/Ee?cgf:{t5 #V޺߄-r\=fOumvlͰu~s8ݗϥ_Waב T6Wu˔w>?97#"2_S2ɐ{]J_ŶK^c\6vf}]1Yzn=kl5|tŬC-VҗC[Yķ]=}+{pl6Nh汣E_!"?n]N] DA=mG91C" Ѐ,ۤX'qZ^W!s>~v8~t?ċB<~5A-vDZO[e;vr߳K[" V+؍xbne"wmcNBWW/w:.M~Y.z_+݊Y^9 H8+|"k;^ޞgQ,%̭˦$Ik~+ q9h,ߋ흿?q.y玉 _{X{L+9aMYbk,i=P"9̏b|Ș&ToŞRVvkc{|m|/N}="{c>c@S{ nYLA>ϛ`]{tudgsOݘmO%%}ӊmoI}A1q 9\CY4i˷kUc۱=sK:^;VxO겒߿6pIB5%šoa-6#E$Mf|g7;~ mcPMDp9+QѺ6#ǵ~ke]::kdEΧ2&?|6w{~{{J@wpxx`X s"\Bh뻲vlgu Ccq{\b~kO,9*c~e ļ7%:ot؟絴=y6$Cc~*gr{ϵٱ=65qLAʷiVktN:ߕ@^dyٵ1qnJz*Ve I]!P]_вhr FrwOB9>a>ܸvZޮ"2\+w=>]l}7 "!ƕшxqQV?pVuk9{E[ʃв9}q?=+ym嵵ea -@ۉC([K1gwݖs[?0x (a\3+ȉ+^ޞ e=ZU'╛-K m#}"<X0םEwrp_G#N퓨>mn/Tm=gv|~Uh^Ẽn!?׷p?O'B_Kݸv|~9`?|D'ߧǥ2\(kG~~>n6"}S?55a[3 :/G>sZ=̓N'|@k !OՇ0%}#Y=Us[9P R{ǘ?sۤ w(~I"ʎyfŦ< q^d=Wez]{Ky9{#n5s\g<f—%G򪷮yC\_ >=aBݗm;楱F-&ڽ&ak;Er߿dSq?ȚLH!<< euЫ ʁ<`"wm/Lg[o}=_JD׶+⾶m(ZC[g\\a!~#PSkqH$͟SÏcwZ?ӡ:[`Z9\Mk { A^O}GI(9|eeӡC\x}{+ˎ'ܸo80٧_GbJ?v\}&ejLWϖsԘxJ1Bo>Dفv#~F̊c?i>,-/ۛ:{e{܈wmu{.]6~s`S8;1pyo?Z{ȸtϙ碅?H,7o*6ٍ[@mϽ{ ̷-i.hgIzaZ9C;!j97͕sLqNplȳM3ؔ-ssSv:nOp9ʾ3?NM:$F~7b<Zyc S6&=-~o_3g>p 9g>p^<^~=~yajzȺ{^MU9 sܼ9}*ۈ1؍J!gճGf8wLq*ǺSQ{YO}0U+;P{GӇ爐{Ӈ:}#.kUx똵ѾiZk_Wa5!rƵ\ʮ5wk9?>'w?{2'jI%8Cw\5_wxF;oӃwu!azذ}|h~\MVYϧ9Xaֲ~VtnᓟEźTl]нOy=sN<;oxSTw©Ix[ Yu. AVB{ۺ^(HzCl;{ 굘(^ѹӢUq4AO~‰?B\Lm(Qg'|H1(7sϒ(^J'ׅ-i / J_ok -=Es k6aZcNs[A`]oc_:>r#>ՄyWa'?Wľ b73›ՙxDԖ_|S p~;m5 :5 ~uEo[Ώk}lzz+kCӲ]P+Xg^4o݅A^^Lb{,h٨ ۄ|6N*~].cyd}5 QGuBKCdnWG~a )_s')V.ٶr!9np(sjjlrO^K9]⁖d ێپ1~Al;\羼f+_Z8Vܶε cvT˳,:yvZfq Mq=(0'Û*oSumGgbL/ K_9ίC pv7"*A&  }i&! 8>N-"Imهwx`qqӁ>,V}"EoC%0(菋M_~B]#r3*TytП7ʷ  mj"łKQ0w9{~McaՉs|sʳ9h;_ rb!꯲B'@EPD1Qh`>5 9j9}m1^c"֦quU|r1{= px3]+WzP@R_8cSt#ѳ+19Uur6%Ƨk$f䰞W傹k(!}ye,v+YdoDzkqxM=cD{}qxێ8~scD=YfIN,^717d]o'{k\ Zġ;0yh)ڄka}v8_ԏ״p<= vT'-~ ߜlg^}o_/mqߋ-|K# s[ax8`/|<\sOV+65\^O:O~`qDrmhzB&G1k5õUc~1Ӌ#R̵ "TG2s$0.,oin^! ELzӟOlIsr{9|WCﻹk[X^>/kOڨ’op <(c,+,:r6ko\z~k>ќ=|UN:_Exg?qk_b6 )z3=˝<;oxIB5%šoa-8#EYvko'Ayzۻ#lR >Ǿ>9+QѺ6Sױ?_^>\aR׺&ۄkP qXj([VcJF iG,-ڥ#~śΎVqENyuZ0Be͜(s):v;ȱxYffdwA*=z1͉1מCkśf<"e1b fU蒋ֵ.2HOZMP CEo>b'r_XK{vT9Z1fê#e'oJ%W#ZLkRNGt!/*[ك9JT 8||;% 5$Z߲os~5G(=ZselgwEw,-<;oxA"!svbG~ƳzkhmD{il)*}u^.ukM&G b?Uuk9{E[Lj!-{ӓaq}{| fN9g.B9lYqgX'`Ƭ6'ݵ%@>|9P1>.}o3MFp[ߦ4Zc :ӛ]5P#[抵{W;VSS G;7ל%{TQznǡuݛ]~9]쟫kҡ̑ySW>FZ^V]o[ȩ=`I}~Uh^Ẽn!?׷p?O'"_Kݸv|~9`?|o>ia)[:yN@`Łw8XPC֢o9ȡlG/0 :/G>vǻ"*8|tʷ3RdE n.!r\obŬ:8{HYel\9ouoCPY侭kFlG˸iauq?oU1:6/mrXn!qbkb^ފ{r$ƽ`K4U(ǹq̶jk/`>|P;{GǕl5峫sȡyNlqz0{m.1;.Cn.kjm>K*y67~5 בBxx0@,=((*BЃ ]ڞm?_|)[ۮoڶ!7#rkoq2'!sP2f6sjw5]C\}Pltu} c+>m=? |c#xw}Tbʅ܏rRhg"Rrn$_BPYw6N}9rlna}5Ҷ#-R|>bl$0H=/sV2S>PI{L\bRXӾ|sa #kn;gmF(>ro?2/vmb_Gݓ#{nWu{%;ڢh[\o(b.e \.gչ.prH\[c1;.|[z-Y93Go}M\oR`8/xg귛:.!CA=r փ3 `k,Io#>lDzZ,o_og}^`wy8R&ۈp )ƟG8}3qA8s}3qx|9zzz.u.rm͋֔ ѳW1<.ifc6G/X΃?N_E>X׹?}*q/+RշjeJ>~OhTvovY[_|~m`o67S>Wk*5D޸6ȥ=U"_5bzY^ϟ=~Tm5YdC { op뎉ԱxW^3fU 6*T/Ja5YXu. AXB{ۺ^(HzPʶ=h]Y¯8_X;;-Ը|_2Fa L- Dn*,hQ/^ !M.\#/s9W 5`;A/]]iG9l1)Q\;Ҿ)޺zy<rm똥rs݊1%T܋uiϟKG7 ڧ;v5Ua25E3^r6^pg':WE з)-ԧ ɾ}ML{+uaЮ}1.I?݉6Da EQs"x-uk'Eu)XHLBQ6/C'>ƜWuܕ̷Y\υ˪xiҳ]yy5틼-c&m\^]+-ޞ3`{X =ʤ!D`{\؏%n\;>z?7܇o~8@mʺڷB],1H &\YlM!ԄBqHdElI ΋Qq?y5,Ϣ(s݌3'J2[9E ~q|>PFU-avcV11/Ʃ}ex1%oi2GP6!l?ѳm>ш0QDdp^G .^}}QPT ݅ſl{a=}^&R]mCoG D9*2:rfq/p%_MN`W&)u} cE%S`[HB=? |c#ޔ>WR1*OTl[Xx gX8:Efl< Q}5]Vl:_֬2kr>Ƽخ/k_G;c`љ]e[c \')5 9AL~x6qΒv=>o[ovqqA[Ogɷ%>p 9g>p /W_O_^<^r9=;WcU"jP叞ZVicja%iަ|zNbm;=sX_?}]|w~t(fz{g?PǙS7 Zqt,'?ϱ99vێ˧Sc~wl؋wLq*ǺSQ{YO}0U+;P{GӇ爐{Ӈ:}#.kUx똵Ѿ>qߞE0iү,^'6V91]Y#|0=_ӿ?/{ٓG߿>PH&4^={dmaw- -}?a}UOtb=(D#Ů=}ookw\؟SL!6y.~zU`^~~!&vR k;!öڱ=ޗ-=%/kn6X,b/J8ɳEY wg5 5շ]!AXnQ9noıBᗷJO ̵~B>9j5|9jL Ż@ᓨp-q_Em3 ?EىB|;]y{.||B?yKs"ލ]wyhnP\bӉ\·SBY??Zq>~`ѶfF&~r~c4N7[ܮ |qqGp5NfbC >fE<_]3(svϺtjy丶'W._7{˅"_^o[udG\:IX)Ja6:uid;{>$uz\'wB?7lw6oyqxsm6}\)o|?9&羴Y68 Kkh!5b,m7 X=&œx3ZeC`%MH!9\Kc3&d|&XsP׊"-qOc~yp̶֜3e{EsmSW1[bsĸ*Io. A(?)g/s'Bv ;K\_HWsaXr!lae<\]iײm_ӂIվ'7.Q}rl]Gf}w}fn73⻊[f_t>/w~+kF$~ښ?z>=ķ̖lu;=\gىuo2/ Yt]ߦȮamG|Ǚy`U-^ߞkϚd:PK8ܳ>C}8;3ߖCy!xXB>Je< 4շO<_(rӎ/}'!/]gaNKRc}J1],1~Gq$ǿmev僘1[(q] qg!q%-m'<ȅVB3>ykΥd_֏~5ƕϭYv*q0ng7ۼE e\X8j>z:~V)4sl\!r|e#{jޓ}>|}_<>"ޗT~7X>/1܋E\WЉǪ+\7m~}>W5uXOސlkam{C?uxp+QQg׿jbn Ihp j:Qm;)բu=C!V9'6:7m;8}+8昝*8Tu̓ R^§ypY9(݄lɃRџ+@fg7ʯE|%SW<XǷț!oW?\n}h?}ˮgt\ uѱi̵Q*) !yf+.??`ԋ6B 3S,+mjѴ/Y;` Y{kc(I\-m#](c"cl0˸m[0~dŽׯ/SHcZ˕7ri.e6enjk$ۍqJ{ZƎIS][5}}l~s0Z{}֠c}.sf ]0Av5q5;= ֱZm$0~d}Sh¯f kb{MN+V[1cfПA_?|3>f+Wù85ֵ R<Z-[ QfV>E9$6j~T"]:g[kuW1:s~v,סR1 Ywv#n^]T֯{oR'Z̸/I_pc'8YٽRu?M_ 뻷gTE VA릾]wK:m=lYu p[Ab!³M ()Emv5SlCxoW7k{L`b?0B?q k-ukA<@Wuk9{E[!~mN_OOs?3y&̛yhs[)}GWktlZcık!'5!p'B3S+c]k /FCk|62ƝqXJ\V}^ڂ3 v,?9?` Iٵf;B\j_k,qlì2ƹy=\[)-r;S̳Y>{W>ع'5٫c|XٗliUGi}_^;\o6a]%jmZmůIxuT>:.sgcq*{pp]BfQO} Dž-j{#|U~us)1s"Yy97Άu !mpY9L#5⾯bqT܏~޾c|y y , /s[g~/3B?:6͘WFzn;.ū bKZ,4(ȗ kHLT"ضL :"cТI޸T,}8>=K]3?dxM/.Xg-νqur3=-8|s#,}KeϹ-5GoWlD9?9g]qjݓo9>ؾ{t{1D1f3MHPN~/ߗi-6 |+z4 A=C۾6]M&u :q_ -F fvNlKmyz-3ٷEH9ќkgK۷<K>wh١7N waD<,-u} cEPA8mhGܳ3_jP>&;v88Ne94'W1pdcYm_,†_Wy3G>!8BXD9nWE".["*TeO?fL?e,yzh?KzxyWru~DP]즲J5siۚz]ͲDe9ߥO2N\/ո\-}涍:"fZs}(pBflt]ZdS}!ewu*jbtnwˬRܘDϕ>\?[pl|,n#qA8s}3qA8sjo(x>IENDB`llama13b_autotrans_1to16_train_modelarts_params.png000066400000000000000000000441501471400134000366010ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_CkptPNG  IHDRn`\sRGBgAMA a pHYsodGIDATx^k }p>A1^`q86*82 V]$7 b%-1q竦1b D12 NpƎqc8nR[^kvgfgvgvV~ ]kf?{gOD H"qR$nR @JH)7)E Tm"_-'dd2ZZ{봂:D7?cb {gXV=ը>"q~PoD^ՒvYSM;~s,WCc+8Hi{:x`b*z,;yr{5b9g5G;$7{uy{!H@A@6m|]JK,f>+j c^/m~aSL6W;YJAl ?[ ׆QljB&&q 7lY5zZo;%ۨR.`N\-;~+Mda[2XjAײ6S;YJAl?[ֆa;+NTQ qέ~: /D:DvCuzƳ^ZFx؎xuY7N$;JDmN(TZSC}?2W.IT']_m. ZN-c֧.9,nz*թ?3mue1O*_} -Cmne-TDO^yzG Ԍ[2^hyIkGOIJlceb1n8 ՆD=WOM[wX~IA w575o~/ЗlC5!rTGܺ,g:u).b_/?b}INt:F.:r^+Yx}$6q#4Hv.Qj?xuשD?ֳdq㖴suĿz(pU3ϨX:"K91}6}"ݾ_r(v7nJcC_//j$?^S~󬸽h6$\HnҫK`?ڦ/:HBJN_/΂3c5?3QS.[:mԩI%Ѳ$7}[ t< ?r{Դ]Bsiϔu}Kq".Q˲1ZlvFܲWYx▕`WZϒm`DMS\q1U\?5nۢ묒o5xVQgx嚲;.{kQDK:'QʌWD"ڳcUπGc5ڰru>:DkQe/Ÿ5nr $@` M"B'YFԴլKe$Dt^BG7:zj.qS;]jrM\D.vDNZZmH-GE}fڻbߊ]VNQWn=5M#73$mϯVzvzGM;#}D]E7Fx@9-mP5Ӆ7mŸ[g\;vq"q }G}AoD>j~,ïuIql'iܲ{D%\@F$y˨vtĵKcYn[ve盤#7qH[j%]?Ung5lc#vEs@QqyaK}l,g['5=Sq_NK&x#>3:,3.U:>~*I2HȰ_ jgOO{W 02%Ѳm%պ-+7_Ko̱ZbD{?vֿʮcm.*bۡB[߷ˉwr.=+'czF,kY~t^om^g#jI]Ϩ*oϝWj#l:&َ+YpeSlY:d{-D۫rڰ y{Bu+/|Fbf^`}Wu#q l=Q_*כ~? _yo𿿜uIi5~x/o8g;UQ$گ:fpD['SY:&6QxkԖ^۹NeT TuZJ@Y/m_ jEزZGշ6R ?i{$>Q}ϯw^rxbruW[q/)NxpmXLQe Ja_j_:ɐ]Q_j=K-.Ӱmi;Q F~jUi法v֤ `H SuGyygοTFX:ZZh9{(yLl}l|>XgAެϝG[D{svLl@ qLƒ>9{(1QOg6 v9&b|=}L"qL_z`5ߘhOуZ:.0(74D Kyq407: {:$n)t("@uH S(7PDꐸAQP$nhub9l ;Ӽ 2@V'ܗS]cCM;4*7R("@u'nο7O+9 q{>9qy;)="mve6ׇȈjtpcZ:+|X>7V_P:?@|^rdCXyL,*SͲ7rbBpF7S X[u=xonމ=:)"ul%n.qIC汜蓩/mQwD |,q3a'e-TRmvab4%JfVz{]P?ҁ.ic}9kk;M'/="C;x) rfHIzr~HQ$yS&Ctڪvdn-XWŨ>%/%[lTNʖCmiv4$Lkݺ(sWB%n\=wQn/9/[V{$wFg1QP27(24qI'@ڳj%c ^j$5VmGaq&/-$:,nD)6sbArzysl|fB e, mwI_MB;Nw'S7Cwltu`XWM1oS4GwırSN6vMkrl-NUwظuXfgX%<ejwwy[r,ԭn|{6d$ՙa33=eɌKGdC/Y&FN~|X:M}*&Fzg;'ҿ5/w5}nR:;tum(~t w{ֻmK-6YoJ/f&vRf5k[;a1mҧ~eU?'g}cf0i5l_a71lOg|kA^2m5~ڇ#mG1nߴYw̨nǴ_ZBN;>I2&^.}ZZ ghT+T*sO̩0<֪(|o:)O+OY4.2"өt:N73&AÙc4Odn z7čhc{l(F҃5zVzOK㜗ppcިp䅉.i2S9Vze"Q>%׏ef0Mz_w[<7!-{Єw{K奭Z>,4tQMy.`cZrg8'̰s(;o{t?=!{zM:uMkɼ=BiCNal.mɤvwE^)\;,w^qLguX)Sys%=eȫYV﬐4pQzBb*o=%^E/mʗ?tIkr(G+؞zky,o.&Ntzu(W?B?d&k;ɜ JNqRLܸެp?M{|܏h,7% 6!0ݳq&rgk£9L'w&Qe75|B@ëAJ-U_/s@U<Ǽ{_z-}_6ޝ=2-R. %(>_Z)&bm},-yIyiNlÌ;lo^ &=b=BM46$ExլmuVⲪ?⾓KQ~lO}cpE3J=}s2vN/Q.ď^12-G4-; ghTϸ &T< *dŁURMGyӎKLT'02AXWMq~vֽdǼ{2L^bZ<^>v{:x,/ݕ+S/17pvn Q+oPKuIKܳ۞}}'ėK;DW=_'J7zUwamM##ѳX.1m>Y0mV }oߎ:|RP[`;_Y>\aZv֡l|nmTMc0x)g9-@I2g$n40jm7sKxwbξpof4 (tڞJ񤴬=|a BPm)iTJb}[*yŃcedYaTN\2ߟw,(Q .^9*&=igėCg3n}t |rھ$D*7kr;kCf5kߛeUʻKXcChq SxT݇+[:o<\θ/3ϸ nmTM7XßgnX^N~.4L76R_r i4m,)^ѻ́iŇ;1I74rAR0ӑ֍Pu҃waw^m.5lsM 1רWM͜;Osu%$^yכz[Yw z0Mr#޺{D$<>})o|ylYfux%S?:o{)wo޹ i/ˉ*lE6YVb\&Sm;HnBsƬ z]JJֲxpƴ~_=n}3nud]$n^c^ଋ*celyTyb }┾a $<^#v-:rlfR\=xէ&}\8EyiG>#GzOޭ/*^:go S~O6*U{yb&L#cc'qӰ|cw_>+2Fu}s^>:c >*bլmLܔ?5IխO2yYw}Lxº/4Hf=M3mM1xrwf{N~<иܜ&\n[9$C!zTp==`=$nAZtY^j}g\gfDC}xyj?,گĨ9&pH*nN 4&n;O>i * al4X' ;Ciݫ6ߗOQ&[ID9""g=%+=A6q킌lsOޜiP8@[%AMQ9ɑ{*:ⴼ˞1sR*h<$n)7PDꐸAQP$n!qLH@CBG@="bT 2>%X}s41ZKsL"qL0000Š1GcOzHsL"qR$nR @JH)7)E H"qR$nR 2[w>~aM9{҈Cc"dv<Ζ`iؓFDk-1Ҋ 2ECLPd @=P7: {:$n)t("@uH S(7P:{~PϿiFA z qE{?Ȯ⡏d ) {:$n)tVƴtvMu[ (Zy쉾,*S d, epRZB|M9s2-7'rҴth?pَɬ<&ޗo&L&}_+2ӌ.АH Si^6,qsv\FOf 2m=$n 44ȴZe *$b!}k-@c!qLpYHu楥{Rط ]&7I.%;,^9.l\32;Y[&MrFt(ܗꦉ?y{4/cr[O@Lw,:'[7]uMGy.3n\밓 fP|O09ٸIC(k'nɈM8KI ðaG 5 Gd{>9q˖GЈyw3ΈܖATQ7Gvϔd27|gܗgqƍoyV6qc-^Kv hT$n)7PDꐸAd07'Cv'G*@mǞˢ\>ѐAd.q3KIM6C@N6.1 r䕩wٹO^6k9zxEv~9Sd 7@=Mҹa\lU˘xРlTD?doPvM_eh,$n)ue(pp>`AƻI.%dʞoI&f20+WףF̰limM]2]oe̿s^}r02Ksݪu&-))˵uXGu7@čeİ7wK-n7f1lp҂;(C'oA*qsվT l#n Odno^_H:|qT:GZx"?8yߗ98)j%?,;eY4z og02ώʖ _4/ݔ޼}E\밴Jq]%nC垍Er7mMHKqP ȖZ&nn'{y[t s?A*qԨ4)/yb.CA z qEd @=P7: {:$n)t("@uH Syy;[է1GcO4DH+7Ȕnq: KZ s41a4DH+7)E H"qR$nR @JH)7)E H SuGyygdИ' k=9&@ZAhgl V=iDL H SXki=D@ꐸAQP$n!qLH@CBG@="bT 2z qE{ȱe{ophl$n)7P:羜 /Hn1QApH@Ci욖H@cOeQI.hH S2q% mG{9ͬ6/l@<&ޗo&L&}_+2ӌ.АH SH[^Fs-ȴZe *$b!}k-@c!qL0/CLJsSNz,׏Ib@M#;}}&[1|2-[󲱹Kgs>,^9.lyar} l5廤ݼK\-ۤsbAc4礥ę`"s#9zkA^j3r܄t;hlvcˍwsLwLqe˳rWGd`KA:y۾ 267+W*%nrSz"s{qKۤE,-ͦ|࢛\dKgd`SLd͢yoK}o -Oog{fyM؇҃6@&a|Uzƍ&`r]2sC㞱D̷<*'5NE$n6vMʜ-'6dvqR'#rޖ#:%C܏h,$n)'nAR&hҜ 򣷆_g˘tl_7|{Z ` jѳ]ҧiƟrD71Wd{`Q.MˎA AAT'V8q+G;)9%nۂOkzN0_xfsi>E5 Gd{>9q˖GЈyw3ΈܖA4j̞7eY9rw=_f~>XAJ/*&`diSOl R5MˠIr ˩h,$n)+M8,u(˸ސs 77A9GѩQi:)ܗ r{|<)oȢ_SӃ:qOV03/co#i} R5w/*M870t9dJ7tMzۤiS^wO˅k1KKK6m?UOI˦6ii'קJo'Z{dke}$nL:;aI^r97K'`xYקn7J6&.de#+R 2%OַZmLܣ)FR{މ0>@&9""g=%whH S8HP$nֱ]z9TcϽKApH@CBG@="bT 2z qEd @=P7Ȕw@<Ζ`iؓFDk-1Ҋ 2[w΂=iDLd``X!1Ҋ @JH)7)E H"qR$nR @JH)7Ȕnݑw@Y54hI#b"ZiV$n)Yx;[է1GcO4DH+7ZZc1@={:$n)t("@uH S(7PDꐸAQP$n!qLH@Ğ+rAvv hT$n)7PDꐸAQI~B͍i욖ؐX'-Ix:+Pyk9*>>,#n0V{/J:pFCuԸ~ g'nA3ǧeȼ-$ɤG.eN:|^6!q*Y c99R&}3+=r [NҁZGIi[1c}9kk;M'/="C;x) 2% $5H HTԨl9tٖ\*Ηʅs-&_j q*Z Mʭ, s-VtYbTN~oqv\MƨeL<hPpB"27(247Ȕ: n'񤞎)'o 2޽Mry3lK8f]6:DuIKn5uk^vOjn-e7w|?zmejwYfì۝y)/waTg-m^w~e'YsˬY3m}.A݇܉eL2="+\ /h,$n)'nLGz߬\_c96},]cOrN-ď L'< I<& Mȼ.e&tJ_ '蝒3uq~vN6>;rQ;uߟM=Lyo^_yTZ'nm@ _AsrUU/'6϶M,~2ħ9/'/}ƹGJM-ƿB\zpM|qeĊyz g]2ph(yְ4i-wAy!a M"~͙Ή{_-`ŵJq%?,;e7IrOW/j6M; L@k~D70q`WHUQO`U1ߒzw1-of:x[:θqѼg5۷-6W7CvcQ~鲼UˇNtQgK<>}>yj 工ee.;˦]n^OGݿ 2qs>{Dr[dwPѾ 2č/g{d莎|p-!& Ow98;P< ]oٿ`Ka|6^$Ý=[΍9rvԞuމ<*-76+]G jyRKԫ$68gd7p04=Hl3ę-X4V^/j:dB\󉌉Ҝ 51PX(u /-d=۬tLܔlK3iEWHFe|~|z-> oD }0mmߎ.m/o!Z. .'ď^2WIeGLu H SO:BNG=uK\NXt1.* ez^iZOa^e;^zT>3ku/Ip:iT`OEV~t,X'6|K‘ai.I8a~_l rM;/W_iƤB= bWH&|gxL9u .C/.22*ͪoze`d[,h[#\0tkmkMG9ƣ =bˠ~qNΔPN=uK{}Onc: {'f6t"r/`}AJ UUzs_ltq;8j>uV߇{qq+-ۘj:)4:_*C> :\&'Jh)CmY GٹSy9ձ<.ݛڤ$o"qcOd9OLs$:OMr*e;r选3f͒r 1OXlQ]㯛p>/sHz]ӎf]IK7m]/=brrmV|$jb"mnJ6J#qƈ> OIgH-˴ޟ/l1Cɔٯ6}\8;ƾˢj}\R 2}9A~p^b$n)i=xM}glC4CHmgZ &ڪQegG}-'F+?2R 2e}$n.  ŝS-Y:D#qC/qV@+>Az+ɤ%qc3;GdX0";"47ȔhHXe9ժuiyq^=sR*h<$n)7PDꐸAQP$n!qLH@CBG@="bT 2>%X}s41ZKsL"qL0000Š1GcOzHsL"qR$nR @JH)7)E H"qR$nR @JH)7)E H"qR$n=o e_ H ~_>%?)yv 2t),}-GY>/O2pmO'r-;8s[J¿YFp0spj@~RDs8yxj@~?%ok_)&UܤMiRE&"q !r=uK(dΥ.?yߖ\7vHz/ْM-p)߷ :[OBEsׄMptDsPHH c.~J~{`Rĺ[y?)Jθq(&k$l|ghbK(ϸq9θ2ɘu H c4+yf?w6*>2č̖PMY:\6 2]orsK} w߽>[{ofR*_0md&t)V! _*qtv\gdCysqc J=8Tuc$h$n9Ĩ\.zxMy/E_ C<Nܔ}HC76)猛čϙ {o>} 2Hms\WEmuIh2]&EJBRF/*MS1q%}g܄O3{i @#!ql:go:ßH~&׾3r{oZ\錛1Va|LK!)o>}!w+dڿx_WOS!gyp╿Ǎ&PJ*ghb~xpCD H"qR$nR @JH)7)E HTasRGBgAMA a pHYsodIDATx^eUyƟ1?{mvov[%1s Jt!dp6"PR9JH(DA(*^,)@yq'q;k~z\sZcϧosӟGsV_fc3010301030103010301030103x7gyyG￿B!Bh%o&&&϶ 0-IGh^_+B!B(y3y4y5y6y$<͓O#B!<3zMY>DˈBw$3;#wkofxB~L6n6F^q7Wڼ[Qskg[F|Wun~! 6ۮsչ|^mu.XnUmj#mW^2zmchiFSIsNeݢvzu^UѢc{hS~2b B=>;wc{.~U'kÚy{ՍǍPs3S?^ys_Σ WY?ռiεKα;mqsc5w__[.N'GNçΟ18M1-NJ՟b_Øxchι<19PufC2Սqhmci]oI:uOsTs>#O0C7dl63oɔGcnd/2Ihu^qBͮh Y̋LGs^yQ] c̼6ڤj{mxaFWS{W9mxY[his7ޕT͹7jc\XmPl\ώqVǪr}Ph"oH޹/XޘշL^UO:VIh<&{]q:&6\:Su}sq8e5ξ|!ueqzDf>m9')~2{׫mFe!VM:Cxycf^DCG7˴e3k 6zWoUL62&̹⨾7Q?6UeV|+b{X@wl:1WէJcxc8JUH]-ΏbKէ)^s޹~g~*6]fu|^rrync1ng1c粙u9OHY'b!V~|[g>p=bz}w43/{ձͷȋb7KqTqw\_[4EӦ8FevL6'a]Isc7+9(U/^\F^eU_cq;y˲r^n6X8n'Σǒvj1)׻,h'΍9oGyYe&91cf>>BosfhW19f>>!z}A@!~ˬ߱3lU};~f7忙h=Kcp]ڽgF^<  fUqwtj 6ޔU^m\m&6qS]mIQ1c4MimD36z;$Uh3hmGByos饗6۶m骫#343S?qڤ8tnf?rHQ.)Jcm24} 6z)xʦ]wu>ޭG!z/\/rG\}5\rI=P'y9yEOgyyC@m~L)"~qԮlug^wUV!j٫odd*/'Oo4=Q#B!Щ<<"`7MΓO(B!B'yE &ǟ-#B!ƕ<֤F~|UuƲyS9;Oxކ1h?7VWbifsͿY7nsf?3fׇyi r&}d@-_kFqjAq<ʹm̕Tml>(͟˳h3%]q-"C`֥kTuQxNu:^^MFe:/2{l]X'*QYcخ߶7|9?mj4/imi 6`ehormw&@yP,_ɹ{Lyql\C6͑ԕK[dq^6X11"lVSU^vDZ\9qׂqKq$yFDu.p"VбV,;<~1zOVnۖ+Xm2qsvyɉ7^K?kyqmVɆ37S7G< oF/J2m̼9]XG1ʱj?X>ܿک}|ֺ^sg)GU+n^}1z1WuMjMCq"Ku+w*=W8 a]w_q>3j'9ϷοkY878={|IQjc3Z95Ҷnb Ѽ}C^ujcY=Ec.k_uZ˞XwTdif^Fޛ3O>jcwU.+n2qsF2ob{:1ܮ}{XyJ&|l} P6I8s]m=^_7k:G48n+r{:B}IxI܆\ՇsS{Ɩxu\G>΅r<4Lu켜#ys*_w;1/c|&Jq"{u}tYT̹|Q]+W56"[Ku6|?U從c׈⻮9\ _'Bu+Ϩ_Xc>˪ZuO8U,̋_Es]z|7x}ěPXo46+nVYܟMmkU;忪&ҵTȐƠq{,kA('I{#{ݷ<^u<'jԱV*gif^>rF6 \S^Eomyi4C`yc֮z/E }ezա>deif^w:tzt2 [U!0o'jȼ3govmZfiH~:X63;􋂙_CyCD 5Iۉ2o4tSo@6^+ ։j5dޘi:X63߀ 1v7oo.t7׀U2ZaNPy?$ҧ>}Yciȼ֖`XϤG7TW\u] >xi2t>̛.= >;;Z*^e,ԿVMICMZ^hnK}^MCmYk a`}99Ma>Ds9cqc|7͉ |f;fe%Hu^4 2Z-`|VeSPt*Tliȼ9*NV4_4?E_z~vq~&_uֿ\Ơl35˿x]76۠N\~[3?j穧4]ryokg|7Ge-ޱjg4vF|gsŘyӵ9N S܈#Cղ "f^λaPу'?yg}OgR;DCMҘy/l]3?**<}2oZkYcבMq|<NV+`Gu4j~᡹8{|,)6棚;ooŗ~Z[/ ]@ߍc>'ϏAsͫ>?s"oˣF˶p?B՗s!h Ts8_)lR|,Kf,棶w؆x˚s8Y?f75ʑy!3_k 5gO6^reݑW|n3? ho(E3V8c|.SEW;_$!l|ug5G6&￸%ӟ~WmkfwRדCղ hU6F㘻μж15ʲ2;wuUPTͩ5O2ZQ:ﱧ'mnز]yU5ZzZSU1TfUd.e4eLɌvQq6rm-lWyWڈ6ʳW:s鼍m束c䱹1wm`zfwwտʫͧ럖kv6]p/ }G{9Kf]ޫ΢ly3/MMZ:cMs?IГخBc1֢sMuզϨGïvMyW]|TV=`O1k,u̼ j|C5P4f޴ie2K .ټJ6늱Iy*Sձzc#i:X2sIfCu^TFgd:ͨFWHe#6Ul˿P.Uy<69~Icv]0/6xj_rgԼx݆~y77~gTڐGtQ6L̝ͥw]֍3oє}3Muxy͝Ĭ}F|D:Q,<*/tCzemuY?h9˒YF=E4$7eyt;l3y.ut͉'n`TV`{l 26z/*}.Ω`ڸ >#k>sNш+=rQ|J9ǐͼB<:*S7859?םqE Yn~o_775O3jL a|f8P{9ƫ?2f8`Źj;*o+q?j-KFfTev]ҼʀH|GU&]Kaa#j6zu 5IcsTeE߷I,Uf3CG4^WƭJZK3F7p>:W`l(6OnfeeL͡Mmml&Zz2'3Z؇Nu~^m33WnEsʹf=6ͷr"|r]˹]/\߫G}bu'},^m/MeymB! :cMzܜf9_Ͱ _Hm9U|6SeisU֦1uZҿ?/̮4ԸVG3o釒MLYW{Y=hjI3ocd\yR~Y\n?'y^4e1Қtz_`l(2_Ud8UG5mrQU6r}p|ևybކcGsRh57~۫ɱP[svu9;_oBＮy~^>z]67]u3mwe>gTw2ˬ}XǘyӅcU,L SF>;y~#_+CU?mt*CpP_Cv1Z5Xzt>qS/ z?vomʲo;n_!j̛"f^sݿyeǹҫG&~f^Ҙby^46ߙ6|mkE}.G=5SUSU0v-d"wyW6mD3*C8 p\eY/ bh;X*o3CqԏM^UG?<֫;FW17*S<7>!d; (ݟכOԋӆ/r֭.ۘ)zX,r^n66zFrCc̼ 1OyQ\k̾+/$G H@h#zR5*S\k3o>3<א} Vc ʪUw5k5VIdJ˦UiI3oZӋ6h\5Fx6՘\UǚoX/!6v-"Ks16^OcTj:Xtg;;:\Gsm+T}Ӹ|\}\M F:y|ܗsf>8G_88#!'s9~qdu-4{_?|m WՍ/r78a~FEYW3}O 3odd6ڷ1t>̛.t wIF_]h/l|+C2o |1V8mEշ9fGEr! Vc GIume4>UfLF.e!늕5$Uy͓58V6ͼ1}Vy)Oodo ]@ߕC̼fT/]V=W;w9_,ҿfXh3D+b9؈s:7 vBhNcF]rT_h_yz1f4OTh='2qP6cfGϹ1q,}+sm^nԦUiAwb}:> 5IcMs/'S*NT4mզV?5?,ϱKC2ocPE#sdy>xߨNV7"3m6}@}422i"s0nq)V1ϑJbylc==rcqXrW9‹3е¡_n_6脽 =| _ 6 Cc̼t)yݕ͏` V2XSP4ffT5dXku GF3;ߊ#)cۨFۆreTEMj'}4xD4luvoETc2>V\?j|q՘}xy̭2m̰~蚍AwƠ`W`ަk:QCMvs< ] T~&1M u$D 7lyNVAũLb7 CyCD 5Iۉ2o4tSo@6^+ ։j5dޘHm:X63߀ m: V5$1o'jȼ3govmZf/y] T~&7mmy| ycs_j^{'(̓c;:or4f}eůS,xHMCcNVA1 7l9NVA6Yyg&[wW)gifN yyyyy437BhjGh5 ZF[AG-"fk'>h+hQ0!Ew5b#m- f!F]q6\E#h5k({;;w5|QRb"V*Hb#m-ʆ52G6>Q*&Bmet+jM{Wϩ<,">g/ 4s|s׻>\ݲ.ssV",- f~Ij0)6#p,׉?cl]DhUJ޼7EwM lY!P֘5.y}jWy۹.H2j^9~ȟ5!?kcT;6y|f\kYguϒBԢ`W Ћ?PwmmN 1tj?F|k<š~Ʒ@^fk69O̫OQ1;晥J̢^tN^^ߒRQChZ(v^uv2ӗ2HQIEhIT&@O6U=owZwƣkMWGn|gr ̹ĺZ:'ܟ%Ǩ1{8{S̘]Y=nww)ۢ1k)~O[kOf^>UsKR}}fF!BѢ`,t?DC^erKf_}?D"$C}O(nbX-񝆲ƬhxIMsަ ׳FA|?U&B}Ŝ7e+=fUߟ?W>tc,*wm˶hjZzkYk3sE8?S# - f~I_WCT_?|u^j'TW?N!ZEmn|l >J߁L;/}; uiWqcZϪk7,y[wfކ"U;q.rj+?s6zq^Uo^hMjsжޫ"?E/Y!Q1z I};v7W~s<QDuOyWؒiwUWWs_֘ϵ?^T<~|Ο) ԧE/YlW/)Pokv5[3˯BӔηgz*,; iAq6YgڊAul"s6"6 T#:LWoUާ3g;󹖴^׺yz֫TyAhZ|F[AaF[AG-"fk'>h+hQ0!Ew5b#m- f!F]q6\E#h5k(K3=!4Em|seif׿F!B!- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!kw4_vI"TiQ6涝[}(V1Wvi.ʭuB!BӒǶ\u.BeÚy}(ym}8jUU2YǛ_:R@!B-WlQ(.msժ{}<:j}͋_+ B!yyU6F>yʘ- f~{-Ӽ3CG8!ivF+'"+7t'Rc<ԁe.W߱mr,/j| $ij=zh-u5.sH\9fG}>ZL'+1+?ih<7R1湫cmKAׯMq^z6Q{2νϔ,3;u9Τ]W==*&B_*+w4g_zMs%42\Mku9ZaӜu{] /Y!t*a aY3o#ns,8nwm~imh]x-yx[uX6p׮UeyFg|ڎ9bsVcWRW~}:g|mεy IsvzU_0#;ZeCU)Mqb:vl7*da˿xݠx2[!d- f~{8c_?5F﻾i~ck1h= ^iO7;`ɔIfjU&ōfބJUlR\+c֭fPyk6jfG=43/\_m۶YW_*ާ*NdwӖ!FMXYw|ul)G]Sq>^կ}N)w;gә|n6Lw?}KΣm>uc5бǤR̜lfo\qu3Chuˮ4}Qw|I}Ά?Y5]eh}(%2}TYFNPF-5=826n6RWǎWطꫝ˲1c}_@TqbYW[uW>'R̶)WMy.=O}xjշѫʻ6RٴGiՆ]nںZ9֭tm\犑׳{̭:g)^*W$ʯ1{kSט$soS3w)9FU\cL;ǶXN%c1'nETsw3>f?<'3[8nsz/_+fﾖmӢ`旨ʬSͤڨUjsޫd*E3 ^Yd4}g؆W66c{F9*]7+<19ӫcOۼͣ?|F^<&S]IUPm>y}QmLTWVeqMf7h*ͮn+զ9NϟG:ϱS5&~?W2nSgt[S|M,9ɹQ竒ڏxSyT}Nz|}ԤGOy;of%XXy3DUfOUJ2٘K2j2uԬXʼjh|'>A&}lӫ׶,qurގ_Us"s=d3R$Rj>|1I*reN7*+UODtnf卨ꩾs!W2]k6uyM:?7ZY(VU^mr[|qLu2Gorr_Tl׉}(9Nq >;3Xg3'C'|P3DUfOUJ2d^ԉFMNul6ccwzUW՗M^}9xrU]R_*SݖgͼW_߮~G%[x9sV~cFhz䍤 b]1uYFV9gǓrI6aqHq}[g!kVkf**O*S˯oܿڹB\Tc>b\9Zm`81V߹]LA3vz^u1: ($̣!ZU>Uqd)ɬɴe&g3*3;ժvL:^4Y6}<>+y l:c:N/gX+cۖg5&|<=\~,׫߷y[9麩qsx!ЩS2mU@onGȣO.jNU"(߷9rDm? UeTYL y4[M| ژZS*G(>?Fz|_K{y4DagLڨmsպaҰI8f13Ӓ%3Vf!BjQ6Gk3~DңzDqTy|4zq2G!BhZ2oD ku}Qv-T=Gl~}楟eo>cV:k!}ͣ,{-Ӽ3CG-67]~IigWno~t;oSO{Jq]PsUY%{E[o= ZZ^+Zˑu5.sH\9fG}>ZL'Ot1ӕ+ύ?gLy*7yTs)wJm1ڈ[ߵ8),3;u9Τ]W==ce_"y;/9kO]yyi~Ʀy 8e\æ9~'/ ^hs7}qCصAfw~qňs݌mu7x;v  %uNo^E^s>혃N19ulؕʯo"s?ڏ\+γdwO+ӫAعBd*'cx8D.220/^7(̼{ f~X̫o\{Lc9fmN@m~d7q)IfjU&F3oBE~-hZĵ2f 6ϋ^#6:_Q>mFm]ƭ|2}cErAyv]8W]TuK.!ΫVܮ-붾{_~βͼ:ܑU_"YZY_9\/iNi^kUcּc0E ^ *Uya42թF^s2lj[Y.cߛ:n3m}UQS1twŗ;+zjW=W5qy5H!sm)s:Tf:ǣ2a*Sc_?Sf=GJ/ʬi`׏1f^xy<ޠVgSYFm#ͤ7S=QݍqHe^[22Cmc(`Ɵ5#Fպn[ymts屺ՋWbb}FBC/ȹXuD_Keu>(vߜ}>c*6QWgNf^~l۶mE``He4~Ǵe Qa_,Jn9T6b{W>9<.}Mgn+Tsmuяeuyes]X_yy_qFV19@mk0o0ҷk@k3YT= kYZnV&AM^ju눶܄εuůL\+G3];ml_:`QU3D*ާ,|9DhddFu ǎm[N#Qʼn9d<<=&뺯|"ϗȹl;>KUxLqzoTHF*hjiGiTv&m uZ93C׭t6s=V3U9]=fGטr]>bj|&I5TslT(E9ͿU:|7css㸙׫ Tf^1l3`0K2}JIQU U)ljahd%wKhc_nL Rد\4UF}蒉F^Tc9U`(}Oo6zjcTӈߊi&3O غeyV9NϟG:ϱS5&~?W}scY;ݖT9_='9WW6>|U?>G5Bqeݏ9n[,4n$T'D2"Mqcsj['UsTУgoο3S.}6z 7t~2`U`He4xGVw=}Q96M1̻^u:V_6zo\]WyvK}\Ow[yL"y+!}ȱueпW:9g|O3Pz䍤ںHnI?a׊Wۺ,bۼfVm"yϕ8\?*S7__\Osz. *cy#νk|Z1~S1Gv %&,g x۝zՍwp96ۯ1?:*/ʬi 6m6j2|62sSng(/㌍anϫsnoC2N&?R}3m˳qzuOWms)DZž*OQ?ɱDV~8u=ت@onGwߓyczdeVņ5ܵ%Q2}Z%ln&*CVؘluv-?>F1}.alX3ك3Sc6jܰsOiۤdyݙn-dfuGs ڎcÚy#G^ Gmv={{7?xG`pcaaͼ/Zڮʼgȓ͡ï6׼qgw׊b|Cg-O^y候_"yϺ{^~ufw%[Ӽ9I8팓͏{`~t"6U,Q~rsUY߹_Լ|֔TO֖̱֔>#qM1j}3ϫ<>5LW~bh<7R>?cbusWcmK ϿWjFM$Oƹ]LeyW3Ϲp&L,{֧ќ}5ٗ\|-gM76O)6Y;Y}OBø¾ rD3>6ޔwmO,q;' M3l~6.vƵ~w8t>=uX6pw|c*򺍨g|xmvϩc<ƮV6]eWmr}~9iF^f:ojM.@uSfv?i6Wu>5iCW~^|sW"v*kk[s\cWc8:=VXu|uu.Gqe8gU:r]Ǿ~z>_"YZ̯h?f oPͳΩlFnܨkCrݍF5⍧6kKh-uX85Xez͟3ȆQnmmb]s~\WL;|Q̜lf7j꺙ŗ]iQw|>g,E {WLzbhild eԢYѓsՉvGs؆Fkb±cjV}9c}_@Tqbc-O}+o}XW**y8Oq* ` &22fWm"T{7Xl+ i6غx5WkIe[yPcn9xHIʯ1>$S̽gRlsT3qNb1ʝc[XB.}ɨnEr9;ͼ^uNe2ey_"YPL_.$TmبJ fh֍ ^mX%wKhc_nLownq+_y;ȋjL:rm>yUQw,Vo2c~S}vB.Z;m;5\6d:>:>};3Xg3'C'Vf~TfOC!wdz'5hm6ccw" uul:;>ιcR7*4c7}]_oc VL(gr\9g߁ [Vn}>#g!kVkf*o\l#2e1ϪH_~}<ر<5>QWs?)iCksǘWt3tg{Xƻc8 ba`He4MIfM-5>Q9ݩV]q0ge|տ9!UMswLs}ضYI8_:O+q_qz9}+']7T~9vV~8u=ت@onGwߓyczdeVņ5ܵ%Q2}Z%ln&*CVؘluv-?>F1}.alX3ك3Sc6jܰsOiۤdyݙi1qwl=63̫ѶhhU^>;XG魜2l=030103010301030103010301f<{!B!B#(ܙyyyyyy~ˮ>IfÚln۹wڨzp=ͥ_S~9pLlU'k;n keʏym~4Q2YǛ/U"ÍƆ5˾(ki*u#O6_q7o1_+>yQf~T=i^xՙunNw$}3Nzj+7?щTGF>UeUQokSOh~}?V"ZS.S=7ZSZ[.2n5c|磪D>~D3]H 5]&1qj.<_-Fq^z6=^t3eeu_<™K2񪧧t L0K2YrGs4g_rM+\=7o4Ͽ>?+~4gmd|V6q7}6g|'m)ڎ9SS6y]y+:wQۜky}W)/}9W.nWi?sqU)NCǎq\1u3eda_nPWnn9?q]]4q;־y0j:mq}dUߛ6 F3oBΫ<[qx=jmE ?/z|UGu#~7fbmcQ(u\uSU.| 8o\[s*8jrq}:6އpGVf~TfOke-f~-pDQ͆vj5/^ג_6]eWmr}~9iF^f:ojM.@uSfv?i6Wu>5iCW~^|sW"v*kk[s\cWc8:=VXu|uu.Gqe8gU:r]Ǿ~z>_"YZ̯cͼƮ|`axZmuNeC6zmuFUe]jn 7o<յy0^[ZmFkk2Q(vߜ}>c*6QWgNf^~l۶mE``He4~ǰI:ٔF\Sqڈ^ԟxs6Pϵf7ױ 6ϗb("<31/뺾6>2yN21W9q,{CnS]G7rqcS,hcmumzՆ=nd*dnڌ_\f2vѨN^hM\ZWżq+>3߅C[79G׼r]2~qD1sw7|fZD3s_vuF  5]e0M0K2}ͧ͢YLfN&mTGFxدpؿƾU_D ty={ܪsFb_hѕ_c}tI({ݥ#g\cL;ǶXNr]1Q Qsw3>f?܏&0MkMo4,R ht_!-ky}6;k׭b5?7Ze*8H_~brۃ/8{2u>Bqeݏ9n[,4n$T'D2"Mqcsj['UsTУgoο3S.}6z 7t~2`U`He4xUw=}&f31y!b_Ы]WRk#g!kVkf*o\l#2e1ϪH_~}<ر<5>QWs?)iCksǘWt3tg{Xƻc8 ba`He4Mf01:Q6ݩV]q/ge|տ9TswLs}ʼYI8_:OgH>4.kayp?m#|uMUO$PcKEƩ뉀E &VEzu;$]̫ӫ$#/*6m箅~(ڮY*fs3QhfCwgkAwOx`1ƏAw/dÚre#Q{J&_%s۝j!ê;:39ٰf3~ ʏ'GG/k<~>;U`6'yyyyyyK3=B!BZL <L <L <L <L <L <lvGseW$Uag6;GImv=aүک[w?8txwj}7KVdž52G6?ڨzPo~[*`fcÚye_] yϺ'C_mv߿ygY雷ZHs(3D*uw4/yKM|y׻s>p'=5uۛF{%ǞR\6'\UcIQok֣֡׊rkjMim<3t{ַ>]ct's#}kMcn8\ y]R[6wm"{2νg,by΅3}dUOO``He޳>}KiξSW^?o8{o߾i}~NWi:K}M_vm=f\1\7~b[Ǎsr>4GkCkkko8{ŲWmD>Ok; |>SL}N1v)뛯O9׊9]<8m`PG>v:<6ʼn}ر=<<8~2̼ '33D*ާ_?1N﻾i~c־y0h7tj#fLM6U27y*k@"1Vh=T^Q]i3RmZG4nU?o|+GPʳŹB]r q^M߸vUnq\3um}׽< zZZZ {M/'&X3Ҝia/|`xWmDU^mLu*}z3#oj @&VxmVXy-Lw[_lTm ]z{_ŊUm5Fϕs9F\^=_Hr{"(csATϔY3'0R0K2}Z+cWE[`a<6zmAm&t1n Gb,/&ڒn#G6Ls0 խu[ޢMSms_+/^渪è,g57Bu_j8/ziGMǪ$X*34GLsGTY͌O2?s26wd۶m3U,Z3D*ާ1Ȕ'> ֱͦ6:z>Fl>Żm|n6uc^8xC{c}w=~]ɱ219@mk0o0ҷk@k3YT= kYZnV&AM^ju눶܄εuůL\+G3];ml_:`QU3D*ާ,|9OhddFudFBu+WطꫝqGEKs]sxc_myzLcu_|87SܘQ8&s1q뽏UMdoWwm4E4Ig i6غx-ڜVeֿik׹bsk\D[kLB9ۈ}.V1H5>$c9Ŋu*"ǜ_*S1{cqU^Q*3Z0%R> ͤڨ@۰ QѬMի X:wh\Fr[mMg58gzU}F^TFioGuw?yyL:] osFOcjcQ][7dy~~![6#,@?'Ciq[9sʱď1o.s6<+|R0kbԷ$FG ԧ/G__?SE?vX3i 9fN5%R> E1s!&[GZ6UhF>܏1PkMo4,R:x*W=m_ә\77Q\y?e~֭xgCmѱq*v^MsF}tŪʫח_ny /^LuP\pz7ϑoE˨O*}(9 ՉǦŪ̼cS؜֫ugt3|~/gc>nͼ X%R> E,ޑ]Ohd(9ٌmb!f}W|/cձԫx8Uʳk\KRxcXy [Gn8R~/oU_T?4c1 śͯ7NH mڌi9;6aqHq}[g!kVkf*o\l#2e1ϪH_~}<;9ܻ'u}G9Ÿzs4m`(q^r|mbyܮ~yWx7} gsaQ\<z`)f&g3*3v61w;6*tirx/W>Z۶<1W6Fqۜ}+']7T~9vV~8u=ت@onGwߓyczdeVņ5ܵ%Q2}Z%ln&*CVؘluv-?>F1}.alX3ك3Sc6jܰsOiۤdyݙi1qxl=63̫ѶhhU^>;Xqwo`yyyyyyK3=B!BZL <L <L <L <L <L <lvGseW$Uag6;GImv=aүک[w?8txwj}7KVdž52G6?ڨzPo~[*`fcÚye_] yϺ'C_mv߿ygY雷ZHs(3D*uw4/yKM|y׻s>p'=5uۛF{%ǞR\6'\UcIQok֣֡׊rkjMim<3t{ַ>]ct's#}kMcn8\ y]R[6wm"{2νg,by΅3}dUOO``He޳>}KiξSW^?o8{o߾i}~NWi:K}M_vm=f\1\7~b[Ǎsr>4GkCkkko8{ŲWmD>Ok; |>SL}N1v)뛯O9׊9]<8m`PG>v:<6ʼn}ر=<<8~2̼ '33D*ާ_?ƚy]~i~c{ fmN@m~d7q)IfjU&F3oBE~-hZĵ2f 6ϋ^#6:_Q>mFm]ƭ|2}cErAyv]8W]TuK.!ΫVܮ-붾{_~βͼ:ܑU_"YZY_9\/iNi^k:E ^ *Uya42թF^s2lj[Y.cߛ:n3m}UQS1twŗ;+zjW=W5qy5H!sm)s:Tf:ǣ2a*Sc_?Sf=GJ/ʬi`׏E<ޠVgSYFm#ͤ7S=QݍqHe^[22Cmc(`Ɵ5#Fպn[ymts屺ՋWbb}FBC/ȹXuD_Keu>(vߜ}>c*6QWgNf^~l۶mE``He4a4j22رͦ6:z>wl>Żm|n6uc^8xC4QX_yy_qEOX} t1)ܦVcF.n #} 6ULճp6Y E 1VkUk2vѨN^hM\ZWżq+>3߅C[79G׼r]2~qD1sw7|fZD3s_vuF  5]e0M0K2}b66r22jѬ<86T7kWطꫝqGEKs]sxc_myzLcu__m<Ŭ\gnCm򮍦6Fq]1ILa9m[E33tLW7mm:Wcn9xm_hѕ_c}tI({ݥ#g\cL;ǶXNr]1Q Qsw3>f?fyAr,/#IPm27>MrQ5u3y#zu>oȕ0Z;]kz}6;k׭b5Tk:gGQʫ>}嶊mU8Tŕw?}V侌Tl؇rиPxlkQ̋<6q,9}o:\W}AW?SA;y9L}6֪IU_"YPdYFMNl613>D+zձX}tw}s*w]5.rotN}C'`(um~iuFRhvWLm]f$LH W kEmzv? m^Z6S~JdeOQ)yV}F/v'P=ر<5>QWs?)iCksX}v3tg{Xƻc8 ba`He4M65>Q9ݩV]Me0ge|տ9!UMswLs}ضYI8_:O+Osq9}bKk뇌SPL v4{=9HW;=?WIF^`UlX3] PR]*ާUf2n5VwmׂN**cߗ^VɆ5=83>Fj ;MJ6םiߝS̬;[ <cÚyX;c̼mKf퍀Q} f`끙yyyyyy43C!B!E<L <L <L <L <L <,kw4_vIX5x`s]-߾sFm׃vi.ʭúC-`~fg:A_q`ulX3/S~kᨍڮyϺ>|ȼn 6l66]EYK۵Py9tf~60yZQ~謅 >70K2Y|wO˯ {Xtww='qSS_}̏jT7O=}`^r*unCqrUe9t5o6;hj=zh-G֖̱֭֔>#qM1j}3ϫ<>5LW~bh<7R1湫16Sͥw+h#K^o}&'܋~̲̼+֙\8wI&^T f~T=WhξKi>u󖋳'qS'za aY3o#nu3'q;' M3l~6.v֭W,8;1{yF3dcc;ԱMcW+Dk?js:Ϟ!U>s\N c_ cS؇cgH/̛˿xݠx2 0K2}Z+c-f^Ni^0h7tj#fLM6U27y*k@"1Vh=T^Q]i3RmZG4nU?o|+GPʳŹB]r q^M߸vUnq\3um}׽< zZZZ}FcQ3{WO>V6]e*6FT:ވ>k7] Mu+6+e{Sm|6j6r|bETO*Tֶʹ#.q/$tN=eU~]W[xױ]9sVs*ugʬyX)%R>x?|_3js*kmFuǀ32 nkKFfȺ~,S2ϙbd(TZmy61O}s~\WL;|Q̜lf7j꺙ŗ]iQw|>g,E {WLzbhildbj|&I5TslT(E9ͿU:|7css㸙׫ Tf^1l3`0K2}JIQU U)lͺahf%wKhc_nLoϱT_maoG~4BtNu*Jӛ箍4"nẔStB.ymFrY^j#Nѷs,cc5E\myVnN:aĨoIU/8_j?OQ_,Tu=~q?5켙>fyAr,y= :?*0K2} Y#׻>Ѩ0 &b݇}B:v]/NʼkhڮP81V߹]Lo^=VnFSFMfTfNwUl>WUn<ކTe61M/gX+cۖg5&|<=!9W?PjطruS=!@?'9 '2[h{rTwQ3vz|_L ?ذf6jTfO$DeȣjTݝڮ=TUG(>?=/C%< k<{pfu}Fm׃v) {Tl3;Ӿ;-48.Ywзydž5vƘy#V| |8|>}^2l=030103010301030103010301f<{!B!B#(ܙyyyyyy~ˮ>IfÚln۹wڨzp=ͥ_S~9pLlU'k;n keʏym~4Q2YǛ/U"ÍƆ5˾(ki*u#O6_q7o1_+>yQf~T=i^xՙunNw$}3Nzj+7?щTGF>UeUQokSOh~}?V"ZS.S=7ZSZ[.2n5c|磪D>~D3]H 5]&1qj.<_-Fq^z6=^t3eeu_<™K2񪧧t L0K2YrGs4g_rM+\=7o4Ͽ>?+~4gd> m/n6m'N zS޵yT?\q7c7ͰкZך=t׽bI_٫6v^}'s>)>m]WDu9׊9R^>s\N c_ cS؇cgH/̛˿xݠx2 0K2}Z+cMsMXcDP7;`HVm3"n4&T㹵lak׊ףVD͠׈ꪍWuOj[jrP]/Uj8U]Ÿg`jƥ>rc溭/׮l3}waU`HeVbbKo;flepX3yi>V6]eWmr}~9iF^f:ojM.@uSfv?i6Wu>5iCW~^|sW"v*kk[s\cWc8:=VXu|uu.Gqe8gU:r]Ǿ~z>_"YZ̯c|e~ca<ʆlƍʺ6*xoT#x1Xk ` R׺exZU\9SlV-o&o:cu?9r0*9t͍Pׅ+ ^sӱ1>4|;9Q9}3UmC3ïϜͼ2ٶmL4/ʬi 2e;iGǸwcM)>ڝVh23ޫ_Żm|n6uc^T9R XDc}w=~]Ǚ*>q:cRMu\FlT2 X:~[r^aY ? 6ת)6y]4ꪓi#r:׶2q1|ʯwVy5o"Ϗgs(Ə*>~\WL;|Q̜lf7j꺙ŗ]iQw|>g,E {WLzbhildeԢYѓsՑя Z~c6l\yԟc* ` &22fWm"T{7Xl+ i6غx5WkIe[yPcn9xHIʯ1>$S̽gRlsT3qNb1ʝc[XB.}ɨnEr9;ͼ^uNe2ey_"YPLZe 6:' }6R4 5uc;6zauKhc_nLS]W_z!_yK&yQITWsR osFڌmeLܴz6ui[߹,@?'Ciq[9sʱď1o.s6<+{ERՍTĨoIU/8_j?OQ_,Tu=~q?5켙>flAr,c<&r*B˸N\ܴVȊz\M=u^1|2ZCZ[.:lw=?:6[k2n\3(VU^q̑r[_xq*e|ʀsܾy|+r_F}*PYC9hIN<65(VeE8>η^O>砫G<ߜgg>k]ql$@od/ʬi(2d^zD&Ci'l6ccw" uul:;>ιcR7*b\9k08/9N63Ϲ~(.fV f~TfOcєlQӝju;@xgl sx^}VWp{Rt:w49ؗ+^am[՘ԏիxr(cy[9麩Kk뇌SPL v4{=9HW;=?WIF^`UlX3] PR]*ާUf2n5VwmׂN**cߗ^VɆ5=83>Fj ;MJ6םiߝS̬;[ <cÚyX;c̼m+? hU^>;Xqwo`yyyyyyK3=B!BZL <L <L <L <L <L <lvGseW$Uag6;GImv=aүک[w?8txwj}7KVdž52G6?ڨzPo~[*`fcÚye_] yϺ'C_mv߿ygY雷ZHs(3D*uw4/yKM|y׻s>p'=5uۛDTWmX#XUFr窲sϿyᩧ4_+Z))-uc}F1c>zQg"W?y|kXyn}rǘ8N5Aׯ8/y]Hs/22ïXgsLz%xS:X&'yj~//ͯ~ojj?}KiξSW^?o8{o߾i}~NWi Cm/n6m'N zS޵yT?\q7c7ͰкZך=t׽bI_٫6v^}'s>)>m]WDu9׊9R^>s\N c_ cS؇cgH/̛˿xݠx2 N2You1gWnn9?1G~ck1j:mq}dUߛ6 F3oBΫ<[qx=jmE ?/z|UGu#~7fbmcQ(u\uSU.| 8o\[s*8jrq}:6އpGVE~?uԀZZ Z}cͼze5]u(^K~CwU^߿礍fѧ?3{͛鼩6mdSMet"^2V<-Lw[_lTm ]z{_ŊUm5Fϕs9F\^=_HrXc_Dס2ֹ8ulyU9u2a>fV`3/Sf~c7}cyP0JA6:!q u7^Ո7z <-6Եb(`Ɵ5#Fպn[yv}ۿXݿhz\1byv]s#Tu¡fyt:yL2:{Nxo>|yALe{̨+3'c3gLm6SŢe0MFyTݡ̟{Σ2{ьꮮm6b*Nޫ_S>v2*󹶺Ʋ:s{3iEqtcr:dȘTfAKocTn3_ի6q# P'ufZ56 E&WF]u:uD[nBֺW&.敏#].6jMqlZ܇O*v{瘏#sp̿[W]73"y;x6/Glcea*i2KcOfD'3 ̜LۨlCu+ c6j'r\s{\׿2W[uW>G(b)f괵U>[=ouTȘ^fSG(N܌g\b6:Mv`\}.$[mmA1zzU>"*'+PN16bKG1R9Ƙ*wmb c1'AAf|^xcw7z9_+fﾖmBf^o`OͤڌIBe~߆j65uc3黿6Jz:<W(ǾV}9oSY)Ι^#/TW!7귊6By}rF^T9UU9@osFڌmeLܴz6ui[߹,@?'Ciq[9sʱď1o.s6<+{ERՍTĨoIU/8_j?OQ_,Tu=~q?5켙>fyAr,܏&0Mk j4,RbU^/U<_n{gq\G( Q=Hη"eԧb >휄cS]bUf^)cylQ|3~ zt>q?|vf곱VEf^OOfDd]WfL0D&b݇}B:v]/KsX宫<ƥT.ʻ-<&ͼאbSR~/oGn+skMc7BjsLUsڀ?;pҐ~aa׊ח֭u5Y,ĶyjLE+q>~DU,Y˯oܿڹB\;9ܻ'u}G9Ÿzs4m`(q^r|mby~yWx7} gsaQ\<Nd`OFSfLfTMwUlEcyY_vm:Ufcb_x}2myVcR?Wb˸pB}R?1I%u"D"T`"}O.jNUX W__F ᶝ6jTfO$Deȣw ݝڮ=TUG(>?=/C%<ͼJ<{pfu}Fm׃v) {Tl3Ͼ-48.VՎwyɆ5vƘycV~|=8|>}^ɏ[9Sf`syyyyyy43C!B!E<L <L <L <L <l毿֓kW%f`iz|: 3~ˮ>I6ۇϙ0=ܶsWs˷%Q{Krkn@sy 2߹Nv|ȼn 6l66ye_] yϺ'C_mv߿ygY雷ň7<|n`9l3Y-Ӽ3CG-67]~IigWno~tQ]bbU}˝ʪ=歷ߞx>|hE\zo\ek:njGU[|^1fCc幑AkMcn8\ y]R[6wm"{2νg,by΅3}TOO`tyif5rGs4g_rM+\=7o4Ͽ>?+~4gmd|V6q7}6g|'m)ڎ9SS6y]y+:wQۜky}W)/}9W.nWi?sqU)NCǎq\1u3eda_nPWnn9?P]?zbah7tj#fɪ7m@čfބ Wy< 4l-Zz4ڊh=T^Q]i3RmZG4nU?o|+GPʳŹB]r q^M߸vUnq\3um}׽ьx#gfy37&lj⺩NkZ]ʟ޴qn:괍+C/9_~XS r1Lj˫Gb Snr:Tf:ǣ2a*Sc_?Sf=GJ/ʬi`׏ft}y*<ʆlƍʺ6*xoT#x1Xk ` R׺exZU\9SlV-o&o:cu?9r0*9t͍Pׅ+ ^sӱ1>4|;9Q9}3UmC3ïϜͼ2ٶmL4/ʬi 2^;La4c22~LfSrs=SLi#{Eԟxs6Pϵf7ױ 6ϗb("<31/뺾6>bf>8:yѱi Mu2ōadFOu*1*Ui3~ql"j׫E:y:-7smk]+Ǒ|i?o5L8]&~6qbxHCx'uds98fx披jͼ]|՝Fu#w62X԰w4/ʬi(66&?hddFud} `[l:;oW;*;D'<ƾǮ>rΑ<WyEs>DFlM6j8Bqfbj|&I5TslT(E9ͿU:|7css㸙׫ Tf^1l3`0K2}JIU U)lj_hh%wKhc_nLovِD oDӛ羍^u|7Rߙi&3Wm7^.FS:GWyGy(`v̯֐֖˵Ƽ>jsǝwϏV̟j-W2UyWs/V|^ q:2Go#9ߊܗQ-TPsMu "UyǦ89GW9g*|37_ٙZu}>y= :?*0K2}LW׻>ьP{__}&LgĴ O(i|Ѐ540M i8e>D@WLT"3 |@ozpu{{Y^ɕ㺮H،}1b1Dυv[]+~kV*Ͼq)eCn$׵wO$'Cυ-LxOsrU<M_כ&o$667ʙ^eڀ+s8٘kEח֭:lb߼fV-"yJdeOnQJM_+;tj+򋘇u_!XEДY f1&Qg1*'j~y0X\1+<ߢSu>M/R{xmYIq|+ƕS i}1/gv#?'/?GX/N}o,C%0+Ї9=wQˊyczIK]+>zl(Xu^%x-f}COgAOO`E,z}_Nv?}L)">wD)[dy=}h1qIꉸp!yɮ}jװmWwQ식D@17AL <@L <@L <@L <@L <@L <X?}aaaa زd`b,-z ,-?ޛb=Ov?mtwUwo>XOđ]}&EYJ{,ak7K3OE8uy}^+y=K]'?>/qRl`){ y9m3Z'apq6b^nO =m~xoJapq=ġG޵b~7qǺB>; nCgޘ؇iyXwkw^`=`}Z1/Q曋uyQѬPXT=ϝ~?tg_s `{` lkZ1˲P Jg{sod__>d#'E>>٫/Հ_{z{fؓ{#VxO/nwߝ\?z'ɼ|-gVէ.mw~3(k邾rի.$ {N"ZS;7^o:of߭ur>c>:wy4y.W<>7L_~b<7xt_}WosREކ>st8/ΔY~g>W;kU_{qo{nE]owǿu'^B,;; 9X?eaѦ1o_ fɎϾ7qS8 w:њg.?6Tz/-uE5s>V|OVjP^q|{0Z[^-׹IȏiU}}cO|䵬YVޢ'ٺoD_~NwGř*O376/يż*9p*i_E"Wb[Ou݇oDun#߹h>x>n{ k >Gј pɽWYˇB5utY,uSԷ;L!γԶOsQ{cT NS_ekg\6{c&V~uqil(FQQY{a ˣu!u~Zy]Uk8>Y({H)rM,0sTW7$}ۚEMq,ECxmOd9|^;w3"y^!> ň; f,+` W(,*>dcMX0Z^bLG 2 C 3--Gd!(B?]}΢?θ61*qr\δ8;Aeٯ5.y_D}_UT9eqs?uݿn"ߺ_z 4o6Fm& l(cjpn8L^zZgV߸#Mܪ2#1F$ǫr-} f?Dߘr1oň9F8'1Sα+P˼fs2rP݋my1t.,߱V- b~¢C6,t/^Kik4~mt.1}I8O¢EG8*ψǑp{E<$؟R1ȱ81\#j(E5}>2ܗz ݰ'~ovY);ƻH4 6_ EJSm6ڄz޷touָۛYƶo {>ZyOH򛟊VK^㑪$Yc_-2nYUQ ($rkBx/GPW/ueW\7֛gb.6eEJX$$$Di: HW&Tq*i8ۈwS*}hn_Wq 15GX:ls5wQqTsm,zIXJ6)`3Vm*T?̟_cYF_4Q?<jxWgof2lξۢC~l,(^ߟyƵS@P:&ͪ(Fo"U}W w/~P[#Wq~9ȱbʷP]4n$&^3W%Eؗ?η'Uxř zu>?q?7|}&곰Ve_b^o X)&XD&H:Oh%<~B3 (^Udj8[Ys^ꢠz_}̿/b6aUǘc^s/ԧocvq>'yT~EW8p{=FKHB" nϞȇl˭<#AU7ĉۂ9Q":o~skYМ_ntؖ`fmcMve'].zB&!uټbShuj,/s(^uo{P~Cv<c<7>QWstϑ/2]V̫^ӫLB^`]Z1cKGI}w4qCM*>dK "F^g ,| v{nC w ܖ5~i(#Qoa#lγQVkf<iq|~+?OjN~(^tzjtOPց_m_%/d׊gD/bꣾRLwIn{Eͳi*T *܍cpKжiyVyߖnu>A#C.vٸ\!vV_4z#Qz {$j-U]yWqeRbch['xPhż8?2]G֏y/|Y#visnτy_Oy6V>Oݮ/f][ mm͏A qS g螋v a=E.ǚk's$f]iy)9>a=;? [1ݾly` ='19(.h=An b~&7|) ȍx仈Ź =%//ӚӚW =1Sm1حn㙏qHXDALļ72X?C*ļlfbk~=y\-+ּ<17~@>1?-+Z3y}Ĭ+Zb|10 qy!޻𞫐ml޾z|ߓxc{۽7F>6!E1 [{BbOϘI0`17}2W a>1/< czmˀy219 0 0 0 e<@L <@L <@L <@L <@L <@,ɡG޵ͮϜ>z{Ǐ-dꣾ;}GOt|^{3so{yc݁y}ZkżDo}~5Qߝٞ~T;^N$ذصb^Oٗe;}C%޳= ݹ7?}{Xq=1.^}o̽ {z{fؓ{.'>-K/mw~3G/DOk6ߨ_+v 98We_k{wGPkEk9ҷn\eݺG>c>:wyj}劓'ƘOFy7yTs)wZ>Zym賉8/ΔY~g>W;kU_!x;tWrwwww_y9r}]h"~'+bM_mg.?A?f'u;' fhmh]x-yx[]u/_p(򺍨goؗoh"jsԌ9׊9]]6}u6FT6ވx1ț>Aɢf^@V^ܴ|%[,Ԧ5νz|+_S յVc\9טc#OABe~YXPVYڷw\c# Fպn-Z}bcNs|yv~.ˇQ]l?4Ρon8>_~^9,&I X*1ts#Cs}ߏ3UV-ogB/om^y;UrU,[1B*>d Ql>ݕIF&q(xɽ2Q۹L>EsOyn2՟=&T"x#7^kdS,*\ g%y_jin#Z ֺW".敯#} >Қ?o5L8}&}qYH!<~lj'}uD>sw; jżzWBbl }u0M+C6EOc!')Ś8lq}cWy\tn_:mD'dciT 5\_QL-,TeQ fcBO-xml*;W(c,U}u|14w8-?鶩M%gbssUQňUQ!1@"}mmTu9ǩk 9orqD35'cQıgFK1|1C˸M̛VWOj t&䍨کsTyސ+0Z;}oz y'W}㺕&M2}BjC}Ͼ(uj|J;dc Odu| eO1F;QnkŲO}sתw[7.RlhQ9=V[q0Y _w?tirx1ߐ`m+jL|utϘ[14.#?՚[9sS;!@?'/?+9$XJ`W$s{ Lm*XV?|RQR *>d$ͽD%ȣoT=n==TuWj"} >:ٵb3Q싘Np`oگ"dO%1' ~1صb"b^m- hU^.b_x}[%yb`b &b`b &b`b &b`b &b`b &b`b &30 0 0 0 ['1011011011011011$zX7V̟>s{?=ztϽ1Sǎuv#kǮoxG}wJg{So{:F`^c׊y=e_xԳ/tx;.c,?|d|ǧ?{R1+ߚ cO>? tا.|[SqS̯jToݟ^<=|J~]-'\U}ɮwߝxCGHߺ՚r}^wk:Ѫ{"+Nc/?1WY"湯1>Sͥw+kh%&'܋8SfUb^OknL"^VV b~T=W8]}7}{=]wo q/X Shs7}qCطAff\>ynQWq7nau5>nu׽|Yɾw6~^<ڎ9~ԾMc_/_R3\+γdwO+Q~Z?0W\{~-DƎ"M~b ]۷Ǜ!rgHHB̛۾y(j +C];"b^Gu:|]@׽޼yI:mqݳ'gVM&QŪO_ͼ Ula"Vh=#j>*(jkwQUOf(E+XUB9(Ͼ/U/?U[f?0y5CRUűVs݊3uV-}< m𣓋 vka,~^jW'jhNujS苧^,jƮ[Y >aM˧[*BMmZcܫΗQ;P]o5Fϕs9F\_=$T܎eU~}>oȯ}sAe8g섘?"a WH%ևl wyѾy)A6*S]kj3鍦TgGO%@?}ن|- 16N~Ok{u:L>`j[Vޢ'橾9ԩ?ΗjWb|CFCsȹZmP *O=q<=4g:98g8Seb_v&%[׿_%YŲu0M+CeUm ({7-6e2s|O_k)8x|̢3sYm7c=qj K>Oƫ<[O?1/u[6C?яsT@} `FfvT׆UZYdjjMfQ^/ u4܄Zk]+ב|i͟o&syy~>8Gby$Ds[>w:"9?Nwb^eoWanB1΂?Y :&R!ŢŧPbMBO}FB? (+6 k 7V{ Wy\tn_:mD'm$B1B*>dcp\HI~lCI$JO|q,hu{_[تTu9ǩk 9orqD35'c1i[3Gqz1T ˼iU]|u^Tv MgrLވ;G p<'~֭xPG{b~7[k>}Tk*3'*7_+y+Ύ~\Gȯ㨝s$[cŔoC9hIM6gJ̋<6/9oOn>3|~~oLga˾nż:.+C6 DV_PIPFhDcļc+ uV,N9xzU}R,ˆS3Id1k5&PпW oYi>7 0o646y#)Al=S_׵iOVdq}֭xg5j17W" /|rU,YŌ 74_~n'sv c<7>QWsd`)BMbTbNO,Ea0cbVWy,:}^7$X+aʳ8_ӗ?#?՚[9sS;!@?'/?+9$XJ`W$s{ Lm*XV?|RQR *>d$ͽD%ȣoT=n==TuWj"} >:ٵb3Q싘Np`oگ"d:>֘$fD<>AO VYD̫_Ѷl8\_]ߩuz~U yyyyyy+Ϝ0 0 0 0l[L <@L <@L <@L <@L <@L <:|;x][ `Z1ǺB>wDw˷굇?ӝ>ƼLG;ڝw<0XVKWQ *Nu?9D {]+}Yw;T=SϾН{'"UKٗo j@̯Jg{'W^k&='}b-oMvǡWO=3Uʗbd䫊a';WUwߝx?x%8V"ZSS;7ZSZ[2n#qMg1*(jkwQUOf(E+XUB9(Ͼ/U/?U[f?0y5CRUűVs݊3uV-}< m[XT̫z4'~fϒ` Uྺ 6.?_nhF}KcLMmhEM\7ЉxmVkOX~S-݊Ug61w{_Uշʹ#~*Snrs|y,G~u8gU*S>ǡ8Sf's1k1B*>d1shfu jyVlZmFUu}ji7o<o `[Fko:?3ȂQm[yVg߽חߩS/.ޣٯz09͍Pׇ+^sӵ1K%U{Nx{htsp>QqʪL卿Kb1gJ80e` WH%ևl$>_Ŗ(bOt/wڂP>*s;ɧuv)7,,s_:f|36۹pxC4ӾR\g>ļ?8H~T|tj O9d"=Ă6jcTo1_6q# P'u~q,"׫E6y6PYk%b^:җߐ!Vy7o"Ϗgq(U,whn+}"1_G3|?n󞙠Q̫w !,_(FY6`YW1B*>dcX4kcQIIH: j[ Ww1ګ~GE?@T~bcc_c!rlշT|r<{qNtk Sm"#lM6/?B~fd6$T|ks?z^2K/K%3>wUy_!XDŘ^L&T(1uc1駿Qt66c1>y*jLqtTEP[˂\Ek|nc9E!/Lm m>yUQOlVo2c~S1~Nk}纼+"'ciu麺7y圲EG:FyEgV}Fm#gbssUQňU5z1{C}}f߸n;|Tk*3'*8P~|r߳:; q:!v;4ϑoEeSb q;'6TAD*1/Ǿ<6qut>b/TЫ^3Q.}z of@̯JX/cQ$Zl>1"ƊBG]RG?u5^mg߸KTycYZym;G}s /?ǣ5WsoMcl7BjsLU2m9 lL\Xŵ"Kqy~6 o^ZS~}%2'(^%bUP~Cv<j3sĹOT"UG9E:9g08/Olbtϑ/2]V̫^ӫLB^`]Z1cKGI}w'(6 bާ+9tzjtOPց_m_%/d׊gD/bꣾ;}GOejNg?S' b`okwDw˷굇?ӝ>ƼLG;ڝw<0XVKWQ *Nu?9D {]+}Yw;T=SϾН{'"UKٗo j@̯Jg{'W^k&='}b-oMvǡWO=3Uʗbd䫊a';WUwߝx?x%8V"ZSS;7ZSZ[2n#qMg1_~^9,&I X*1ts#Cs}ߏ3UV-ogB/om^y;UrU,[1B*>d *_c~'2Q۹L>EsOydciT1 \_QL-,TeQecbO-xml*;W(c,U}Vͧ7Cj3iM3qMf̯x*fi\p%Zd:?]W:ϾSS?WCs}ܪڨmLb{Nr*6q*?ʣXz|Cq^q u]qLXoA^ 1 E1 s!1& AD1χp"0 MQ[Eo.RX-R[%qtԜH@G.bLۈ*~xE]jcqvYܴVȊv^MUP> h_!-ky cO̯6u+yMZUQ>XWy1G};ϗ}Q؏wߡy|+r,-Tc(9 צ "Uy&?9Gm3}q^OOrw _,vכC7b~Tb}"1 (-6ccw csV׊eax:Uʳo\zx[y1,u6#b9'Ős *S<容rn9<o646y#)A6QT*\񜳟ąEX\+n]בxg5j17W" /|rU,YŌ 74_~n'sv?..1ykضUBrybs17AL <@L <@L <@L <@L <@L <X?}aaaa زd`b &b`b &b`b &b`b &b`b &b`b #[bf׊gv==2Qߝྣ'[P=t7=`<ر{y-صb^7>NPlO?woН}y/X'lkZ1˲ۡٞzouǟ>ٽŸ̞ogZʾ|7^Vb~T=ۃ?=ѽ[3A=v?ԥoyk;uzU㣗|Ӌ5oTɯ/ٵԽu赢[ZSZ[2n#qMg1{y\4nE]uK1v m/n6l3ߌG o?m3:nwM3}6.vƇ֭㺗/ 8`vyF3|2^1O>uڷcS74__jQߜkyIs~:OkϯqUOkx8DL Uys7Ob^mb~T}ȶ b~XF̫oڏN GDP={rV`Ed mYՉ̛PQq,ncK~1i ֺV[>d[db Q(\SnmCW34._[k5׭X?ν?8SgbO݇Pú@̯Jvَߎ8>uo7/ y{e?/~ ^ U}a4j:FS/͎G L5c׭ڬjzS-݊Ug61w{_Uշʹ#~*Sn2*>JWx׾]9sV2s3evB?V+C];Ǣb>yiǂj2յ6z6hMpF~4^#їm'ڒn#nG>w\c# Fպn-Z}bcNs|yv~.ˇQ]l?4Ρon8>_~^9,&I X*1ts#Cs}ߏ3UV-ogB/om^y;UrU,[1B*>d Q\vj~k)sOT:W\_2WeYǺߌĩv/DK>Oʫ<[O?1/u[6bnj۩cgkIy>gōadh@k3YT; kY *^Th-Jy,nV&@O^uۈVnBe.yH_~Cb[79G߼j#([m%:c\a1j~"U1ۗs[Q9d<1V+O}[ȇ|E]!US)cֹU?+E6^U߷fȏ6~*&?SmUMvk Z83c׭%Zߴu.y={ܪ2#*ǫr-} f?Dߘr1w|+F1R9Fw-_Mr]5QCrP݋my1t.,߱V- b~Tb}j2P[XʢQ(֍ [hEn4c1>y*jLqtT1B>#Q< ƒtǤ2՜Tc|zܷSژFVq+՛;o< ؿuyVEOƮuu 9e_>u|14w8-?鶩M%gbssUQňU>jo\פ}rUfO(VU^o(W|W^Nm_ pQ;Hη"2)Bu1rиPxm "_ylc_sT8::V1}g*K/wZm}>y 7t~3`] WH%ևl,d񉬎xP0O1F;QnkŲO}sתw[7.Rlhg!B#,1śͯ7MH m[Ou-!aFZ9gx",Y\_u5|/ľyjZLE8>x(yV1#C ͗㫟\w3sĹOT"UG9E:9g08/Olbd$ͽD%ȣoT=n==TuWj"} >:ٵb3Q싘Np`oگ"d[Hꉸ<c׊y>y+6WwGˣ;5ncV b`yyyyyy219 0 0 0 e<@L <@L <@L <@L <@L <@,ɡG޵ͮϜ>z{Ǐ-dꣾ;}GOt|^{3so{yc݁y}ZkżDo}~5Qߝٞ~T;^N$ذصb^Oٗe;}C%޳= ݹ7?}{Xq=1.^}o̽ {z{fؓ{.'>-K/mw~3 Q[|)FFFqrsU]oyykӋ^kE/5:S{5.=tѹˣU[D.W<>7L_~b<7}7}&1j.ϯ-ǵN#?E^9vAs\OGi_q;<61tmous__)#! 1on=I̫-:@̯JvAˊy{yAB~/w1FP={rVHVm 7y*TX 4/Zz4ڊh=#j>*(jkwQUOf(E+XUB9(Ͼ/U/?U[f?0y5CRUűVs݊3uV-}< mv ,*CżSBby~ 1xWm*Ԯ~eFS/͎y37&E5qTB'Y>a!O]Dw+VUڴЗ߹W_/;~+vWVj+s{$?HL{ʱ/ϡ筲8oyU9LL=G< mα7}c[<oPͳT7fj7P^O%@Qx`m}g㵥u0ZK}.ɏi V}|?N|GBmu[<Ns|yv~.ˇQ]l?4Ρon8>_~^9,&I X*1ts#Cs}ߏ3UV-ogB/om^y;UrU,[1B*>d l>ݕIF1&q(觺2Q۹L>EsOyH2sYm7c=qj K>KcUޟ⺭?_.͸_syrsx#7E6zjSm,ƨbjWmFBO%Z5Y E'WB]m:mD+7ZJżu/!1߇CZƛoD8Q?X18VDϝcg51~=3A-W[CXۿPmV b~Tb}bhi,$$Ƣ Hsu-Զ1_}clW?*K繭2dv[CExQW\rQdйbjYdWm"T7X+ ʴn/^yUeQ \khA>zzUeFbHW$Z>z~1 soc.Vcss>U[b k1' Q2_|b^G\Y*1/cZ %R-$r}Ge2:PEE4~kLncS :FcyLG[D U? w+ |Lm m>yUQOlVo2c~S1~Nk}纼+"'ciu麺7y圲EG:FyEgV}Fm#gbssUQňU8FYhZ-Xu.8um"mU.9sDz<8v_1;}xE_jcqvYܴVȊv^MUP> h_!-ky cO̯6u+yMZUQ>XWy1G};ϗ}Q؏wߡy|+r,-Tc(9 צ "Uy&?9Gm3}q^OOrw _,vכC7b~Tb}"[ŘeO1F;QnkŲ`9,pVyKT/Ow+<&żטFx29GŐs /?*S<\9koMcl7BjsLU2m9 lL\Xŵ"Kqy~6 o^ZS~}%2'(^%bUP~Cv<j3sĹOT"UG9E:9g08/OlbM/R{xmYIq|+UsT i}1/gv#?'/?GX/N}o,C%0+Ї9=wQˊyczIK]+>zl(Xu^QTs;VOmN *:X|5~}aZ1ٙ(SEL}w'RL^z'Bc`qB{]+a,"֯a^hU^.b_x$.y b`b &b`b &b`b &b`b &b`b &b`b &b`b &b`b &b`b &b`b &E[OLIIENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_ckpt.png000066400000000000000000000153161471400134000344410ustar00rootroot00000000000000PNG  IHDRsmu? pHYs+IDATx_l#}t'$Kq촮mh,q[4 n%SuC'KQ?Ku(")̠!R G61ڢM6.؉Dqw'N/q%R~`ofvhcchr9)dR)dp~81zksykN/GpB '_8m 07Ƙmj(4K4s|j_sOC9Yd4vNaOc]4yrV*zs]wtn˩\YFF0lig;;F rcLW:YY#"Mv}Pi49K&e ɜ5)kƖ}i-_jJ!̱MΏ.QH9]DwW $ZCEY-*~<َY[ Ue߈IVUъh/@5BUX(Z"+d:L!Ǐ64Rb(Uj]}ݗbK^fYs_wjOn1V2w-lZ۝kk]O!Cnň{>{PEQUe껉yJ;=Ƙ0\fe58f]?;);x*@3^)ZAg-DTg=enMDfk5jUΛE-KWZ ":)!W!<,= F܊|0{(wU!F%CݩRhq!ykz 2dwżDD$vMJնv,pq*u ODDNk+/h˽7Y1N aIM5q'5t F 8OwR;վ4|${:պ:3_y7n7޸qW^Zj5yh=eJ7[s4#~f3x[Wp'1(Fо4.V43ͷoߣ^8$4 emm|Ў3g =o`<_.Vڨ#2!{ׯ_W}ƍW_} Z !'UnXyyd=3VF%7m{JRf:F<>1BwziG4RXOustwTic%b/zb& )b2f T۩n}qh>;*Xz?VH랗VmdL&p6#wMQa@Ճ/),q\c_j5e8nV$(SU2'o yHNE= %XEԩhIVَW&N}&3:@mgfhW;$Tq* ʤ 37fTT* j\L腚5qB%j/<Τ[]p'l^IesjSJ0T;|wt9s+yKWj<&8C?fr\hs[ip p~Cckל쬷pʕ+WdVC7\Å9'M8pg wX+q.Vs`T,- G$sa`n l +i׮];zD= 'zOI9G7 p sa`n l m 076C9N&'s>|œ;-?gjn[t]ɪgWY#*.oPrcn6MEgLW:;gt/4FۙbvR38AP+1X3Չ/ZG]I\-rUm0T1QtVe+{'bIW*=dJot; UA. AG~+qW~腦n}3_ UeY4/QR\PY޴ˆTH=dmXW3<n*2Sh unĖ͞v69mMmf%ϧo9[&eϹREW0iCc(=kS1J|8B ;w\ )V4)>|5`I1,V[dc޲m]`lH iE(Wqbcڵ-_nMv9nwd,ղ&]S {J&y:-tF-LW2nekoϨ}gٗ־F.R]#"K)n޺""96I'.\ԲO-cl""z8]l̞yT!)ۧaR1X9׽WTPGx/NymZfY{tj2V?2պ2 勝}n:5թkdw̹03u*l-tKW.=!aG× yhL:[v:PgTaY]qΘL!W腚tG'(2;-}V1ݮ5s/)HHQWꝔ w<+b㼒Ϧ/=x|is{nh=7 he7ֿԭri5c?K{J'իvޢN2TY5 gsFuNF]KqV13BMTD|@;;B'>Wz/wRY$"KiV7=v-Ut̢jѻE%*Y)E_&j˂u=&{n4j'5vJd-jdm/mX VG4*w^'9KM~f'~T.Fr]ɭlݎ;N*jI>*'F[`q%|-Z̈{ `R$*%]ق6@sa`n l m8GgfM;ͽe魩pM?l?˽/KS/༙~_zsy~qu88;Xԫp6M4_ƅ~u'1w%"z_.~BeyɆ['hw?6x Ɇ'>zghhYgFsc͛7oL'wg΀%Wٶpʕ+W$WC]WWM3T`m}87.8]~> Ȧx7o6>Ç?xhs'~K-g3ئ?O3S>5 UKKy#S`sa`n$&AmϵG_ @D̟Pj{3 [ߦ_l?MxbJٶ/f[ ow" UU 3kv٩ W43{R` UZn̸4?WaĩC nUƕp l,5cLW:Ynxpz7d1ƚj34ڢ+@Z,R&%.*r\:vWʞJ|pf%lVoߛ4T&{_ԫżDD >JE2R ogi[n;ɻsRN].|t%3fhdqO4 "v* !k#JئvgU$wg.j2,VުYj]Hf?87F 9wlgN Lo)\ͅ1& ekBmꖽBwۚ(ԶT#$~RMd bU2;o+˂H& /uNTQ 'ڮN,SxRk4*u:O UvЂ-`w2j5s]N(͎N5If&l; UjW Q1Z!dn'&vP‘'94CnۿuFѝ"~ck7D^&5c4j 0"ZMFHbٙ'xӐv+իR>Վ_#t鴛f'F)_l`]t%٧tۚ_?~MX$$[y@|i; \2Ti9AnmY8dJe4jmj(@;f?Mx,RV=يSOzUm"r*R`!:6[ne*zX Bd+}Qo&ԄLFrS?73NP7M|qbt*t.>;"5g%dg+ͩ_K0V9"`x0@K_1 _?L}O< [gÿܳϿ[WTؾxx?_m,]_~yKV]ZG_ы/{ 71 =Isw C _^Nt3)z,/= ~`۽Un+[GXEkGyQjһm8{52R"K<$yZQ;9oЈ19bGXMbdDw6.oz{^vzAx^o0pv))3*"{iN*'.u.1pӷ;=ƚjvB`X=ttsZiuV4Tq{:hZo(l~Av ڝuvY-w^!'bBTD3mOhżzhτTah ۯ~եVV..,^^}x}=dg"|1kmG]Y|>pB*Z졥Ɓ9hVvY9)Mj_3 DQݏ;9s_gnj<7yjfsi8-65ιV/MM7Trfď cjr04UC^;{Awָ~?n!>wYO5pt=Օ--~g:"=1휂Vi3"Pە<{dͣ.M$b~) лuͮdGj$n&H$l8-,^^Y|y啅ťԓ?mF<J-OY|⎆*d ֍LK[˶L'7<2T|ќt? lZz*rg,֌'IGU漽϶',O*cM[m/mDRJO 59v~+ ޻18;wwCp/0h/m,W(zkx$2y$ss ƯO]Uxas֒pΒ|:)RM4ԭ2)v26W&"syJy4TYlfƟ:|{h,+pbUZ24GHqIpOXKi:bԟ!HNl֘(zirMɩmll8Kp?ޫ[.]XX8ƈ Ņ"8|ۻ: o'FOzd#ȆyT6@<3qL?lu_{ƿIObIENDB`llama7b_autotrans_1to2_predict_distribute_ckpt.png000066400000000000000000000156131471400134000366200ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_CkptPNG  IHDRnSsRGBgAMA a pHYsod IDATx^ϫ5YoB7YĕlFE,BD2daȀψ 2\Jdpvc4}Q2!0Cfl_TuUW>zOWuO=]=u{`xw&y/ FhE/L"H$D"ŤID#h&Fhh&Fh@4a8Op25׬]~M_.b%wݝOdz?o8cX\>ڌ%ט75]ԟ {$o|x%\k,{bOL>,/Tjk%xJ頄^<s{w4%rX/3K[WD0sijf/1^k\ x_O6׊ZיKx*{~ ݢQAA d¤tzRG\-OK|RhB8L[w->NW(\eTUJ:naA~w$vhLTR_~Okqt(e_M?Bm<%äNQB<0mjsWca}{u\+>k׭nخƏ$m.@h<.UJ,rl hIO]0k&ۡmX{d{;T5|} u( Sbf9PHGXlRu=7i_V~Pvj䒾W|Ph~6}>\yzJ[Vо{Xt9;R'n6~؄>'k>~{)s6Q&Jtұw*IV]3|Ys.&aWafqՓfmd}bX'(gW]|6JSwT6=0H>I哊g9Yshžle(٠\x,fuIg=r[K܊Mͧ\/@}K/ o/29'u4}QOj gQ(g"ru/qUV?)W_(ly}B[;e㢬| %{GK8s竈F@n' ˧=],+m{/H%N@> O=ƒWVIEc?J±&.јLsXk)va{Z_\=틞@帹)Ldum OG H6&N~m!=6JƂUΩ˒KѮ!o[;l~S(]>;gDȯsHQ>جkuv1.ؘ|,@ 9(t8I LrɺP;knrާ(ފEs@'5L}5d&>jo趟K).f>\'㗯S'AMtʹ_1Uýv0u*6IAkp$q5}2Ķ?K6^[zH1V M݇δϞEP'j`GO6v6H;9eMէ,#9(ttx8'DElD2$^ӕ2~sMMdV4H$D"H!ih sۄq&F ¢qMmh ,}ۄq&F ¢qMmh ,}ۄq&F ¢qMmh ,}ۄq&F rE~8;9pص? D<ҌIfo D<$=C)Dscn )- F<_'9 ^HK4T݃n=x8ǃzzC@/sf#2 V9 $v)̝nB4hF\W+~}p\Ţ$I{,waQABO`"4;^ъ=\Qq ΅=) l;U?^ǣkk.KEcq q~dQ㕔blO<OeQs4yB c,}Y9O1fZ!@1C!Fc$G3.>\ItZH^~?fɪt^IdnьXb4Y'عϋyp"),ufca85I/t.K}B1XފIx>jɒ?s7O䏮 НCұ. ]dsݑypڌ;ei#ۡ w+H\ug]_Xl-nt|IꗎvVo̍hKKޖbwC1} 6a0vˋƊsCzB sgGu=EZAtC" ǯc{.kuOGCmߟ`<1m[1?s6. jB#7l]=sǾM`-tQmnϫ]3]^s/2j;v8W`V4,#o<C|}G*F<:V';gn\+U<{N6[{3dh8ٜhy)]:q&F rE.vK6a\ `hAnu 6f6a\ `hAnu 6f6a\ `hAnu 6f6a\ "?{|o`"_>gom_*q}AVZ"wI$uຬ"Kb7~Oyur("C UD.7 Fym6uihh&Wi9TDw:SL4g'yWD$G!v#Fh|7&B7I[.Ə$mຬ"`  #?p~Z^OYG4~K`ˬ#E ïӻ_OKh,%)/?{`F4V9ǻ&~u`w_ p82w7X[vOmH%ryѾ^X/| #q ϖ};4FhS :W3z W_3on^б`O IvAXzX?Zg˘;.] Gd=kҺﳑN塿QM6|ifw;Ip|enn[=~9<]ݧԶkס,m/lL=e fSٽ6ޟλ3iT_#}8(|ՆǶkm< W4ڠXTz4TGoGF.c_cC}x@Lʼn؏ߐW./wz6v~^}F$_ *ypem722Nu~Կ?UY]@ֶͮfM>:.Y{E΅utޕ(ƶNV9^;ݰTma֏ebO˃n5)w2/]-K`E:~~y~֗(U6Y#u06+,.kbht=K2qpN,^nQ꿴SmA6^f{=|,{XwϒuD Ëo-KF^\x7^oFP#*],5tڕ?;׫H֏VVcRwe,Fn˶Xfg^f{=%$BPN@zR.m_sy.|*Bh2E~٭2$"K(.uheJl|cy?|l| _~?Nc0zY=-ȓtj۵Xd62i^KN뼭VjG-qPv myدhAU7v),i`GBGmavqe׵k265WP3fֽljGol|+.P2~/*o߱ITUpA'@ǮcXtTeuaV+XZ7dg9׽޾I1u7tB?e8zvR-~[?^]<-Rغ۫Ԥ HK8%M6K>Kr[\(eJׂ]c0-]s$6++ž˚!c%vT~N㬿nQGiۂTmz#?JY $i &" -JXG4~0x{~ֲ$mȕH~u;n1F5w"0i4}p?|k^yj+h ˪?bφo#\ƂpMј G.D8$uUDrgM+|)0{]WvDNEѸJ4nN4~a!R)sG',u7P[ j3#)<]q\~u:EM՝5lg< _: (;&^kCx5FďƯR?-=쳊qzE7s曃Gs_;*;cR49's)}7ol?(KHہQIeMl27x't|L|o$lUn~q־& 雚(u2lLl'U/ i H gɼRNv>*>͓%:|=a΋՘.zt4s\[`2p-cg^f{=Voïc>Հ y=)|Fc} mMSi~P ײ |HoywkZJek]ƜQ$t ߥNXl FY+g#:8cߝ^7#l_EC`y|@|x`־_9s\CvVL ؟+/]g'\SƺY܋d$(&t~.#1 g^ssz}Ѱk翱,m/lL=%,M9+c}u>WH}>n8_~:Wiݿ$rhAV7s8_9R͵|.-zkK_av!)wa&F5 0B5]iW} GC}Ϙ_'r!/K֧|.g[Y5.bp[liϾ|y=w& ڔLq s~ٴQD=?瑹Wa6ێ{(@2I&y/ɧR2:lNQ̎K3o+kv^os~cXo\iW( _naMږNlU>XJ͏ #<\r1(~!Jc=Vm)`oPo(k~GNR) >-{R?/uDiN&>pcݱuL幀=r)iGj机++ž9-c%8'AC_NqU(Ty[ ^Ebkv$5# ܂JF8nK40B65}c|;OYU4~k^y$uiF7Fhh&Fhh&Fhh&Fhh&Fhh&Fhh/Ζ[IENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_log1.png000066400000000000000000002744131471400134000343470ustar00rootroot00000000000000PNG  IHDRzk sRGBgAMA a pHYsodIDATx^=fǕ|a`"%Ӗ ,4c؉6j '# 'Fn4&0Qp~vڵ^j/ٻVZ~ߗ~gN|kr&եS|\|}u07./_r6//4-gNFu>L}Os U^r-ggoW̝=Lr1|0%4/o{Ό^ɴ 111 ]-߼Y޼gy~j'JA.o=Wwm\\?91?Y>.|h_|HﳖDi+O[]Bڶyv|Ȥ~xGuu8~}磧vx'8 v> 8;Y`(`/_kUtad_E z/_sbpe6&$1Q@.e!n抐 y)&F.MO=Bw9CmDoIwדx|j$G~0ca04{>$4~E*W_b=1˯XY?7O 'vI]v 7?bz% עYm=7EʯZ-#T+5FM9>X Twy*V옾G0ߛ$\w.1O/^]/<%`Ol~ Rz$'ogM,_u" ,6kDꐮYL8Ӹb ّ}"l$.U\"n ﹈#ZLݨR7j> *Et*i-9O*_E)-dBΖKM'`ዯ߾k@E$ϓ9\ۭnÃyΠx5i#rlʉ@3-g1 QW⥅IdmYb_ &Dnj}h6G?\ >>b,LcP@jԖpx +̸bGz'Wx]O8qՔM̈ zNX ӧb0~3∆}$\cM-t|:_9KsגBdbpNox?3Ak5Ѻ`N `B Zd=v $`x#u]^\\cqȿ=~S^wG}u? D/D*x$U7DL=m -ۊ0f!(,LY1>,o8+4ؘQxcS6+)eǪ#&2u4_.VK4_}o6&5c~Uu_n1 tc 1p!<\B2+'/Y#q2-RSB﷗g^?OQ~6A5Q#kwWw[sf,옾>&25Yࠋ:cr!\]/IЂ UE]o7i% 9ldV/pO?sbp" AcaAqb9m]V_._Y^p$:{Ʉ6&E,m\27!/"E)!Y|UN:g-]M"zRO?[m;~_|}bo~f, ƙfO_'P"$+ 9Z2B\}k"k~")I)TImyR*ZeGյ F5~BPO _x^ NeWX;+6r|Cqmٌ)EA2fi, AM :pN18# pVF|blfme[54t^&`\~Xu Facl|[з%亦mb&ezā3)}m~ei8G=t:^ulm0$WNǎu",$r"5'C "e?%DK~{y`hӊTZ1b]qwE߽~ո:gB<Ύ`bBD"& YzLbH`)⍩7afm|lYeoJ _&~XDscz+pcեk X۾L_yxYb#Ƣ7is [[7kJ*/Eo3Vw3Dz? ѐh|__>(wRصB+0b0Aꗽ 5I;Ogxh@ pK@ vq~&ƈD? b0< b0< b0< p\/^13¨]j\~Y^Pogj`N O ~d+};Qڊ='ִwm>=<=2!+*QCj_). 9B4ΎsV1.o ?X>,?_)=|~!yytZ_>]~ig!/EWϭ]{~5l_qA ^\ cNllKvΏӎٓOv v,U+Wn{];p:frg*'&$1Q@.e!n抐 y)&F.MO=Bw9CmDoIwדx|j$G~0ca04{>oByɢm+I /_WLC,~O侈Iuܕ v߻mv11G,i܈\EZʿvM@S{mggww=#Nv&wٝΏ]3L!*ѭ'^~-Y1֣|SYB,>Nur[#jlt3PL5~'`X9Ύ} r_ 7/E\I4B_HŋB@<5jmy^~oAGuM![%pW<>!1ݟ6G- iV?G we~\v?3ϜTW:O%3R=^w?<]Nu~/Nv\ST܄(:aCcM~/ҷܼN}]Q1R}Oq\~Ljis϶;~]Tݱ'1,ǐ|6eUvUq6H塾wm˾ʎޅP`Ol~ Rz$'ogM,_u" ,6kDꐮYL8Ӹb ّ}rHrJR^t^@rvɾI~|uWt(Vے˘ i'`ዯAY"p$~H폿)5 O b"PoGkg׻a vA\*E_ %.<^xy*/W>|Ke[\Vb#}woL/}7E2{FL>3)$d#hNJ}|3ǔq'IKkO\ 69lQmX?usTWC:﷍+Ѯd[I\<)i73Okyߛ>߮~;?F}=My/MlRUlxk.?>WihjWfftg2fn߮hV=xݬY<>ٜUC1ÍdLvkL)wEW,E,<\4w@ a;5\wcB1|h\b߇ 9Y#2.=vw/-{!`e;~g0y7{=[^NJop$ȏ^F `;Ne09]ۭžٍq?Jq᫓焄p6;>7Kn:GsEѫCظr͇uxX$̖kn«7T]Hfyc3S?X?ٌKaS.Uٳ+ ]w~uvIeJY\pW]Q9L}/.SKu^]~Oenfۣcl}xVX+uv5N]9OmutU. ,9"#TZ|1ꊒX )7Kהț8MGqvL_ (׽Kx`3rQPy;VmVS6\#R]{'g,oi1V3؁bZ/8R9 ۘ#yf;yNFǶ. 揻hFs,}\cC^PَmIL!TGu#-Su_ؙ,}2~l[OmgBN75afxVZ+3OVKy׺ yT~/jO/f03ߊJ*#Y63h1;mخj[c.b4LLhT:NB Rt䷮˫üZok,cokO1{hy h ]E\\$$p}˶ N\\Y)W׳bp}`Yq3+{ѷ4V^_h}[b8P$s M;1q .~&:~z]5/^g9τQig8Fu?m^w~c>ʚyӛ#3 k0?}=3_Nߩ-\7xGC~nK߰ve7/ӞK3^-}Ԯ8w2Vjݬ/mfgЮbUڗnfcB ."F|2貐ʉ.tZaO~ ?d[ הZV汌Yg O?fʯՇA?~84lq궫뷝XmzY&#^VDGCM { nEv}#ww4 /y J8w:no>]?@ 7HB0 qb0 JK ;^@ o4g7`(n;)? O{n    Ϯo,o^]<^|{\x}rLpK\|YCr9m&okbrKs兛f\.quNỏ}\qtGˇ,wE{x|'J[|ݚҶӧ緛G&=${E;{H;=;!X1Vv휜U ?˟.(WJ_-__~^/>]VeOFE!#9R{眶wPo!lmߍq]` 6O ߯q߸}}{vsow=17J0!'Q~$Zu抐 q)&F6MXgW!;霡߃tIfc7Iu<]>l}cV #eQg o瞾Oxb U$1|{bo_2 mső\'+$k]:0i?/?즽'x58:U 3L!*ѭ'^h~-Y1֣|SkYB,>nN"5FM9>X Twy*V83}K݄/WNKj CjĻ(zл/+˕*<5jmy^~oGu.1ç#tr|w~ Owq4hxoƞdڏ:rriÙp~ mm뱽ߖf||N ŀDexL`mBʛlsv=1ڠOmJmfTD~mq|SmoFڠ_fK 8b 瘤g;Kʏfj.WƗ)mmi5 y[9<h1IYh3˗kݮ{8c%̄S?;.0gڞ2vrNrJtpח%v9.}!ˡ|SP妤ն<|PKLkk^:k׍WES//ʟ *6rAFcIQ/7#lZz}u|+6f+31ƷO*T ]ꍽqpYM:x*/^9ˉ_9~y)*T]D},_.6W/6J`}+_O f (V3bpZ\ь?7f_ve.1䙄^;GcX#❉'{G*Eݿ5pXǷGS2~_bF>flGq gcn؍q4^[w;8辭1bծ5nM9ƶۻc)jL}cƥ9fƔ"o~WĹ> vFB]_bEQQOy~W5_̶U\wcB1|h\b߇ 9LUD:}#ء]ˊ{ae;~w.&y {*,U\Dȏwb1;ŗ?Hvx`&鍇n Jq_$c:Ue8sj;#s2Տj+#pRmOym9>~ۼ؞IcqzF g'>2Q̦;ǶGc36S@#b}KIQcD_;Yf-&qi;co/xLYs=Gd\QR$E^_f~1y3j]p1~gR8޾}Cyøwr/ZL~/Cw<[-~[M΋]{NX ӧb]^LESϨO#6hLhbf;y"_a_';;7R @6csB)Bs8wi\{l1{Hcx/֣fdG-ݏ-v߃5>Θl 8j ۘ}\`GQ8gJ/jΌAP&`E8cG!WC~뺼; %+׹z?֑{*XDӋMNÅyءK{2yȧ J=K@$ޡ\"g+]Fܶp߲-R"]|0!Sgѳr1C}) 7ס/-1&5m{0el[4&Asool8'?nm5VAM># dugھcҜ gݑmjW)[ ,<;~u;lo>?p16nsrz)#W~6qփm 1p!<\`B2+'/Y#q2-R:"$};^,c%?Yut#h]Cc@vwbI! w/rN#v#Hryvqx= *N^>pwob0A׻ ՗Ip  HW x % {oƈn xKcʩx4~! 886F]n[Ԇ     Mjyե>ͷW˅>'^/WT:}ۈ)8޵r7NbrK)miƯCX{ɜŋ,>@y|/>zWYKwk{OX߭i.!m<}z~y>{dCBWT:SJ;]r;|lewY|=ϥm|?篖/d/?/._˧#"|Zp&ax1ԩ8pvRai^.4׮חNScΊcɄDU&hEM+B&ĥYH4bΟ'{:_|s~%͏$r'tնGIX'GaB/{>8@m\Ib.bef <50Is #a.nr_߃qxRl}Y] ru{/I lQ)R~5,o!@7B'Pc˦qb;<`%nB N_]/W//&+_koe^]/;uէPH .teeWmUyj:uuw/eQ~vݬ_|| ;kǧ\;'-[1G WVQCT.>ZY>w8GI~}"|ry)cu?O~vg[ٽzGJ(Synrշ27ime{eZ^*fbMoWo>^}-,xE^-{b##18 ~?+mbfRw|Y|_]1Xpq`5\SƂSS.Y.υnLs_rq՗}kPr\.%%I幈a/[]EYn/*د|wVLXPb;rCXeD ƒ ]7#lUQ>[\\s,}m֞G1a{&aN_S]SrNv:ljVkYi1-iROC2~]7W$9<ČecA7YcC"f6.v$z"](-^ OPm"i.=Ϭޘ@LxiqIχc<@758k׻Ir e',be=bowU!^ 4k~3eY"lpE/K_zb0MD̞τBȂolJ*Zw!|4x˜Y{¾DŽmON_>$[~L*y߆bbeoNeNeeۯ_^gZVq1 }fƔ"o~WĹ> vFB]_bEQQOy~W5_̶U\wcB1|h\b߇ 9LUDЧ %V.b]L]\Eչ؛squ/¼gҚߪ<^ZeۈH>zM\ `;Ne09]Be}@}Rm;}OIq៓`I7[ά=a_cpb\59έ]>և~64Vfj}^]ZoXZLUv.;F:ьljSa7bpa*Y9W⥅IdmYb_ &Dnj}h6G?\ >>b,tǙ3ob.{g/\z=!s.lUܶm5e;/ө37|~3SʯgԧroEؒ"ş;1!{}BGq!-;]3E~ke$Vc5mҘoc٫ҍ f3SyghYʶ޻ȇa9Sbp(Z0"NS#w!u]^\\cqȿ=~S^wG}&o'B,o8:~Y`3g2! |zyZpmy0lo)Ouya2C4g 1BXNeslTLþDŽmG2Y׃5EY=S^v赙5J;"?U?)ϡo坈ˊ 5F䷘#)̶ z&"i'p1 ɬcdɐ"gH鈄 ) OѦ8c`m{|quΌxqzBp],Ӆ\h2_*]ާ Pqkcunmcn|xZk ? W t"iyVO?//yM177 @ 7HzB0 q b;˝Ty_[ _y>'_:oZGlsLG{hg1?ֿile7atF&Px6x\<ȇڙ= DZ."O o?833f3}AY Ԛcƽb8؞\)d-˿޾:jKs_b0gǢ!_%7-fe_vQ1Ըm~8cʬpGBakekO%nh+7%s"16 ZDºL{x&?z_+W!O;iz~@ 4QW1hߕ=T/bp~7[_^kqA b0gVhJ39k=xB[|= V  nїC1G bu_o?8#UOd!O 3iHzpYmmQ/+7\ ֺۗ&zZ]OfN _G_1͛ͫK?}˛o }O^tbᮕ9ޯ_^w $h |@k 1W_NTx"kB9]73 뽻b0KC~8~{bN O ~d+};Qڊ='ִwm>=<=2!+*QCjܯSخ6T{u_Vft<gYz}>?X>,?_)=|~!yytZ_>]~iF@¹^.? UM0#<2G[zܬoK«Rm_be&f5O\D_R4qJ sE(܄ @Tv^շ'{:_|s~%͏$r'tv^gO)gկ*{'p|wAm\Ib.bef !׍5%!Cpd}pNV0Md_Vl+=s&`V&E~uG_:zs}/Vw~Y \X\c {b9KP(/EX]v#AwgACbng#3/F115̫)EZPQ%-dx6q\ E`)i-9O*_-d6ҵHt,?|UT~˿D&Z4|pU:jYzf#=y.ik՝JVi_zc~'Z[̚FbXI5FUkG0li {59' `',֖ Ot{Pݎ܇^վ)_|[)8vc$G' qc"P0A $,ؒ!\xA ZPwT ;>|KŒ-D"sE/K_zb0MD̞-EF^sMEٞDPKsIn~2ʷ}e b^kQlGM>v֔eW?E,@= &|3i<# !wŮ#r֯eqrL ַdE$UHqj1X2׭*mq4:({bpOʻ"3bpn1%DSCeZ b#:b P% CaI_ =b_ &Wه&ns%R Xg$1w\ VyŻ@,!ܶm5eO={'g,oi1|WDWg~1h1,&Oe_9KsגBo|lhe —`byz,0f57A}S@곂u{µ.[ ?W춯aEbmKb}MϱT[;b[ߚl)^_a1%0( ]v ,kUߺ.Bp$ŵ+b`=mGS}~LqԖ3`!ŖTfOHqDdzJ&HA+me[8S|"0!SgX`cFAI NO0W8SˎUG@d1h="J]nϗh5XvdžQ!T#j?11h,uD?r,ne ]xb&'*|FdCS3hצg3Rx\/=Z}qwO5_s8bB ."F2貐ʉUqPuY\kȂ{cXXyQh H)eE/+1bpm/m,8N EIuA["$xˮS=f<ߚa| .Yxcy|Ynv8o]ۇїMT3bp:׋hY}:<M1 ^8fʡb7`x@ `x@ `x@ `xܲ|\}fyI{\߼Y|{\}.^^ެ察r9w?NɄ&t Ĭ(my='^]-=1%xkC}zz椝R } Nʴ,pk#Ŭayw姿fy%w=bqB ``O*_VIlZն_c鯟wϗ'2y#+L\/ԗCBb)4#7e&7#솢V7z~MTy҆'>PuB}afĢ+˱QuT.E۪4XN%d+׳E＾.2a|q +$42ķWތyOFA8cDolL.~]zaOPt1RvqLӱQ!*+~_ .B#._q-j[z&DL.rF w\",ԮFqik&Do3zX-nlgս >ϟ&[}}\~'b09('˄[[ݺ?Bb0XWa鄀*]Ir&E_ {@@,H!ԊRp͢l7_Zw_b~28o_ ւ&ێ|iD~ZyD`~*kP2o/Ǚbp/V u~pB"LT*܊ Bp"]"մK*=ϊTg-psA~-iϭ/2n^1Vee36/ާz  1}b0< b0< b0< b0< b077˛W~,ϯ7^-dx\=S0m-9vg@~_}"?#jܯr7N; ǍK;̊}\qtGˇ,wE{x|'J[|ݚҶӧ緛G&=${E;{Hu ՆjlNw>z?YSY|r,?]PZ><_||/.?4hM8t\}{ܚpNqub}}{MWw^w 18L$ъ\ĺMK7hJeZ?| ~%͏$'tv^gOS[[jס"]ƻUTz[ɢm+I /_WLC,~A躢 `'ؾ>~8\ >3+`!*!'0VA鲀9+zo_<[G2MnSWnkC_Sy-P'?K^WFuQc[P!z|%]O5WW@sKWۗB^FKpZHp^yN~]M_gۺq5//{%Km>~ZU93W+icѳp`-s)#18 _sf KݵnW%q0v; ˏ [D!]R@d#ہJKg,غu]%64r.dEIzywpR/t">r/y]sն<|vkk?j~/?X~"ϋ ˗gQ7ִkV\w:uzM#4 ֔X϶2!%ƧL]/n̶zȎkײQ{=^mw}5Zbgn_gjs嘾涼|'u|x}mmӳ{b@n18&mb# %-^蚈OkBf.)ϡ_[ϬCbK\㙘RuqH sc![:.iإ.] `/l%%ѹs.<RVf%!A٦ꥷ w~'f$W~8Q,gd*4AJ$涱;)ֳZxEL;q'횏Uڶrl_mY3X[X[Х6~ |kX_?kʊz@uwvMGVR=hW{&fO c:뗏Ǚ_g?/ Əwd_+etyJy` c@i18ꈉV,b]O, K}/\N6qkJM~|u}h6'*?k[ҋ)_sڵK{]jgfXgo)p/137 =|3T^w-+zxMEX"yOb;^zF:="عޥv}Nea"^6Wp/۳ᶅm\#dB <_-_ϊey[Yі$'9C+Q:Q6J>Q s[h)!ic?VZ)z\BkN_H/!_DhζuL6?1kzi~ב?n7|g֖ĮgOn3;E;LK[9fGc|Oqߠs۵_v}Ti6c΄\DGd.ey여P/`3/۟(T?EV\Q@1xv$c٧rMh,q#Bpbt͗C:3!+cmϵz)L} 0nԋ7afm|l7i% ldV/pO?sbp" Aca-\ivvԮj 5o_d"8G}WIΫΚg]b2ltLj_Wjk_n7fXbmgIו\ߣq&q9nn66s^V=׹?ظSb.ۼbƯ} 6=nqclo~~bQ:ی:r 79ɂ /p nu/`o`  b0< b0< b0< pkixNbm +ty5/mOo7gLzHwTZ ^ݗ> A݉|o r,?]PZ><_||/.?싴s]2rxs-qͭ#.syᘓZZ{>= 3sF'\oΞε4o| ycʝꃛ#÷/x,ܽbc}\~jr QD >4 1&.tbuڦ}ST]kzKk|7fzvɵg\uvyUoC="x,HY3g#~ưw>FFu/:}vWU_ﺨ%QZ1RcXؖq32q#ױ]p8tˉuԻ`[~LZic73Sݟ1\N]\mk9T'-]To$ۦ][{[~?,|=:ska~`cPctgf~`-s)#18 _sf KݵnW%q0v; ˏ [D!]R@d#ہJKg,غu]%64e!K,u xNlPxe] ͇ZwO`YKZmKΓ _з5(.7GO _x^ NeWX;+&Jc^b\zr6 E}?df%y錥S#]G^:xy2}?5k'xv)*fQ_G Y3fLQX1lNI}dv)O3^[Mynuϼ<=Zꌑ(fr,u8+~Ōcn.;'ut"?y_u|k_ucڎ뭬}o~̵;X&3g7m{q;EɷGbǬQ%*1Rcn}!߃9?#5PcR}1ko:AݪˎT3VLV~f<<|ݎpع&k%ؚ?F ~. Yڷ=|{3=:ӳjn18&mb# %-^蚈OkBf.)ϡ_[ϬCbK\㙘RuqH sc![:.ibVX6]̤dʄ3D/1/Oc7gbE]IYb0'Nx+1Ll/qmUƅ7|徑}Lf˿Am?ayج?lulvWRYi>f#l_u_;1D;پC;kOO61>"Ƹ3b莅cʒ>kJcjlT ;ѝ_<&GM#M?fUVgb>5cƂS,d+*kr_F1&>hmM~(]f gcx}ˋR"_<Җl?+TfʺL}K0ݠmba@Q诙8~DsSq{;3 '&_^e-b0}!\ESk!_g>o~ۅ7fV1KUeqb%D'RTt_kŹ,<\4w"6ޗRpߍ!as$R t9Z;lsbpXHP܇ΝAt]*S,A6E]ͤ$ξO,. ^޳UaioVJof6"?O@ H_ ]C۵SĦx>ڎ3^o&r+M9tƚ;߼m~gٶ~ʷN(?X4>wez4t[{ YC0LGw,8Ec#MvҗLׯL\~>fnZ^jsL-l\>tIuwvcFkV8sK?jW{&fo}po後5;3}m6Yy^!$ےm\3[׻V}ks#a0ZS[㲙Ⱦ^ hakC{}l7_`_CmލQ3ѫ;iI4XbVBs**[{ELOm3e~]ƖiZm2zrޗ$ g=F}Q׀+oZfw/q;-gT1 E뉒xɰ/X C~M ɏMDgmKz1>'T̺(۹,c[Ԏٺcjg cA87)?ȧַ+[SƉ~Ɵwqb|u1_m[smH亢cax[&zW~bHX "G ow %m4o%ȶ7n|Mϗu@#c]ƕӮ83}vaG(~zvRl^)>AyaGQ{}(M/'>foַx G!*ɮVꎳ_q\F+q6Dnkv.f;"[_JJ;W?W$7goik?Z[58<<ޚ2g9mo=L:sSU}w6f۶ Tʺ}ҤiidZ{yFR6&&ZoQvh cf|}M6m9]2gO|6fX0e8+Fo=#r߲Dqk79Fx5Sa34D?sJl0f'~L79 0N|eht#F?:$TϝؓnbaDj̺bpmٜ nu/`~0>z9pC>}c>T "R1_ }=~3`QLd+};Qڊ='ִwm>=<=2!+*QCjܯlX~磧,ݎñ0|?g(t7?˟.(WJ_-__~^/>]VeOFE"/LjI}MlbAB `B NH`B[6ѯvPobd"єT,D>Y*{'3{Ж.I.ʗc`0‟P#F¯Q9+zoϗd fSוPcmDT~$JD5_1z|q%[ EdƈiI{u)yWZ &/HdbeUXyj:uu/ Mbyߺ].t._n1/0by60Rʨ㴊?'6QvЗև W[*79 m{?^D7殗h h-kH,풿moRw|pw"gACbng#3^$='Ԏ1ݟ 5vw-dAImax=d(ߚ 7j>vE)i-9O*_,6ҵHt,?|UT~˿gƂ+6>dL;~tU8l>xcXk.m1;3</YU7lN߃jff?#kJ`-m %-^X(&T,Fgo*]}fm"@[.A%=8vЄ^֟N?aeWXЂy铭!ߴ\ &zb+а2<8"D5sDP WX},_e[H$HE/K_zb0MD̞τ׮8jĨE>O0Ʒ|_c۵He{>bx9ma1߆>pogl CޚRb3Xv$,*70W"b_bÅKQOy~WBxl_z~K5~7fYA+݊#q|cmKf!9Զ XK1{o}Z=@ &#}XR8 -ΊS\l9"^޳UaioV@b7Bb0Cr2.Wd>bƠ?2|Ə~z/3jW ՗Z^C芽+dwe$DLxCaZ B#9b h8Pd*7KהX:+nV#*pqCiXHQ?p=0q$ E0bOr1X<[-~[MNqD37|~3؏'bZMo4#տs x1K忹\nn3l36-uhMQN̼1 `R `B jep}}+3폜_"_e5ѺW1.~1<&":j(p/ ɬcdĿU/)!Z3/۟(T?EV`1xv$뱠׹qi)]_ aYfo{宗$fh!'TCכ0u3¶De5jDBadV/pO?sbp" ƓOO"<)uW6W}_l1#A$oi2Uzބ[WRCmٶr͞m-6Cׯڥ$mM1!bj ׺ur 7% R M1111bo>l?]>m\߼^yiWo7;/_/_^i[MZ2ϝ<=jW~,ϯ7^-^]\zig$-CXno&G} ߟth}y 7=#vjR]Pvh:>eѳMhC9ƿ鷅Y]b~3zM)z .|UFg\.fCrGCrm븊=p|~>_O]V7OS_?=)8|y;/oy}_cy˟ayې֭ۖ9,f6aCkh]VnҦ0W n'kI̗AC%$5X}r'Dž?P<>~ty/eB+$ E_MZG+S^^ouҦg%Xt]ZY^(lߖ'^6 zB lBf;=HiuCz|[mATw;8SAԎ^D|(uJ/EMlqɱjmҾfy $iܿN.hoBAM/gUsG]sP/>z7]_=+!5$P ["h޻I99=YH֘t|(}dp{L}'wr\ !;v#g߻zqW.:R Nk;ߖ;mw[8mo rb9xg=`_18z/E&Qx˧`5J:jb v(`~B| 9D ~uµ>_?wMu/ҞpX7 (FNT]-%No_8QͶSvd̦nZ u'Tꐐ;7I/KǶmKv_^-eQxy\=W4W'+b.ѡC{i^qPY9?Ś!*,멊]Kntݶq݉fҬ]MtgD鋏Yս϶ƈB=c[iȇOm'}A$lzϣ`_u‹NW֌//ֆѭȭ|_vK=w"ظuڥcI1IpFx_٘G񻖳֤ͭwnF&C&R}n[ؗBcz!{\v+/=d2mvpM:Okm8q_>Ơ8pzOfWbp]|ΗyIeUM@G ^mW ]2qڀt'nhMA_xMFFS_JnZe7Boz荿z8$fZGB!ɥlI;©n {C/~+-Of18X|Yzk]1-΢^JF!xRP^Q"Ub5Vymڒ7*Wr]ӣ/YǠ/Jg}uDZ,OE_ VxĒ}Eoa%=[FDb޽bpYCSzWuPkֆS|OյN}_.{fJz(j'_vqAFmUtE~l)lA\Nc8?w5ޭ"ţQԘ<>>GVF!IdwQ-? nlg^^{Y.gJ<{ݭ{.5.5&5дgwwWPD^kS{[uۦ%{Y{i<{:2q8dӬS/{ g`[?9Z1\׎y$Gb6mոK3Z6 O oVkE|'$+ktesEmͷH=$f$jja`=p6v ")XWNm1~V086 /;./3 lmcE罳K{_sRo 2NWLfҲVώN3]~MJC'̶+h{gc۩߾; /Oj~'Y珰5Z{klψi2?ðY-F\,t=bjŽG;+fe 5]q:?_ _*>3]6m94vM2޹l|mMdi PvvOJOa+]cw~?kyY_OTÊ>_cc>.fvt?=^gK߲ ]_h ,_ş]Nif̢m6 Ɏ+6xuXPHcX jQ6" qz{ٲ:Cb;Bݼy>)g'QPk5bZfO?i_\ @_6"L;fAem1O{}? M~Go,m^9ݴfٚE$景sS퉲\L(!zn-$n9+Yi:;3g@y CS9<1wGs,۱:nj;wEkUUBzsqְxͱT[cd}䇘;q&[m8gֶyEs7}sy,:͎)Fq@}`VA2mN*6ݘv\%-fvnذn=<ع'qlUz5hT}]g.=N!k1Y\#31/Su{A,%kY+:?cm)J"4qW[̗I!w'&1k dk[{]?z [̚c׳r]׏h m_g_; rsekOB byO=3/Ow~83Oy F+3g;?M8|_콣,r,bp-_ݣWm;16];`{ac"|ߊog:>|k_g}2kU5Gu޺=ZQp-{o%F~߶wl N]~U?ro=[1muerw(ݍY= v*ydG\̎)QPxY#{erғּ!ksJY{vl+پCbOGKi*{|opQUҷ;Y~8^ƚu?7^U̲gkDkhkz[l+B6ۘˊkc`WhbrPcd>.f0cgq}捯!9rN#EoTѿmQݜ(MonIږ c? AS ꍲ '8bD[m>Z!4_[vE<-*Eʎ "瑾vwfz5g?~ FF_" $b2F _^fSR]k6Z١MWAYYV x[ ۪ޕO-0be3R%a#hd+Jo nKǶM+(xByd[^8>cC=9<_{{ @~y`[{^}^1{s-֟_s-:پ:{\KStmm1ap.$v]/Wo^.#ƙI٨c޶݌a޸XƶDcœ\)W\p-➸{m1j7g~5ٚTe+)C/v,q=wZ v-n犫 d,?&}{Oq޽/x yfTܔZم{j]X1u{@׶Y]v-{!Mio|Ѩ:6m;0571rp!}ǙZN4i7y0fA{#A q弴ߛFBDWǤpy6B%g%1n& @,7G9.v1 pϘJbM1Dtc+`.18D|~kmƅ p< XBbppg 9. BsOE&}du' r7yGӔ _ b0< nz矖sy ;,O~ ?/jw ]uNab҇ZjԻT//ϱxψ{sNrNI\mu)pA_eKgB"3m")yEW$jwn-oͽ|SL^|.<m?}||\y\=,o0w6.^^^pȳ^e;yz.Y_/oZ. zI[;Lj`?%63'_BUi=bp.Tu\}[\=p|Y.I+|ݣ_Î$oeHӻK?oV` !lݺmsX*m†&v黌?#*^_%xu _I̗.UTa=j>tO ~?x}wy_&>ci+nh·ȣ֣k[ie]AGBoowV6磧'{<q{c@{݋r +|><1?Nwx[D>{?g]kiΦ{JkSw3:8R O`Nu1V~R;z1aF7*5}跂b%ǪIRHɎCW|8t!_(/gCTsG]sP/>z7]_=+!5$P ["h޻I99=YH֘t|(}dp{L}'wr\ !;6'oU_Z[wsW.7R Nk;ߖ;mo?5kPg1ChMkw]k%” bޟ.3?=v }?G7ԶHTC8R ޫ/p71[>-WO:jb=eb2>Xsj=Dj}~Ud9`us.baO89)FN(U)O$6܈)|;2fS~7RAT9 ߝt雤c۶%H ~y\K j>0Ug:NLxbr+AXhyhmCXJdA|VL难Y1x_lk(0V m|!?vxkd{Z Nbn'<eSNK?R^nҜ^׿f~yYր691"byz$׍ޚ}kZsoz=_zYTbTs/ 71Q!ʏ}Re7ۻm3[V VG?ʆhWJ~_S38ۥs`^y2)>w_‡9[?ٵ3:2|lʛ@~<,ЭqDŽ[}WώosyǬI!3g 9}Sοȶz}}^ުMƯ}Doe;W#W<-/L#հG\g OG (.gep}o#}_*E_NJ3 칊}w|Y 18_<-Zi+l z"%;Pqऍ98L%*cJ./{ ʆn K!vqĐH.e[H+H>붰wn;D{hlh[[-K^M0:W^!u_tHpLH\TF`Tv )vE~e|%[!F׭ %ݡhFZ+_їVc޾G-fqM §/+Sbɾ[ǰ#c"1XRepA;DGBV%~X֞|sݝuZSg6לzy'.ž̯\=3rPmiuyW/[;ϸl~ #ԶGu?n"q鏍GqX .D?Lx_1g^oi9k.&x$عgoNog]c9Aw܋VOعڴYuT[[T;{/6YqNGI-Ltl5O^Z׳r.6˨}SE2F].w,ʞ^ 91x\i~$_jqbHkG'?=7Cg9Z "<\ f_7{7 ͇Ne1N>xN1x[Hӂ,iqYYŮ!&ћ_Fnp qY \;l0&=:ͷsH@!PT>m}-/vCUzO;q~ ]?kR#;b;$ ?-c/7b8.f ꀣH~PBJ~v@trE7cHD&Y H)mT Qpߪ')ܶM`lg%&n,CHƃ|-w 5!S[jL{bk|6%ͷB<:pZZX5g}C8'ږ֗2˦9DG*ͱn[KNg+Z:]1I~[=;N; lLwM6u*U3ۮ= wmm~p~01b4'}h}w;VW0SV%\eK[#[^]w!9viFoVbęhpcbPn9㬗nYb x۹f=yg9v-]Imεxc@qM}|`q0gW9gOyy~LUYۙ:<Ҥ˳c45}{_As֮g^Q7C!ruٕNj}ZPe>2ť s}?V1]+yԵl~86z5\ϜUObuS])5mKJ %rĚW&f?D__h>Cѽb&fWD-UsoLB'_ӳK?&ru6 18}}Leߵo}㇪+dV6NJs>h凙~e87w]ƫ? v^?4:J[-~C۩]ŦΘ΍vi!˭>0ع'\faĻEyH:sA*kV|#!x$z&%c"X"Jֲ?Vt+~ƶ R0Di蚩8/sBv5NLccmZ(Hkz=9 xk~A95Ǯgm-y#u׻:蟿κ$v)F[ʌמ7?߅>jK{&f^4﬿qfOFq93g;;͟T%&;l |s{_fh"]Q TVAƴO+3!ތc/f{ZPwg+3k]N-ekXgͺ)1k xC~NԢc_,6(:~"_{촰Lquo{+jj_5oo)?xItgYIyGd{1&t!!1x8&6IC7稞[`ɾ;AiA1pCQgN/۶lE egK Uh }(IOep֝y춹kC]n-ɾkL 9 DvA;,TGu;@u9uW|qtH\39#q ֱ=GԡmJDk1MlC Pyk.G;;יXzkU5Gu޺=ZQp-{o%F~߶wl N]~U?ro=[1muerw(ݍY]sg;O?/Ϗ ;ٟ<ښOۤ|%dws( *׿4T*Q{C%Ըx> C*&4Sl>k06ź6ZZ9kqUeEWCK"mc1;Z _zn}м4`TzN&6?{Ev瑾vwfz5g?~2ݟ~g 8.i\ȃE7_^fSR>4{*  EHA=^BuZtBn+Y%-gV60z, m=zm+ BUh2sXRi:DŽka+d%غvދ?eǖm֡LϾn Zjok[ '-v%z2~r1LlFuDf e2`Ecbg;y˜\Ѱ+ws֮TNzK.Fh=v0&n~0ɘ>Mc=Y+FAo=+keQV %i+h-LyqwH` _k%5ږM/qGI~պGbpQڞȜhq[/VO)(68 ~/8LO(i4zt{l?̃ A!۵!g=voG ##1J8mCO >b0`?z1 $4nvIl6t^%3Q43b4A=+C҆NN)C D$a7HBruSRQ^&"gUf`߶+~"/\O u1*ȇs13QNO,_6q/YeqSk-qcF =&\ȾXZ -{ _7!o7-)br Q`|([|DmU `Z '-:y6w͗3/ qsg 7 ppDȡOSo?3?_˓{iX~J|<<_le<x͵<5f+pK G-W|b̛okzIyJgo(o]_/!T1\buXmr>*ayפ~Mބ|{ucV>xz0/<[8oCj{[n=@hI].cLH0W n'k}$|(eHq尊*Аk>xx‡lix]ǟ}Xڊ="[;F-('mZaWP:.;e[ݟ?)O{G{:{.q(X׭*3m\J!z>M菻H^yw|k뒷ˉ|:yN]kiΦ)^19?&ixsn*'lGnc.s1υ$lMU[78cHx+b\!QZ^o~e[|∾{`^^ouqLJ mYxؿY^(l͓_yz+mҝAuZz6b!CG!W=vzχƌATw;8$PAԎ^D|PuJhC-.9VMڗBw,OvӸKݬƟoBC?d98tJb.H`Ԣk{΂]Gዟg5dPY0s[mQ{76'1 I~|bԾ/,~xDN+!dSmZtp9ɸbpZ~qn"}WǍRs]K:f u?86rTM*xx)GZD-+Wk5^1Ϟ21x}} 9D ~ub>_? +ZP׃J {a̡O1Jnxjԡ* DjNc)N~N` hs;7I/KǶmK:;Гj.s9IA%:(4}`Hs/ͫ2t*+G*0:XvW1~Pv׍6;;4kW*Y1Qkc0g}uﳭ1tVvZ185Ɔlwcn_}Ei18>ۯ `֥uW}; /7iN_Y3^,k@[͘yMgK]G߶c -m[ eC+pLVΉ٣_:]ZZV`-_++CE^?^@3}RxujvcˇD0G{s٣yNWz}@/+y $VqU2ghsa"W[&;Abp^<-L|8`%CQb6WA|!*cdI_J aQ]0V?X/_%Er)귇!]AGYs!CcCb\mQXj:mrH&mԗ|Yʫuen z_e*P v02 C:I++[.k>8&tݪ]T3/KCgm*Wr]ӣ/YǠ/Jg}uDZ,OE_ VxĒ}Eoa%=[FDbݔᕴ>!Z`aY{ا;zU lm9VOյ]J}_.{fJz(j'_vqAFmUtE~l)lA\Nc8?ω/5ޭ"zʞ90=#=E8IM]l㏾1I\RIx-28-e׿SAb0_k_GA/@Xi8{ 繑7pc{t R6L}ΛqCpCa*m݃h;w5N|}S{jw-2={x~3m~x7{CۨAFnL!Æ{Ybܶ库}? M~frۼr6Iք\. &1|lOeB 3uk!q#_JЩޙb³m wLH>1b4'}h}w;V\E6NU 6WȖ{׿:g ZJ5F[I~Sg9[A挳^}fm[QuZ/߬':ϸ]`;gnE+m޹vTw8?^o |~8N4*5gl8>/Ϗ ;|gtyvUam&[˺.fuvˮ<^\"dq/7ݮ'N}t_X۹N5͎]1t鬕0¬w>;׿:\a|estmvb0 |Mb,$Ïʵ/\[\"l,2)2/@XHV*asXm|Ce?y-C?3~@@98zmc?S҅e±%QΘ΍v!6P]'qlUC Ow}d1n\dq͊oVsD"DPdL]CDZNJ{ضA|qC H8M~e.cRȮb&݉t|_ͮ369 xk~A95Ǯgm-y#u׻:蟿κ$v)F[ʌמ7?߅>jK{&f^4﬿qfOFq@)%͜6^Sl찳xv/oMvӤXXA~Xv)FWSqX{.W?v~X?k:tRTOެoys.!d=9ipPt<`}ki/-kmQ~gxF8@u6j;7.ln2رUQ#!9ǫ` c1x4NJFtVw\Z`+6oˌufo}.־^l#mukn֯8}T~%\92!}ɶQ߮+[ǂS׸|n\h_}y ut]܇ 7JwcVpםNczw>3u'<3??S6id#:&e+SbH7翍ۭr{ICŬ1Աck~˱j'%n^_ϩRwhx~=ۻsڥF뽚B-]m{J^CS"kX;?P&ο-;!fq Z TK) -N#7K[LzŔƾ",_7j}vJY~ l\>* Q7mE)CX{GvA)o^Q[!|ai[z}p[&6?{Bv瑾vwfz5g?~2ݟ~gnmG4.a2_^fSR >MJ4!5#}.7rjHEsYv SK" jXDٌDI*/YJ*Dﶹp[ےyfm "^+Pcbp0Ϊo+$똹P&3\gaw Y a;\eǖm֡LϾn Zjok[ '-v%z2~r1LlFuDf e2`5'|e֖l'ϕ^y3+vc{?ZzwWۻ*&˼&&>~KښK~V9-JHŌ/tZ۲2+rpwMZͯov$y/^F:X?yKmHmambVm6T\6Kev ~Uw*ۭq2}LeY^[Nx<{݋r +|><1?Nw|~ٯ-h|^Ns^Oi=+WR\xozc!&86|Ybq&@(E,F_VԦ節-NZRK؞#7 YaDՏys[[A1cڤ})$dǡ+iԃBM/wq!*L#Q9 vMKk/~ž֐A(g-oEY$Cۜ,$gkL:>J?xSx|G ;9Na[YkeZXzqW.7R Nk;ߖ;mo)_zgٮA 3O,)cpo J/LVAyh1XM\1ɂ'rwxbzK>p)(Vo1Cm#wS(bpptj+S&&3/5CTY_EY7"*ܙCb:pY[P5%gmQͶSvd̦nH48e=^K$4/۶-鰵Eb˫\/^^/WX'1P!ͽ48ywec]qEFKmםUڮ& UbL,`Ίg[cG!HbpnkC Ƕݾ[ bpsK}_=A\Qbpoʝr5˲!ٌtkΣ#n֜%_ ՚}Y]xL$׳h͢| ?_?֞kaQ~.j/+޵m1p|Ҷh:P6DRZ ڬl=uYY90?Zdjy.Ɣ|Q_&?!THq"״`3C߷KknOL7dDSK>C6 }Tﳰ?v11Nqhb=s'N+;lZ߇K:OokGfQ}>Ҿ̝+?4lo+ʈq"d:-R_ q;kfe61~B0'ZU)΢iaZT`%zA,WvVb#1od{L$K[1bpwʃ.Z`aY{uwjMq_s孖X73k2^Ks]̔zCQ^Ol<SM躽F?6Ra݃01}-p~7j[{E ?y^tbϺ+g`-a~jogGv?v!/2?EA}2/Z^폙u_#z=[Y+kc~nϼ7^$cۥk x_ Ti&'4i9TO o/:Ǐ _}B1/'-/Xu1xgp1}SZlD&4"S: ;;m!M ֫@ŭ.fe_< F Go~6f6r+zA>8R+0&=:7w搘сFB̡B}8zlЇbokG=Tw Ӫ/E=2#Czn|n12r#{`b<@8G; %gD-WPtc9MtkP$R ~AuOݑ: ~mvQb2Dnsk]Gڅ[cd}䇘;q&[m8gֶ_<vYOtq2wΠ~lWRۼsp~0<qxS1XiU8g]tOy'czw^ tV ԲftyvUam&9k3ա9޺JE>- Х s}Ac&!صM]cWy>]n[?$YZ7uٕRYOƻmS~Y+,?U~yaF_!P4M=KfH &YdQSo bB_u߿>{ھR2ZF7>?Co2 +yc9LvL@W7Y[Foފlysez!1o7Amq굱{L||e #۩]ŦΘ΍vi!˭>0ع'\faĻEyH:sA*kV|#!x$z&%c"X"Jֲ?Vt+~ƶ R0Di蚩8/sBv5NLc2NJZ\apN5=ZfksofͱY[Ks^wGa6篳/oֆa2' wO<'꞉';o}SQ\к֕cmn9~wzgsiR,:Ngp:)ib.}5A=Z}?q٘ x1!Z#.ϡ7\Vf^O;[V'ΚuSbmd#Zz؟'DMEv]1ǛK'?+J-@LN[=J%k]$&'?78 1oNZe*9G/KޱNo-M?s~ضe{-~L/ߨ?[y诮oF;XCI/;c\_ ר<:u mI]c:mȱH&׵4!go:{|)Ffb˩;f⋣s6FI͑U<'m7`urb /7B歹\ggbOf͑j[GW>*vuY͐d(oWCcko{c>SG.~g׾<tMLN1+NZp^Sv#y?uOۤ謚H>:ȗL#ߜ6n% T1Y.SLNVB$-SL؏^}&.)ֵzYۮ.>u.7]i;;x~ 3bI? ˤXLi].r[qgm篔a%wMO$! 0zn}4`Tz8/Ol~|;>fAV #}Ckx .1?ON=q]>Ҹ:o^biͦ}_x:yz ӥ"MVD+}jiUA (( {Q@%\]6W.}Kp[2Ϭi1-]˕ˈqfgc6 ꘷m|7cX7.8È(|EJqJ/^gb4W4)[y~ۚl{}^ucf5-s'vGw2^;kjVg;nX}ѝ3`d1S}{V9 |2gzV>vSAKMj3W>[F{l{ۈ;!3B1!S >)t'9t@}cָ2]ng6{}xL zϝl#($q1.I Z]<- _|sbb0!u@}k!F%wJ-$F.[NC|iwf 13;_Z!-k/gon,@ @{e@}N]xtO'>r~zz"._Y`l\|~yngW7i\?w_]<^|{\xiwjr饝/wch9)~Jҽmgcyc/{XOoG^˓ߖ]o7Vfy>rL{] `hK!H*wymkgGW?[#xq{Wd7'o?@[tϖPj>'_BUi=bp&Tu\}Z)|U.I+ ѯ,ɇ|`_cy˟tq0(0߆n6{<,f6aCkh]ƌFXazo:O/$H?DZ x/`w-V؅L-|4CC:*@f.A(gB*D*R7U=]{y9dtOuULu͗y(NaUJڡw<*|'[ۣ'-|ծ}D#{V獯,6ouG׶ie]Aïc>^veYsb.ظwvo['ObaR{tNk@`c9x/=/Mx>α96k\4ξP+Ǐ0|lz¥sz\OY|>`;^}wD7g`NwY`<VA| *{M ~+ \o~j~zcG+Kb&G8Vbh{AlI/>=7NSr7qF,d8wƢF8=ssp݊*Ecs8zs$SnŞjhC8ۼ\/; zR7k]XdBV8Pb.P`Ԣk;& upoQ5~Xl ArV 8Ez&P{8g!0gO9kNVP|G wB6 ;mA/O8 RKpٸb0hg},ל+y.;s{5zkߩ)|;*_WR ~7S Dbn|Wh8Q Ϋ/p71[?-W˷^1 &&3/5G7/"$3 g (J {aLѧw7<[[='p±m9Q1b}Aoq:iIzmZ:m,P|+Q ~ry]^x|6x>)g ڦ x*NL^vW1~POycBڬ] UblG0g}cﳭ1tOV@V X#ܠ펟~{l޺$bO zZϓ^^\Fʻ>9CrJxi-73Kk">:ɼ9S U|V_D2E9 9XN_?k/kaZ='j=O*^vf{ѷm1}kƖ@]@qA[5Ƭl9EGE#j վqi<Q~ ȧEUcmAAպ d\b/]j6|l{ƫǜwþmZϹ_gzWqC_ϕ@0G{s٣@|^\V7 z_TjQ ŭ0kd!EF}: W@k;υ-T(mTւ_umdB(=EGh;$OE_ V8%Jl{$l`iK;E *ЗU]!n~XrO>5_ռr{rC?ԗz"oc6RZ:Xf2_oX%/}e> (?6v.;6:}dñt_kqR+s)=9ӵ|mryn>VT+7J=߾Կk]Eͻ,cL|yK/Ţ}^{Bkz]q#.b_p[znعrʙ\{Aj@?50.CaK 1M<^_8GؚCb+󵝮m{'.cH;"\֎`GԖ\׻5#+WĥċRw/#;w=9^k$6sy&1om>/‰k-㓺e\gc<B/(t^ԿWr>1:bG41x<9޼ߏ>q|+MA\RQx-2ck׿C`|@+dykXBgp7QAl56^(pP՛(/o?2gަ"WsujQl=)<]_Y1? o6X|=06j]Ϳ)[lgOxBϩri>ߎ ]'qIH"[r(b:7=%dO\˅K@ 3ck!q#͡dMOT̽Em wLH©ě9<178ڡi27N*=,z/N8<r~g jAcdp pfֶU'rM>}qA;5hw"ͫkGcǙo\;9 :NTE`e|U|8vbl6o}J͋~L 7׊]ECd{z]mZ想&b^f%p\#Lcx{>E8 pJ$۞ x>H=b0}Խd==h\ \b'agClAQ:ϞxV+jxO ,+PV?w^Ɛf7+9"`G'0/Sc(Y^3m_$N_Mo`eL _ րߝ9A6{dx{Fd9/W+:LαR+ ̅4羙1W̥K1jZM̓ Zm>t\b^kk[1#{8OʆFe{n>E8q0 x>P[PAb {EkRgխ:{.1x;2kfy2F'I=_]/J-@nm%¢u#b0&Syl7A7Ơ(5lXơFϱМp€|km/FZ,~o"/3z6{s>El@~تtKw۳Oc\Fcq[`mKlщRY,#M_I!^Aw@c9cW|qtH\g?9#q ֱ=GIC(/iwlǜWU{bϭ2㝳:.r_'?#mkѹۣ_ s8 fH_ȶWjݶsz/8cmόxA5ɏvl{@cc;AnY';ZP/SVߔ#[MMyVJ5??S6mh#Uюlה9ctQt ^PsV+ڗjZ/dž*?O nT>h{<~efg]xqܨޫ]KOO9˥*)\/[kQ{>1] 8[8^1͕_kds~zG V$;b1wmo}KFڦ3~̋*SLB25ꦭ6VsaXqFm}$z]8-m#=NPAdܵ`>7w.[Ě]/zgn>^>ྨB-/^fSR kM%JHߢiA hs^yAZ-0be )ǯChd+ 7ے}fm `1_+PB/bppUVHsCgWd/zWXc=P^~leѳ{̥vqϺ =aפ .Q\o~!3>;g bywsz˹h]{O䖸uקW/sFϊzL5[Սk}b|aL.ֱ[37/j oMܷu\;/.{`:/EQm*2zb'@`O`{΁Ḟws%Zӣ.&7ؓ=7i"\$ɕq#Kh%qSN<&?t@QmOE $K gi;v͊k6&@G#G ndf?Z_$ɍ+q |KkFIrØJoz bk/DK$z%)UlMTb%AuܾZ܄[u n$jP^vf[g_୴uw[9)?Q|1 W$QI\J^Sƛ!dm|RjYmI~r ?qWH18ހPs |Ѱ Ɔ*Έ=[\pa1oPsgH6[Sa6޶}ybp$I$I$I$I$#]#\1 Zl_*[@-¸^=M =+Fb9cI[Y!~u/)'I$I$I$I$;b${baZ!`:ф!1?]^vykK$雟x-I+OϜ>=o<{:^zmY+#KDQ smb=.wyy7'rw>4'AQ_ZK |}} $y%:{ٓ=~|{bKo;o&xIDAT+~>@[vK[mc7/! g\^xB&˰f)\nȴ> /mnQ mTxZv[0}vQoCj<[f csV j^Dw9g X5Z*]"v^ˎzvgω`]~۽70XN]/ЧԼ%r⃜*|41x8z\G~kyEz9 ըr)^1Eu@6|j-y<$`8*/ATbo ЏCb\AQ@m^o~@ͯ[y숾{`| yݒiG&do;P;4wXW,:O-tن]icQyo9b*n8O0DZ9V~b9cFVlBsbҾ߯_( K-`wQta_({/ơ" \&EvL]>kٱB2[@qn}-M2͡pB`/Ϟsqל@ }uwl{՝'}%lv 1*<Kp]^nn 9ڙkK5*s{5zkߩ)x`m71}u9CL18I>,N ML-OEb:FW ~̯;K`1˵*j=~ *2X?gݜKhO8)v䪂REՔLp#±m9Q1QaY v>BTw]&鵹kض@B1uy %,j.كg^+;2U`c{]qE\ul㱁^F_WitgDٮ鋏$`Ίg[cG!螆8LF>ԹA?6 u HĞA-uqE~s ܷ5ӒZng05"OE}t>ysȷ<}!>d>r^s{*^0~۵zNzGqqTocֶ-!₶=_kY 9sv_TA#j վ6I{DR~?|TU+y̠j]zs2.1.}D-N{Phlw p=ԵsdžsVI.9؆NE /ުMW>"7{Ib0U\cZv\EX8_bj/ `z$' ,1@M!gV *c uC "lt1VR8q3:Nurmw|ֱsnb|[mms\m{Њ!x^g`R7 Na ԂD>[ ,nI++[u,+ڊ=+, KC'PQ>[ ~+׵= } tV7H[hhZ>}1X|0+~z+7=&-ub0䮖>I/KkN8"*%PMau Sc9eyb_׹Ʋ5zC(yyW/[l~!ԶGqU]s}Y #c=fZ{Foυ?\tL׺+] MX]~SmR{˕zl}߭w]+"x/Zme9cγu\Y-^S~`~1üSDQ^̜{ϑ}(g+3r΍t|Y5j{#K3Ę$Id1xgK}̾7{7 G`lk1>xN1xK޼. [Mf%ɮ!hzh/*pq 2+kZ|~G U.Wǁm-{[3o۪f-8r~GlEZLw7l1kˍxܢyA8 C@Xl⇢;cPD&$Ć$Z$=cG$sܷ3I 7 m(seI@_j1:'ҏ=X1 u~6eoυt19濶A.ٛ$m_lz5t!"cn˥^|VL銉ozv8wmyj>'YqϏ5#YvFmp}T/1vaG۩]6;@ňqq_bQ{p={Gw&x?}| rɍ3Ę֭_/q?okضK '߻ DSWxT^p˿ oh;$>2X::В/Zt ,LM]p"q7QAlB6f<ެ}PUqb2}V̙WEWhFTg3z`u.N}j?Q#Jyu {/{;~lLv|ǾRlu { p\dkB.A3_kp (!zfl-$n9 ꜹճm wLH©_ω7Gsxb0_oZq,{u_ĞY#{9M)\g嗖#[_Iw'99oNAD:3Kwͬm}޼(xӂyߵmlX^hc?g:znj.~<_ s|/.7VOo:>[ɕO9[rȳ&oJLN>Cׅ+l[n"pܯ*|~Z}Hh *&q1ņ76?h?׹u>("Z> PMe7ZfzWWg(2}.m>s]E]gl-!_ӉYk7TiC&o:{})FVC^9#qݟI0HH*XX 'e m=aY#%Gr^sk^sYgc|E'sz}9:yy{+a.{R j#WCv][gUw=/ю^|tuLz,p}' 9+ߠSzyZzOXXg~*ئ mD*>1PPFq+{ICY%Ǝj<}F_p;SAyi9=zls3Ʀk|?7uǔ}+qImkSh.ߡZd݄T%$vqD׽g$ڹzї3ȺoD7sןC7t[|:{byoX[]lOBk#cz֗4nmJY~ l\>˼IK"w릧 VsobCo7^f¨$zl~=9 -D] V #}qwPqM_e5٩EAϣ/daQ/^fSR k N1W(ˊa.m¶w%*lUA (KHA=~R@%\]6WHܖ3+mvW=(Zqȶz1 B3=Ã[ޡF+d,b}ų?y[:_/=:\jok[vM2{ zyi>3s6 Ƙm|7\ﾬZ0"!=a+[N֕^Q-_=+Vb0Qc 2ԩؚϙ#y]sfFGs₱GdZD6g{߰;QsvcrԚ릱=ɕ}N(SY!ȕ}Fq ~}Sڄ8{-p&x*Lj!ӾWt|[ >$k/́'Ē_mZsOzWD\t1C`|.pS OIrfm7PtJʸ%]M6/7w1)=w2Ƥ |1xNhet$D"?o u-Z _Z!οDµ`֏ӛs\װ`-m7وg"1b{U1In_S~wМ]\3R NƌPzk__~k!)'VdmWjX8|=_7{O-dk"3zGb0͛/`_-sꭥg׵~]bpC7>|Jt$ɵ@7C|$ ԆcMGWk)^6y𾔶?!A=`$ _X4G~=+!j{s5nlrEL$I$I$I$I$I15i$rlpY)ֻÈN nm&ʃ@o>mu,ًSH18I$I$I$I$I.- h?step',?㈨x~(zWU n㍿.}ح"$I$I$I$I$dB1xR~W ַ/WX3bp;>׿RCCa])'I$I$I$I$%s =1t#~1kb/0^i ܋o>v--S\?6sbp$I$I$I$I\2]]8חO~Ο&kjUpoW#_8R烾7lրbj_~$I$I$I$IZTD_"v$DJL1ҭ?S_ B^SlokKeZ ?);^]޾x%Ir<| ӧGS}gry]g0S8r~>pxh9j~NལU'?/w?r?/|<[n'I~sr9&] p=b0{9 [!S_rĵME5ӄ__bp7n7ckK}*Vo/>{osUL}Mps|\hk.|}z |F%_նˋO6T`yVhU@U [>2mOMg۹_ԍYB^FL}#ky]ېbֱ؜UDl†/&]/FV㷷V[ 浮tAxS8rXE#Rv?/ IyG'_k=c{D4y;o[pbѵ-bZaWP@]oozV 6w[y{ 3Ԝcα|3Zm^z9 ڱyX{98i='037uOߓl'KWzlBdxw) O#b)ٯGtha?쟯h^_9޳E saD˗ r\bh B (E_^/7H׭\@:gXLح(] 8v0O7G?zV8v\>[s UIR~-?x!.~q| Na;gϬEXۻ9ėq#`]X9W"}]wJ>j~mڿS5!17R nkMo.zr|]I1-0+ޗ(A)?b8q<)ʾ4e0nۀw!WN ML-OEb:FW ~̯;K`1˵*j=~'ș+JkR^4{aLѧw7<[[×d{ '6~ N)CG)NMksұmcB~_Ab"33IAwe*]ĥ]6BfjHЭ"e/>9+{m{ƶb0Pmwc`/5 {Z 1gzS v[s-`3濒3޾xZr@ Z9G2orBso:<Ǘ5gQyS}oWO_o~~VωZq(.]޵m{ p|ڶ=`ctP6D\ֿck1+[9gnQr̖wX\DVZ79힒#J9G_)RU1f03zs2.1u@KQwhlw לzYkr^]Lz{Jt=YQ} >/^vjOֹ^Du1xu2'{1&vSH~N(ZU)>VݎQeUE-|ά{H L9$& ' ,1(JM!fsE2(RM@!6 Kc(9D1&EO /6-s೎sK mkj OK^'&xn`~Yy>V7 z_TjQ ŭ0kd!EF}: W@k;υ-T(mTւ_umdB(=EGh;$OE_ V8%Jl{$l`iK; Kxat7~n~XrO>5_ռr{rC?ԗz"oc6RZ:Xf2_oX%/}e> (?6v.;6:}dñt_kqR+s)=9ӵ|mG^X]~Sl'8[\`5.m$؆ٻMh>"c;^w}w @ZXYHL fWorhӤ,&,7^7z-0k[I3,"x%^l` ]ǁm^.8j ݢlE- YK1FniOQe_nij?|,fO$zN8 C(W3R⇢;cPD&$Ć$Z$=cG$sܷ3I 7 m(seI@_j1:'ҏ=X1 u~6eoυtr$-O|(r;IL<ˌîsHw[.r+>e"OWL|3m䇰ճĉkfthS[P ;q(\Wlclk9ۏ3~~<0|3_ѶT؅zoV4נtb1XـjOq^6K 9|{#;w=9^k`_q }l/}e30kN\82+*2x/CMؗB/{W>^^h jOkn_ m`_k_GZ!-]d8{sܠ.XqDʆ(4h30ہ6ab6#smn!Z^"yE8*w~K8f{x~39Vco¬Cۨu!#7? na=Y 1n[El@~تtKw۳Oc\Fcq[`mKlщRY,#M_I!^Aw@c9cW|qtH\g?9#q ֱ=GIC(/{8 oe׋}U{bϭ2㝳:.r_'?#mkѹۣ_ s8 fH_ȶWjݶsz/8cmόxA5ɏvl{@cc;AnY'Tz7뭦&zXJ5??S6mh.2z ?:(:nŮ=CY%Ǝj˼?eh$$Xnڊ`SjEX;vnP Q[x6-^θyKgϷl+Tw-X%MݱKC5f|=~9pzgn>^>ྨB-/^fSR kM%JHߢiA hs^yAZ-0be )ǯChd+ 7ے}fm `1_+PB/bppUVHsCgxrj Y{,g>u2^"zu>r/19Yֶ'운9e ͯ#}fgl4vA1ona}Yc9` o S_!q])+QWz}z2ghX@YSUZnck"?g` uΙk[|9gqu ū9Qs%vb7׃mqtֺ3xYǾo}̸޼15؝+adB5#qk' f4hN<#7q 9ocoHXx_D$sZ, ]boo=Y^G˷wڹ*&˾t&?n9>. Oi_ľv=>^^}#jۀqŌ':*TkzGI [pQ??aL[並vؑfOmnG^.>kkZ6ƃرul96gŋI.}s zU=l}y2]P !,VQTz?/ IyG'_k=c{D4y;o[pbѵ-bZaWP@]oozV 6w[y{ w|rS_nj@`c9x ANl{>_<|c=#wl;Zmq Puj^1 eN{l[T >X9,b _Q&C?fEGLbpEa.ŷziE5n#%vKJ X_\_@nTb\P>e,mplBfw"EE:qzG :T<XqHbF݊^>[s UIR~-{I]s/=b0WwO<^)l|_/˅|q]^nn 9ڙkK5*?ܫ]kNՄİ^H1}C#;{s\b𻧊 J //_ ?QQٝX7__|}U;g}hg#2uo?1D18:" o\\!_,߮c>{W/߼\Ϡ"Ls͹=nSN(REՔLp#±m9!-%TA QS ߝv雤殥c־ /Egjg xPm`gr>x|?T!Unw##|qFkױzE}]jHЭ"e/>9+{m{ƶb0Pmwc`/5 {Z 1gzS V6'5j—xWrOKhYC??fzks`wϹ|Y'5~gN>`+;V7\{7u_1v4PO&VOwdQh s`}/G?m,1x#q̊VgmpF1*JWv-;",/n1xcvfCb0QaI ]Ml z"%+B\ .BPqA篠lt1VR8q3:NurmIs೎sK mkj OK^'&V >+o?Hp qM$Y FOXgq+H]_٪c1_VTU\aV_: *Z+_їVc޾G-ؒ@{D´J烹d_sXm඾1,msg!w5wb oUK>_U*,7sN}y'f=fcm/žs)ek&8/QjG_8/qBmc?n"lG8{0/Hg=0wk:-^1 N̙uWk M3yn>VߔGrow~EʻދVwٞ1mmY:.տGQ9YS\;XNGz3sw.,ϑ}(g+3r΍t|Y5j{#K3xD"1 K`)rMTe z]}_"y~Yj?qWuuCʳ- Eȥ>\ f_Fmۄ#b05\|GR9ݖK\'3rL!l8q"97V?xl\A< >ۆf`x} ~}N㌟ak&Gth̉ ?N"w\Ԗa!wj-F|Okb0 a#/H&zrqW牸z{ȃ;ͯo7gj,޾mE n?>(-' F 6Y1yJyI/$q \1|'%虱WҦ'ts^W϶%2!Up ='|ioDZn9p{&*=,zlNϾS89,96F/\+xv &Q^>F׭ '_kfm%#sM>}qA;5hw"ͫkGcǙo\;9 :2SVߔ _ȩaYri#fUa6wڵc1t|׮>޼(xӂ2;7@5vIp/ҟ3yo =75??M9>'7Dż])㙜|MN8|41M ھZps`W8ug.F"ϟb0 ?^Işd`}0]lqQW`xxѾRk1Eո7ͱ}!v~u[q`]_Gl+d78z1v ?`]u)L, Q^nu`=5Y]6=8Q V?䘃kV|#!x$zC15v,Z+:?cm)J"86\Ƥ]` ݙ9c`B&I3Xr?L9ȩCzZ.2s|0c~u #m1 ו _wOsyO=3g^笿83O}y夙zymj7oqvꗩ#n{ DZ羙1W̥K1jZM̓O/L+3!~fc^Ǜ3iQ`^oVfr/./#5'ǶZ ~}&rymZgUw\o:}E:.M}{ ;RE.Jk[EKsr\ /rha63+Gq{_N\G8 k I} wblc_rҗƍ:M_)˯͗gbU5iIdzA!ъv7p6 * ڸzs"6F%덨zdsZG&%zX3C˜k8z=ӳSvG/p_¢>7o_dmͦ>T{/E=\1>u|c?-s} d.{LE{e෵ &mqz4~q9]c6{Xw_Xmo-FD0Q_gѳa+C5yP+Cڏ 9-;gm;Z|wOlqأw2~WkSl7='w΀zLΙZ4G:I0ej^>+(Z?d\/o Vgz؄8} 'oo/+=Qh+s^z's|zbh~i?v7$7,n} E$I\ n|C~'Bs'cL .F sB++'ͧ}+E*zcKW1xZyݦcocyD"^x+ci/lܖnw[+XB3\\ =$7);h.I)' cF(5|{/`/P?` _?W&~zǹ ="?}ѻׇ5aν}VF_F1VZb O)o_}•$,+Ik/oI!J %x'Rll?})m2Cz-8( |Q1_"ZW)bhg6{>y[#"" T{\´$I$I$I$I$q| Rmm?hN㯶{__ƚ$I$I$I$I$I*~kdmmï[;bcr5D"$I$I$I$I$$(?AѾ ߇%8&ח$I$I$I$I$I>R N$I$I$I$I$H18I$I$I$I$I`Z ?);^]޾x%Ir<| ӧGS}gry]g0S8r~>pxh9j~NལU'?/w?r?/|<[n'I~sr9&.$]boo=Y^G˷wڹ*&˾t&?n9>. Oi_ľv=>^^}#jۀqŌ'*TkzG+*vS~~*-§&ų/,b /^ #n >\|. mHclyl*"6aC͋].Lx{[+ǭZeAB<y)Rz9 )X;T|Gtwk{b哯ڵ}wOXۊ="[՝F-NQ1+vz^vˮ{=N̿;yQz* ּ%r*^JO!z>_1|c=#T?zw\ҫ2@sʫӵcsQ՛d~x~gn*'NbMY`<VA| *{M ~+ \o~j~zcG+Kb&G8Vbh{AzN/>Sܤ;OuنÅƢ"W8=ssp݊*Ecs8zs$Snj/Eam9Ť})_Q}YB"2$7á" \&EvL]>kٱB2[@qn}-M2͡pB`/Ϟsqל@ }uwl{՝'}%lv ۍ~>Jpٸb0hg},ל+y.;Ͻ_۵ȆFrzGדSJIqW_nbj~Z ./o1jbLLf~YG_ko^ETIVg)g ڦ x*NL^vW1~POycBڬ] UblG0g}cﳭ1tOV@V X#ܠ펟~{l޺$bO zZS|&۶\"ϸ/7L䌷/r=yy*rC=Pg5eM$YzET[S߮sC\?eow}߷mly4 ZcVxΙ_QmZ~o\/s=%GzsASj_cMK%gU20{c{b9'}aGTCƦx9'#Ʒv)}DUK9abd=憳Gzyi=YzI!rF1*JWv-;",/n1xcvfCb0%d*!19Q`AQm+B*dPPۥ&?h aXE)'H$PPH.׶m1+u,_"Vh[W[8^j:19s7ʛ9?HЛR,n^# ,2곸$ŮlձT8zl^y.tNnB1o}Wk{%E鬠o(:B $!Ѵ0-|*b=`.WVb#1od{L$K[I_GJڵw\*g}kyC~Ω/ozDެlؗu.leP4J^^U%/}@H?Q\lG\wpmtcc鬇#9wW7ŸSz.:skݕڎ/S{5s<~SV?w򬮕w6ﲺcγu\3T>zW .kzX_Dzȶ {~/rʙ\{Aj@?50.CaK 1&[Zy˲or$b&!p1}c on"l؎bp}Dߝb0&"H<G/ įω~~|#lyEN׈ E-'.cHKv[K_ҝY&sc +R}}i'^G0wzصs2.N_soIl"z?Co_1Lcڮ}_0['u-˸8xu_/JTǵ [~B9|'߻ DSWxT^p˿ oh;$>2X:: Yh $++0mvƊ R6zLqA8(M]S䗷3ow +@Ĺ:O㨘r"Y1e?o7gj,͞Suh.d-6'^Zb C[_Rx౟$B/;Zk1{vOmci[+\|ha>/gKW'q-Šlg_8=ӹu_ ~ 8Wٗ}U|qG^^;#2jR8Y Fk;  ׵/\-. 6L/WxM^#ƛ8}< ,a)G8dXm|Cא?y`/*8~|dxR1@)L^2~1.~J{plC\Rt lh>bs*yij8*^Qx\ga_:1 Ǹ25+Y<=xɘGDzmⶇL%p ~jze.cRȮbY=g {\xr^~39泖K?\7u(Xh_]ȿ0G[ u%Ck]S\mmc̙9o?>SCh^DeT!^6_/Svoq[3y[Mvۻ\XAkp1Wh? hŏ/qẊ9l!ZXby@/W60"?o,s/‰k1w~D(g_8=wGKԯmM툵U{}sbyHG ~hQt}|+1 qױ(&LBFbMܠ\TLDhhڬo9y/!h-[X`͙8* ȷ֦9lldŢw&2gk=+ب |Qd?J{=4kt=Gֶyh.B0RDP!h}I Z=3v!xGlu~&Ø#!9ǫ` c1xt1aQ>-Or|bŞ[e;gufo=]N~29GڪלsG+n]rp.uY͐m6"q5Ժmص^p9[p׃kG_Ǥ"h w`ݰݝ[ j;Owj^*C^o55*V 6iC*~h@_k1?:(:n^/i0.¯k^>ྨB-/^fSR kM%JHߢiA hs^yAZ-0be )ǯChd+ 7ے}fm `1_+PB/bppUVHsCg9^2Xiϰ}ų?x9ׯ[9Gmd|M%}7 }rsH >H\1q )b&{g&m_K"$2nsi_4X;^#n߉ǤPzܽb;?JIr} Lǿ65x=I1x+k!~m$7);h.I)' cF(5|{MgM.~2y5S >O)o_}•$xiT$׀f/Y$2=lۯ;oR>:!9'6U$R Nn9{} \ " |$I$I$I$I$Is:S~4hN<#7q 9웗ɥh:GH .1+ϱ8/ `3*%j|?W?ZU1Y7xqkwqQ}J%1|UG ./fGr_[( ̷!5Ďc̱9؄ 5/^L"v黜3^a{ook] ]G~#_Y~m߂D[mj_e|zz~׳ۿ]q.z^>]2"N}yKI/9U|ibqPdqjK^g_iF+Woo>kr{k㚊+S ~g1σh[ׯI`}k&q^W#9*/ATbo ЏCb\AQ@m^o~@ͯ[y숾{`| yݒiG&doAzI/>=NSr*نO\u|[Qu0v+ĩx9̱͑OŌ7}`~rwG[x^j? BI߻~a4Q 2F-cP]sg̎֐(gBs[mqwnmss}|dd^﫻g{`\,x>}'+!dýSn Yw/2WNp]^nn 9ڙkK5*R}ڮjBbX/op#8xo]wΐ+S ~#Q1x&W[MܭUo(>jƐD1U{^}Ei1B X]Ǩ}^01ug=} 9"yQ%YǯA_E笛sy) e}Q; BUpY[PWH|nD8vm8gS~7GTX"N;7I]KǶ}E/^.ˋ',aPt}Z܉Cܫ#F=G.?֮c 0JՐ[E>+&vM_|$sV 7>#> A4mi`5 Ƕn_k@" bnϮ'(ZU)>VݎQeUE-|ά{H 0*!iAIy 7B!'|)9(AAelBnrWP6weXEF)' ^:ErfbMca8BR׉ɡCʛ9"n=,@\S|FY bWWXW1zlWXtNn4o}Wk{%E鬠o(:B $!Ѵ0-|*b=`.WVb#1od{L$K[I_`˒1O;UK>_U*,7sN}y'f=fcm/žs)ek&8/QjG_8/qBmc?n"lG8{0/Hg=0wk:-^1 N̙uWk@|Vϗ~|,<r[7\`; ]Eͻǘ,i_C![h-.,)yˣI^]9;h>}Q糕\9KF:noHΑKsbP}.z+v R645qr2b&!p1}c on"l؎bp}DߝbH!ay]$$J_]C&ћ!7ކ8,a#'T8S_lRLеba|oY#sF/ _Q\j8mC%boUp}w]-~[׬GHX |~ϟ-{}Ϛ[tX=O4>G~b-WPtr u9(B[G;$m_lz5t!"cn˥^|VL銉ozv8wmyj>'YqϏ5Sk;]#6>Z#rk9>5-ݹ%wj-F(Ϊe1E~>+G+`ܢF_0>Wcgޣ\qW̓7Ûo5_fo_6~G ^#m^^v,ؘ}aؼ$ք\. g DO rPBZHms+Yi:9s/yūg} 8Sě9<178ڣ,.'< `*/-G9!Ns#1r< !uu+sƪ M'g1okWx&'!$I'(>xm'1^o wE]!IEJ)kxU+dX`9+N1)>pLFoފmo>2g[!ʼnՋ[zXJ[& CWi瞫o҆φ(z/`Ϟxˬ.xO(Ͽ`rB5+Y<=xɘGDzmⶇL%p ~jze.cRȮb;:^d9/W+:LαR+[ɕO9[rȳ&oJLN>CI4#?{(>_[@]N8JEXG`Mv!^#1xja@dj!o8[`Ѿ;pb84gz>("Z> o诮`F+,tQep祭\ҽ۞}5k[B9ͥ-"!)_n"tLAu4 S$b;f⋣s6F??ȃU3x mm%8w _n {nQp>zϻ}drU9G:/oW>~%v\2!}!^mDjuαk5^s>3]%?ڱ׋=IEa;gv<䶠^*C^o55*V 6iC*~h@_k1?:/Vu0.s-JCyi9{zls&mGv06E^gq]=<N_K"mS_Bs1$Iu|c?-s} d.{LE{e෵ &mqz4~q9]c6{Xw_Xmo-FD0JQ_|djYѰZ[Ս7K7)]Ox~Rd$I bp$I$I$I$I$IrsI18I$I$I$I$I`('I$I$I$I$I$7,? F{IENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_log2.png000066400000000000000000004055411471400134000343460ustar00rootroot00000000000000PNG  IHDR4ITsRGBgAMA a pHYsodIDATx^eG>x@Csϣ)YrtU4Ʀq E^H3щ5+k $6 ey5>#bεk9kĈ1b7YWybϖ,|m2i\wlaR>L|0Md9y]qYQNWIo~y<5]]ݫ?//~?]>o./~Gk( @]˖,_v7S&[#ԥK <@Ysu,(.k:˱lCȌ a  m%vZf3joIb]m$ke;}aHmw-loO+-lտ$WO3MrcRoC)9*㺡ݿj]+_cQ5 C:3Nycr#py;gEfg}6:gKS]hEKG6uK+eRR;n%KO(q˨!:)C3/ {J׮m%h}}ܻ npT~ 8mϖw(R Jo-]0 N ewh+PtL@:t4L,a:JyY&%6Na*ޥN$sa|HȲl oϔrag9ʲȠ s°߄ܥ]*'2ϸ_ [gyuoֹvԶ>Vk|Z~(q }s ;㹱韗TPtGyt~vij?*PunƳٮIeٕҋXN·B۪:?{:.V@o_1FH{zئC>j/m m%h~h{cKY,ӕ= $0 lMO^X`P}ՠA;EY,*~W,~~vվF2w)ڛx`;6>DB誦YO=3+́;-oSꇼ^yJ\Ϝ6^ST7Cξܖr;mϋ\a]}ʦhB^K/(jJ&Z\CP/Xg s*P7ʨcLeLNibjжG׮m%h}߉LjvxGeA%p@'Y2 LOϵxº׶0r֠WJg3 iGZd:-B䳝 hҁ޺xM֨ 't&a>{ig3.=ݷT_단+w՗ү7-Mσ修DHg / E赇e:Uu)(WN ]7'bژN.U 4/66-Q+(Lbmm :FE[?a,A:?6ye,p}[kp`+Uގ=v.^nեNuRپ}R6U#meGO_qm('_Vr,nl!ƭ#e\0Vϴ-X5yYk/پl+p qtSg7k9U g=;ۙ:㞇E_ә) m4;}GYjCͯ~ \/iJ3=#W7mͿ2MlE\mec,l4ygL{&܋x#[@[Ac @Җg{l&&;}\GNAApv.-I+?-R?w({#@i+-xu׼鋲FiOםF_}/ԎhH;(q,殳wҦrlNKK|H': # ecԛ荇Ę-BO} iՓ.<8II󖤵:9š3 \6*iW t.7KcuvW<ضO-M;9"A 64`-bi:U=gmRЏкx׸nnvVW >o 9䶂gG]P|[ˎ`Qb^l`>Qo9ns[׳C^6|/T1Qܺn9'4VM%G[Z=oo#uNm|2Z:kh*yugRycuuf mPџ@ 5ޮ%0үV[ۓ愦oϴQvMsGVy{xP4^l<};=[MӠNv: 4cU. xNŭ \pQt7[]]*w`.Es'dt&8Gb  ;9/(b/_Gy\f 0fC~%+XA}XC9v}K >`z!+R.l|C) E~W%٣?XA}XAAAAAA                              zj/F_8>^n>_>҂ xC7oiA!.?J\yz?gY>vyj3嵕v"܁ܰc o[1.37XJ3տc偕zyT<'`zH8֞|FΥc  8*~~@|jy+_,?lJMk xYKv1(UtΓ+[.g˻o\,%*ΖX>b:8t+ɂ>|A3;k|2蜬y|Y9^zyY3ۿ]v-Rmm{r`glC!ǺlЋ#N/rCT=OXs$yw|kgK5>/^DLg佁Ы {N1:AshX{9¾x}+`AA`{Lӓ+$W.߂bԇ$` @e[\)(G˭e^0<|_׃Eltܺlɉ{嚧dC~982 U [.ПVr3%WgxYC`ضc!'#?9vY ҳ#\҇r PU/IWOR Oˣ 0z)8| :-ndL\$j ET-8Oiʻ/3y ږ`d6ysM2]/^:ȸI'+X}hb{K L8;|eD>}[v[m ȋ#цξ/g]f}*<:UIA r%`4[^͡3&`AA|$ _ZR|2V,v|Ql {vEhD{/ɞ؏o\vk(3;:g9@ 'G9Q:E$wp9lխd)s%wY? ns84s,u_b%޼]nZ\yak /J4*3GD 8V 6Wi+V|%-]m6s^ӿJR6tK mk{[pAPu :PtC;ݯt ==VG'`;l/l{kzf:ۀfҚMug<qѳ9I Ce!>?q{,,Om0"{;5=BPrԉV~/*/)_޴Vs 0dim>x/HaXcE];Cg6Dus`:o3XWeJSG>><(!T6RgkǾ4κx6Cb;a{O:C8M XCstB;e~| YM/5_Mӷ9 y=62_g廾n{ގAAp_9q N%]<xg Jx99K*WvҥS&ʡm©:8t, q"Ύ!'R՝cBtK^F>C lK:ʎ=:K?0g bKC,m[!]F,Bޙ`sYWq R'X7\˺V~\:oQ[N+S(N_ՙ2ܹBǪzY~QWiȚ4CV@-e;sJےwGJ po_3K:A/f$߶G2H.S(3`<;$_?-uXzusĪK><~s6JiVN`*3^/ aSgb6\X1YI[4x~iKS[ﹽ4]^+,'_*"~Sշ3?O~yr8& WW\-sN5`_&m#:{)l>0{$m͖ZLZի'b&N!c{ŜgqP_C/߱|Z~(!nS<#4(3]Y!99[:{I.ڨ}gQe[{RI+VaPH3?-dž륫,//_as(a96F_RW=llZtc1 Ow?[AJ%`$Gl**Xڠeߐ+!}uAH!8ƁMGSe<7Y0GΈ˔6ma:.́LrYYN(gC/=%_φ޽-IB<KgUnV|Ӝ&z9ՉȈysİ {ڱҨ ۿ}<ڱL}qƗ)@\ ^р0T pR}, "6Ӷ Tܺ))봂;❚KUƕ|V&T/ubh+F;Y2&uV.ٔUYľ4ZMk86̷9ڎ-@v2L48GY[}N3]7_< w*'2ϸ_ [gyuoֹvԶ>Vk|Z~(q }};aswsAAp$)nq AW\ dS]U ߥ ko⽃C}XKlp;:[ 2IY`$'Pڱ=:ӎ7-W]g:`8/[nЯ\Μ+Fµޅt.ck .< d{]rP;=Nl6˘_8`QlSm`Ȟ@L @ր_}YՃ8i ǎ/2UyͶ'Q;&f|<6\aVIb3li?6hő9ڞ5[ru]WȎu_'_ tT0,7>W&)GW5͒Yy`tiy̬R?2S"01il4^ST7CξU6Cn~":w{Xi@:ʦhB^K/(jJ&{C;K&>v6 KcLYX͡Č-Mx{ƻuר큍W2~"GcA`/j M/g`P X2==vD ^Y^)(6siw xBDhAٺxMĨ 3݇]:0=43ٞ[Eߕ;KϖAZ_׃FBlj /:Uu)(Wus,!R;^ _Q@hCkSo $*ٖѶϠncP3) yn'NZƒ ׷ᙽŽov[Bz:o}l_>)*͑β#姏ʯ\/+C \Tu6JVߑe.mmgy,H嚼Il_IHcθaiw&~a  >j"; WApє#<.̈X\3K?Gl W"(:XAAXC9v}K >`z!+R.l|C)rVJd?cHz:SސXAAXAAAAAA                              zj/F_8>^n>_>҂ xC7oiA!.?J\yz?gY>vyj3嵕v"܁ܰc o[1.37XJ3տc偕zyT<'8&>5{k3}geo@=8s{rA_-w] | 2/?k鏟;.Jy|yKly-DZ˗Q fˡ|$h'3/˧*-󧟔clM^ADЬ\=kKgifSeߴӼOg7VЋ'w.;uGP]{ޑ+[qZezr E2ҥ֮ˬ?\~E??>i/Z>=MtV+I[Qu2W~VdsʱlQ^6ǛϿi;.evi#.WGvx.n}0N/Ǹ4>1z2,6>1H:?)6:{@:{}Xa4d&W rkFkk _zg"h0H7G_ t:Nnt$e< ]yG@Z[ɱϴl^eSnR]F`ێ؏Xdɶ%ҳ#\҇r PU/IWOR Oˣ ^%8$|P3Mls, 4h x~OS~ݟɳ@5dж$$g͛xnaݔt}|jIGM:YG[P8~_?\>dy 3-#(!sa~0˥EF;8r9w8/%yngiJo΅ӡųsޞ= r8A{/d,Xj~m{Ј\_=߮|ZoᲣ\"i~勷^89#bAd꺄#9Knu+9llJw0iqoOQe\6N/Knp9{v77kyak &X{m ;Mt5ڿl3޿ym@ i&ԺƳkfsdiٜ$kj ͡I{Vm=l6J|֞j!(ZVD+?Iuo?u9״6 Fw<QOϢ._fBmMIV9|%^9{s/Ƀr1Lk#u+L~{Iz)Au h׭x!/y=oI=,EsuF0e*:ZS}y\Cx.О [/`kv}mV2o_@{g{jߖyz{JZ}>4' ˺V~\:G)Y +S(_Չ22 eE]-"kr Yٶ bҠc[W)-k5bI7(4BRa e=~z{G~ ~ݣX_1zXuwom>.2Ѧ pifۓzثb7M1mۜrYVsfy&m \?/mijK=їFk}~Kő۝ .{ԯ6CΓۙgxT ^x(ѱfs8=w敢_oiZɳ.*uZysȹNg&oJogٶ^7"ʛpG܏}zȟǣES")٨7@讫?{u$Py1 }i6J[-ɲ>21}S陽& $WW4? ઁ\t,(<8Hxnp+|DǻC}A}t98tY7:W,0hqrJ&;Gg a:7(I'M;~)>;uHmyss}Y|9RnkyXv~G ՉN ][vZ3!-O:a|y'<>ˊrv3t _AWc08Q5-K'7A   [HDT"3#d[ M}`2pL][0$,k~m`˚=VOwMF.u N^YfDvR:–1;]PIKrzD1:$7)V9D{~9r2kA@;Oe~X]3}?͟miP;/}BzD.on99[{+ ڨ}gQ˯ߺ2O\>_Ohg\2j.@V>輦 <μ܂c/6#+]:V{|')s*P=}Sv: Ow?[AJ%`$Gl**Xڠeߐ+!}uޡ ~wl8Y<*3!299='aDt=:BQd('N+zJ\=^ {.W]s=Wǒ`-ڢc 6:9܉qf:2z1 }J`Q!gFIOK7G_͒ʵ 8@C @'` o3mIŭ*N+#ީd9VW~KGY:S *ԉ94gpdgHa:vx Vd.mw&ѵoC mGEecx{ ; Q=FVh4L׭/&.R=yJ8ߪ8vͫ}εAXsG1cGd Ȳ+9]ރ  ,N?7\«A^wX)}UX x}eR7vt!m|4y`ؗ~O:?Luv8evl,}uvAnY ] u:̹8$zm*\A]IGF=ְO:4M//?;}I+G~=0`Cj;Cbj ŐGڶ|˪ru0 ϋhP7YKq|+S=vuh+@c+NfrP==*{?  ._Ԁ^>"dzzֽ'4R: lQL;<&ӱvpn9"x$A˖ o:F]<}إ3sHl-M>Sq9辥Z_D]~lizu u=8$zm4G:KxI(@LGF=7,ӹK錦-@Ylvf9Itt?/䯨jA~Q!9h1j{e%P ȶ}u \I^  tzgisx]|9Yv{dQז=ree(7˒F~b;RmL"9\7ўV>ɶi W7L7u1ϟTPt^G%yk:]9muxgUdm}!Pu:`=C0Wܟ9F }[2Ml,yK&#xcgRGiz:Gl´giȽ1 4Fd/miy`f21oSutAg`ߒӂ s 7dFҢ'Zw͛(ktk4՗B0ˁd9Lqr,`:[~!m*@F䰴4PyX -T9{"9<\6Nx~M)V=sN(0/Q7oIZS]?sy89erV`} Mr4VhwsmҴ!4`CIA frH L/_Nsv&l ~Vf0hgk zu˹3ϊ Ln+x&yHAڦ1{;<]DQ3`zݳ?k GͿHm_%z~RxG_rct8[_7[%7-`Llv1_]a>ּ,7lsqI9Wd+v˔Z B3h[Ko8+9߷u&$Ӧy=zI}}~A#~vrVLһ1:36O?ކoWXWܭPuXsBӷg(;չg+<=` (YZWdciP'EuBh;Arرê {^jSqk(7;"%\ݍVy¥XKap> Α  NrG| $ʿ؋tYApfD,.Х#b%Up&D, spg!>%S_D0= ?s)q6 >fz"+@QT NӃ PF, sP,      ΍AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA`_|#}/^A_/7/oYiAx!N7Oʹ go^%n<=J_\,~y<J;G_@ṉ߄~Lh},d r#ftyvR1AA`??{ ?X~{Lߕ/w%5?~h*yr:-ugݷ\.ihgK,|1EG:ܕdAR>o~X>UitN<_^w,'9I$tflCJݶvze:h< };r[c~<^˸L_bC^/]bi%ܟQT?//2_QQM{i_/^}wHڊ{׶已&S}ѭieCR-/A/=<,}߮H߉t)涋Msb3#lᴭ+9eۅ^Wv}9FŔ|8x#ϻ_;[Zh|^>do7S{3WK9}n.AshX{9¾x}+`AA`{Lӓ+$W.߂bԇ$` @e[\)(G˭e^0<|_׃Eltܺ=r 2!*@BgWПVr3%Wg :v.%eXȉO漊AKU듔%~h%^$<߆CN+DKA۾<` Ud= E Nӿ}6L!% &A?kcew 릤tWg.WN:2n _?ZX%;>hiD3|3es{Z'"/F;8r9w;qs?۵ę0X\}-y/  >Nދ/j)j>A+Z_(y="4"ŗdOǷ.;5\vOY )vV89ʹҡ7("]p$!/_aۭn%'@MeK |~'HqziXuŶK yd雛5$^4yikUg2 @tqhlzW '+Kd7Z۸m缦E/땤 V l O.)h3%2ƒX@?u,ՓFv_7j1H3s{z\ O# Nemt{V{kzf:ۀfҚMug<qѳ9I Ce!>?q{,,Om0"{;5=BPrԉV~/*/)_޴Vs 0dim>x/HaXcE];Cg6>890qK9{s/Ƀr1L#u+L~{Is۟|km+~1C^ʝ=([{` XC:#Nv<:ͳ_™kt3-x{osw{@mdh[w}# rJ@x"%sksl뫗T j KtL8ܕCۄS u pXXDHg;zRug8?P ݒےhqc{̙b]gglk Y2r<@fkȺK]?>Z}>^ֵy+:G@pXBqZ:Ve-V',J[Dm/,YSؖ;UJ~ GiMX ~94'= ApyE`%oÂ_(׫{#V]<ǶɆ%?7ڬCL[0۞tk\^!lLlC R?f73i+ٚ&/mijK=їFk}~Kő۝ .{ԯ6CΓۙgxT ^x(ѱfs8=w敢_oiZɳαE">r.DlR;J׶*y7|yVބc:s?[#_"O}ދ'0= :Mg}R{Dc(ڢϊP¶36CaS]ݷs;ҖmAsX HUC`+RpS a:q`Yv tm K#8x/S衷b)Z J3QNqYفtY7:,׹&:ẻ,&t*]ʃNz<2{ϩr3^;ȫƒ;:W~w/s!F/tZ>©cI'/k w*A|qiA|] >A R`NV@0fҶD&Ff&Pu'Y'9M/l#cu؂ a?\0/>V;C˫=f&m#:{)l>0{$m͖ZLZի'b&N!c{ŜgqP_C/߱|Z~(?3NS.GvΊleuΖ.fhEW ,:~V<)sE噙 yܧ9Mr ~#\I=a^ceQ9ۧycf/)=RЁY`XYDmm/g۞9U߉uY~.qPµ% kU`č~MݳJ&^eRAs^~-^ Js]4O髲wRgWk4,2Kia/M4";W K@e誦YO=3+́;-oSꇼ^yJؕbszٷfo|^Rn= Bql&+eTi`Be.Աdcg=iAAp.v0I~ 0%smG=0 5Y``38<y{3/ Xt~m$IXgL;V;1 ^7G׭u!dkԷ>zgisx]|9Yv{dQז=ree(7˒F~b;RmL"9\7ўV>ɶi W7L7u1ϟTPt^og${}Mg~"+ҰB֡lWu,-r}!tV+sgdε/:P[:TzY6_7:mc-zK%.nw_W큍2){u2c~V,1XAtoIZiACY2L#_iM_5J{5K}v}|_NK8I 1JrZIԲig`:FRvΪ~'CpX;&z!51ftINޅ s$[G@CpcYVq IpY[9޶}jia. ` 3h9&KB?h~\6ut33b\yk̙gE&<ut|_eFXAAAA /{zYtv @Ne`C |6P+0:XpYc1]go}kʗffހzNq9dK U(% ̱r&A.{[d w7X}ouӭ򞜅"ϖ y)+1cKb~.s7?7fm>&,sum^ 4m;43OfıSe'K_-w] | 2/?k鏟;.Jy|yKly-DZ˗Q fˡ|$h'3/˧*-󧟔clM^ADЬ\=kKgifSeߴӼOg7VЋ'w.;uGP]{ޑ+[qZezr E2ҥ֮ˬ?\~E??>i/Z>=MtV+I[Qu2W~VdsʱlQ^6ǛϿi;.evĭL)c,t)v# Wv}9)94HՓdq8Gɰ5ó_;[ZgyHg4N h WK9}n.[|g%=;!`J"v/Q{EFهt.Џ! ̲+\f 1;LYcח8d`!k1VJ{VyG)-A*;D{n@Kn{s[]땁-h1nX2'\{}/`{Lӓ+$W.߂bԇ$` @e[\)(G˭e^0<|_CEltău OCp܋=rXsqr[#)YHHbim%^<~Z{uSnR]F`ێ؏Xdɶ%ҳ#\҇r PU/IWOR Oˣ ^%8$|P3Mls, 4h x~OS~ݟɳ@5dж$$g͛xnaݔt}|jIGM:YG[P8~_?\>dy 3-#(!sn'ח`6sgrr0цξ/g]f}*<:v-q&L ˕79~\8Jg={"v[.?Gu:׆ , /-v\noEKS_}}P\`~t&)ysy i%isGS6WqC0l|^V&iZC[?kydJmAOn_D7pQ.g4?euZ.ڊ3r,FPNK8SpVvΦnp!ϙHf4U&e̱4}'p37o<}sN 1'Ё)iґ*3uűmJ]10a~FK} .0ӿ:JR6tK mk{[pAPu :PtC;ݯt ==VG'``ϲudoa@KXӫ64ېlBko.28ہ-JrkJ^Lis _b2$GV&?9ge?3.mն13PA 7#wR&褾ޢkO-!gr Q滄О [/`kv}mV2o_@{g{jߖyz{J21}S陽]zti>Ƌ֍#E,^pwY[/xqՓ(0κ`C|&%}MS/'%˙ʔ{AFA=3]LFm#^p` e.mAӓg\><_]^3Of`ml0I`5 W) h~LKA*UO-'X eQ+е1Vxp&@WY`w;3\޽\s q '8P.F:dr' YA?L%i/gn=-o#ˀQvCRnkyXv~G ՉN ][vZ3!-O:a|y'<>ˊrv3t _AWc08Q5-K'7A   [HDT"3#d[ M}`2pL][0$,k/w`K3]ĶK+aA9#osmԲu=e෰eNTfҒ^=F?s̴N7ɍuJ@g_N+輦 <|^mcGVvuh+@c+NSUv{d{ l0^ʋY{tߖ~mΓ4iC˸m2`9mϖw(R J~~6h7 oH|&ewh-oC8vD>̃z͙>y|;q^SrCgT:?;mỲ>v9ItA}˲+% e!r\c1Y^鄎U/`cǃ]tl'Q{mch+@Cߓ;\bdٕ}G ^v.ʥn/mE}yemONAw|[ƣ?Ɨ4P.jn \W+۵ٹJ)Y:o 5FתOiE2ucߖ6'έ9w{}S(u6 ˍϕ ?ilx(3}UMh{FgVwZ~(3y= ؟9!mcٱI;23^nեNuRپnuH[gGW\[I.K:%jqH2U3m<xrM$G{ZKs@/$J1\}g0Yƴ;ϟTPt^og${}Mg~"+Ұ /d6je °_ ~ ݺoQo9ns[׳C^6|/T1Qܺn9'4VM%Am|'`AAAAAJe/x,"AAAAA*, Z/z?A@}p(                  zj/F_8>^n>_>҂ xC7oiA!.?J\yz?gY>vyj3嵕v"܁ܰc o[1.37XJ3տc偕zyT<'`zH8֞|FΥc  8*~~@|jy+_,?lJMk xYKv1(UtΓ+[.g˻o\,%*ΖX>b:8t+ɂ>|A3;k|2蜬y|Y9^zyY3ۿ]v-RMmNwc]]zeOܗc\LO̡A' Mc9O~fYh|^>tFdK< ̀^-}_(t6z>C+ړ[ pN`,_A r|> 2"g"m̮uy.l#sAYJ>=]KA r%` ~>LΛ{As茉Xp>䟗XlK'qY>_?LZn]U˪k+- |V[s?Z9A{/d,Xj~m{Ј\_=߮찯7pQ.gw4?euβ 'F9Q:E$wp9lխd)܁sQf4O&e̱4}ˋm{v77kqk I%,+i JG bic ~Xӫ64ېlBkwCkn`b`+H~>ӡcSt|myΧm6'87v{18<‰`p*%yt>cP<A^Rog6+.e0pWmN%18caf!GH՝cBtK^F>C lK:ʎ=:K?0g9zQt_岭$w9 ygar d]Uǥ.KN `݀sGX/Zq]Gm#t~cd8qL8a|-Vgp eE]-"kr Yٶ )UtlKr*%e#ʹ{_,|F U =Sw>H9m匍6Gsm6O UP /IUX>쒼WY$XO2={ؘ/cXiS[ V.iSѾ-Q~Y믺%rYcGs-G#Yk/Mؠ $VR9xǴr\5Ԃk{`eXe]`m?z-X ?:HLszi $%9ktY7:,׹m(ٛlcGgѾ~PǭNɛ&]ʃNz<2{ϩr3^;ȫƒ;:W~w/s!F/tZ>©cI'/k w*A|qiA|] >A R`NV@0fҶD&Ff&Pu'Y'9M/l#cu؂ a?\K}}MjIbȥu^J1!}n){Eu-cv7`u1cuInSrX=r#0^1Ye\7Pm`w,?xf`vӠw^6؅<\YҙgG-M]wG|u@^_Z){ܻfLOuzl/ =9m ؜Y#/)M֫d ~MM`6^ yR1xF4Pޥi_c@jZa{?e0 ? F1*uZ\`AH ɖ`(6S'qs^RtVɞ,#]>;I<_o <5^=i`,@JH0TU Aˮ!mW C 0)轃iS]XH/Nz1G ++ʀA'>ͣ7g|I鑂ʵ 8@C @'` o3mIŭ*N+#ީd]~Zuk\ 0/ g` dKX: Nc`}&ǴIQf#MC)1]Vdldk6}b0h;[6g[9ɰ3ecdPn9aKstlq0hߩʞ<~%ooA]YQ Xy9iyƱ#2%$ol8>/ϩm=f[msտ0Y6iCI߲JcE@,'@Yn<sZ;<7(:9_@(|-5[`lїX'T^oL ]eWp' apgkz @|`P}ՠA;EY,*~W,~~vվF2w)ڛx`;f)DtBgk ]&:k;2V;љvxlZ}:9; kA9F_<+ݺ_NS9WgGvQk9 =\\yػv*{@]4m1)p #_@0 =H1сnq@_emo D0 Ϳ6}ܷ/Y\n5Il=0퇲Wm86G۳fKS^nʑ1 jʄ՟46t<ܙ>誦YO=3+́;-oSꇼ^-elX*bl6E.uj=_ʦhB^K/(jJ&v*ЯYYY2l_Z{}dp,E͡Č-Mx{ƻuר큍W2~"GmR4GrXKthͬl>v(ڱ%`ehp]✷{ v2t6vK ]/ 4|E+A1 b\'{mkO h g zt:¼SLA*|赓-[:([/XucKgR&Z|8"s}Kr'\})z$]jg ~!+h_m~mZ^Aw| Tdq1-mA; 4W(hgR><%H09Or%o{ KIeC f躵.wzg\u쓲9Ri,=_~k@92eIu[gD_ 1n)[xܶ|mTɛhZ^ied[4O:˘Ns@z(wLqâ&]9mu3Ȩ6=Pu@`_:+ p9V}s-8\is*=:ļϲ\6l=BG$[Bvy~x!X/3guu|g!R\̿XQʋ}r%!@Po[ #ha=h%FM`O[+K\J,=z/ f9][VZbPF WZ4DyeҞ;}ƿR_!fz5.98΃t(Hr֮tlji3ŁT9{"I<\6Nx~M܆CKi$9zDwB/ɼ󖤵:9;C(X1lUkq*\nxmZrK0DB&l(i< Z) ti{.ڤ urk GͿHm_%z~RxG_rct8[_7[%7-`Llv1_]a>ּ,7lsqI9Wd+vZTymr?6aY:tv[4̇ eݞ:6zE#rر<}-`pE{wEK<-ltYK; ]5@Iا3 pY `f `,.Oű`/{AFxִ/m$MaNA ik8Kxإy2z^foZ}RI9Il!lKQ8* ~IqgF˜ ]>`[AMY XʋtKi 3;SK`l Pm J%*RΟ @ϞNR|F y o+Xidտ,'ʽW/ sLA|} 8C0Vϥ\8$Rڥ-MJd?cHz:SސXpyьɷ|!)Q0.g(BTM蠗| ;/{;DЃ%OOXtP\ֵEwIJ"x-:      8tY\)_{"|eQ?^ܯiTNzLJ.G_6R?2C> 8%y ʞH+G3ǫ[K/ꭷVT~cG    A|ۻiS"<#f.فT/D/i䐧_WgWaV+]ݧ@'i`htM=X9Oi1-sʖ^97{9=P~"#Bʵs0zD,    vۡ\^vnSjHDE ȬRbY}t/eYd }aji`d8w>Oo#f2瀖ۜyРL՗wA'2'va.Kk`$J^:ӳԗ5o hZQ tsas|%U89_Ӭzc3N5˚aADI 9GAfyy z~yJ v*O߼_޿yjAp<{*qsmQNf婕v`oVډ8rr>g&}do c~G_cy'+?,W_VQ-V|׬/<+͞92skI nye2\ v//u?~ !l߭}oz>H\#j ,#/L0>tv<-]z,=|U9nYח6ipn2b8 Ykq1r-/>VVԿ5=,_Z=ۻ@ySd^~?w] J<9d 3[. r %/%A:̖C/ HNfg_6dgOUZO?)$39:82];̡YzזΜm˾iy oO\v3괏N0#5V赌%8/l=ѡ@d K]Y<ˋ|T^|w|{׭W|+-d|pÁcyDztf}ZP/CKl K7Ϸy<|F+w"]ʶV^+;}Ҕ|8x#ϻ36- stF>;l5VZP&l.t/x윷g#"6UG9] Vz|6;¥7 хξZϪ?:rVA kh'4i:(3wwsɟ3=K<<D[`uNz +@f]ޑ5D䧤{,{N`,_A r|> L |xv۳'`/i/ cVu< NX=mzdd+,` <"i%00 |5o"Ȓu2Oۻ96f=F3X=ueK3Ǵ>ڪekYX _ZR|2V,v|Ql {vEhD{/ɞ؏o׃b>D7pQ.g4?euβ"JqbF(q"u Gr*2Vr2ٔ @#9;߬ƥHqziXu뾀؛Mv`Y^ ̓ [K4ڃũecX6\NG0YOv˾z_z%)U:%K n aOK}d(Q! Ze Rn:ܞW+S0_$ elmHk65E^3#H{F$^S[(lMڳbhLgNS (@ϣ^R}zmu}ʕ~c=A]t.#0N8Pÿ!XT|G8BL>:T>Heu@`[-NJvN~Yz9<=g^Dglk?i9V]Fr;}[ :.u\judМ4X/Zq]d#t~b8,L8B|-V'(tZRuI3ddۢ_X.ImIN#_8 r䯙t% Co[ۣ@J)P0]:,u`m|9b%as n!́n: a=L׺*vC؆)e<ngVPҖ s{]}idWGYj?OTyrʺG*m?4錇k6sl^)neFy.luљavF\d@΅(w:+7ySz;?˶*QYy_OUZ<-%}}}uou]5=T-ޫ&鿕7Ƈc86'I٠+me{7o4&Llwg{!XNRjeԕ^5doi{tpstҴC_Sߖ7?We(w)7ӊj,Y;\#]-YF;plɏm'ZNheE9;:y慯1ߥt `\jc-m{AIk"*[|admUwu-Ϧ>r8& WW\-sNʵ {iNp]ĶK+aA9#osmԲu=e෰eNTfҒ^=F?s̴N7ɍuJ@g_N+: X~u/yR I:" 퐺θpe\m|' yM*z]ʯ,{үЃ%%7Y[EǾշl urtdcuJ2oCΌnїtoV9%蟕k+p @8N@fږ[7%UxVpGSsr5uO2N,ax1># @@ؤ("P6c0m@jym]9İav]dY6g[9ɰ3ecdPn9aKstoB._gܯc3hwܼ 7\;j]+5X>-oC8vD>̃zol^s@z(w:ߣ`gT:?;mvG 8k3/k!A?[])y,h( Ṃ?/X tW:b XfXzv}^DJ`=~'dYv%+{{AI`zٚPk_x5h+u@ <ʾK]ѰL]ʰ&;؎.=P.\%x/qNjr:dId,}uvmsv<@*rtxrWl=uCrN;s^ sPzzґk5,3N"Mw7Nicv:x)p #_@0 =H1сnq@_emo D0 Ϳ6}`w1;<& DzPڠőBmϚ-My+Gfd:篓/ :}+VPrgf?ά<0ﴼQfOzzF\Ϝ6^ST7Cld͐ȥt!{MdL_Q:#զaeru0 ϋhP73Hq|+S=vuh+@c+NfrP==*{?  ._Ԁ^>"dzzֽ'4R: lQL;<ӱvpNJA*|㵓-[:[/XudaΤ>g!M4LqEk}wNRƳyց,%12߰L碪.3ڢeʣi$YLCҥvEцŨ}Ǘ@E7#2-Bsu~&>{3/ Xt~m$IXנh0i{B.zȺU6;I}3wf:I^W)_lu/?}T~ŵztYMತQ߯-shkӶcaG*Mrg-տ4dOD'p Mel'9U g=;vNadҶJ:|dmpu(ۀ ~ !tV+ݟ̹ҭa#eJ=YM6,16Ϥ<=3=H{GEL-t1J |iK3=n6T}ם>Pg  8;N؟;e 4 <ѺkEYN_jGY4w$aÝL~lG@84csiS9Z6x'`>jvPhyٓpUNΑᲱ~wMCkbx ND>I?tuByQ$޿yKZCS.t˴+h:E+Slۧ Р Ja0CJ`B4*tڞ6)eh]Q|Jv5xXcX%ɯ焲9d+vZgjѶpVsjo-bLIJyiMpI{Փf6R_'e=f_3'՝Ius3_i(O36x:tHZnmOÚ=6GX7ͯ=[iCn@xM'?l5M:.G  Η>6VE]'R+[C9A9.nlȻ.U"]CN1 Mp8AAw"; s_X%Q^n 3#`Apa. -KWR9   sLA|} 8C0V ϥ\8$RKGwGS}/8O*0@  A       87"AAAAAA+"AAp}ޅUWAAA  {~l"`AAAAW"AApXAAAA#`AAAAAA4"wų?gY>vyjg"=Β_>,<5҂xci\[ic@"a$G7߾X(W>Y *ӫ<>-JSbH:yJ4?6q竗eZ~~m'Ga93c{Ǵw~b::wj}$|m{a'.ǽwoyy2σ o߽ji/~K[u#Rj%y:OfW+зxy,6gK\yx\:%' ^nֺ^~\/;`?=忋_իO߼6߼I`hnVZ#a`: q tw;|/G]Fl=v&vX@ m3q8̞#l| r|6^"fbҤ~KX=ܻ}8ܿk|tY wP/F{H*:T^9HQ`rP aۿ]=Z~'PN:e(/] ɍ@Wֺ;e %׏/Vl B`)0VƞcX_o\ J|O[cI3FdnD̉u{|"6d{ėwq r11NM;c[86983q.Z?]"^hڛA]<_%Ǻw~u9o j݆]>s~|{m'E 4*p:JAnG&Y,@v E7vن|l-kO5W7[68RB,fJ]…폯SCy ux˳ b\o[_sv䮗$[v`Y\`_|ufG$u=Γog29eyw,{L| h+L/Lgz= ^1da{ws)~{cgn }y0+4eqz'"ݜ[Drq:Mҡ΍TN)A;bYNC=JNZuf;d_q%)?EDnHٜ~t;8ɹ.T$Xs K:Rٝgbyg^Z?\<\sj_QfkR^Sy<7#u-u[?G9Dž csНgt,p=Frg]w"aH b t`w):t>\?r^ O9+_Mc~a0{CԵ9Y<k~it*u19*?t/uW=uΎDŽ|@:w]_"?O{[6D<:}R-awb^k8O`:(4u7][@\2HDŽ-\'.i%|.Qw–g]PZ´ /mk d&c\v#v0Ѕ_sY0rЫ%کcƈsorn=R=r^ͳk}!Jf_6Gs^\puе9Z.֤׀^@N齺TllgJ&^3cP.p0Zq<̧wʱ$i Lګu\:% &~I'Bؖ/>})e$g( qZ!'8N\:"Y`7ԧY&\?`R|n67 L/[KiC cճK>NfF5e;kN }}CB>glxxg =n<뒕˔<^l*?k˘xu9pP, @Ab2À G,tfցk*r|]dh=W~nL$l Dx0 #ݞ6$N_VҺw^Leךּ f_쾪{o8O_`Kq*SsOiuq yuwݹ0gU7]3 7EеiLgۙ tMr)~6OZ~]w͡tL t֝s?\cmHeLΞN MPN郵N5z{ y Vaͅ P^o~yJ3QeL&B<(c.wj{%HL u~ve;&ֺ'y͋Nu9/LH.ŷ]}K_{oƞ,'a"y_l,xyIc |jsu×`n;:r>b|2~|/g2#$Mdhk%ڲN;㷣9LgS̅43Xű>t&dºhxIM>+_``@pRM;6Nv,Ў~]GmY*c&鬵u)Q2~vYGoN:cd׍ne-0|dqj2>Mi{OkNYt7=y5{Jmݟʾ16ef[ 9;:2u-Aҝg: u~!}v!@'/OG s yqL3aGr] {. %tֲJ4^IϴQϟEPcЉ\SQuv95Z?†6%Qڳ(r&Ho?Ui!Kkުa{<_kIY18&m޻yNw%cnOnc(lj>͎i[֞RL{rXq'}q)y3*\Ħ՛gVƍo'U\i?eE.̵yJ_hdL~gFy;.t21l}33ϖKmdii[O:.ÞhZ'U:X ;5@^~.r:0,A=g4Gd:,0T_DW/yPN+niiZenFʸNǨ"s&G̱}VGY;2m@_CjuHyLՋ]?ҾHA#c]1sxlϊbtG녧ڞWolʾ} 3a@Y7צvϳB99͝Ws闚XGoJԟ2vBZH^7Q[׽31](`~siͷg{yq~ܾHV^klHu29 PQFkEl~tf+'€]I'KRez2~8x0uÞgBoduY!^Gmݘ8;u"biC$s[@څ{WO_uoL?c`Apn 5$d=f9<CO3eD9=0{s&\ {1 ػ>>b:bbwLJ+_A1mҿ~po_! p9_"%pΗ @<&}F]ww}lewO Cc.5?NXRЉu(/._ |1p ._Gk?VbػJ]K]A#.$`AA`AAAAAA     K'`AAAAAA"`AAAAAA"`AAAAAA"`AAAAAAW/_-6s}s˳]/k+ˇ27kU|MҀM-ayHOO Nͩ<9o 8V婑G^u}c7/3~ ٳ3٫عK:FusLHlFʾKvk.wonkNzSbaO{B<>[\P;_%~Zi Yic$|yk6[[/<E7.f+;y&|t3;rVl_)2>pzvxL99ø=#>}d0= &Ah+oA1GӋg|bh ȲyM2/@X˺!ݩk^(zJ:=.iLomôJ'm僸-Oʹö,^N;tlA|@S za/ ? `^?Peϟ ~#>{З0lr v0(Zwֺ!0fC΅`!:mJny![pvp;6 iﵣϴ@:gCn<.s}?0Ǵ3˩le.C*0 d\ׇ+@߬9Gb'O5 HJ޲7{ku8V whyfw,dؾ6 vwdrO<5a _ZR|2V,v|Ql {vEhD{/0̈Cph^33 ip? ?>|?2_HT\B~NO'uA)p|Jx`z )#gwMe鿔rp6YA4(esnE:#*vZY]ayj4xԖiYt}6Dzw s)nP%.D h&i"@v ¾zz6@RA9x?j1\`:AX>C[?jH6u \z%`߿ymºnvҵO텶dr>nhy#2n9\~nm{//eKzr71GnWq.c <1imr & u'\X13<:'r,I!K^1%OnwfiZCYd\}a|hn:cMpwƹWn5dLy}moH9IK+ψ|'W{Ҟ _X<zS]KcEt2X6Uj$< Ŧ'>G˚16u4*+m5_ӫ۔ hr ճVz(QcRY<~]ٶbIzkga@Mmӱ}zC?Y5/y֣oǰy`yEVdk͊?4b.=5y6 gsj˦vaη5E'AqJ@x"%sksl뫗T j"F6Cy@RF5ץ6сO3ݨү{! pa=;;:T0'6aG9gG74,Mis΂ld4snE#g#YW^kn%+mۥ_l]#v!=tD<B3#] @ :;ȕW;2AhrA@Dtr`J=$ 8$?FN\nϫ{\:-cB/d,??}M?E,لlvmm-us%m }.6ǮL3;.엲=i) ǫ31{PwâOE+G*A_;o sknY37ܸ!eD[A с!{Ss8 mM7!try7hnw.zmwǚ`nruۨ1~Q=c Wx{w Xúx|Xۋo7ViQes|VsY7QOkz-ҵN^uJӉiT6K{d+i{2oҿ'^+$t2r.g5~ sot&Y7AϡYXљ7Ym9'R]i356:l㍕?G鬻=qٳKgtYwsgEnx92?MtUoC,΃2_ dsmH eZAbуY;uݴ9h]^Lq FZ`XR`-dP,VJ%˔_Nː;勣%en,:eezꮺ.诮Nd"Bg5Vy19lo^嚱ǜ$6ө׻A%[ m߈ PѰ<?׼%]к#xuxڼLueynU%\h]wsmY2p:ni~>21zrd /omOw1vt:;^m<7w'l <+}t5+C]&\5Oyc}w7M:Gg^ʧm߶[@_L6:w+udžlUA@ue@erظ,9ܟ7Z &< 9I"T);ccj>:]RN'W_g[+u0XO[hҝu ȺNY?7Ap9mϖw(R J~~6h7 oH|&ewIƍƑo Bo(7.=fwU'U˭egYp~Q2gKaFnOBo =MkՋmј3vt:+}";3?&<´aՓq_Ȣ~+7^c[<;@ܵI粳zrsnKOudb.F:J%'Űvf~pc_J݉`/v;s%] uCY5/ Ugl sf>:g+s\1׿I_gf{RŽlšf3=j:H6:68085= >n0(׾jРW뢁,xJ_}+^??j_aaMw7 j & `׽2(A*tΒm%:%/$NaU 9EAuֹQ 6[F^9g3g;utXsSzyLt @Ab !"Mmڶ|r g&r|]A#ԫ8 zDJ?aOgFl_hށ7l#`]]U76V_o{o-.@Gy,\.N˭[ԭk[w:*__:<r1P<+` JOamvX:;e`PT;9Щ[ ۪E|t:ub{VwƵÃ~le_ٔD\Tʉ!پ6 8P"wǮ(\ 9erl`θ tU(\nkpc4\X?hԖ|.toIZiACY2L#_iM_5J{5K}vgZ 5hQn Qˆ[WyAQ[Z C TQzAFڵ5փ:ik1=:KtuZO% m ;ʪuK3C/3Gh:Sd{*ke[ kqX*X#[_)Z090!)yKVK*88<k"'`}A/6Wls\ ak``j`v<9Ϋ<}콙b\m?E>Qu۱G}Iz~tGq;Lmۯ{g w#DYgwbAj!i[.:jy{U7t=o\j8'|Gu6͂MӃ;3~>9-'H;֣PfmxPm^[Rn_Jf3Xij3IW./&} X̧~ 1.650Xl}@^:iu@5-xP[6m{6dzߵXQK9}zuZ_J@_/kZCyh^R1?yǖى:g2kZ6 } l}qܸ981x!9\A>.%W3G缟 R6q)<)΅Ҿld٫,8Y{ Dl/ 냏!)Z#?Ao<&.w<[AM̳[&`GDؗ9UO_wN Α`?xNtpt%<-_D96~\M\A̳[cvJ狏 a_N6~!s> sXAAAAAA                              zj/F_|<.o\[i'O߼_>|Y^k^nj7OEz~ZpjN=ay9_.O<=p}b{}/of0Ϟ]ޟ^]\7gA!c}p]˹R#qƛwxK0 ،NpTk/ qpgׯwxTOmR%ONO]¥9ҙSn>Ͻ rK1u"\\yw."j=Z~GpyZ|G?ͪϧ_>G=ѓs|{#IG! .Au,}|[s8:} R> pt(e{7RDž^^sxp[oеp FgCd:{qإrʅO%-MKXuܳW#ٯC$[qrߊxlq(v0[ܡ3ᣳޱums"eZN1_͒s`0rO_Ap8Y~OIxz|$[P 0bX0lky^+hK#е36|06ln!/7r0k`? ?u: +=ڻ\LG_4S5V0`Ɲ h }uCS,Quz\Ҙ'iNq%#{[ȟ/sm5Y~ w,sI5ؐ4K ^@~~Z?@G}.02!/= `ȍ`PգN٭uCaP+͆ }uwNl<bm`׺<:c .؋Μ^;L{sv=d287gsL>V\]^J?$N,GC P7?-/3#&G;0Ç_/ @5_NO'uA)8xuRFL)փ wm>2!iQ ܊.uFU> Koui)-2Lme@R+,%7ݠJ \x{MiHE@?h[}\ucm89r~Ԫc`t&*}D7&^ X߼6a]h;ڧBTi{9u^P7CZy.?6콇i=ޘ[#Rm7ѫ8б阴6xͅb.H{y }SL9$%ӒSg{k34-Ay,ac.оC0>L7&k ;\vƫ_c2MO۶7OΜ~@UB@3jwBI-r??$\8VMuV,q0n3c}Tm5WhP7g:N/ZB-koQdm|Nsd#s6s+>kɺXv([_i.Jg+u& 鹦 zd%OAn/fڑ @v "*3;> T!n $0t\/H7r{^Sՙ %l{`yx+d&d'm\lk(icmuY79vNgRqamgMHY8^܃+}*J]>=W{ XuϺ9 )# oΘ9\mCm"оў[λA[Twӷ\Lvћh;[gGwկF1anN`4dƻU ;Gڗ k?2`m9k>9쬛Aݨ5cZ'Z:D4*]%r==7_Os ~A:K9G?d7:,RW LY,ж.4ЙCFy6OVʟ#t]Ξ8Yå3y&O9 C]\ $' ՀT^<4F1-W <ڞc'eG@0mن" | \MNRoK$)<ȴRW {.;8@Q_`9nhzs"7M_g[+u0XO[hҝu ȺNY?7Ap'˻ǟ- P0# 6`,fmвoHߐ>L:z=\sC'#$Pouy6-N@8㗝)s 4zIcbnHOǰwX9sew:w9Hw,.~k Yāy 95кT1ClwthU  kX;5XB;AzO궂vж Јwn`g9J Tҁ%F`eKKG[ci{kA jkө׻Z봬}66aÎ3;B1*)gK{@*GkKO{~]1ǽ`Ier&mYayi;LeeZ^VL6_TLi/sR(~ۓkgOZa"zuۨq4L<JȎY=xt!Yzk[cq 綝wX6\v֞ZOzmaݶћl[HrtԡX[d”ծOyK;Qaag.䲕ks}(7+Ys#;c G_g+7lO@ؑMX쟃7yVcV&AA`r^~-^ Js]4O髲wRgWk4,2.]W D]CĀlW#HA,I3puct K>0v/'A/ʭ< oT9q0$`r 09[A>k9Ś읪׃5crKE/SlyoWn+ж593'c2 ^)?]у &R>c.x:3e D ~{o`2@QQXmE6S(;BEi}Uv:}35>"Wg<@esk8Z?yBXۛg3o4^.F^Zꗧ3ue5?_)_Әg3v zBKw`nuYw~Dֶfbl;&˃WITݺ-=}>,TUc&mEo31g'G7Ao00/Mg-2d@G垲3` γwTXsڧ3-4<Lzl{k*AnηA؜.v0I~ 0%smG=0 5Y``&N 2h ~_f5b!v_p0FX5P%[9CS"| !w4ްS}vJm.1!:Sp%{?b5t?z}m~>1I,z_ $g=d>밚kD_n/6/FztKn0&j.tesWihyYg]fȺ5I{3в u#l]i]^[i===*bI+:Ϻ]57VA=1yV¶HftVNtobɏe:mܺ՞3B#!hKʄrncMͧTl?u`bʶxnۃHuo/KkC+\|q ;c/P?mcncXo[r.Pt@n'lu:҅O~dV։e h_dzwYg]$;GYƝWdfddDd9'F\YZN)u1XT!ٿ7<ԭ]q|~D8efBO9# llLlߒg>W?fq&bacޣ}\[<AnbCֱ= G༐ `PEVk?>Tm,c{۬iů RMO&q5^IBI,6?uy72o`g ﳧg3$?=ld9d`~УF uFs:I"j](鸽z7@}5Ӿ uWg isvQ_8ΪJ=oi#.LlH ;hݻ?\?%bt;N[A?Ժ՜6u 6vR:#A 4Xm~}^L>U[\=)1nSh | L&З0z{FҭiW6z Cl}O\@$m7k+q~qf &w 9fclJ=KjM@YKmFcJYû_ދֵlϩ6D_i!m5ř-Ӝ#lt6h6$ `LC׽~8'g̓I f\I>q_]58lx/o1^ v$£/Gpvc}pٛ6I!cn恤}[#o~. /we&~$ɹXd2YK_ָT;Y5!u,' `I$ |s챭ȇ 2nIr ?ˋ|qgLPcyJӴ/5/˙_\n.9,~!sX$%rR,I$I$I$I$I$I$I.,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$ `߾*W5<4գ[+qÇ*ׇ7ثo}/D{~[rn2n$ ۻ OD;Nܽgյoû^3/b]꾊۽7{b{_/eHxK䌇v.pqsg܋7ٚ&I__>ן~ӧZqx2Okt^,J>[dLJz"h< `X$=T#Q{`2a^T7x{Oې?xxϭ?~<|Ԯm |sF9x*lй_M85I~~%\ ʭqPh7||6Һ΄`q\8ן^}_D|>ڿv=mË?m옏3z6ShHo'0zo0{osOispN-Xls G=@pBܻv?Kv} .G|$봘/B!Q~D^~%'Vk`!6zym,óޘCnl/OJo6.AN0RHz `_&EW36& P:?E?KZebajrxl:ksiN7JG\&[_!x:/t q8@!j4^X` O 0/X_(F% CX?%~ Eb϶42b brq 7t/oA7 ?pYg=ߖy=%Ps{SL6wer1}V9W>Y];vNXKW/gɹ  27\;%O+f(jwgu/qZ@0qV]_\ٱo6/pr. CI `uK^@Ojъ~h}~zЈ^O/$'6 $>n<~fIx܄B9o&7??Tj$<ȃ(/J`% ^,>!e0薱RڈG6#tP\F3 =ڌc}@& RԖm)l}6}V~, \xUJ+L>+Lyn/~[N$.z.\gn{i$U듢VO(bLaUM/ދ ߾9a۷;ާB2E{utPwvCx?N9쳇mHxu>"=w L}2L? "6? 3,L؛df'ʼn.RgFzvP_h_nd/]/5/5+T+"6 ~[{S͡H_D6\4 MPN]CQkC]a 1?$0(8/t>?cQ|n'^^ `o_S(gu9 \,MT,un8F4 yXC<'/(奒]5|_&!?x@y,0tQf̂n$vnې(8vhZ8/e wHKc v+6 Ԙقd\nZՎ,`涋\PQe޵ zO)hQ)p5"ձ0Wd/1j3Y,+Ӣ /^- rB;s-6ٷ_'a ډ/e,ʏ?X͉y"et "mU@yfd׫2&,ib3CAX͕fp?8g .?8`SWCʪ6s=uIDrX/HUC `WW?"ꅧQ\;8)UQurL'uv{Ӊ; %yʛȌ"+r:{엗!P@-D+E+|y&,ZȢD/\(XRu*/}wyG%ul6ec"Xɲ WW&LE]hY/=.}^_ҍ}K7mnh<h?7֚}=K_;dSկba6:޲tX2bx&[k'Q\?_&F_O elر骿b6JO#V}]I&Xγ\ֈ ` 9<*Hs;ow{o]{?m)[r<؋Fcc,:Ooo^uˀq㬂FہvlV sqN񋔩)XfK8J׍2ebRd?[Yi} ξ9R6A)[DZq$>ӧ~BZHԊU`o6 mGohL:RN&G~H>P Rx$֢$_H/(7]t)[!Z|WR|bH'3Y702.)ׄ;2Ƃ56fd=[c"n8,k-wRr'> O/lkPt;W90h^ab =诅u"*ڙzܲ@#ᾁ81V JqM# ^NAGFIJпڗ_|0Η/JPgwX3Vsla#~Ѹ1nqny׏@ܽIc{j?99mىa=M-b9Eގ6 ȕJO+L]mftZB{zS:a ͗\vm3qzq+.́GLjm6g}2s636ydVkNf 'Ibr^sТWʢ,Vx**X)x51l.݁<i1`<'p,{c21Tұ>,IA>)pć ZS+7‹AV7mLb+Is$8e<"2qvkB>YO$I. `?c[͑%%)e-`7"65ܗ70 2n/߽;Ƨi _j^*,3L<\rwYCN/<$IKX$I$I$I$I$I$I\YK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$I'}UjyhGoWV>ToX{Wo io_›ܜ;!d$Ir @wFv>?{oϮkpÇw/Efg/_Y}߷{o&+)p_lXiבkuę:Nu 7dG7yWh:|3㏇p@t8᛫Ⱥ}Q`7j_`h EtP젴bǹwؠj|6Q77{XIߟ~'i^od|tdӳqBDDz}>${{Ü{Jѫ-frxaΘ_ν2x>8m@Bum>dxx~-c<b:\9Xyp{3/봘/B!Q~D^~%'Vk`!6zym,`,_0BP__A_4ބ>߾'s&Ihbqk)$= `7{ߌ |lxA)eC)Įq[?8[%Ij^$ѧU>Mп/sas ]u ח~:ah^UhE/,D '&pjjr/vʂޓD@p1 U6b϶c `ʑz]w0xvnkr/oA7 ?pYg?\6n<(9)w&2o|>+W,Ɖ _f/Fs ۜ#ʞ)3&t9vw=7}G]<x{}t~8,9->dΕo,Q;#' Líkpٗg(y/zz?YE+V{~a[sR@#z=ݮ{\4qC\7q7mΑHMn&`PM=Tj|vO.x鿼!?}HnpFJ+Tb̘&iHEḒµzv Ƃd ;Eݢٹ肂،w{FM*o՜[̝t|spڏ::hoUe;!c<ןp'gKy[w<~:]} |>squ qq&Mm`3'җЋ6|3ۛm*Cy^Xk?t 6 B`MX1)@wDYOyi9~%6,>vLg)?9[L=c?H;Ў͆ !j{+PwUc6zҵ.<ˌLm~ ر/Hk.PtJ;5>lE7㭯k1=>zdo> e _NbAMϷU6L%~찹]8[ُG٣czZgJ}_rxUA>ۈA=G<Ư|/ .m-dQrva[_^Oa,X|5x#_V.[g,pMBOZ܃c߶t%|3 JWE)z-z.m}M*բ9 `&t >nj4q9xIKBVۆ!0^K%> R<Flf+u3DEm GEHn f̂n$vnې(8vhZ8/e wHKc v+6 Ԙقd0a͋ Y_vKL!ŴOU `N/ѢF/`m.MdS<X~> a˽%ANryN9&(cdns-{gy\/[uu3R'0{"cSd<\%66hz{˸ $ _zֱA/ گ =1vn&i2&suao?3{gf71wkmcEŲmq]kr>L`_1;AP({R=՗UkGab#d-^BUW]V䐽ʮmM]6/\+&s#Y_!sexn'/Ѵ|6.oM_=:ϕۖsYl0t6v [;]iC2^՞p@`'bC nf{+̰|[ 2G1<RI_>%y+093[W564G凊1(cP瀘yĞKc`| {,|-Jwwٿnl\!fnz7w]eZ,_QϺ>X%ˋa adFĦ8Gt_D(g>kzAJZ#W/<10l1N:X `3 8Lp^%V]R{Y ? r\#vXsg VR<πUX ///1ʞG>Q}qcEpsq1@8]bkߪ]d\ω]\Nm#u,5>Kh_#or9ܿ0XraՆm}ا_~OP(R+Zc _I]g] lE+w. F{2` /g@G{%{MvX%&sYcIJsqYVZf@5k8zZr=m_TJy~) ?x6= ^>-{Eb6ʏF\ ښwpeB ^N˼3)~oPxuu5O!:vZ3hr39b Ei:`ۥp񠷩V(shiyTom-Z''Pl( bC|;07(n Cw_`ޗ9-@Yg %;c7}edn541/cLEa/=6 `2VaG}oX'{fXйR؟}~W0 C6OC_zb\#d1 t+=9@q{0o}^sТWʢ,Vx**X)x51l.:\J @BnDc, y0OtF,{c21unv NzCp`vs"7ĉŤ_8Qo\f /FjOJ\IL32fgZ6/V kJoRPE$ȡ #?'UV!*,Ll\BۮbF-~iP_'m,Q"D5fYj SW̔@c ` mrDx{rwC:V8,2MuѮEsz4<~Ytn+g},EOj<8SC_ SۋK}y;]ׄ8^i4˶Qzq"73azI2Sg~^C|sYy99sgn0 j>o3X*1pc_M]f;Fl]5ֲ+)Rzk:~VK|MY6kN/dvk^^Q?WbXĒsGP610,UYa `fd'ss#`[`'IJ7ܤJXyake9(WxoAV=0*$I_9T&?@<{=m Q1W;o*Qpyd~?.EoGvmK]Y`س+~d?[D2ccJmqZ0z˹]vejͮW[oV'Şj,K8Wr;_l/.`os,tESۈ\)e+sc͋7W9?Ǵ <̼O LعfaU25u`"{!g8p.ue-[Q,qL0wÕ+f󮮣 cA%uBI{扯 Q<9.e۠lr.&{%ARऔ%`-v/csR cݓj>o:t]d[_y?C+m6r#mE*1p(a-L>SqPUl^rG_Q{ E>7.M\S)| ևTjL_-VǫS3\ڰk_g]^lM `U9NK3@ 2ꃝ/s쳙\2?E0?md|؆:9$|. Pgt^}Qmݍ> ci߅ẫ39/UygU%~}xG ۇ뷴 @_s}& do6mWk`1]'̭ j]jN:eʊKau) Iumoe^HFeRILYMM[ q$hns\MAK>G\9ρ.#ץm`ړ}Gz]'d^y`Mw1O(gZhT.۴?ٛ>v{+M@͕, f|ۍX[`y1GΟS{ .=r0xu!?!l!K⤃%Ip9rsn_.58l˴c{eA=R's=>bhr(f~/ `ǒKdz[!m_/~c]^]>jKιc˸IûFܾ}=q^ @ٷe#.ɛ=Cb5GL2j2.Sω{}1&XK>nvsub{ ;xqҾ>S3ʑs?'gsD<$˼q0\)}ww=?7gq[J]Ű,gX!gy̾NIPx-hE1GaWR|bh mW)qۘFe@Nɰ\W0 Ȣ|2quk~ė^ dLx#^`6\[h+>R6| v+LħO`r`NGf&o?d>sR0te.KssQ yFadi*d;|~G/|YSP4HPբ }ka;d&Q]yvl `;C.|mYwd]| _r9~`KsR3<fr-ݏ3.w]:1qnn{9}q]&ţC&swkNw|wf wv ^\7;;{"]vcyeux_hXی-,|Үbs8xKw"{[O=vDʖcXʞ\I4QP݌xٿ.:2> *Q>kLdM"q޿}Wels4~T^>VLX r))mSoSx[Wl5Îmuhvgߛ1b@+A~wL/j,^"P|"h7&1Ÿ==Ѿވbتq ؗr,Θ5<c2Dm+6}`ݷ*6x95LkI޾p9;7 u g1gψa:γ0(8/t>?cQ|n'^^ `o_S(guf0,B+A2g[Sn>tA~F(@%nZ,пDMEQ0) ٍ8x,hP.A9eQ'k$ ) ?9se#nOC.]vKGڿSeFXIXꯙEWKNZ /_/ʜWq; `rmaf6=_LIJg9 `>.7Sߠ]u|,_9̆mWoc1>wN>)fԅc۷=ݰ(cɺ^&cfgk :x?t:6bU5^Aŵ">Am+̈́M7]wjٶA_6 kùR{q0[W<_ ai#;<5Vx"Vͤl)K$Aق$ZK\aCKKsCWn3h66c}[Xz7&My5Ƃ}Rbs-MY+>tԳ0X]?krl 'Y6l]v^J\2?Mu:_35kH+_NnBLCb l^ n_bN){ b659p=\EogM0>YՅgy¹ٺ"[5li:.FscoUf^Ndľř퇎sdysOlmǝ RxUmH?zi׎)a;q `Uw*t\+s<ھA \ pNq `po&XԶ oVǼhײKl\YHsE'@Ɍ< dh`@ B Q]euqe}0օ,o79Kɦ}Q_1s72;֫^,v|_ _na@nfٺ P_bunuվs3=C)32Ǿ;cX=M'VvB S&̽0ma`[v׈[9y_6a,Lƿ*Lmf5S`,ϞQTB?(}14~ "u:}6;%ܺw'wE<3FFe8~>_75=RY,pY~,8ԋsy9hCrTO۫w:c` X1_zsP>Z7!?3_k^lC_Ab8FFӏa-}ا_~OP(R+Zc _I]g]JƝAZ #52`h ځaa' `f78X ~H8}d{` =/:xE" N0Sc]"$e.:':w &M+6jAɖLܛtu][o`@Vjf|0aKnc'ցaqDeOf+W]#~7eMk-_ ߂a,L6&L7 ˹  ;=o?D&kc"7LJ<#XE3OS ?q _St"0AcAɋ9?;W\LښǂXKe3/35y`b9[O'Kq<܏@WIi11g_}1ctlšoৣ/Uh!ʯ~8V ^|>~cm oẋ>&jA$ rn* ͌1 ~֠Le7YO 5 ^]mnXc'5\cʘ~PzIdӾ;M_CD)D^g1. 2+֯{ltA~ 1汊b!lUά hb_ϰ @ch(0=FW}yDz4>3G9G ^ `ܬfڥ陠Elb٬X>uɞɕ=`Wͳӧ*Tm+6X 0ax3*FB8~j Wl!q L'_66k-;ma]m,ưk$MYѺ͘{'6 ca2eS^k~=;^5o>ԃln&?c `{<k/l1rB2o<|nۙ=(oC˓f,õR⽹¹b8F&KcXY8~lzwHC92l#Ң'*-(k4/ ;) `x%O .<by7*Flvc|I5o|H˰0>쩒-| 5j<˚ k7l}d.ʹ5i"]ۃ&r: m|ߛ7._Vy$8Q 0m/JE^Kba&[+Ivʓ]wx)u-Iٺ_SU7K+>vdO! r >-"'TVreg,oPC3| j]kyV~}x' ea#K5Yc-LmZon:A1kbx+=c H6{7}b}o ĂOx6#qiugWy8C$>fj{~3m00=&ޔ-Ǔs?*~.[S&'$Δ͸}ho.qAœ{"{g Xİ y `N =AP7[:4q$ܔ1kPgtX;?|l]wۋCݻ$mv}l?i,kqqs_Y;,iՍ֨tg¾bXol\KN82Β⸏/aQ<囐^=As6WW`^6ڴ-/]I~Iqs_'&ϱZsIg}[~8)\:Q\*_C^J {,_%I|I[\WrE;gg_3Ì6\x<="h' Kc3YuR,I$I$I$I$I$I$I.,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$ `߾*W5<4գ[+qÇ*ׇ7ثo}/D{~[rn2n$ ۻ OD;Nܽgյoû^3/b]꾊۽7{b{_/eHxK䌇v.pqsg܋7ٚ&I__>ן~ӧZqx2Okt^,J>[dLJz"h< `X$=T#Q{`2a^T7x{Oې?xxϭ?~<|Ԯm |sF9x*lй_M85I~~%\ ʭqPh7||6Һ΄`q\8ן^}_D|>ڿv=mË?m옏3z6ShHo'0zo0{osOisXz=9|vXo)ŗvmjU7d|h}d$3ǫܻv?Kvn<~fIx܄B9o&7??`T~'( (ZxiO/iRf3эnK) Яxdc:B%yl1cȺx1tim:,?khπ/EmV2:gnӗ aRFKo‚M\QEc_-# 0 ٹFE)?/2EA O s8ߵolK& \i0{Qw5'ls'4_\ƿ(y㹎7|n}9Y8}R2-x>GnbWq_@}tO|܀ba]vAG}fq {S,،lj%#x >lf=i[P~^09"an]Mk?s+m|L|$4ݹ86sS^_Z/lsG}W YU+"6 ~[{S͡H_D6\4 MPN]CQkC]ad^({;bbB-xIXJHViL .؅.r.޵ zaFhQf+^sտenVΌ8;x2 5?@0I;d%s q^yαe<]?,r+Nr?n?@d&zOdw 6{*aFMo|Ăk_Yϳ:61f~Ml mȮ>51ܰsC7ߘOCq7Sd{i;s7C_l5-+m|vX1dg +g]=/bR`kf0ٷ_'a ډ/e,ʏ?X͉y"et "mU@yfd׫2&,ib3CAq\ʏk6 s&z#3 :e|5jC1\$I$g)T-^>F~uEc[)R^xŵc `؎cXu ]'tRgHoq';HCP̘!b+8-t묨&.8VRxp+{1 \O"b7*`z{x6m%z! uzq1t `uc\˦\'QjZm^ V 1F[D+TJ!(`uܯ c~Af ʒ:{61.Yأيid7%c8/Ǫ 8wti}~π9ޞ9*ѱ*gTr+t~#:lx؃Ys}F>}K7mnh<h?7֚}=K_;dSկba6:޲tX2bx&[k'Q\?_&F_O elر骿b6JO#V}]I&xyYy.PkDlZcy)\'5vڱqޤ1J?5(۲S<x.Ӎ:=Ytyuppԭ2 /\ՍaY1Ǎج:q!`)SSz l5͖q+-JLř-k#e3u7I9o/?}'(AAdMXEVfsбvT /.%ћkdrD|@E I^gLX0Y `[+qC_}@2C{`]O)|fx7FƠ<@<嚐cGXsцl],uk ] ǐ} ==eBPYJ'>)^xQ(Y)XBBXE:mULYɾbƘ*(ZxCdL⠧w Ʉϻ%lm-5Q0A^'^{jӱ*gFgy5VczsuJ;>/g@G{%{MvX%&sYcIJsqYVZf@5k8zZr=m_TJy~) ?x6= ^>-{Eb6ʏF\ ښge_vQ!_4n5F[`y7woXvO~e[vyXϭvsXFѣ >rғb SW[ٟ0&㞞T~lB%׭] m\l\{Acs8c~$81bf?bay `2Vo̜O@]X`4δo>f29 $Y `Xz^ P ~: `0nk@^*X쇣 `h8ưw\O|P7:ĀIx(R$aPߐ?4ޟXY7^ Ŀl^^qcHsC`1>~ײ~b]Sr};X~S `gd&$z aD_0e[*YsG"JaE8ްzVeG/t& `ޗ.Z3@ɑH~0riAX3\ȼJ6dFX|9dy&Ȟl~o=^[_[|fLdk LGmVnNGg'_NRo/Vo//vn_xX_c,Fiيܰ$ چ!pX\XZzZd_N k,kmB4_@<>q{oXNe[v}=g7E5>c |".tZhHFl7ɛ,6ۣm@J{)?8sFL6ÿzյ}Z(?:iF{j`N)]sp$6+]/,4~ Ű%>()lv_Y$ a?W"Êx&i}/_ZUa)̙mY.~m`-4=]:["еKoP't]ލe| π쳙\2?E0?md|؆:9$|. Pgt^}Qmݍ> ci߅ẫ39/UygU%~}xG ۇ뷴 @_s}& do6mWk`1]'̭ j]jN:eʊKau) IumbtoƯI:;ɋ>|\ްb9\߾7?mɹ9wC ȸIo/>|8-sޞ]W׶z^{$^;w*oL6'WAS|}#旱Ү#y ;s/"3͹&g7s/~ߨgk$I'~}D\rOkOǵV@}^d^~Ŋ\*b{^6PD8C8{iS!s|͏79:Q_10Nmd2ٟFA3ڦ@7h#'p%&T;{(bǹBmK:_p$_~x}OhO/zYc>:|w2krٸN}"">ca=aQc_~$}T]?V},`:_O*{Ϛ׳w9oxS9ݻ}{p̘B^j+>r=+4~տTUVd3j Qؓ7o-DZ`}+z ז7C pgBΎ{+Z4n=Yj>bqg#J `uZLoD \u(A?" O+5F ]@mJQƼ6 ]GYoġ 76WЗ'wy H7\}c Ln`' dV)$= kcy_&xmX0[Y^ 펬U&v .V~)> 0Vdz#tuo%~;prONJzA@h ME^Ȃ}BQK/׊2 zOáDo,_?dHtLQ-[ڷW.۷dg j*zuϗp8j_8n~#,΂;{h-G{Kߙl!bL?r\}ܻ|wYl' TB9w>5@?~9f2~Sʕ7A_t*Khc&1Vu:/Jޫx6_[s2sywra=|ԭo˟Ezϭ鹛UP_/g17`XX]Q@gyg& 6q"}I(/lçO =YqV2?W%LvCl-d;e(9 ?}wn/N͜u/_ U]=;yyZRˍ@%H!&~q `W5o+>uo9)8ȆⰁ ʩc(jm+hsT7f3 ɵw]AWWD' zMJF1x9PlcMu4V6s1b=1 \kn & hjͪ?T\ q|Qcq`ڋyCƂfr}~1gǏ5U$/񣍏-B̏=&I8s N ].xX"[ ע&T7Y-jFCy?$K+&*K3P=(K, ]CǴ$6:<+?u3ZV:7K_l#RvuQ; 2Y|7'6牔M1Ui{5+_s˘({T sqN+#<_18cf~pV5\q&cZUm({$,^ū҇ȯhl+E* OvL qSVN@  :dwJ7s#DV,{cw `X}`|Rxp+L n]'T1n yn0_Er=`=Z@ՇstngȐ:m`Ӹv01ef PYJ (RBE-H6؄ H-WNcNZ:_UF 3eI=M٘f,lE4}撱qlcUW_knbo~`gW,.={&mGdFXkѶ:x,}yMTӊl\{ҁc\lo:ǞFq(}==*vЗ7۳]cǦe(=8[bw!'9=/s3*BƳsxem䏣t`y4r_+6ÿ\i~ڃ>8Ӿڝ} rlfbSS&I-}O?>"((,a/l:vڎ*>uvݥ$7zsL}8I" Z_&_H/( [+qC_}@2C 4Ky"%όfY\2{ b K%nGl_*Kɝ$؇>  1 x#6K"Uh^b"^'7+2p@WSEbȢ,S>6ywQĔ-ƝQ\;s6|)X/b4*C^cU0[ͱ%<9k<66w b}^. π5@.K(7#. K&UMƒ8  >̀2/k:p {tX?0N6/R~le{:7"_|Z hŲm\ bƵ5gܳn9g q=*d0/7#-0<7i,;{O'g?-;<ɹE,{цbRt1r-Oy\K]hqOOv*Qa6 b֮m6Po6`٠9Ph1?13 $NJMbI< +9xƙLY5'A$19K C?K OG ƍ_x hѫ]eB+<_pT~}| Կ6u@뉏` B4F0I81E*,jՇ% { [?fax `fU@x1F2{lx1 P~zƍ!bp Wl Y>sc\˦ŊvM`yN-8‚UE.X}”oWohdQH+-QVxþZ"vVZ8Ӆ0 /d4[>_Z^'G ~9cUΈs!"*TmZ4WGc` E"{Dz`QzkZo?m1k-3?Y9mI|;eH8[\Tڗ3ۅM|MOc}lg+r>X/h_ KX"S.<2ۚXix6|qޤ<{Oʶϭv{X|o7\k}sE/\v%|?nA7KYmG5ڀ:<{=m Q1Wo5s "4<H]\mK]Y`س+~dH0d{]_JSb\.;HгfW]0鷷hgLkc/KN3_~bX>Kdߚ礽YX\4DyޕEٛc`fا\ly&m\3ڰ@:Y 0gpͳJuY8YÊͺ(8&Vgz3IyWQ :ޤ=\ׄ(muƲmPY9c=BQmYw/sbkZbq_,sŸ11e6Wj N8W/d:d³' d Zd{{&ɺf׏õR(b b˵aCo6l?)y[ӻ(bqˊ̈́r/GgW_l&6So3Es 81lѾe.yfG-I+}GO T;cQ 6+-Zx{AO.FRP;{K^c>FB(O8~j]z 7Q!_:KF*TQUm#j:mf6+Drځ/1meN5VP6S?D|;93,NJe-w: KůF-0IL6,6HC|M._SbSڥS `(ߍ._bƞफ़:_ 1v_ fH~{:sɾs4G c;;tDպ 3@Q<q{eoDv7,k}Ҷ좾TqU(t{f3lF,}]j3_l?' ِvкw~^AKfwt0#3u9mjm++.ץtF&1if]|6+&,:sSc6(n9M?/a:v ͥ[Ӯ|m81&0PKk/H6n5^$WkL$ssb3@(:y<{:"0Ԛf"uیGŋwkٞSm84e3Bڒk#3['9G؜mjbm/I)=v9{pNΌ鏵uvA$}㾼Lb+Is$8e<"2qvkB>YO$I. `?c[͑%%)e-`7"65ܗ70 2n/߽;Ƨi _j^*,3L<\rwYCN/<$IKX$I$I$I$I$I$I\YK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$I'}UjyhGoWV>ToX{Wo io_›ܜ;!d$Ir @wFv>?{oϮkpÇw/Efg/_Y}߷{o&+)p_lXiב[wOΗd]\{3oԳ5sM$ `|"?9pOvᇧZ+e|/2?~Xj}j;÷طn?}bE.ցyvx½h/HD{FdV !9Vo>Q![x]/'jk?62GsUn٠smS pkpٓ؃K=[1{n6geamu /fq??߉}|{g,1>;59gl\>fO`1 `0zr(R/?>r .>\m| '{w=}Ɏg}|ٻ<ϩ{]Oyؽo8f!i/Sڋ{?_XQײp1|>J>bqg#J `uZLoD \u(A?" O+5F ]@mJQƼ6 ]GYoġ 76WЗ'wy H7\}c Ln`' dVy+(6y/ˋg2)2~ϿѶ3blhwd2k05hp9'4/ H,2}Y7&o $o>|OE&U7 2?X!?}H=D7e,+o$ă@6⑍T<8LbnC6#jyXХn@>>e[EʠnM_F7UKVh.Q 6qQD9"yn/~[N$.z.\gn=pX)`?r,̥׾-_ӗ '狅jN-Ni9Q^so*Tݲ1spne~[-;?|nHĮ⾀B>S9ӏº삈 886YKGE|e>m}zӶr/arD,еfwo!0&W,FI is{q"m˧~9 qajgyʙZWKo^kr*_XԽP{h"g&(~mQݘ͜Gdl&u9^5W\ҞlS'`5]W+hQ T_@A5me|ZG@S"hg0 r容/@.\}'6Pq s,ٜ\i]6bΎk z/#I,^G[慘{`MqZAx]꟱(E>\EMַ^oZ:GI&WLU:Tg7#yGo(%.!cڌYЍmsrEGRVX+l)|zɼP>vy33[",BӑvӦř\h 00]R]k,B¢\g/ѢV-I=H7-S+gB&z{7!o9+Efޒ '<'rks127y]9ǖ=tLƳ<.ʭ:캙 ǓY=1|hj~gqq4e\ }g=Z ظ&c\A5!ܿ'p D`~c>a[dN=-|q&}MmplX1aMnǐ +>xz1p `V\Y @v~'{SĬYŰo&N[{ڵM]uʰmT7%փz=˕t>ٷ_'a ډ/e,ʏ?X͉y"et "mU@yfd׫2&,ib3CAX͕fp?8g .?8`SWCʪ6s=uIDrX/HUC `WW?"ꅧQ\;8)UQurL'uv{Ӊ; %yʛȌ"+r;0VԊ `]/=]nâ͵<$2*ƍ" @,ѿAh0VbPwC\7Ƶl:ތ*K}yZEV8aIcEJթr P Vл<d,g)삟=OVPz\26rs `-M_x% >039ۓ#`>] :V J.roD-u4:O{g -ұn,JL\'5vڱqޤ1J?5(۲S<x.Ӎ:=Ytyuppԭ2 /\ՍaY1Ǎج:q!`)SSz l5͖q;re+6ÿ\i~ڃ>8Ӿڝ} rlfbSc&I -}O?>"((,a/l:vڎ*>uvݥ$7zsL}8I" Z_& ^Al|E>U̽/Vv.'axR>3n e}#cP~brM#c,9\hk.MS޺5.cȂѲ~w,%w`/<(|,ڔ_,MPXT}x!SPS"xȶ*ެd_}q^1cL- !R&rLqӻdES춖wFq{@X: /JPgwX3Vsla#y(Yb$yNXdq{oXNe[v}=g7E5>c |".tZhHFl7ɛ,6ۣm@J{)?8\+#}bb>-Cti}So=z5y0ͮ98I^^Q?WbXĒsGP610,UYa `f& NJȘG$yz X9&E,yh;_xXS*n<ҭ݄˛󡢌o(xqqM><{=m Q1Wo5s "4<H]\mK]Y`س+~dH0d{]_JSb\.;HгfW]0鷷hgLkc/KN3_~A|g͜%gos,tEm"WJnI8_;ZPbܡ׍_i_ |Zv5՗ `^c!4Dy[FWh4eZ0*B%*Z62FP98FhmMoBO_`.І9]Tc h3Ct̷Ùoh;Tr%ny/9xy7+~Lst[g.v"Qy>ylCQ`gwHZAx3a:n޾(6F|ʹpC]ԗ*μŁn>slC[߈K@m>d76Z+u5Vydz.`5}Cyeť⺔H$:6V:(T+g }b3|..\wm4[asx&{Kcbme{Bsƴ+_hr=̅?6Ծ'Rk.Zz Mk5ɕ8Z8r;ܜ Pv3o6r^%OŞ? &٬ȥp]6p}%Ql8WjkٞSm84e3Bڒk#3['9G؜mjbm/I)=v9܏99993?nM1{J%g{)|Rx{$e~9{޼I$,Kv>x4$ysDXx/.ӗ0q$IE&XƥzwsGdl?YK$DN.Lcm7G>D'Hp߈LK_q_^68eS x|=|yX2rp9le9f$I. `I$I$I$I$I$I$Irid,I$I$I$I$I$I$yPd,I$I$I$I$I$I$yPd,I$I$I$I$I$I$yPd,I$I$I$I$I$I$yPd,I$I$I$I$I$I$yPTׇWz~חݚow}8\_Ymo>|P>a^7 xsc?޿}! o~ےssq$%^}"pZ=m޽I|wx!gRU|lNSF}/c]Gg;g1ܐ lo}‡v_:<AuxGϑ_#?sxg<|տn{߰~gp35nwי=N{wpN.ׇ>Wkx^bQ/b&}>>׋D[U b,&M?b$Č$ [!sIJ{?M=7or 1Gs6gL}]ٓZbBҊamGay~'ë+zhݗGxzx>?|'͒ᓿ_sOu my~F@ f s)mwcޝ657ڥ?ܧĒ<#l}dg]7?-Cƍg}|ٻ<ϩ{]Oyؽo8f!Sڋ{3x?3}y>h)EfE,O-:`]JٍcΜŞq)̹pL xE}v{S!&5ntWZN{wpVb8| $ZGQ yXxzuX0Znx.Urt6Q:6l|yA}}}yx*fϙ$I"eWJ䳯z~3;K_h_R6rHnL'UK8,aN}*\Cd KR=wV8[p}S&PxņZ,L2z E8+O}ŗ>NY{(=&zcagdb@gUΕO{gyD/gU28cəms`eO?lzK x;>#.?_>:?csx27Baf/v v>vFN[׸g]9/;P_ZR-~VV6G볷礀Fz]~%97h↸&'n~o3ۜ#@$ M—z@\ry)FCzL6xꔱb<āدn*C6fszlYW3.mMv[r m5-*R]tLm2qX*`aD!,\RY+Xp_:m(H\ŝճs=vyһ 'fsw ߲^V"M* /N3ɘ0X,.X(ό<3aoj8$}^ϗ\'ެ8m+P+&G]; k` |Mp~nŲmOҟƾ;'fκ|K+,<ŧѮ=<Vx!:1:.sƷf{2tHv /#B\_%WS^ƵLjm|٥ k]vsyk,mQ@ljc_\|vk3 | Coo9+[_b{|37A}Ă?̗omLIK-asS_ qH]YczZdIo+|%1~f{yNpټ?mms\Fمm}]z=E{oXcq;pY{[ncnбҗ6\S+(B ^3Vx赻5իrVZ(TI$纩ipE%- Ymvy33[",BtvmCNqL! ,,.rRQzׂ'X*=B_EIUqLzmXn/ka,D#9cʃRxs[r䄚 w礝[nmy9F8K<زGxER^']7 ux2 ';=MV0Ub#nwk@b/Yab͠@ڐ]|jcaxo"01 -n2W>Ӹwnvs&6k8[W,FE&7cfS6OWVκ*R ` r1|? ^urK(׊BUm+kEdn6k}_vv9~s,<φ=N]}}̻dLr37|-ިg]0q2Vh#bS#/T}m35Y ` UW-_]VTFqVuGB Btt&8|.)I=[QX9:0c=l^lrh7 p>7dv6n]fOcqDBqڨ>Ls;E`Ӹv01ef PYJ (RBD-@6؄Y+-XNNj_zW_,ŚafS6&?GYi7;c^a]frT^A0ۓ#`noDǪ!sPɥ^F^g>`gW,.={&mGdFXkѶ:x,}yMTӊl\{ҁc\lo:ǞFq(}==*vЗ7۳]cǦe(=8[bw!'ȗ r'Y^b!{ğEYyFhX1hCGhwյ9O!:vZ3pokOcga [ڌru[,躻lf67ö[GKԛ1tuѲ$$C6' krb;0@ [WgXrUcrx^t•yrZp~ol} }ql({fEpsq1@8]bkߪ]d\ω]\Nm#u,5>Kh_#or9ܿ0XraՆm}ا_~OP(R+Zc _I]g] lE+w. F{2` \'I_%ukBfcAD^ui֭1tE,Gl_*Kɝ$؇>P0riJFmJqbR Al+NKR^'Up@WSEbK7,jE.\ב*SoiFKm"VCh1px9hUwX3Vsla# hoؔ K@. \_e3yn,X]wyĎIb]X"c9@Je;c^X, l<}M[5T3_krϒ0kqr\b!5Y `Xz^ P ~: `0nk@^*X쇣 `h8ưfs*5y0 Y$Xsc\˦ŊvMpR ]dgtѥ$V~mlX}oWoжSH+-()@Vd2qDʞyxa+,ǂ>X'G ~9cUΈs!"*TmZ4WGc` E"{Dz`QzkZo?m1k-3?Y9mI|;eH8[\Tڗ3ۅM|MOc}lg+r>X/h/3%x5D=Y{􇚏0=J52ѡ˷13q C;=f39zZ/@lfv@uڃ#QGeS; З6d{T]]$F8)G7YsTXIslL3 :yz#7bvޚU^!<H]\7#@헺f97[[IgõW*",<}Er.{nůfYDZ]Po^AL_Q&]g 1GiNYQ_ч6wM0rٷ9iwC:V䢩mDyޕEٛc`fا\ly&m\3ڰ@:Y 0gpͳJuY8YÊͺ(8&Vgz3IyWQ :ޤ=\ׄ(muƲmPY9c=BQm_pRJjXh\T˜q􇚏sTFAr|E(}&52&m؄);f3@6)X(6mySf[/ $Ζq09y 6}#}Xi&,_ٿUez +=>FShl_AUZ~`^vLv!t];ro)۲=l`St!lf ̥bnG YuK:.-!6Du}K\v9 `ߑӂ"nȰJ޷lӲ˯/^I0`,mP; R  NZ@LW CbKJJ²y偧trp^lc۬˩7bSrh쟍l c[TaYbCǿe2֖;z؏[-I%qiJs-R>QgGb)*ʘBG 30DorkTӓ BGj+,29Xo0ĀC1~ M2?>i a%!c\6J6'O6H 7_2ѡ͗m؄V;Wuf0" & N`ٌrkc^M")shZĆkn?~;sbۛW2n 5 VlFc/YXC}T P7?ol^]e#2B9}]= G{Ǚ1?/8ubmN՞T{rM咽M c=Rd/\̲{n?nplfgA7݈1 v3lϞ<#J \7OnbC`$N:X$9 q_]#7fW3K>OML;^#%ɭqR<'ɹؓ+=?K .]޻^,ɉhv,Y $z>T޿}^8muzu}ۻ -9;y/&I ro^}"^;r* zy}3eߖĺ$o~0]1hyl?N='Xk;󐽶Ⱦs~G)֮:t\k4޷E&ǣ/ROm|q63^%:0Ϯ B?{n2-P}_ó?OϿOXm cG_tU/m/'7 -K6K(:8Pxvvzc!⭾ n +aOGO6_Dݗg߷?<= r;zt3=9|Fd=!lJCu]_ ?}mҟ?ig)2Sǩz 1z X >sq.~>mgX9^^D&Trd\(c.0ѹtG{6; {m#⩽'a\ȍ>L컳i}YΗ1>?m=zVd*%g9}9 `uZLoD \u(A?" O+5F ]@mJQƼ6 ]G.}pjM yd@ئ擉7\_#"_>q oDT!B[L󮗡IB4> LIdPg cS&lԥ?sqn@9 "(,,4"l/EY!k M\X ؾZLٽo-б `d3Jk3mc:a$$dKuK_.|_rE/{@yp_Ddsh} Ś_Ļ"xf~uB<{ǕW}כf~Lg`ߝ>ޟ-1>n>pW1,9KLܾ?C̻.~i HX-ZO/vŗKҩ/|8T#m*xXHVfA }7 FD2iG9 (hZ|Yzz9!{9 nzuy~k~O<Ve=Q״oL 01U6[2Ik~T°C[)2ؚmmk@`BtP\0@X_R҅bgۖFTO% m&Tl¤`i뽈׀zIyg `pw1W\M|59CWT7(9+g{rl`\lGvڍ-ybFeq3䋱LELqЈl"90F3|akn3>I!_-݉썾ngWt>ꯋڙbo)[bm+{r%]k#c^"FaBu3f\) $jD1"f6l&u̗Bgh{]lQz]2NX9^3#c!`o,ȽllnMm^tnNmDm)_7D ;nwL!r}obtC;f~,gS%zS1Ԫxe7R |Od EneLc?{<-{s9,}U[hţ/=Y|1G)?ky']VeAVlFM/oU,l81 8sk\+ A׮'5 !۽}[zyq!svn*bx1ΰtg/ۙ `Ppj%^}tƢ Opv6Y߾zQjQ0l`XAW?d8A|}RYD7-loO~GF_[3P|0O` )Kzt߾.iNaEY`)០[OɂV-BHLËTu/Ne4+u-u*]R2|ٌ\gr]@3K T7hWo4Wg)!ecnzۘgϼ|g umzd79Xnoɘٙz^cm]?=~ͬsU@fǶWPwq!ekPb.i3aFݾZ-sh Zpn;CDl(E,cXȎ>O>^ƴn3k![6yP9ᾮG"eؐRRЕ ک @_ǖ<ލl)|slkX}`KSʺ',1 jrE8&s[mClG5 )?({,W'9w1)jc]Nŗf_9R|/Vg{[,͚?k}ޭ]K)ebxA,؆6&B8X;M좺 f\60[87[Wdm3{XgӥH~Vq.}]w׸Կ+!)׳86pu1=q>3oMzAJZ#W/<10l1N:X `pns~C8G7H+oI@"N n6DT6_ 7>uS d/J9qRdx2ay0~ !wȮ2:8>z@c̥dӾdȂ(D`RpMX|E*Hvʦ8.CCtq;]Ab)$ EZ8Z*8qkW<;D/s0=?gꃯ_0r/>WX;y>Aɦ`![̧dӼys6;l(=]qm/<dyns3} LGh{c嘏Eֵ>f&uqZoOSQęέxf6+/ % [0]'Y\'_k:70t1.n{3EЃtne'D0eq  6[eq].5edn[2f*.`6X?r{K%wC)[ǛSwjSw~Qj,c}QzpBX{a#2_?\/ ,,^?{EĹ]Gc9؀_;^[{1ld,/f kM(FEllf5fen//K l1^_0^>o/?}'(AAdMXEVfsбvT /.%irΠN-}04lsZ0Y\,VM^^J8}d{` =/:xE" N0Sc]"$e.:':w &M+6jAɖLܛ_D_WDZV Uٷ]5 眊Jh3Vc>ۭk}LLnx2l}T6ױژ}>^Ǔ>j=Vl.{g.Jh:.-{Ϳ7?frm+n3>W0%n_l׸\Da"²'an}Į&5ٖ/oA0&scI}Lm}oAރ܆sB7"exuұINmf{x[!Lfxb cڝ/4 G N^عm]b2\/-e/$>E*] ,Y|1l-̉:~:?XF~JOL?8jvc,0,xk/(?07~5EwE YT~UQRk4Snc| ]lTx7i,`lDV `&vS}enfV]8c_Yc&2RC'^bHߚx]s/ʮ67`1̓Qɱ}e̎a?(=̤ZPi_Co~M/!tAA+K-~ =1.",ˆ,Q*Jc(6X]̊z ,Vy6g/X g( ^Ί˃-YFW}yDz4>3G9G ^ `ܬfڥ陠Elb٬X>uɞɕ=`Wͳӧ*Tm+6X 0ax3*FB8~j Wl!q L'_66k-;ma]m,ưk$MYѺ͘{'6 ca2eS^k~=;^5o>ԃln&(o}t<7s4 q O gz #8א7kqLxDЖ\'#WYY5K1_6YL-̙͜iDڰ˨Ww:~pLCckmzի^B/gQ`Q X|:4гJ;+lQL? "0`DnRWst Y7);E`Y{}{ŦV9?% +:E1#C'BKrm[R'm{'= /muqFf}maP/sjQfUŘ"K_OI,+(~mزWdؿcc/0Y1W~(3EǑ7q$>뀹ߕ\>WqpJ~KȖyWFl'^!>n!3|]^sxyp 7͘,\z& @a2Md3KǮuevjp]f_.ߕ#5@# F=Wy[R|Kev׆$Vೌ + }XAfC%uCK0_W;Eڶ sYc"ۤo)c8aV?9q~o]du y caiU?6[+(s5WXNMrVel6 iRʽ{w s3N h;52 ł(_9f9)k!(s6ZG붝OxcL81-qXȞƂQ`D0GȜb-"'TVreg,oPC3| j]kyV~}x' ea#K5Yc-LmZon:A1sfr2~gkLO$[]. qH[*["3Ekkass\ӽSm"@&֡2]|ceJn=c ~'{Y#~#jn wٛ'?![ѱ*{do{6PҜt܈Y,L&rim&(.g >?[T<ﵽ6v}o`z*6"A,X1MwE]}/dET[E"qʮ~l2^?5b1r v>)+aیĥN] &Cz ٓ `l{SO:l-Ooד8S6Ybx s1baf,>)=v:%x$AЏnݗbI:pSCك=bxh\yxuݺo/oc>;uٽ"Ʋ \g[` 䦱|PEeXyT7Zҝ b q/9`8K>GI_oB>{gYO^^y]kN@{w%%{B\ ?.k%ӟ1ntrDq| {~{)I.G;o˳|q$E8'Ynq}|^ (}].+|3#NGڌs㋠;3/ad'OI$I$I$I$I$I$I$4$I$I$I$I$I$I<($I$I$I$I$I$I<($I$I$I$I$I$I<($I$I$ɍ6~31'Aif?w ̕M:Id=2&wtܩ68u|r?86f{cfjl_i={\爃ٜdS+sY}踧/e3q~c%YK$I$Iơ"7[ =׮y7fp.V1f5 ٟ},T*mC?{(v:ne nz1eB}/%>{X(cZж$o=}%nj1VjƲ)u^ w\ќѸ)9M$ww>>n_wu?{)ړ$ɛԋQ~6VEYfLJ__>ד;ï?cۍOS___^=?j{ o3 |}Fb]lGy3c6h:bE_}n"k3}9fe5^׬>yrOZmu yud՟B=q/6a1{QVYX<ɞlo~|V~?1gWNi񖼕>7luj?j?=Ǟ~Xs6^|U?>O_~h{ymZ?=#>Ck߹l'&{Cۛ{JC/?;M0;tzx<q_KoxQ?%O>kpMpNgS$:q]wZ ~qxۇ/6 (/ٍ,eY۳c]V>*YЇD^f !u/XX=CnjagOyrj1xzJdշa߃'Oʖ+95J^myc1k? 8Os 1AqVwm=㎙G}e?C\kȶEOy[p{ϴNg0j7oA͐+}n)&i׭+-kwcYcV,ٙH ԋx п#jQ+N_ (AAu_,L% 9CXd=כl:V^.RF],؁!K~ <Bas#1Lc⻛fsqk{ɦ[%Vwmѿ=V4h9]-f}Vd4Z_oL$ Z_YiErS<['g6v!:n9C?{uz-’m}׭^$C7o~B+v .+}Vi쁎dk>[WEyOmߒ&sk>V9ݸ "S .hZآқNdRù\e6|Umモ"G3N>E]%o {/*Eo/A> /ANSNh%8Qwc-Wfz[vً\; `ֿ f7̊{R:-@ЗIg_"Al}4Dz пzN_Թ0U1*U>k7y $&nֶk㡯<51 :_iյg7vavz׵7lnT'y\^vlL@bߣD>*`ݨYN̕kԑ~^ڱPYܙ+QNoK&Y},f `?T}YvG\<ǚ\*_^[Aʔ=hwZÓڬlӗBۢ~GXg[u׿0{L]O0[Xmc܎=DiQk=pd/KlgЛ( 06#7@ᩴ6$=o^7dflh^R7&[ɪ7zdz {F4}P!N?}Mmcx6wcבod\_It}/rlAXS{6~AAGߜ\t\G$t{?Xe;zMSyc[[k)frL5]?:U=V{q'3'r+Yg/YslEo61$VD1L5oOwm\g}"ۣX}{_-(~bo'fP۶k͒}Xs 꺧g}"M5ۂ?zPmt%{sEH # Xtm^ V0Y/"M)֔b,4NEһLֈ?ReI= ZfHoZ.7|⌍<E1ylH7_߿^3^oKn27& B!"םh]\V?Zc$Ik7x":`g3CшMcԇwgc,7$'} 3(~*}?7sgƳrԜodK7vޒ7&n޵V"ha`4nHx&}so]|B똟usVZ%1zc]<f\Xӑ켏c o7Wo+!pje5x?>~4n+;ZtL*u9'+~0Y6w"-9XGa?s-tjYet::=VYE// 'kOT7A梣n3:ke޹=qkR;zm,ztV;<k6Du1{wyt8=uڛM, Cr.Sm}1䎖Pe^K\żVWI%ڇ}r֮~D댡;6tss؈;0UWs7eٜ=֢m`bx`ozz-no-_kCkڸ7n`yevkxx1k}2˂+_zcIPwM| ^[ <b״V3!x=nyOM׈#_zuv~&x^n'ZǺ.?g=?s,u4-jWz]8/QW|#+O,A/?nͽ֢V:OVu6=e1?:^NŢc /XSjiu_޿7:.:ijECUt< _" aS}I)&u_VWiyJ|{Usm=Iƚ~< llm y7 !Y fǵMps,<9^ dCYi=T_9cX2FK(b wokfkмuX=.a)ɿj~//F~Wi2 aJC[+?DMLMHzC7C1jNaZ\ŹpņcIZ$SyiH9o-(11c?KCl1<[s_39VMǮG:IRЛzbs^ު:i mz6ZG.aKڶ{_UF}ޫGV]\бZrXO7:~X,֖cuN~^Rj{[FD^ DSwK׫F.^.?署=z?[V?* z4{W[ϣxD~HL= 25]?kBkBRkf-,ZDZvW#v['F򛱄0MetMoZ `d:?DU?F}9d;c-oRXkjDNYCE }0u4$XTO7L!ϯ=x]:F\J}ט|{M\CYg;om1s_]kՄ=uzqj,XZŷoɅuZ_N.#Z@(o)EZrX3z[[fkZ9V_q.*=u'>?Tn?"{?[=ư#~N9VY~UzigS޿;(^8'}qF[uZ ]!hvos7n업!xևB UnS~04>zc~~uυ=xV]BvK=z3͉XV]>ת_;OʷPkĚeVF-x} N.[uZc Xen驳Zޜ${>?hzmVV*[Ysx}UZ6%uxݞvkr Ǻ8k  0w)lH%TR * }85r/ֺxk@u9ŭM-N+[-A[VG\hxkKmmxUyeX:㝷 ~ӊA؁Z9meX`=| _Gʭ2FC}^ZM8ˢVNqď>S\S߽w{zx5_5ύw:ujyu/]ܼG|{}7>T^}h>+kR;#/]敐C:KкI8TDߕ?mg/_ YۧR^|Qvp[ȡBÃkx{l+Gƾ K`D#sVcuv/fpX)P`2! .pdD7''bBO"tb§wc,Ușh%~BOy[5nI;R؅x 4L8[%pS+H_z1 {9ss8N=`0 p;}bz G.K8¦.\e?*7E :EBm!}z9ȶJ"E9+R.+W Ou6:Fo~_vF-_a$UE{m c֖?FǼZ: 韬}jGnnaŘSqL3 ^Gȕj_;8$t ^c\>$U(1RT9sR X!:F:Ku3a p9Di9qDH,$I%ɜYv'ƾMHCTy' I'ޚT'Q`Lb׃iܵpr81IGjj6m_ڬ:ms7o㼙]1BmxʁqS?o(Y<:=m8'0G5 o/ۤޗp,XhS>Ϭat'\Q 4׏fWn1 ,_8^u~Дy5b'Gf}McHa}Ƴpy2x+ܧL"JtbynbÓ9"^3t<˟$æ~HP}:㚤Jn/Ǿ 5ft<5*yY8NS:v,*_)"sSD}K"a"hQ2O6!a/fqdMZIXA7ǿFX:g@ MqoѼeiCLñwӸN-/ 6Mkǵ| R!wSF =_,9bFl>ҼQ{9,U~js%-“Y'Z`z|1$RIRYRrZB$ 2mCL#UWZcJ|Ee%Fbv`ro;"g)XwKPawԈ9ϖ VbǷYj\Nd$0!gJ$kv<#ծ3-I9} lm[z7w`8g"̱c-nT>v-z[[ֆxq״qnݢ-{%yETb36^簈UܫkHGc-= lOtH$źDae.-YRSMH10Nqj'Uբ>( )2`_x XIRɲݽ_RdKd dG?O.Y&yb|&,d$P_u9Y#jxU<]dA#åDgS>d -fr.&m_W/f(O1B~)K"SttħyknM`ɴcjw=\Mb."n*Xt߄&0nz9IJŤ;^k >HL= Ib qLeB Ibh@ky_k݊i\륮{5xi}(A+#X2_$Zi+.,&Q%* ĮBSYD r<2>ɶ$c.dHdDiIőbgIX̗<1>Z94}m(`j]L''bi顩l=*Xn $Z}Zk?{eL&b-%-Zs w ]T  8K֤s F'z$K5BƒR֓Z4{!Hs\$깤VwJs=?p@͐,?S} 8tNx*H2IuHYHѲD-a7 j`{'h,9Ԋ5?dK~( EIDATi,c=O,`O0OiG}z9"عy't[{ϧ5LOşU8' ؉(68M&[EKP,u ƅ.#fSwX8j=sBNl^+\`艚RaR!ck &oSl:us,bL̇3Յ`] o޾޾{yv=j·7gp,y9\{Һgϙz8j"i)$8 ASG,fV^w`Ӝ4k-Ų_;?F;v]|s_|(\72vO>w:ujyu/]ܼG|{}7>T^}h>+kR;#/]敐C:KкI8TMgpx߆~|a,U8~1<}//U~|<|edz[zQc?r`@C=Ԣ&}RQyDV/zϥk?WRQm ^t ,&r! IFdy3P0YNx(] .T$P*B')|y7R%9VQ-'hUsTc+]gK@:Ĝ=ٔϲhaszM|wi} Ib:l.r{/Mf3=yF`u}LkƑڟZMYNZdY v.Fu7wbڰxʁy3;ݡNhOi#|kħVy߆1؇_d/Biχof fCqC96)<É="8>obBFrD?U4 PY =>f1Lh!r1ys]Cxª[#M,ғ\uآmʅsaA%y) ~4$kТHª_)g685%9|``pl#~(N\oeB,ujƤQ6͍bLX 7ޭt.c?VoΛflp.s7OK]E_,,^! ,!ugy1X{VīcQ.[3SyFxy[kWS?"^:Ӓjݗp+6ͱc-z#ʆZi 7^=``$|YS9:yo4EGO:ZT5NY-h .,|T6coH&u_VW)yԌ?^~ռ\sO&<:]+l͚HŐxE!iϛJ6ͱR,W,`;^ dCfz' {ݸ^А9O'ߑ{sR'c\qꌹ^urW17}1#c +X!dd.%͛\k}-Xm묬w[X|i㛵)Mzb4n~:6Nuo;/ĴX me6ËI+w},${20 ȃ#cW>M_>BNvox"}O#꾷Ƙ̊Y>蘳0|˸ p&l`BdT8g COpA~` Y 0pV@ `xg|)IENDB`llama7b_autotrans_1to2_predict_strategy.png000066400000000000000000000137321471400134000352630ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_CkptPNG  IHDRhUsRGBgAMA a pHYsodoIDATx^=4UBlK@2rBbĒJXByɰl6[6qr + fim5uOUWuUyU}SL2>sP$&~!BP( &IFhBMH I#4Y4a8Op25c{]Ӗ_חŗ7}l˟cV#{3F}"Jϓ|_:}y8^F-._p+LZci-xK5P⃰ȾM\V%ۚ$^Z[cu}[.渕^Qd0?G6G'6aokL zs2,%B8L_w%$AOW(\տeVUJ:o!s1M 6֗2>dM[kۼwKY%ž->W2k#CY/&2]ϏLhbscM}ƞQ˥%6۠;i<[pdk^ϯcp k Fwr[9 ۶UӾ>U_Y΄!Qmjb~Zt믥bߤaLAkm֋ԒgPx.av =:ף>xa^sk il:Գg]6cdԞU)uU1Pɳ>=kRu|dgId\۪~sdXݯ:4ΗZz~e+܄j>̷HM_Ҩ]r KYȤ]%ve{@ hk6[ڿD.oV cp>m>mt{$ڿ\O@k˦ʚ+'gΙl֞S2_O25=Z(t%ՠlɇJV+Ӑ-ZE hymvcI KlA`s]mB \$ SC={m~ESϛ1Η%חȩQ!_[=jyLfSh1t{nGav]n% ={Ji?=)8%BWҨ4@.5 F& 7 B L/ q;Ajq #8nkl_l[/ח)%{QG;]^O~D=~uU#֚$f5)z9l^ E ǯee+ ansLX*{JՂn4 nKHF+i|.ZQ I ` H80 $Ŕ&mX_~4@FhBMHlHP( BI#l/_1a^ I#!hu4lf1a^ I#!hu4lf1a^ I#!hu4lf1a^ I#_|NxN0^}}%v\d>&e.hƤboT%2Iݽ|3s:xW-G3pk{Wdn^m˜<[5X4zOweRM]]i5:j,&f뿵b'0Ic!gxbžsP) l?% {iX#y{-;jvS.Il>J;C'0ҷEkt}6+$&Eo~=_2)> ]XpPI+mMmؒ> 14|M%kųm9$|VmXd$,M82|?9VJq׵=|^F_ߏf.LiH.\_ńQߊ63y*/}lQ `͈MX̙&M=|2yNF ASg~c'/K&>=n/].K|ZjZV;Q[6Q'K|\ϖDO :N?ͱ=)]6Е4Z|L:vit}7ei#{CޭĢBl/_uKy?|:kr;dn^g=m/Mp6sif]eYYwӷ`?'o„C,o/^;+aW3 -uK A>IR:$q"IΙW&SVk -OFh[1Op[mBA4ʼn}c¼nF B?=&6!i >c¼nF B?=&6!i >c¼nF B?=&6!i >c¼nFhJ1'?$o oL-JͿEӟ|X%iO'֮}Y%i,% {9?/]Fy{( c(@2}K(o& ߞ&$nIci$ nIu)PL4^W'u wMH(ߢ1BpVI?'DH_/$_G?EE}Y%i}CMVK___?Z} &{(|7$dSILE648l$R$q Ed,˽!O ފƔڽQ$LХ}(z//Ë- 2Ϛ𨜏~ gDKޔc+}=ƾtT7jGG#n^c&iҾ{Z m%`=H֜7G&jgc<$y: KUϮ|:=Lb73~s^=`m;gn{Cw4'pf8K$%g7lmn'P'yFN?MYSDNDu)jAP/]+v ?kcY[!NXg7k Y"=FN^3|^;lv2/bۢc>LbaQ 377lKwltǮ42xZV6MJԽ05.$Vffdp3vH{XBK_+v/k+;g^T^_1oODƸ92mKFVcW>l W kWMNNJzlqrO?jO4?򯧣j'fUB|ۜץW-rOǵ1DѷRڛ10[gTyU\vmX{-0 廒@w!%N6zBZ.-dYW=CL$I*[T:Ga-svP+gy}6Lcn賡 WR>< ʙu~mb\Y#I,}MFK1~N5(XUBvBdv/ӽJ{YTޟ#<y,gN?|:ןJou+Ewf.I=}UDޜՒF/z(߿{ViχOCA64~$[dsI##L(EP~mY˟{);{SI.7IKiQ\ly8rCP"kgDϪ%oJ>ZKQGu|Ck^}Bv~O#vh7Ic/ӢxrC5 Dڟ1Q*]^Tf\19WTnj=kء;KOL0lMNgow/䓕՟d6O駱) c"ȉn7E[<Jش+sŎpgm,Wdyxެ gG#Uϯ 𣏕S/ۭċ`c)fXc{kM"ҝ2>c2뱫1M ށx6MJԽ05.$Vffdp3vH{XBK_+v/k+;[h{'T^_1oODƸ92mKJo_v[ tFv&Yl边XO KŞƆKdWIN'!ק)+3t,/{%?#B>-,d~;{*?~7u {gR3.}-\#7n/7bo% tJ݋uc|{\;Q/lF8qsY~z,\),l_u::6l]59վ(Ϝm4Nc<TXL(S88V򯧣j'fUB|ۜץW-rzߟos{7c`>J4/nڱLZaA6w% dC*J&yl' \[ɲpt?3{ITju*U[xIgcP ͦEnQ֕M p^gC9') xTy#ea^R\.ݞ˕5Ă>D[?U_+yST6^{=4p'9h$?SG7xX Ϝ4Y(t? ޲vMW\,F{EI#9/04~o i&$Є4@Fh0 HFIENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_1to8_train_ckpt.png000066400000000000000000000160021471400134000341230ustar00rootroot00000000000000PNG  IHDRl!osRGBgAMA a pHYsodIDATx^ۏq7💼kp5@ȋ /da(Ȗ%Xa[:Ql6-R"%ީm%;)#',ŝsZgj4/"}8 8H2CvA :%w E%L9H2@ $tN$sCM'5Md2յ>0cs.@󷶢5fJ635̵iYVK5Qte.꘩]B;vJ' i|f}k6ϱlL'ź5eBr5kVI }z>:]3:(] Ðd2B yJH];F דړ?kc:yOsSۮy 9G9LJoAoX6o'bxe(1Mi3N 7LMuV~Z=nDlIN?ݲfCzۗd gN-Gt|sxy {I=Ҙ,_eo~>K6&pv1~d;!|i "19v95Vd{'3Y&߂E$n jȇABк鬚cd|m@;PRI'Kt{7ӟE(N6[N*iM͑ yhk*=$%v-P'C!hKͅ>@!s:ij:yP0[OJ#ӑzV&աْ=r)[SռJʧ[r+|!E6tͩ$ѭ)#>cGt99'vuj=rZ*% Kݻw=ܣviZ Y)-D vU9JDj..d'%﵋[9=5~y jD> ~$hibRsldiO'fT5ݥuSjlA2yKI[O}hVnŮS} O,;D3`! ~6|,oLMXg>9Eu>&a͇KbiyodX|p1km/$/MU==Wr{ 9Ûo _7'מfL&~,nN)U"2ҺJ[:ѯϱ*z,f}b2ѬJ0FI;/j`]_wE>}@\mzZ^6jJs֤~7_S\(tF+/9y/yMAdDBDGܖ7nlu$d֘Mc?:ycOI'<%{4cX>" hBON0gX!lY`R"l J.LI&#xKv%W(7dA :I&$sdΩL2QPPPPPPPPP-d\z\*`n=1.qLj@0k-縀?1.R@ @ f̀Ǹ?EH2SǸHI&5 5vs\) fN{ c\"$Ԁ` ة[`q c\$ fj:YSk2DM7L':oLd2`[j )}-!:.q?FdNt0ӱ` V f>MBnGkvc:PI|?6Ddz7Uvڲ=;X83d̷!|M9ǸH%};p1$sl"l@ U%!NJھuLHp:xQBm]`7- ,j܏)}$ fN9[c-ף et=퓚`ezs] Mg[Qx^:Xzd"\a*g7a=5^`]ڒY+F[KHgp.6]rĮesPf?.$t`ILq9]`I&{' TM253RAL*Tҏmt XzI2:T)qlZ/X,zI2?Cuѹ E/I&X-dA :$:/Ww`'|zeF,$Pxyw Q$3\)]@}2U KâƷex?(H2Tָ_`KsiMMv'&Y#y]ΌݘN$uZ7&MmH ]˺$ fٚgLYSYr% 5ē%^ܠK'qe]c B%PPo&1 * /r郪 Dp`!69lRd*kڣH2"q,=M8I]vd{zM<:Ǣ+N!̛æ^k ]^_BYF?ldzLu,V \8M6vruA2Ŧ[ړVt&]4v^jo72➩:y>n 2̥v[=$@$TiHhD {mP􌜅{8)mٺ͝XM윰K686gR>'Lz.X۪/ GzSYH*6zYcO0py xH>C0Y?7't:i{8&=#gqL2 a9H8,Np(:s&eu,/#~6zIT!&osݳul'7Q6`XRJy=$뙦@ [66vS5zFb'=CٺhorËu{^_$aL4mtƓ˺u6JD`dzBu-}7I=CϦBOV4ÏdHV$(nl8oG;C@H%mLL_δܴoƦg, z:ֽvrP];XE8лd,>Kʛo=Gjҁ\%cQ nշHlT^/iDU4iOa?~@6Q7v06Kژ!d9>nΈX'<I2w?O]Ts F}3se1^C@7mdف?"O V $sK2}ţ$ӗɟWo??U/n\ﯿ͟]=:, &q:wu]u{w՞益/Ptɷ7{^=Lo(d^gO^Tg߽oUQ]v}StyzJ?{S=~L$k﨟l^W7/W_R^Tj׉ 껇No?o0O2`[Y} 5|eumuЍ;:|WM]W3V>>_7&M2V77 ֪^ftUS~o6%zcPN~X|~Yky=.Wo1_8;rF=qC/ʞyI]y{$s=DP23o\L]J~J؃5vvo6 7_r?WgeQ+$ϑwUÿ9ۼvb~ĦzL'_y-~/ɤsؾ}6r?}$ڴH{|9sly][LW$4Y۝[wŨH$dhd.iT2}Cj|Kɧ?\JLe.x='yFO}nONw$V̻a?"c^]S_>YF?ldzLudnz%um_/W?9rZ} .[꡽?>u~ܽ[&37qbe~LJ6wuNDZçԿ\W?{{_9>kco2CGQ],9:(tPk릺Bu؟U8lA2+Sh +٤~uz~uaM\u61MT(ǶM$|ޭu $2'Jز XBIճ d>zdCv 7lzKo]T:zN=t{17~$N,*A:Bgۺhwz uf+GEP#6}@*AZ5?%!Do%7h$~O^E$iA)ЦBOV4Cd Qo1Ϟ~\lRw$Vͣa? a} DD.5;?bf@=:vzI2oR}:rV}3FoQg&@ЁA`咍͋IA !ߴ-e I{&gL/(Qw ~ @ $tL9H2@ $tL1J?u9rIENDB`llama7b_autotrans_1to8_train_distribute_ckpt.png000066400000000000000000000323201471400134000363030ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_CkptPNG  IHDREsRGBgAMA a pHYsod4eIDATx^5Ibxƕڴ;g14pGD- a|ۆT]\5b-yebďXF=iDi̦SV9byzdR'HRG2{cm̗ۂJY$}-+k!Ծ/mg:|%S\?O+w/&[umZ6Zw?е0|<~뺶,_m&hf{˿//A>e =s{A^ )AOOpԧǒtX/1PPKObA/qCo*ds:3c7ɪy crt|e;/\u},4tQ14&aٞoiܭl E>5ͮc[_t.ꊟ,Lcߡ"b ,qIjO$Y?6qykL8?ќҹ "kajmRtLE5 WݹS /P d :xI/6nlg?_x;ՂbDd0}>NQ'(R:֟tNyH7D]~_jlV]s0j|/FZkYww$E% =7i\JI5>v@)1"b8)FmcŀIt*-OE.x:nj1^>ע OT c?LP'Qƒo8+ }1y'ͯ129?u>AUbz;le!1v=F.kj#}{./[DnQԣ}\1o)g`ĨIQZTF:`·X|`b}Gn똍a[/o(i: %sIG@;ݸLY񫘛GXҹ4_8]swieu':鲕8G>]WCz_Fhr^kʹ}W9zdRӱGV_6,ObbTKbRDA6lsP2TK'EGp1THJ,"""""GwR")&E2~m85`}n )`>`}n )`>`}n )`>`}n )`>`}n )`>`}n )`>`}n )`p NNU~ lKCn/ctRϓ; .1!~M wo}>>H _M2 ǽ@ԟ]t>r6a`́?n϶$rw\cR Bid&:Tؐڨtt(Ty=7lF|Oa{rUR?~7~<+F箅 h.Kˍ)~8Ǭ~tRO fNX;}Oz>'ur8h,9&E v6Y_Y+|MsGI hdЖlO>r0٬3 0dƢ-uYpׇ_j!'缴H'E _j܏X@> jA$d(}YB2 5KTtoFl=O9`a8'\\ۓ~y?C_>2~6lEv?8_zvG66k-Bhl_Ӻ"Z>zpN2%Dt|+C3dsdJKe.q&? [V`8K1nuj;3-XMұGu]jccc?0'E,[I\ />Ѧ%[*{C>I҃s{.D.ISJ*6.+)bf}|pwdw,) }jLCZ1\C%yEqnyP>|u>Oo?KɧoZ2hruArbbͣ@IC?-:fή~q"A-?:XB>MV~zA=ZjRTgq=:|un'F[unOJS#Ή1Wf*mJl_(R]fcžr]$\~l:)`kнHwoZv~c )"XoSO{#~b!>_ۃ);u[I ;u[I ;u[I ;u[I ;u[I ;u[Icݯޝ ,ֿoߘ? Y<)7yip[OOhVvYH,r[ޏi,`PB %HZF{zRDO3NQ> )ppRDIQ"gtKIpR"*d)ux~zu8BuV`3IJRzZ@RxR_d pO~:B}IAR@RX>)|p`O|~Ѽ|YX"J_+>$N'EdhzṇٽGH8^_q[ Zi=:?J4Y+ke>:Gčy_ȚjtNf ؄(hXā8UN7tsyTN z/M WNJ*q_};s$;l ?xؤtt%#1/`]Щe*čwYɏ$tbAmR$Jc=Ve)9RU2d/O+s$^;}->66ĈW={Mƒ3OR$|Gu8'@ĺ+$8:3䟏f'M˺ETCčDoR߈+ԇ ",|?kg,l$E$E哢ߝ߼ ,}y <'E=>$N'Edh܂9Q^ $EC·=._zlj|#|WxI+ zKoʮ._c|15GվaGH:Ip%9ov^obBU`U=_}* sd4?xؤtt%#1/`wY{],Ht֬J:R{+ɨVB_YDsNJGr.KwbkC`L1d6-j{L7)M: 䨝Ɵ#% U57ycӟDuܧ(QYoJ|e:G1'JsyLA9%qC5ъa=ڋ't6פ9/P=㡟e A"S)ف@\7R^3wO.Ԗֿ_$;Ҍ-kvYE%oySa{77u_c,s51Y>"6 h%Nv.DoTW4[k퍟̗sKX>e+9RU2d/O+s$^;}->66ĈW={Mƒ3OR$|Gu8; uTHpkx埏f'M^uFi/so9G1eFHѶrαڛZV9L!1cq8t H#9,dްĈS9Ѕ^KTkѷM)Ow^eN3ߑ_fc͑*:W`1g,:D}(^ӱR!*llccz哢B}\hCB<;Iч`|P{7tk8IF|RA8)(XOw>:/B+|R'ς,QTT^8N,r _`G{.@ 7*I+LKre7e_B1?+wȚjnƣNw8ᅎ\(9pĊ[K*0*HκI P%nXk}OYЖaΑaG.;-`L/ idt %'ɛZVЩ[I9&G/,N9c9%E;pS v5xM]`fz2Fٖn>Mm^u? ̦^S[_\LǓ_$;RҝaͿfa[h{:˩[7zySG52^3#)b㈠ V2N)zi9)˅6-d3,}x wCg6#dO'EIIcw7B/"'E_~r>h^,,I%@nO嫏IQ))`eNy8IQFxqCOrwtɻb˲қkqcW|15GվaGHz8r<(zݯ'Jb;'bKS䬟H P%nXk}OYЖaΑaG.;-`L/ idt 0č^+3Zɏ.;R{k]crR$Jc=V̦^S[_\LǓ_$;RҝaͿfa[h{:˩[7zySG52^3#)b㈠ V2NeSHIV64>=Xϑx#n_5?K`?I-|<ཅB믰kx{ץ`"i!FoDc}<9)4zpGsuތڷBaj ߐ0S7v1{M7>a'k&FTfNtAO'*~ۦMޔ^eN3ߑ68 d+ԇ ",|?kg,l$E$E哢ߝ߼ ,}y <'E=>$N'Edh܂9Q^ $E`X Z']a']/+-):Gy_I=T(*z*`GH^|i/H9C>oS׷\͘b͑4?xؤtt%-`L/ id.`*!`XÑPqYɏK$[ }1XhЎYKC7`k~^ej f'am_tP{ec(wIQm`$G`4lBG ^zI_ :[{Jf׌)t{Bϑ0Ѡwd-PMuojbɗ+f4}5iKA$Thx'E$!RƑrh}k$@A-&*ꄔLꓻlx0t)wC}Pc]T䓌iX=c` )p )p,}|u^VdO7ןY>)|q qX>)*%C# g?] )B.bh4Z3.yWȕޔ] #Xbƣ[[o^lIQ0ņ֖ɠ s+|k)T^YEJ*q_}* km<&aG.ߵ|#L'pF֘Zz19Z })fq%ϱ+Ms9Z1kyb 5xM]`fz2Fٖnr21;xܤ6QJtd0XvrT0!Ch2&Uĵ(6|$H :[{J]{_M L8cFݑqV-j|S{O<X7m9+tIs^ j'D{fC?)& E4SF39B-8r-77wO.Ԗ$tg99\)x,G^rj@-x~qׄ dH8V2NeS99ۈI>K>uN_˱ 1bq^ :Q2h hK9( H!x 7q*EAIY"n$zF97ÚczpgsuތڷBaj ߐ0S7v1{M7GR a'k&F$tS9 =עo6ySXE.zf#m9 d:/B+|R'ς,QTT^8N,r _`G{."\ĻO]1_ W[zSv%tNc~*szL&-./ &4Rr9C>oS"%@a>gB[5GI؄&Ed (10wm!pcz9_8H#sp)P?SNJ219Z })fq%ϱ+LsbL~-ڣ,:5vcle[oȅʸt QqD)ѡ`IQh &P!E{$'GbqSoJ|e:G1'7dch0/O:Vdci kҜIjўOIC@#? PVL: )ɻ'wԫt_\LǓHOLtg99l)&Xd-ySفk[#f t  qc=dyI D)k. nי^39f[z}S"З)Sq1'{z^N#Zk9!F,ݾ?ak0~"Q;=[=-Y@ kh *EAIY"n$zF97ÚcRѶrαڛZV9L!1cq8t H$#9,dްg^2jZZm&ohDYo\c_+e1總4٘PceBUmdMݹָt 'Eo?-';=xІlg`iv哢~X>n 5<5O2}` " |RyX哢/?9|4/_g? wb [02'̦^b:DzBw~7̥;#QaKq|%,nɛz_BM7\{MxGVo h%NS"nEC*ʵ;Cs[z}u]K\zNzZO,>K>uN_˱ 1bq^ :Q2h Y \&.^S(|<6;i2#ZčDoR߈?cP߻3Xq/$#`X>) H'E;ǯ?, =X-)7x{.??܃m'El?)R=dRDXHTWc j{ǣ%&EE E.z)wIUލѫK$mدK󮥬K};_ae↭VUZGw U㟼;/b>)ZPtƃi{oHԫKmih7 =R zBrVr_}* km<6;~܋Gw ՟MV*֓wrt&@|ɂeg;4j-Lk da|EF,NƠ Y=]h,6vW5a<=~b3hnb}3oB&ud͊Gr慻ȉK?ѩO[ }iNc=:r::Dͫo[q`L1.㶖mW={]|&E_[ĄØ Nj9j'>KX^ԗ_kP;Rq5R!\c=~QczIPܕtL$twD-[=%JT//ŏ,Q8ƌ#a AdqC5ъa=ڋ'_6פ9/P=cդ_hHO:&h,)ɳZwq,%]VA_k/]/Tg\[z{fuAa\&۸ r=> ǿR^3wIaC/E/I*bC|%,nɛz_BM7\{MxGVoOP4Ac=j,TyԷZ1KwIWonEJ#vԭJa#:%stXRtc)+Mj9f5joO}4]sTsr=Z%y:} }F̌X}ú`,9ՓVK5)a%>7b 0wtt뤊v\ŗ)cOcAm\m=l,rKN:ϋra݅u埏Ѧ|+e;@{L/sO;;SsvWhmm-6S7v1{M7X5)Ri3h-'eM2h$r\<69l,і=ke˖gslٲeswgggVO?75߼skι\O_XmӫOy͛'_\N59.r ϩO~ynw~o(_8yČ7NᯞZmw]}i"d,8I$A! N,8In/.7o7/Q^Mo~έξzBJ.֙Jgs5]ȶ!g2 YݟΟf&sh?꽬K7Ov;|`v٧_=rc{)>qѾ}Zo[.|{ʳlZױ8 WO3dzo~yy'Ïj66|5}!&ZϜux8inBFR0ū(F"? Qd 7~ 1HӇ_hHE(vS'l>xzcoQ&N|0=|[CwǪ&;Oxkw NIJ8!Տ/sKvZY6$/xu>]^+u dLRS8#oe-dT*Q)\}[3iLw=Vi ؔbtz_!(\G^{Nպ>}5]tS~u X">ͣI b Oj~].kyds!,(nFYCzhnVYO-o`5/8.!y:3 YldCwdgIҎ:{(MUsVgDZ6`{GN{b88 ,ʷׄ,K&ri>MAioa#86&5!kyMȲĤZ.Bo\,iǽ_̅NlC,Buaz[Y`ϱU] C;Xmi- ǡG5S]oQܘ՗w?psS9Y\0ɨ?7Y%Xs8f(lu,kW2|IZg}x-ѵi!*oGcsP! BlŌ($kqb!bE K6jJ!| }fMuQb&y<˨ ˺ݏ"(<[^\e<SBn?>39ޏ+~>o[ vVTׅ-dy߮,X(^cBr.1#Ϥe=g%.dʳ-ǘqhǷuW9hv}s;2߲Zkӳ?|l1>839,/N(!yvY/'燸gu f|~s{>f4r?mG/9Y~o) cҚ u{kH[vB.OjOo.FDqfO¿;p8u&9ϦV;+# ] 0A . q+`9uq/-lm/5aF&!X~g S8 61mTe+a^k1*fxe,W?#I鞏}FΏcwJg˹bZ{kMYm+#]9[??~5;/ߛ;2Q#x;R RbCao=~o3!{h.ޟf]H?;ӽ_Oqq{_ϡrudݽg{:/w]rf+kxl0.bm!m1i!vkj0M7_[Q[taJ";^Ѧ|:&S`+n^Qxi[mUT6"} bpU6c2 d>/WBV=||9Q rFv?jk9}<:;3,0s=o8Ƽ; Y/{Y-;Z޸y},jsJ{z6vrsL[u M_~1΢XWaW%:Z1ݼg'ؽU̽wO~ۼ]Ld|KHF h/wkiϙڶg؞O\>˱m +jacD8 }H5FňtΡNyr,̓eۓcsԿ ]㸰r5|?yR5l`n} 1.D |Q h8F.,m]i+Sƥ&di-1f{!ˏJ9;gMqjͺ("R8W_d٢+͟ӸuA1[\c^G1oPCYC߸+u1!,m𹰋-Z,߰P·h ,vvq1}AuBb"e:?wNڰ+fK>쳙 ɒ\#cucQ;5}{eڸc9Ԟ<ݼs»w+N] ,8ҭ)D,'ܨʄxƾsG7r*ޒA}Uy AwBA _,Pe~+* 2裋-^} 7avAKrf8ŦႦSY{YE󖍼^qw>$ѠK>Xb#EfcՋ^\ҦDڷ՚?Sg6Y"֣u|K1^>״bkO-v:u:uݼ̱VbرX.8!ލSbTRCZ_9w(ZW}n99K~M]w{X@.r:BQg)bK*r U@|UYל OKM*H;G*,ɽjZ6!EO]hxޯa1z'cܳ<FaêxWr&Hk ~lKk?3π5׽}9}cǢw.8MZ7#!t|[+ֺRnz! R}U~-r1fRITzBRHR -@3iK[炑gC`)SK|v[6Rm K;ZV^o _in#yXl |iZG/"~i7.=`>e×ck)\>sb,D;;_>xsπ̝4H'?xرXpX:؟-{|jƻt{K! UN,J_U_N >}ZBE(-ŕؒ ^LZQ*BnXSq.Pũ%E}R[5C( ݚRIw-k| (7^|M],|~fFb;C}X˵b_֙i]3ϚMN_ïUs׈=>cQݐ]]2eT em{$/ugto Y7g2U;*j,^K(-ŕTl |\q&jŭVѮU mBfMR+JE:mŬu:uEDR{@X_QZbWc=KopZ_c+O>AN}[E]+oڻb$c_Y2,9_x3'}_& }Mm_b|_=&އ¾VC"+sJ~ך]G,_*t7RQиr_"\h\Yhѥb[B*$dQi/-X;YWdԼn+>kve٣QK{~Ok]g O(оy mȹO\cvSybG} 撔Xb~_wzZ8qzZdۂmb䜺my5k4a7NIs0<] ~Ǽ p⤯*~=Uտ=74#EQBh)X]Yv&qX2Ey{5ʵYhޫy>Zg1%/k{_+&"j;t.H͇vu|{;WnOܚzqty^_kO5~\JT<;A]a<;K.BEmu{8V=:ʻ\E{,͛ }Wӛ.s빳﹞.PPp:7s~1]qNeFߩ/cڹp8Q|nz[_B]EX߷*n՘}о:b!ϲ[+J">EHk[ie }Yv9œugʩ^0̙ukF'V3OK]5*p(u:es"S残e3>ybٳ<i:K,z4B5c1Xk%&w\k߰?cvXV]-z[NBG翙<\`z(Ͽz/kͳ,?>(7},]Wܯ؞}Gx\ou ^&L] ~|=.P%  {o!\,"`,Պliο 72MU#t- ;^j뮱c..IP+9Z>dw] ¨c)v/5jEJZۇasoYdTG^%-ɒkB}}S1y$$kG]yd>1zy/ {K}\~%ǎE-!oz|ڬ};v `홲zOQ;7.? ?k!y¿y ,'1 `4WQlE;RnbDuўuߋt]QdӧN||pޢLͯMp`zxn?{y\>y)}Yo n$`%%d};}-,hQKn'\\.QYo[׍o;R۳ߟE*YE5XDؒ ]Z.+^me7R lVU-2Sk`jzlԇ|QWloǑIX~>;|5:OoeyՋ}5]Ư sTKF_Y儩^>ͣ,x$>ui-zM=E^(kYMb !ɷ '| ˭ hq,x͂q6j4NX_s܇qm?UN-jKD,{Yӟ,^oAQ+O#,_w[RPcb%-RV(*aU(-{=ɫz=hXkMp>.vo,ĎM/;$7R}"ƥ3=~lD- ܑBߴWc?߳ |ѷߑ}cǢ>YS<.6}-)ܙb F<do҂wMg YZR|{Mj"F0᳛6bA4 YmB%&m>֚pd{ߔƖǾX0vXuKibk!e۶烳?md.ENj زBg3qZ{j}й̻ta"!ĦŸBo ;Nj"FXD &>72B,~uNm;5ߤ/Eb;˵]`sV췕oyvV4{N[U5[g!gSsiv6G-(Ytj,7"kGڹPya.X,R,uƯ9S^C쩣xsm`BVra`ެ "C[`BI' dnE STq;( k8R{j),-.4m8U:1`ټqw (hBX_8*KSYg1&<P[Fo񖭄h3~NDUOH5?*8E\t,ӓCrv: Oy`QBga'duDy^C~9[ Bp.pM= Y ۳p*Hsc:?-cg\2 7\\T֡pU6Kɚ"Ӛd,ht|^|m:>sߠsF4GeZ~4v;,gj蛵K^8vXwVRޮٗ%K{;r!+`C(,GN])@~kh2l qMF9v*/<rs6(rUd6 EZMBl?>3`f,[ s-Z!K-{/d߼Ѱod>"0<3L8ڛX;n-E2G;v}KҾZZݿwkՍx,n1phZD UbV`ijEBdnP}KLc31~)ZgZG/$ E)rLNͣRfWN:O\AƏm{Lr< g4KDOLy}M۹λ4maUIWŬ@V]X`SVKMҶ[bNB׹r"gE.esY!EnQ|^싌PZsW/}""Q-[`.fyB/gcEmq)+!>߹_kAvo xm}ֵ+YW/HXgscfٴUƗ{ryYs/lOyP61Ǭ #S c.>3G8 )6p,pc`! ‹v@Ȃ!ݚByͣLȊ7n3˦r dՂIĚ>! N,8I$A! N,8In/.7o7/Q^Mo~έξzBgo-tau9.z3]=J_^wn߼:&!g_mcoo"?}>?|}~mGiO?JmM>m}?.0>nߑ3[ݧ)wa4RcY~N@XloK: c]{8 Ev!-efpS؋Mf!xf* ϢBj"B R73Һ{+Ƭ}}vԉ?x8>456vZѪYKg__W wuԂ]%NIJ8!Տ/sKvZY6[˧cRa?j{u7X,펲IRb?YsT!9# 8VBHƄk7Nfbm/DJA?7$|ճsg{m[yӊy=&k/]%d+ZEJEVQ/FX*>ui"BlCzhnVYO-o`5d-<˿*sͤ-svP^ƦlΦe[\gO;,oXd={u~ZLK?R{ #1kSkܶCj+ײ[Wg56prUٷŏ/͓|k7B+g'T*Kj"F0᳛7LAg|&{ۮ .mB%&m>֚pi^p%<=Kܬ&|,N]aLլgi\DE6s,bX\ -QeP鳂^lrl.'7,Tؗ*rs61cy"}*1b|3 ߹_kAvo xZElgR.Urf*iMmJ9G`ʹ=vkgsG~~ouO̕QR$BY Y Y@.x135۝u 6湊Ecla7gYnxuqeRXm013+뭽rq8^$ F sLmB,D[DLNEGqsĊb{uDH ^Ftngq %VRxH>7'!! Bc ^CJBLEWe>^ھyvzֶӧO>}us\`Hzۂwİ^ڳ<шpT!b-|2=?zo{<ݓJGg5Y:aGI,0l{u87W ۅ,gNcoCYb 0,'1 `4WQlERbDuўuߋt]1fӧN|pǰɶ.LVg/ςX:ڴr.aZ) X 'dxn YN_1 fRy ptB1l3Gmﳮc}_eQ6 Y)W.w9p:`%fNEH1ÚGpG gӶXcۋ0zҼA)M!-dlG*\ec^{헳k+G} kZH|g֌x*divKEK&ri>MtFk<&dYbRvcI߸(Yγڳֹ Ukp df=Kcb*ЙcL2\mه4w iZB7>B<\K,еOϞոƔY?u/mGp^|/QT(_->]!eu6w_^ u gw)%ʘYs;خ΍)|tsacƬCw[ckM{K}9h^zg(c`Sߔ/wPi3|)!aK1Is)rB84 ^m5a?"@9Q"%X5qN_v]ů.6Չ"G :l[BVmgoKr?gZU% PV!zV=@i|q,7=dM'-~ \NΊ8-dy߮,X(^cB;q1m[sCBoqb{35eu6unsoȘXC絔rEc]\1w/*}u˶aV`^GJ1L1KM991MVcxҟbGrp0!+@5Rt0oVq-$l7udST"H\ضķU=H)diqiؤoũXZXy<{˃wk9, DMBY,V7Xż0-[ Ѷ*H |R[ĵKG~1=9?č,gG>}][XY^EŶYsT͵9D;fr?lٶ"c\zM4ϳq؞c>{ǔ9Z{ 󽕞7bz|rgͽn{]OoY;ة濋/U6plcr|{Lax/11bދ BV F|\78 [2:}o mQfmc|! 37|65ǪX-" yfUZ-aЈ(@eE,ZQ\Q ٳV\bWU` Y74,d-l:Ogkvd8fnvK[\guc ,&@IqsyVX6?˅J0e-{m bZM9rlbfz+2/u$ql3.bDx[koj[oL5}l{Dݚk,WGE&AiZ{?BD* EŢhB[,E'9n(͟[3s-ҷNqe1{.Bma@}^LF{Xrs^DQhO/mGK!X8NWϵŶc f3z]Ҿ=<^O׆}?_"C9{l^7l7cm8Hn׾7kʯ+,à߾Ͳۜ/{oC@<~m6(h7y)bqB|H#h4,m]i&6Մ,m% l/dqX)'y{y,l}QpE v & rYOtqP4@us k ", .eq%';WwcB#YsaϽ=HTəm<Ҿ5Ŷ%+}MBs*;*6f=T*U?1WGI e1_#,Rl~f3;g&dϳǬ^lwֽs,ؘ*vylB)ݜy~d6a]jFMһ013+뭽rq8^$v G_@؟cnf!޺%gJt/?$&tm7Nh!QJ'q]! Nt  jMv|fPn&e1733i_qr d;>T+,?{넍sX7n64y{򵁉[,&Yp dI' B$Yp dIp|q9]y3y}ajztn=w=` <;Ǵ}um tӛVr+w1 9{hۅO3}yP^gkm;J>X0}Qj{oYnq_=vh>-HqL Y7i+UȺvG 6LO~9ۿ}~1=9$~>4RcY~N@XloK: c]{8 Ev!-e?{IZwB( )`0 ,vD&+?NO~ {=#b*gOSΏa1HSkm'\>>ޭ!­^tbn]eXE*ApJV Y~|![Bַb̂E}\>z Q~D5#dYmwMBV>ovpT!9# 8VBHFr׷j8?^~c Z9ho I/h!g;UA4koʛ"泸ޔb X%+oWMwQrEfm]j|\7RLIޯe-Oclp,[f !YZ!d=NMh:ض<*s,1PfRWTv9b;^W(j/c[gӲ-b|Yl.H7?:|Uҏ^iH+ulY 9rp7+ە)KuuFfš?R{o\k`GX?;"VYČ%T4Z ܦd :#5v]Zo,1o;$KV1g峰[^C,T/©+ꢚ, |@g~E3Y ˘E gɾ,ŶǍO?`75R6 t-!/x5.d1/~Oci[Q,ng9sT9ʗ>b[PrZ[gs[py}zPϫϭ5}ȑkG7>f:te;Ƹܴzcuw2f 6p{M"ŚRtK_I5.A,/pijŽE"%5q1}fMuQ&yjAY[%We_jlٌnB#dzQNwҟbGrp0!ˋJ ϔ`ެ J[ BI' dnE칰mo=!f{RZI#f[SuV-^Zҟ1_G I& PV!,NE+Y,ryb^^񖭄h[VkJF͏ +-ڥ#dF#پ-,Tb,wq9Zv3^l[qygXx5 ;c3S >+huv/}H}CN5]});g~Lu^(bbļ7;A "o&pa=rHI_[C[f_kBMClͱ*/k5 ܺhSao7QʊnYY<@i3j~g{XG8اGB[Y=BͶ{${)h;;?[lGcZaiQ{b Ϟ7g+gi׉nk\ c[ڲF>k\ >{.Uٔ)ǖ1Z.fVy>OKB??s͸m6 YmqATZzQA0TӧͶGڭ6}~7lѱI+}Z+b|ض/㈂(@YDX/Y,E'9n(͟[3s-ҷNqe1{.Bma@}^LF{XrsA"otО_d?:BlOcCqX=km@f ۭǥ}Q{;f=1xny "b1vcE/'{s r!P:nn|^q ߓ^}?oaמ_WY]ǫ硏a9_ކyڞm6 YQ0d7n>R)o m-DC"rgoZLM[5)ڨIݖSuncY^|}d{*BV(ع$+e>]㯦 WAYӤQ-[ /L I4XhG߸+u1!,m𹰋g.v R%gHۖ|s4 & qlhܓmv6w'VPa?\Wq\i%ELb.|H)%>Ͼ䂧:{]9[:ϱ`chQ]= vsXwEzitLJۘra^tljp8od/ޔ=vPD6 uD,Q?SSE|1b07Nh!QJ'q]! Nt  jMv|fPnZe1733i_qr d;>T+,?{넍sX7n64y{+ł[,&Yp dI' B$Yp dIp|q9]y3y}ajztn=w=` <;Ǵ}um tӛVr+w1 9{hۅO3}yP^gkm;J>X0}Qj{oYnq_=vh>-HqL Y]yqq*d]LW;ۣwO'?vW?_>{??^?hf?k{[X' 7%mﱮc}=JapiI :lof!xf* ςBYjB R73Һ{+Ƭ}}vԉ?x8>456vZbѪYKg_mYwuԂ]%NIJ8!Տ/sKvZY6[˧cRa?j{u7,펲IRB&ՂJP03"p$@2&|X(-Vm&K4oC{SHrxG Y=ѯ0|{٘&7YoM KWD} X%+oWMwQrE[#<*b+oڽ_Z>&X=E^bBGwXBzmy)|&ntmyTX~cbͤ-svP^ƦlΦe[\gO;,oXd={*bZ+\Y_]w嶽R[ ?b=!Cce#~ Bٷŏ/w~b|kXQ,-VN` 1o Z ܦd :#5v]Zo,1o;$Ko\,iYV7@kpꊃf=Kc*ЙcL2bamه4w iZB7>B<\K,еOϞոƔY?u/mGp^|/QT(_\>mAOWinWorb @o?>2f܎2k(s#GïA \X1ڻr^R{1ס;ʘ5ؔa7;kZI42?rF4_qb,w*dyCVvs-,)e,"L]UW\S}!kwoT'\%&mo YtW 9#+VB(@YX8[=@i|q,7=dM'-~ \N 8-dy߮,X(^cB;qu1m[sCBoqb{35eu6unsoȘXC絔rEcr_9w/*}u˶aV`^GJ1L1KM991MVk<Bۊ51 k~y{8sEw Y^PAxDfUWڂOJ>I KvC_L7E%-dυmSY5ۃB֊M6ZU}񯅕gW}][(LJb,wq9Zv3^l[qyg&8OZClαZͽc=J1q=C>eP鳂^lrl.'7,Tؗ*rs61w!7PĈy/nwYA0UsYL7L#[/-lm/5A!&!X~gn]U̯u7& PY-b=+Vg("{fBoU03bg8زwpBͫ Yoodt 5ΓYZlkҖ?CF?|9[y<+~NÿNw_B^{ݲ݌ז6YbSwlͦN9v1bne%!^Ζu$ql3.bDx[koj[oL5}l{Dݚk,WGE&AiU_{c(" nebYQ xn\4ڢ53GQ Es`fE) ,fEh8 Yϋը+_qT|.Q|r"('j϶Σ,PyzFϫb1.Gvlqi_ԞgY`O p^kȟ/XX!H\bk1߶e$dkϛrص~gycvQÏ/ccjmΗｷ!cbbŊ?(h7y)bqB|HA_iY"Tbj q YvK_:RN,o_)GDj)+I\V<9jpE.]E=M,:յ| Zqv }K߸+u1!,m𹰋g.v R%gHۖ|s4 & qlhܓmv6w'VPa?\Wq\i%ELb.|H)%>Ͼ䂧:{]9[:ϱ`chQ]= vsXwMЗ΁)Xwab^gȅ%V*[{!б]`pH&?Ǵ&BuKTT"`tq\x;bG)*t=w,8ҭ*D,867]v!%v:BI$4ͤ}IAhRjO6υ2c ![b4^-[,&Yp dI' B$Yp dIp|q9]y3y}ajztn=w=` <;Ǵ}um tӛVr+w1 9{hۅO3}yP^gkm;J>X0}Qj{oYnq_=vh>-HqL Y]yqq*d]LW;ۣwO'?vW?_>{??^?hf?k{[X' 7%mﱮc}=Japir(!9噰yp(d9aD!^(Y,2EǮT :}<}%^1lG_ݟ>uO-泏;? MͯMp`zxXj?{yWm}V"*r ~W SNz c(j}Xc>fg]$ '>'!jDry=] BIpT!9# 81ÚGpm`m3ƶ]XyVڛB >Zَ~U6yܭb~kBV(^"W,~ (pPm]j|\T0}RrY';K3"BlCzhnVYO-o`5d-<˿*sͤ-svP^ƦlΦe[\gO;,oXd=+TEZLK?R{ #1kSkܶCj+ײ[Wg56p֚pi^p%<+ݪ&|q-N]Lլgi\HS:s,bX\ -Qy%1\-g_)O!d1s+cf(cb<:7r:ͅݽnَ1+7/yꝽYM9^|SA5y/e~::~)&i<>Y;T f1 &Z8Y(GS0XD &>72B,~uNte0o+Pf]N#msUƦ͘ &t+e<//9*p$>8Lͪ  d}@톾oJZɞ ۦj),-.4m8U_ +ϖqvys-j6˜/#Å$Qp(i', e1,񖭄h[VkJF͏ +-ڥ#dF#پ-,Tb,wq9Zv3^l[qygXx5 ;cԄz|rgͽn{]OoY;ة濋/U6plcU˷>!7PĈy/nwYA0UsYL7L#[/-lm/5A!&!X~gn]U̯u7& PY-b=+Vg("{fBoU03bg8زwpBͫ Yoodt 5ΓYZlkVY7|b Ϟ7g+gi׉nk\ c[ڲF>k\ >{.Uٔ)ǖ1Z.fV[dk1r٠$w~z_Gr7f!-.hJ ?"VO4*TӧͶGڭ6}~u [taJ֊X9rn+& PV,*EEX-\sQOuBՃZx h4,mK*15myʸԄ,m% l/dqX)'y{yyV# nHB$YA.+5]".ʢ&bnAu-5yaVM<9q)+!>߹_kAvo xZElgR.Urf*iMmJ9G`ʹ=vkgsG~~ouO̕QR$BY Y$֑ 䂧:{]9[:ϱ`chQ]= vsX6c\:`ޅy!X\o@ǦwFA"ᶛlp8vPD6 uD,Q?SSE|Ctm7Nh!QJ'q]! Nt  jMv|fPnZe1733i_qr d;>T+,?{넍sX7n64y{+ł[,&Yp dI' B$Yp dIp|q9]y3y}ajztn=w=` <;Ǵ}um tӛVr+w1 9{hۅO3}yP^gkm;J>X0}Qj{oYnq_=vh>-HqL Y]yqq*d]LW;ۣwO'?vW?_>{??^?hf?k{[X' 7%mﱮc}=JapiQry?^lf!xf* ςBYjB R73Һ{+Ƭ}}vԉ?x8>456vZbѪYKg_mYwuԂ]%NIJ8!Տ/sKvZY6[˧cRa?j{u7MlX?BvG9u̽@؏ ̩I <7op~;m5*r^BVv"%^6ﵷnM[BQ߸օ/d[E^_"ll"Wb fOj~].kyds`v\}z]7[mYMb !ɷ ulEQcmņ7'9BQ{:m3bsA= Ybl~ ԪWiGCxsa$fm~umwߕ:}HmeZ6xƆ9Gʏϣּ:#oaM_A7Y5c0w YZR,Ab#>MAtFk<&dYbRvcI߸(Yγ٭ YoBBtQz>U3?Ǣ,eR gɾ/BglKk }T v\sQ.o@>>{"WBS޾rg19澘uvy8GUS|s >]!eu6w_^ u gw)%ʘYs;خ΍)|tsacƬCw[ckM{K}9h^zg(c`Sߔ/wPi3^C_tuRLR d pP! B0/pĶkdM(ca ߘ YێkŦ:Q(1iKj,JЭ/͑>w\*D*Zʢg("/<]g~,Do+)Y17gut^~,/XU%~ūyWa'N ?m+w}nv^-Q_;?[l/b柷ƶlΦ߭1.-ۃk⼖r_I KvC_L7E%-dυmSY5ۃB֊M6ZU}񯅕gWJ\(abS3㟡9K3!+}}O=fufs>ucqѾ}Zo[.|{˳Vj4zUȺvG 6LO~9ۿ}~1=9$~>4RcY~N@XloK: c]{8 Ev!-BV|#&! Aa6 YNb0Jh0 ,(D&0+ ?NO~ {=#b*gOSΏa1HSkm'\>>ޭ%^tUvHz'\G-U^B q ω`kGȲ([,?7pT!9# 81ÚGpm`m3ƶ]XyVڛB >Zَ~U6yܭb~kBV(^"W,Y$sE^_"ll"Wb fOj~].kyds`v\}z]7[mYMb !ɷ ulEQcyyͤ-svP^ƦlΦe[\gO;,oXd={*bZ+\Y_]w嶽R[ ?b=!Cce#~\FQgd-i#:w1Hf>˵f pT!KX*%H[Bbag7)(zM]ۄ,KLێ}5 Ҳ%KyV;T f1 &Z8Y(GS0XD &>72B,~uNA($,1 }3=M#dlR YZ\Z+6ilkq.VſV-^Z>lTŭ(U4EJ\#re+!Zei Q/yvY/'燸guk + իضr0˝r\qV9?hLg-V\$~l+Y7.g z uv|l<ǂymGwX,r͙Gocf-΁)Xwab^gȅŞ^tljp8od/n/ 1m Ed}no]GU3%:U_Agͅ+&ZqIGxAȂ!ݪBcx}eG"YbǩC.[IYLM#LW,&Yx+ʯ6 :ac\(!9 %F %;mʿ2! B$Yp dI' B$Yp d6_\NoLo^_]fzj& ݾyuLB^<7v!LD~|0=_xiZێv?L~ޛ>}.y~\alq}ݾ#g<.ڷOm R܅z/{yCC#7Qjg{nN?'ӓwOG+5Mgmo u&ð=uL\P_) n޲9QY~l>[6{^lV=]b!xf* ςBYjB R73Һ{+Ƭ}}v /x8>456vZbѪYKg_ ۝ܺ+XE*ApJV Y~|![Bַb̂E}\>z Q~D5#dYmw-BVħBw ߊ I@2&|X(-Vm&"L4ohC{Wz_a.1Mw+o<{-b+q _Ȓ_ . \h"a+}dGZE=_,cu fOj~].kyds`v\}z]7[mYMb !ɷ ulEQcWe.6Ըݿe?yNα4ٴl9_ ҿw_g}oTEZLK?R{ #1kSkܶCj+ײ[Wg56p֚pi^p%<ςnϥl*WŵP8uA]T1sOϱh&as0DC;Xmi- vnk.ʥVmZB'g_Dj\rcW_N,Ɵ:ӶΣXr8/s*/}.Ķҧ+l+᷎l9N1䷟W[3knG5ѹ# n.,}̘uuvq]i~?Pel0E5ε{\/$Ű8OO!ĦYLGl ;j0At VM\ B7_]lE.6V:ςb-1)D|"U% PVq,E+a*PZE._+ y@=M8YD~7WSbn,"~fi Y^,!K#WXaN~A~LVܐ[w~^?omnMwGC[c\[;2&96֐y-x\W˅JA{ݲcy(X׮on'摶R SL*RcSwf̅p[cxҟbGrp0!ˋJ ϔ`ެ J[ D' dn/E칰mo=!f{R6ZU}񯅕gWeP鳂^lrl.'7,Tؗ*rs61Y9=&0f1 s!+`#j. )zőT뷆(Ͷ188 >lؚcU^=ju}8FD*+EgeJUrόBU͞ fZc[N_ya!ޞnf=y=\띟-#1ssݴ[:+ko>_g1gOϳ4z~^.T셱-xmk#5.q=E|֪lc˘l-o3+փۡ$}ɹf6E*-X=Ѩ ֚SM6j`\>yCm{{ҧ"V6{+(U@4E*/1s,h"\EA7ϭ[,ΖEh8 Yϋը+_qT|.Q|r"('j϶Σ,PyzFϫb1.Gvlqi_ԞgY`O p^kȟ/XX!H\bk1߶e$dkϛrص~gycvסo6ې1P11Ϣ_3F!+ &MGA>-R!Y-\DC""v-T0TuncY^|}߹_kAvo xV lgR.Urf*iMmJ9G`ʹ=vkgsG~~ouO̕QR$BY Y Y@.x135۝u 6湊Ecla7gYnxuq7y΁)Xwab^gȅ%V*׋xcS ~# xpM68}aiM("p{:"⟩E|Q1Atq\x;bG)*t=w,8ҭ*D,867]v!%v:BIi$4ͤ}IAhRjO6υ2c ![b4^+ n;o dI' B$Yp dI' Bmt>ӫOӹ\O_vXt:SL{ v#D[v!LD~|0=_xiZێv?L~ޛ>}.y~\alq}ݾ#g<.ڷOm R܅z/{yCC#7QPоz*'ӓ;sΫw_LO=M~|4}VYu1Ο#Y5z斄LY7{V8.:޻F! #B F)Uf(?vC/}ogu"]WYe;^jp6}<=1,ij~mǻ5ĢU˳ ξ;uW*i^BZَ~U6yܭbpﹶWӥ/"/+!E__ĂGytq8ҳ?!oBڽ_Z>FH${.>oQ?ŪB֓oXM KȽf^̾BVeU{};S iG=NJ <64} }/ YsoZNYx& ;t<1޿w2UbXe 3PMh&m J3B^mׅ6!ckMBl/8EɒvgA W 6`+e(Z, j}~q6ه4w űZB7>C~Yk.Vmz[B'g_Dj\rcW_N,͡NA|d YgQ(beBBu}6:ufq##N=eO+w>6W>kLݑ=pP! Bl/pĶka։D `ĕ t ޘ YێkŦ:Q(1iKj,-s)̑rH-B()ğX8 f(V {VQ*ngEd^'-~ \QYΊ謋-dy߮,X(^cBr.1#Ϥe=g%.dʳ-ǘqoguU}h}w}U dl{1/`r0!ˋJ ϔ`ެ J[ D' dn/E칰mo=!f{R6ZU}񯅕gW6j~Te1o]#dF#پ.UVGE'BK_cf;f<⍬NշB;%`Icۍ*zD "!؈fG!d9uq/-lm/5A!&!X~gn]i_5+9q!K͞YZc#BѴwpBͫ YF։$S=3 Y@J!ەb͚sJ0ŬU+)BVaAҋu7fnZ@BV[\"~EhTok Qm덩Oma5Af0Mb˼`=LXZ+[ｕ GBV,ΦBu,ET_d5 RTuV\-2Ž^!_a d@}^LF{XrsASt"/rL+f3Y`z\ofN8Ƽ; YMZs̊|\ʷ=ڏwuYn9H-BVL4ٍ18!e}[>uBZD4EfwaoZLM[5&di-af{!ˏJ9˳y݄Ec_ žp ' bLao]M X)"Q-[`.vξ=,A\~JOwDŽ,G.Pоby{P·h ,v5gC1 =lٱ 8od/nk B.hgUg4f dnU!b:9?wEVimA]eb~/Ϻy|h9WO3dzo~yy'Ïj6Q♻~y>?xޫB;*'*dC~1x?3v?{d(d9aD!^(Yb,2EǮT :}<}%^1lG_ݟ>uK-泏;? MͯMp`zxXj?{yv' V) X 'dxn YN_1 fRßz/+ppBg??5E{~=]ǯ&Gn>pXՂJP0+"($GZɘaͣ|SX gӶXcۋ0zҼA)] >Zَ~U6yܭbpﹶn\\M/].ʿ/ !WP~}1VhgV8N7FX*f>uioD2be !YZ!d=9S7]zng}{Fƽ{rzqޟW_#΅$lK_l"59">Y_n߽b֞Y~ YZR,Ab\-9MAtFk<&dYbRvcI߸(Y3,Z\jXu\W}S1X1$XBЌ*,/Ey!͝"r,:ǶǍOEi]DVsjqZB'g_Dj\rcW_N,r8E! ϩ~йۮak$ξ/yE_:jmo}GJQ쏃p㿜bwNwHfHp V->|fQ&/Ë7!B baěRq_~"#~))yE"]2^#d,Tf1 &Z`Xu"%X5q%72B,~uNo[vV"=[~,/XU%~ūyWapcrtsp9-%碟8܏ۮǷ/c}*O뛟yȵy̸/bٯY-A6n=D>DIZ#b.kVVn-L&jDEy_i[];mV1DR d&dyqB T̛UA\i >A$,1ME=-'dlR YZ\ҟ"f[Su/lyj>ֱ2n^y0\4USYZYe+!ZeiN#m|q_LOq#ّl_$09akc41iMx|OIeq}ZHv1<7q7xN.MRMIḆoհEdϧ_Et։X[w>{p8˜ㄬ:zBV&56%Ҿ՞* kNL#BA "o&pBSGJ"Pڢ6B\n/lbkUypO_yf֕e^ܾRڛTe+a^k11"d M{/d߼Ѱod.׋&N2[YxDT°UY庋o7ǖӳ?/{dglj\lNܩB,B)X-"J]̚EE "I[PG&P8?,A(M-{e D~c5U_5APꋞV5Z87l3.bDx[koj[oL5}l{D 2qmX{_ [taJ֊Xz\8 1ھm->7ZoV=.dYq|ӫQ!+V>\9$ BP/rL+dm(^=:6{/^">1)Y8yvL۝Ѷ?>韝q?~9~BKa8 bݾgáHZCDV_ U#C!ezh"ŬE׾Ʒ5bwwZ{[]D}L!Z86 YQ0d7n>R)o Q h"݅ſ k15mĦm,?s+D>o/#wv"CE!v)/2 oQ9.\AS3}RNP=[P]Kͷ\օPЖc}K߸+u1!,m𹰋>cg]Kʾ9<$OU$VvrMe\im!ڳyq;1Îg73m(N~3,7 zB?1p+I)!k"ĂI81*f㱅^~. -dǘHTEM//Y ,Em7n--Z"Ǵ}81F 5^E+/dݲ`}ٯgo #ܰWg⣵0*$8!cqGz Y,bH)-R)OJ{{$FJ],62W9<9?E+c'BUo3S&6[3捚U*%d[JCF|~'n[`}5+p8ogIѦ&P.DBVZ`%o,އ}$,c=C mEm}(.}[BB!I_&8tL'K>AWqV/vHȏwz{ӿ=[iVl0V!+g $2W@j%_䚆E1O_ȑd7A,[Ȋ7:98W߈c㆐pn;RXsPn4Q"vj4E@n٢bjHM,MQD~#C[.&u-ĭG BB-s;;nXs5c7Z"H{WvG-r݆j./kXbMBa>/kٍUbZYNs(Ze V1`շ[esB֚u9dQ2l.K 8JZ>k dA-/W}_Gdj -71}qcM|oG l$,?N v~?5d‘ YUc! n[;\~ 685"&'dEdwD(!nMp>.V;7%d-y$EA,;ƿ;gnj?4^ *(b>CZVZnbP^Ҥ=k=̛ǧ'dI}BVj{=4iZ}Yp|q9]y3y}ajztn=w=nʝIB7S ~g?=Xȯ"qA(BVKzB!khĈU?*jD(G-LZoih(EȒ~:+coo"?}>?|}~mGiO?JmM>m}?.0>nߑ3[ݧ)΅!R\T<,=i+U r #|2=?zo{<ݓJGg5Y(]<]vBt!A+,aŬB"d%{6Q!k< fDMY#-HJs_=Q&de^O Ϝ?2GWYQr€!E ̂EfqŎ(2Ԅ?vC/}ogu"]WYe;jp6}<=ɶ.L }6v?=!+[qjE|\wcwK,^Bn|*Lz\8[}OϞոƔYpB8leQ &RϢ;$wD|uE[Ȧ#| w YZ /,Ķ\8Y(GS0XD n)pu YێkŦ:Q(Agk>X5~]*7-d­+Ƣ*!+QڳBn?>3 y8"q`-o`lg(H߳7K[򂕿]e Y7Q {7Ɯ.^_MW>7R2[z.ys,~|_gzh1I[g;Ü_X"&Y  ESJ[ "H' d"$oυmK| Y5ۃBHU!dxʹ)r(ڋ sij!ˋ:ov!VֳvABVnJoVD/+W6j~Tz1o^#dF#پ.H,/ϑar#:aLZ|dt~4#wS7?iAGп )"dd[8ĔM9N_[C[f_kBŽM0em2ǚckybdތqB=l%̵k-:F,/GBuZ\/q?wB*#G1w.d¯cyxT︙"&Y"~EhT!vl1i!f; KTZtc ~:qs{)mcs.6ec YQnEm_6 ls-ҷ`Z Kt;.Bbz5*dǗs "g>H> Yrx\dY/dZ}~B|nSew4mx/ckt{yv9<ӳ p(dEDH"`b\' 8Zxi2 lbj q YvKЩOϔͽB֎"w18!+sy>qWӅ+hb/ƼbnAu-5s[/[{Xƕ\ ܵ Y fi7υ]<DʾZ$?g$iqjQ)m7r|8x'=/|oۖmnN7Χ˳ᙞm;~# & <+ų6i'B-o3S&6[3M56ipH;tL'B$Yp dI' B$Yp d6_\NoLo^_??MC~{Yoݟޯ(mwc1GgOz~;R|}޶ ]й0DE'^ Ydzo~yy'Ïj6Q2qWr5]Xo׶0}I!&ɳQr€!E ̂EfqŎ(2Ԅ?vC/}ogu"]WYe;jp6}<=ɶ.L }6v?=!+[qjE|\wcwPE)ο) X 'dxn YN_1 fR'z~yn.NL:En,w`} UJ(7}*Lpʛ4-V)Q)\B`m3ƶxXySګ"X‹IZَ~Us0:AiS+,,w\jt7+޸EC(֮"̨m]7O*BV 7RWIޯe-OclPxN3gqMQ~u썲,V|[j _p3ԄrF %ul⌶6iW0yxvs5,\l>bi7Bx>0͓;4}r lwO4#=~9Mghݽ{f޽_lrg{1w ˽ݚ[b _x,>=P[{Q,SpKE D0sLAg|&{ۮ YmBV).؎}5 Ჰw q3B{7H Ʃ0 ֳP.$Y|>®f-]gɾ/ǂylKk }TPEJzof%d}2}Eƅ,7}b<5hRuy~Pg14b,τΞdL[bp&wDly3`I휫>&}˽X;"?w׎J1}ܮrB™ޯvώ/b{_3","J)ЂJF}lKEy=k㹳|y|mT,fxa!Մ(t"@9Q"%pp}LK_v]ů.6Չ"G :lX4ENJRi!+SEu^X4.P (s̏`1-|sEdeϗs߳[-dy߮,X(^cB[@1Ll׌<*>U˅_BQ}3/dI?܋_bMy幖 :V3;[/boJ_p?34p5_K Y{a c.#kjH|SKZaivWۍ ",,H!|IYb&)*ɛ>sazBV %y-mpUY1r.;:g|)\ZC򢎻.),}oEȒvoJ6Ssx%wCڨQżУ#dF#پ.( 稌qz?[S !K_>Dr&eQ(6s39N C ʈ_nㄬxrgM(#ĭ[(N*Jc Y}i@#v-)d-^cwɎіߊ 8CB!neLSg6:}o m!kmc|! qp 76N”Eȋ~ˀkڎ[ {3w gk«,bEeP_CU5bݲwpBͫ YF֩byMg%ay6A}~wIdK`BVh_z,e,k4;ؿ{ 7g}{FMkCED8 27?qgGĜU¸ͲE$l =1ݯOVD(|>e9!I"Œ!U!tN9 Q5\bn˗[|3-c!s~ ~dUs{5Urlnc̳mlӳ?(Weܭz{_~ˑ?1v YE8Zn[I&ذkl٦odDŽWIZG(Ha)ģZc{#IvZbC]hϛlbn_F!+ &OG A>IWKK]X`SVKMҶ[N~B~hvTY8닌X>qWӅ+hb/bV us k ~ފO;BZ=,A\~JOwDŽ,G.7i*er>WP^+r&1U=iINc];14H{<{}K90$OٔgiGD1(%W${ӿ>?J~u+uY0R&F"_kMsܣk]n\J׈jx=,[`pUp8D)^DEb gZܘryٯf~)~׷\r1~; ;PrL[iZ$ʛB -,[WEI7ڷf'Z8haHǴ^~Z$U i2?ֈ;;.! 8A(_uG7g_fqܿ;k-[Y>*d9=BpWJEOKE*YqX!w[,_kjDao5Ǯ1nײH~H?Y'?=Sr"S~>;n`[eJ Zb@n7%b􅊄}!"Z"AHE<> ?:ߧyz[.b{9𮀐ZߟbWg>8obeB)TCDmPrxBr(#=QhDArm,ǰ  +K y#K)>"Bv~h-Ī.~ d6_\NoLo^_??MC~{Yoݟޯ(mwc1GgOz~;R|}޶ ]й0DE'^ YS l>釟s^=ķbzrI|hz鳚Ȥ;1Asu>]t>vX署x'.YOg_UhM0Exzc/Qkm'\>>ޭAm qi~zB>Vռ}VQ :) X 'dxn YN_1 fR9+f"dmBtx d,w~S/w*,6-|Zc'd5u{ͤ3v#kYbTnN͈ qofs{p9W WPӧ"$IbEr! v6kl{']U7 ;*%WEK>h_0E2ϲy Wӥ+ֿqE _a\1;ݒ)ڮH"+}dGoX7-d\4I=k[fE,V|[j _Y9r^ӪV~ow̲O}3 `_F=B)f8"VBViEuv3B^mׅ6!Fl>֚pYqXm YȅCU=}BV,´+л".ZҘL@]s,*E u,ǶǍO ,Qsj"vKd(^ YnLy黟[z?9D!ysR&%ir|V:Wa{ D:3;esΊَ=cYOknޤwJ.;f0ƟeT,fxa!Մ(t"@9Q"%pp}LK_v]ů.6Չ"G :lX4ENJRi!+E*`EH"{?>Y$8QL~7WV3 u}^~,/XU%~ūyWaG NHб&AdV1 [v/5>W+QSzLؒxb GWwoJ;L "B. ‚L"+m' Y$%f`"S<-'dlR YZ\ҟ"f[ W)cjsȡh/Η.̥=,__DX[EbE*`EfwՄ,}KCl%DZW"ZiG瓜 t,ӓCrv:wϫseSz*91?06#ǕH~%[a̎kϽ+vݿ]g?wJ>+׍p`^h2ĭL bJ̦[/-dmm/5A!&IyQۏxcM1.簚E$?W|=Q ׻95yNq:}jpd]v6jXx.us k ں m9A\~JOwDŽ,GcSSug?<*WasaK9_2grvMi*{1SziW>mrlѶ؏w5ѿXE5ȹպ3;ō"WWUY]&)g(UDo+Yp"bX@* [Pan4@!j#/`g|%!$A! N,8I$A! r~fznz{/T;-qٓ|t4~`/;y:)!Y8P}.bz#?Bo]*w.^'L/UۿsoFu]h]t77>LEWe>^ھyvzֶӧO>}us\`Hzۂw{WB).*Wtힴ{ql*d@N?'ӓwOG+5MgmG&iۅ*uB֮!^W_e[\ֶg[Ǡe;ge|ߴ-sn\W=\~{i:wL)]d! xCA8(6?Qd 7~ B"Ӈ_hHEسvS'&l>xzc/Qkm'\>>ޭAm qi~zB>Vռ}VVao;ApJV Y~|![Bַb̂E} ~m\1T t| Í Y~vy?\kkw,2J/Ery5yvq3ϡyvBVK2הּ_76r 2J %OEINyơŊ6<7?zCm&O4ovC{UKx1I Y=ѯ0|2Y'(abee1+\M X( .(fB/T6.dE/(ؒqOj~].kydsbc=D^CBGwXBzmy)|V8Cg"/~t~qȹH39G!'wZS:U۵uDL޶:rEBϝ9{cr%kk a\ǽ2vg]3s?f,wlVgq/{xקDo{ouc< ~xj ΐ#k{(υD{B'f1 &D#N2&.Aİc \]B,~uN f;#s#z9En9TlMB CO~ϖں˜kn؎a/G1[ 3/˘'syܘxs}1s}= $u%&j\su-ƺ}C"ui1<ۊ]Ozm"~~ /p`^h2ĭL bJ̦[/-dmm/5A!&IyQۏxcM1~vlW/s^Bh/Q-6(kGs]>ceyy#v6W˹ji&mo{ٕs9Ng]c+/dĿ5=Sm" ?"~EhT!vl1i!f; KTZtc ~:qs{)mcs.6eNA5 E /dX(*Ά.9b->s ̵BKp"I٣2 d>/WBV=||9͠Falv.d{su f۲ n ((2x3'^3-F!+ &OG A>IWKK]X`SVKMҶ[N~B~hvTY3.W", |˜*FTR-0?v"N6q)+!>߹_kAvMOם\YWҹ̥ v_*<.d~u`*ʵ 3ݳ ߑme|sʘyF j­mQ(Ιk*9 ێ"\Ym?ܮȝh>˱eM=; h~tG}|?ZSd"jY#ΦS[<\k3wCjgb/Xqjwy6={l_b_^>kG--+84,PܮwP4+7@8BaQN4 -H[y'ۚg6'wYp"bXE! Vo({Nu3N$Yp d]_;7I YW͘_pX_p dI' B$Yp dI' Bmtgo^4K{tpՒ%ej"@hP!CiCjh5Bɂp0Enb<(qAh@`h(wrl߽eZ+Vg䓙gGĊ+֎<gñu;·T;s;gǝ{gÉն-m_&u]8NΞZmmN^_ /Ͷ~ 3ꪀMMe} YESGoFr ޽?nIy6o|~c{;vHO\oefĸ+c":qQP6׽u2w-,|/%?^<  >4S/ڶ&!3;,2li Wbo0ln@/90xPB}}8P"a!NI$PIP"KI`pcgAQ o~{]u׉tMg볻}J G?v_lp`xx\Ҍt?|l9yUBb Y>[BÇlLB ֧>ې\rfap͢)B* [K[>oX6!I!d=8*|Uʮ}yN}#\CdS!3H`) PaeOT"Ms) k/rqvc mgc+˅d*d]H(PQVA$+ĢV,B:,E[*דqhEB#ŵ>4>C|&G5XbkԄwUEc>]r{W^$Dq0}s$戅#l>)qsTlw=0ykZ~c67ߜ|۶{bcF9-}vc;QYMp}b>zZ]뢘:\wk5x (p_K0ojS~6ڞksk1dU!KᄅVv!$@UHD/xnĔ-dn;+MePOD--۱fiC ǂtRr˷AP`aE'pbSBgw–($ƱK~3ߨHlJ~}ԅ,'X,!īi, aa(dW+z(_$v㈞7t*[9__ZT}|K=}~}1Y9' 1snl/m;.i^Z?O]o<ĬȺΧ>y(0]__p:eosI_F8U`5!ˋR .:"WꂏAD(E&I0MQɉWu !d\~=2ZʄO>Ws&ˡ`/仕U\Q"J+XxU(d lEX[[+)p%? X*ܿ_ Ox"kev].9\|mM-*s3\{q=-(vim{d8O,;7/eUd*"|,ֱ~Ss8 Y|i~NxFݔu>VR2V3ian F95Ulo'N9\}3?2{͸aCqB!n q&h'-) 7ִb_ʳn3QN($Xk@Q(.+ P"Y"ZfcEW[~GB֡FV-H%&\NbeOW⃼+9oPtdM~Y- E~oe|]^s9szmGwnR7b֠_fk]Z4۪6ۘKs3Ũ9܌Ϭo|378T YEˡE*-X-(#<57ĎOLU}ZlGŹV},َR]ҢÅ IQkM? vֆXa*Ŕ! H?\(Xd_^ )ZoޖY"fQѶ Bbx+dÒ9@Ͻ*jkn*csAsmK;a91-uNqyX/ޣ5= y=cUX~{`z0G)-v]εvl'5Wh*k33cy*N{|Ոw˶؏=hƗovޯ-a;1fpkږoX:ا80 YA0ш'ad.\x} Y-,4DoXRiER`Tջ1'{,ܺ)u`W*/&Bc*^1uw6PBL\P0=/(g*4OH#Ɩo\1IΕO"nM3.Gx;DŽ3gOt.9 ҾG۶ۍKϟ d__qm⬝[B]Q:^uByX(.G1vv}]77Ήn86.7Vǩ}1"s#Q-qN~UOqkdؗrNv5G.)ռΆlgݮN~b <9ֹ j*dOWn&lDkXJ*Nliu]v,xQ owE9|k! i1X*R5ŵ@)6m aӈ˝E}s8sB! V1FW1;v𴶉5D,/HŹ9:'[6!d/A! $A! $A! $l;/.'v{/Oφ/Nc:wv.v^7Jli2r:Φsb8ylL/.V\ִ񶷍_ثpׄB[w?> <|W3Nm=;Qjm7x`Vl=&m}ڼ7N)#1>qѾ}o|\"ECѮۈJ\ΦB/mSpn=<g_J^=۵)~1<9?y^?^}hxd_mBq=2;6 Y[bYw6Xmk]'W87em+du0dEC F`IDT/"J b,gw$&A6~$6%&ťqX sVN _x) X^}]٘DOg}!(OYv b!Yy@SKaO_ J %I8[pbE|囂>!D~z;u9.*l~^"NLBVv E^k?>ok]_َ9U } ~ x>i?~=w|~v3qX\?oսkmfJ6 YAKAT -{akLA^mt}K=CkMLl8;s%7>_Y.לL$T!BQ/,yM@. b:t!ob*yRϿ|sG v-""aou흚AhL'_rAze+rl/ Y}s$E#lłz|b\6m{O?)uk?:lģاn -?SΘ14D08wb0q2\u^w!f_ !ڶ}_+X^˼$xڶ-b}ʼnIpBh+ ;AI`$H"]b 7~bJv|Ŧ2Ar'ٖX4cAtsu ی͠"X(U5 YAٳYp(Bnwx%oTSE0ebF~ YNrOWYB ,"WXf/6kQ,8VnPȿH 7v|LVE)uk?:l _l1uL?re$̷~z2ؘͩl?7Oqѿee/O Rgi8VrX'> Y^D$,p!R|"D,2M oJN0KhY%۝BpU&dx%3Y{aw%߭\⊑i8 +UT3Y!ٌd?\3nflEX[[+)r%? O|,MBOd~̮2.lܞDB`ڋ)s0E(ضIo)uk?:l7Z*{ݴ\yG9>67&Lk|=8_6wmwdBO3}VEb~ ys^c#k0`! `&CLxBH\ǖi@~s Yl3q}N[d; S>/JQ/=ni;ľg%ZgY;Êbhլky!ٌU4kF- ,[s-ZB Q:m!էCZL]?=-7ݓv 5f=lukk߭XA(MTa7w?$Jk7|kG-Ywmv,ksTcЈY4֘YƦֵ+06aUZO2ZSCXTէŶ{0[kǒ(,Յ -:1\X ™FYno`\mmXQ"Y 抳Y*Y1VbZoVB+8|ل!y1a[òX+rYeLz.ܼ|],mΧ}77o&BF f[H}{cbZ~tV}$qq~|/攵Һt~oa%?>@ϦoᚈYv/+:l9=B1e-w,lce{alB!+&$Å /qO51ˣH-K 12y\JB]Jwz3Du5[<(FŴPK1_"g*1uw6PAO EݶؖK{ݱe}Yi[ccg/y(ceٚB[ܨG>'} GNaU@-j+f#ٹa;#1JzM2!Zrod UI^,*vm,p ħ b+d]OD d8@ ؏=͡D,p@ ,p@ ,p@ ,.NӳغN{p_v$̳1Ue} YESGoFr ޽?nIy6o|~c{;vHO\oOq]'eڶ 5rAbe+16NS~ ؼ=<g_J^=۵)~1<9?y^?^}hxd_"]pb_4X,p͹=>{}ܺa[f>u[oG~hrַ_ogm`;Jѿ[kjqB!&8(&" A'Q! %ߋ(7u.D4v>7xgB FUmO5gۣx& sD+Uu"^pVW`-!C6&mQYW8y Qe3 d_0pv~(2 2ɖi~] 8?>|> /DZBOԶ3.dAX[ YC`", DQvϣ|SP^qv6#sl;&]U j(E ^wXί0lcG!+帮 = u6ғT = _ޏiJ'ɌBYIAȊOEaviZGߓ(DYEzhdfYO>Ο _`? .ݣk9ݽe(/ao?h]Tv_QPuqθAk~l7FBP{5<+ٞq\e,0~oԲmShxE9Ksp YqEƱSc=BT{MKA, Y*Z O{st5vYpIח Y<;Ū!KqHTOw"*+d]"q(¸.Zצ,xzY62oh_O>Ĺ}'>phk}h|,"kb]IMz{x ^ Y4&/xp $6⏺ǺL8q /{ߓ#>>>~+M3#ϵ0guDϙF0B3ϟ}wϣLĿ}m~ݠ-d]X+d<+DSkYP*zyzz 氪ʼnIpOf)$ F$x (>OL B _IȪD.|NJ]3Fٶl! !RC_k׷e.+Ǻ[yvE*n@çNjl>/|- A⾋.d9=]e Y7 ^McLHu!X7 GE^PG%kt/ ?/6FV!YqK%+g [ƊDQKWÂ">škrdT='SA]a-͢ɤ;\ֺT o=5Vv7Մ,')_|:| uNjO"$zC"No=J;Ʌ,4~=3f]Q,O2*{(5k?u-/BqgqyxlEX[[+)s%? O|,MBOd~u4/ޏ쩩Ǘϯb5ݓ'u!]]}&Eg cbe|#;=Wz2yj??}s[_}) zRF$H'_Ř'"'jk}[ݗ)=7*B M%ęO:',"PB2L\cd`\PbL..JBVݮƁ YV(:̵kkV-d߼Q:TJq1ܽ"rX8I0 ՜ڇ_K^3ϒ@o~—ƹŌ[[sY Vv K)oG+?sW6KIMSj$u.DQ.dۈ!KK&ZBVoL` "~Eh֚7|\ŧŶ-GSz"34’=>Պ0~E,vM'5 YUiTuhC!V:O]Y z͵p߼bayf\l{Cr}^ z{Xr|>Ǧ\`9p$ruRƺ\5I?GeP+|){/cB(7):IDATZ|lǼ1@!F\ZgeG<󱸔vk]}gݭwdpt iݕ(XV‰œUkӢ qX+kTM?Vz2.>86S؂BVL4 ه ^cD(j3bG /~vZXv)-?OmX-(BVv~Ɯm=c\jZbPa4  XdqpBM]tEB:@D67oP-cy*.d22K~yc4OOOj\Boi.%k"OPG7#zߎ:w=_;`a+=s=s)n!'EWUs,Q+߯k [x<9%<9uvYCf` W -o*$AE+fſ( jeK*=Q>"E(ZS$,%ZUM?Kq.^~.d6Y! *Xa6y៎鞬6$$Qfx>۰Ev[&"qpʲk1l0Au{?>>U.d d,fp0zFW '(CtFzĪZ[v,W/"PgVkYWi!+0_I?[I<#j`m dg.[ W\7"JL$O+l$[a!zFZ$)_U 5}/J\>vXY\AZbؗG_?:mōEį#ZH~=mBSA!蓷̗m- \I-/X'r1lZCY68B׌?=ݫ(R]?=* 3[24VU#႑_1n Y5 m=f1 RJOX/}ϋ7,տ"%?2mMMJbq@eAG|eЕ_g?ѧ8/MO6"/ D%U;S_`W?Q}j?@Y\#}a Y+Xťړ@%z%!+2 _<5!+Mz'T/4K$q.biqk! k7;f۾IJʂJ'?a Fu!|Dc y^ K4*v|]Bx4bkrBpph gñ^p1?{jTb\ȴC^a]!+y5/~bԮE:x?.[ph_W >_~QS'_`A=c~|- ,<*&̏Z{u-f.޹RƘbQ;w_ 6VǬ+-R%`})Fa}tk?/Kv_pV?;B^ajI4{Q(b@,ab}$d'^P< /1^/boIsmK8lj9|3(F"bbcǐdYkqrBN~1sҺވ|fBpɬDV,9/dr^\z2Z55zHHkGr&&x\KXI>N8Q7>ON%Xbو֎W/6f-!k~xCxuY{ uZP]^B4 弸mS_W`Q'_7;ND;d1 |]a*&aspqS`G!+CB<467W%]l?2o|f ddXܽvE{V,wBk-Z/Yp^r9E _D<^+!k:9jBaa 'z?kzO'96{Ţ*d܈9"n.vEۊ$UAe]:R(mWW`NXPv|q:r +䗅z 1˓Xo0h:ס .bƮ{d~H!Ş{n?#UCݤbX|, ("~SBVu?vXȬPmyĹ ʰPWL8 ״@åz" 8@ d 'ܓ+^\~@p@ȺQ_eY Y Y 9`!x8b8yl&Oφp˳꺋p¯` ,_O. d/7_w3ćNџLl+! YuBW?pG[¡U6B/V,ɋ[ӓK-*_Oiw~?{ʮ=7ý?N^(t JƺڦPߩ9cE7ľܯIc{6: {u$aVv9`^u<"$tl~"TB',b]!+$^ڣ0!p!ˏ"9;.Z~߆B+$Z{mGZș{} })M4,7gc|i>Ltr90ͭ~b !dod>H`.vDT-]$!=,)(D^()䂏TG/-L}5Ss1ؘyBVo]b.ŗ|m}wso~%ڈ~Y}#͍ϖm5pַ_Xpoo}@3ε%u0uаDMx-%|iX5D(Ӷ7'BִN,E\64s=E,Po_Mp4/ &DxM 6IʟbY\Z)+ޭ'! @#gG@0W[ O)yD۞Ġ 2Yv -2WjmK:nb>E[j N>sc! WY?oG_cْDʯ3ŃX"dMK4LD#S:LBV|j5]>gW%E2K\j}_Q[_g)е-H p@_T[_~'n}{*_5!+D$&m [?5v!,Fzm1+OBz⬟jo-k`@A_%H" X$ D6|BK?}Ħ-/\8k,e9"V$](BVT[';5']`l;/.'v{/Oφ/Nc:wvy:KW-J87^1gOm9ΦX1o2{pb5j{S\'ϣo8?bw桯^'Jb>XGȒOp7ijv}]!%r>c\*>N%~-f GITŠ,;%e0ci|! r!뭻 n7({0+?>m'ZwhljOʴm #:jtE6ȃJL/M,_DQ'bKɫo?/'Dz'OG+5 6p@VvgVeg–ō{}z}r,ګoz]S\",a7MaW}BkX%IB!Dɏ_*c_-y{j\$ۅ]siMB\DҾ9R_.c? BBZ07&" A'Q!!%ߋ(7u.D4v>7xgB FUmO5cOu*mZx) X^}]٘DOg} Ղ3,Yt޹B d!!~-S\",a#/>I/VW0/!+.,]/n|m#K,󕄗SWBLJBI9AB^YnQc"+eysÏO3:CgO_ J Lpll <77Oon32ǶiXy(ۋ"X Bluj sʶl?vy!8Na}=<hd?>͡ x[|, viZG]:H([HY Lb !XU \[\3R#Cb6~pmK);RY/܋˹Xcmڷ8]g~9|;٥j83u쵷QWbE넵545g|Fy,jڮ $0iQ)BdK_׭,8~o~s%j_B'`!g u!,{ B%[%1>➐_eB+ɷߓl\HskqneGsT{MKA, Y*ZinSЂN7r&;. Y2!ܟwX#di>ɞѽ {_Dm.0EŤPS,ֵQfvaK(ɇ8wϋ- i=jp]{&d=E,?}ɗl>MMlR !Y ϓUߘ1~+ǯfXU$f'B[I BeET#AxL\ Pݞi YkEJBV r9cŮ{Rl؇:\9އV+vbYbc5^2 Yp(YE01ލ46fQTvł](rR`垮,XD^&E,LwbF{Iۙ-옉ڙyVWxej>qw3Hn;ߗ{]^{98VPyx߻;O_U1soÞ^O)6b=E"H#bo2) [-zŁ$-)2ܹ02G~8,^鍩%diJc\*p\jګX@)PM:| uNjO"$zCt "No=J;Ʌ,4~=3f]Q,O2*2f PȊ%TR3 PFWg*>޲am$W$gFɏw~a!dD,_8H0iH>~g`a3eD~ypsWVEMsa!l1y scҽyՊ"Y;kXY4f*l"jV5 e#Y s-7oXPk%vawK:,W %slI /D˯d/Evk]Cݏ 77Α'joƙcYSKo_ĺ\uL&?1KuGڶ/䏷[7?]\kbW-klK[>iޗ|^r)N8wyO.XcKۋֵˉٻG1%__HO"Z,/x$M{{\)ͻ<1٥k̆_ڋ,w3_jA+sc!+5'9" CfxF…8' K!&XRiϓǥ$diۆ`$f Jղݷ1'd[X9?<ۋE v dqpBE]0q Y욈E=/(g*y >"([.^\q';W>!b7\.bkKۥuE˾l쥼=ݟQpߕoҮrfγ*I1itL^Gk _~|n5fVL1yet?Vs\sxM Di?Mߗ^~?X 1kW(߫^{0֦sA{=7'ƖkKۗ~}oz‹^o$dַ|Q!BVOŤ4L4)(S<>¼f"X\/E==~!+eZm /x-r!Kl|Ub`z" 8t|yi\ l! }a,+ F" YI M_ZzDi9b^gŚxGԑk-o3Dů[$%}mh!g>+3An)z^CB8U\ d$ d y=elR_5s;-!?9U'(IQL'a$[SJ+.ڔ9B֮LcůII,ՁunXt*`qX`BŸ|AL<+2I-3,MH%H.Y Y Y ep~q1\>{yz6\|q:[ɾ|8}OM)o.ӗr2|,ݛ2'Yp1<6ppjFq 'V=0{m]/ϗ툛;>xNl3򳘻~8Ӽzvwx6n=6pv{LynkS\߱Gb|}}r=)Ӷ-\ѕ "Eg,f{x{x/ϾzkSzbxr,$>~4Rc?Nɾh ?]of\kvւV\!f;QʽkJ5{m]=oV<[7 qsW8ۥ>ug\z:o?Kѧ{ îqe.޺t?㯚E<{o=gޅ,JUPRgc(d_Q)B䧷Scۉ4r^PE,!?Cr~9eۋHz?vX%ZLגyB7pJBƛIMJ[<C.| rb }xC)|0rfPfx:Mkyia܈{*>o)W珆O&j ({{}ҝg<͌Uaf~yƪޣu9x5J̜Mz#ڒϑZz)}^y{W˞mƗQɘr1b^3,Ǭ#]1-:ֺx?6Z\9stGCbl>*{S,~bf5֦}ƻ5vKg|5f|}<ֺYsnutu}߄K%r.HY|Zbw FlS.ܛodW{YVE_npbΑ aa[-/ic xD1Eyhci1w߂Du9HI KXy7Dg+"&{ӏy/g?s3Ԯ}/d;[qv'~w8jTpg{(qm1#TΜ?D[!jмU}k IwYj'c8;UvYd/_yd*g20䞦>[u!O8x<ﭳ[E峄{ XR=U6%-t#k/y|wU)^=B907܏E>N~} YFoƋo^@1x78p÷Lo߿|sGP UhKoԇWsM\orEMz{x ^ Y4&/HEcb3\nMo,ͬ 7]o8\-%esc|kQ]jG)f-u k6˳Wkyưu?_&#v&unjlubRV}/;_ gXZϡkƺܜٺ`=ky7^oY8;fn|O3eqs-[:cߟ<5 f3}L85*!`#^{ۦ^#sho{pyɿLhkU!K m%a'RIQI12q%@}LI[v]+Wj!+vܓe>ԩBȯ7!} Yco ցfr7~`ZMqa~o>Fġ"XѾ_BU~4˄b+E7<Ƚkљ#%­~܌d~~myc"U1C4۷ּ o-h|3ƕ\#bŒ!;v1xQn0_13ҵ &{SHSƊyOkYgd`ߋU徴ۓ#[cn/6}襰.zcvsx̩{Njjφ+bRڒ^ψWGrl {`aK!2ud,&Vs>7[ȌbRʹMN~8}?vgޏҽ33?{M /`8fzܓYQOi5!yB,njTcĪ06Qυ}Dž,e}o(߯l$[ufbF~sA!֜=Xnw/emw5l?hd빲WEXNౣcH*G*//N>Q Lvk a#8ɾRC*$|\V|̘ub#D?ʨ\rj+wXo<Mކݐ :C bp8T7 >޲amd (Q|fqmܿ_ Ox"kev]״cL(Qeu$Zk4Pa7G5J9eA}gEN>13*u`}h`]\ |O:|qo+}1s~D{<^i]ay.1ڄO]Kk]m=}&>|]&<[;r-rxٜ(UXC#Zc=%u:&x o2n]ּb=滕dž/f>dv|7f0}v 0eŤsX-S}ط)imzg޾5Z~1MR?+m=?oak/π?j\.ߍ,?<*۷󩰎 Nk}\1)%1bZz9ss\Gύya嚜cwۄUX_9˺qsƿS4-;|pn;U,/X qK3^yBDE[*kl3q}pʊ/o BQ3(Q, YZ,؛> <@3ꛎfe'?ܛ+Y"Zfc;jV-d߼Quc|_vz'^7Lx-:sxfx*fs}\Kè4ݯ}W5o>F{{+)u?fct;Sͅ9eAmmh_Wul̒巻%~[8+ܛ՘xk]Mc >TXJ~v/ufcT=+5M99g c4-N=Ϫ19tgLCJl)k~G[Kv?\N>vz)gFVukГ3%< z1bU?Q˅zۉR]殺܇487f@g|Vs,&̬ŬN?Wgۣr+B1XF@}}<|dh݅=<]V౳X*X-Ri_jFA'>-m?Yq>^_X7m]jM]b5ƶ!ٛu#\!op3ṃ;3-:L90}D. Bbx+dÒ96G<`}Е#}6Uy{Πޤ,F2g:ْoޘl3[:Xƹ/|*ޡbVEcl˭w&asA5~}9MP.Χg#}Z[l;ק+h1b|s65~R1pF;ς,W~6by.䂈5޻뒶|ud#m7flM˗s@#cXWZe{}sXQ?b?GK{/l,ߟ'}v5XdQz1QOξ77qd6,&vYۜo&ݓGETɮ}VbU3_帴GaI3ޭGUT,s߉hǺs8[l_(dD#}pE06N􉢆>4^])<壅lbjq 61ڶ!舵ق-dlg 3V_~ҋA!nXop>~< B /GТ%ŵ|Lo܍72:d$~㊉Ow}B,mws\yf߿qeoZ$vc"#a.昑&&s>IOUUb֢&"}y~&ٚr#[̱¯q|g Q<:/Ș?rjF J>{ތg vvy +{ֱw9^ˡ7gֽI͉Y6bX1[1|m՘H1w5,xU߻vqȮ8+dr,Gzm59:[5[[Zʽ)Ǫ{y6zsCI_)rm)f LH_^x u<^J9܎Y{s|,$gv̗f{"ݞǬ Xg;fjv9-g$+qVjbjaݯ f@"(>sBlJr_Ř1N9~yrݾ%'<Nڜ gp6XmEyc} YESGoFr ޽?nIy6o|~c{;vHO\oOq]']ĸ+k{ݤgbncK6|A0ŀC>RۮOɱɓJG;%m;@űaEBc̈́D9! -Y(dH'(pxĈIpT/"J B,gw$A6~ Y /'6"^Mp`xx\}^gLN3s? y!:(`EHz v>>dcf>}'PO)n.l d9p_-蟾B%("Hz|囂mFv\8l}{S(rbz!b9 BPee^h]K1k?AE4u4s'B0(]T珆O&j ]{e2/s"uF!uoOm;=vi]O{@쉹&=ww~kD7fRgS!K5 ,*n(-|h'<4m  L/y|wU)^E! ڶMq U c>JPdum*r~סo_O>Ĺ}'8"uhk}h|,bbuZk/b]IMz{x ^ Y4&/p` %:|a/kOE7̏Җ^1qQs;͎[9%{Aɦa?K5Δ>LCp*d9Vu-N$ F$x %q*|Ti YkEJBV r9n>V욹'%/U!kgI+W`~̇Wd2b/ A ܢq0c7*+ۢw~}ԅ,'X,!īi, aaY/a/kOS_of۴KXu7V蚎?P$:8E_!.Šv9Hy YNPD*//>Q Lv">, bKo# !d\.BGoŧWUY2Vq1vW'CE`V5ş׺yx!KxV%Zy"Bj湍\[,niY/'k_,W\lE̵k3 ]:m!էC,;c5k_+‰4,cŌi_~rѩ{ "|1ˣבBq.Tm) g3+XȪ Za.e􉩪Om/':}l8*OE QlX#xa]SI}kyV]16~CA:+Βb!}Qzz]+ky[z"3 -Bbx+dÒ9@B*aGqz짅,:Y.w*'SI-3-(|DNs}0w?|{\(){@+P F<$pŋl?c F5Wj%o=+ ɿK15mmCk]^qu Y Y pe ಁp QBOo0 ,kG3oɭ[Wnk~%pn}c=™sk㚷i>Fv1˶3'Fz֤g5|LFpSpuxׇ?=پ~$kP^TڶE~8/L 6S~]TԸxcHcn{5n{}qU_G?*[Å,:Y]! ! bx}blt8}pj 9N_m쏓:}Ο=ڐѿeBczPM/S]pmI'OS+Gc-~{kg(ih#Ņ1fQB#*ZO.bo\Ȋ6@6iV&AjÅx^^8zisBg} YESGoFr ޽?nIy6o|~c{;vHO\oOq]']ĸ|)0̈mBp6޵0zx{x/ϾzkSzbxr,$>~4Rc?Nɾhې d+DɷH)7XrgpbX!P %d5sjWG4cc[J,l^bqD~;z9O)E YQLoQ~n+ :'I% qRKyRyU"q)5^/.9XB!&8A!&* B 7v{!qWq]wHdf>;'pc $0r". 5X7De[+UTDA| Y>[BÇlLB ֧>|8g!}>@;(}cE"I K\j9/U!íͯ$XZ|FcO@ZWE^!kx~rB]=W 2PRgc ^ yo +Nfdm'˱z&߷Et >ί0}v{,!ku= u6a:y⊃)+R'G|dTOE Olw~< }Ih[=Y^~(Y Lb !XUkE^ Y.ObO9ee<˄/6i U_)bվﳹ5&!1{9ԽWg͘*>'{Ŷblo=[X<+bb<ږ{׌;_ZO;f< 1y$Bˇ.ZYl)dW?\ԿZFV?0Gv+Ĉ>񩚞v` P[GM$?0B}X*~{/{o!dL YcL c[yf>VPIpGh+ ;7E I(6,qȄnB_$r9n>V욹'%/YM~+#7ogCkyٳ VQQ>#:8^f~o_m7(hKR`垮,XD^&mŴ_~~пeuV{]˳Nvd>b }HPߛ=m{ o #5[YĬ+ϊ5)1Od~p}d6_z 3,I! Y?8[_~+8kv)IT* $ YP" 5@ [8]jm\{$~q~ޮť Cl!dD.'/>/'d9Ziy]_,b5! R 4dWꂏ?D(E&/]MQɉWv !d\|AH"c@ W~5Ĭ] ^?R"MsOMMCR~*>kAbk/i[TYSȲE֛(}{EPDߟ_4O-:PU#YY^0UsB ǖi@~s Yl3q}QamBp㩬FAEB^45 VQ,W\(͵k"+N[ryn!~#Ǐ)~;ߴQi[]JrsF82!oR5/F7{asO"3,b6˯ _O$7Us| >&rZ#fmЄ]y;Nѷ_ʢ5~SG3㣙%p|$ ICH(/-9" '+ʼn@d=S+|{&fĹs(8Ic$$dя@THwuFs'u_>.R뀐v,"CTZZQFxZkojKvVz3!EǒQkoNjstRX*hf2+b. ɆU ت [+Pk[AHo %_6Cr}^ z{Xr|>&xJ$p{f&a٦'mw|u>L.䁱4xt2:'o"ÿ^\?FSEnY<+bu6(p?Vl<^7b|rxj̤m\\8D! ?~~I'ߏk巳ZET.9Bd a )ɷї)$6'fHAen[U c'C,IףRDzOjk >},1(k|no#g9u!V{n,`O>\|"'OGՃZxY h4 $%߹r_8 Y{O#r_b{e`< 0WhI?.byPn_9GX <,Yͩ,c`*tsŊ zBV/?_gWL?G.**Ro =㼽_)X%iqEQ*ܯ!d/׿c Ĉ7skk 1!n->F;\䩢Eϯ]AOwN,׋!h\+żfkY@OA,p=鮟")~! Q=ݯ=pgED"ߧr\Lѐ_qnn4\i I*VkB۳EüzrH :5/W-4VQα>Z{|Hm 7:@pn0kYk muǪ /HƐ"A~˲ =bJ,.#decCdpkCME?QUd*_i^&hlL>cw\;-bͮ5㓿O9Y׈eB؊?=ݫXP ?=KH@} s! L!0D KL1[77|"6o//g˿ii!+g6zl\7G?Q+HNZ5VB5D]lu \V$q.Y@)6EBz$>XBKQm%z91 1LV O(F/|1gbd0G`/|~w|<}bg& u+ٞw6ؐy៏fj_Tp(sz,~"?/8%EPdR*†g sټRĊy?:NZj :(VJ1冋Sَ[1}u d?mkYbH…(D8-8-xI6Iȉ%D6?&V>n.<'drWZ, d(N[ gW)JH&NOɅ,짧X@A! bx}blt8}pjsw_=Umv 'pl[+ٜU/u~'bkr!뭻 n7({0+?>m'ZwhljOq/>oS`/W9*M,_ܪ|x{x/ϾzkSzbxr,$>~4Rc?Nɾh%9;E!+-,."XuN+,d .GOvN! ןB #L qB.MU1 Ad) n 1HBȽM/| :)̲|}vwOO)dٽAH`D]ī . k|oAyhMWK,!!/uKxIhA}-n$n!dQ?~y6I˛'d]9cVjA|RJ Lpl $BV>MA{۩ی̱Dw9Vq^/qb۟NUn/[~%dp8y⊙_EosB{"P i,"%bOlw~< }ܕ'F2ڪxnS?>ĪB֓'ؖ\˯c~V(xaxvpgF1e .d>m$S;BV<8-ˤh6xwvgxl9+,ִq 3bS!K5 ,Q$O6-|t!dl,d˄,sߝbUWQb¥M.Cz,GT!+C12,uA,*&x1z 7]$|sG -_X]UsMjg;5!/x/dјOd u5 &<ض-9鬒ҘBK>FƳ.d9Jt鏑AL1~u>zmzC>>.v?>;wU,'T0`3JŽō$@GJ K\!2ᣛWo 4\fI K~;{#>: H*/Yec5f/@EQ2 Vx͍z6fQXVWT.d9=]e Y7 ^McL{$ǼG~K@1ٶ-s [hi]:s%"hg8&;FȖupbb%&-(=׶oe!'Մ,/4H`0U^\ >^}@t7E%'r_Y%~4Nr!K @!ߊ3-\kXZ:Y 8icVBQY|ϋfᓏlEX[[+3 nGWf/6,Ó5";Z W~hAW \7ٶ-5цQӼG3G38GJkWc$aBj 4zY'랍m[Bug! `þjNBCxB!>2mou!kmc|&12q# UV|y{ ,Y~Q^p}\]}< Ƭ F*Ⱥr^e+b^k6 \!fouBͫOu YCDy?5۶Es am"GcNm9nls4΅1d _M\kMm^_ !.:X#nί||;^D+ kcVI3/Ca7+[EZ"+:am B^ Y|MqPA\>(;JnR VN70sv^Xj]1vY*fn ESǴ4J\:[ht.u)lroG!k{sKۋγ YA0шld.>x}|rᣁA-,4WKĒBLM[~<.%!K6#6[PNJ9!t.yU T ETLTW0cLF6&7Ά*œ+ %ŵ|LE7o ru+&>߹r_8 Yݽra.kW6==> m)UD?a屲MBʹφS!r~}Jhc~]>z778s2"}nٖ~<_ <|W3Nm=;Qjm7x`Vl=&m}ڼ7N)#1>qѾ>u9Ԟt^}ʧ4^3C/׽勜(_ })ymXIx}hx~ᝒ}4;͂ 29MDDiD? YB XV^> dgEC Fx!@LBYJ;C RroxS8+킸Nk 2l_R|vox81G~9gj,wbE2*Zu]B:x) X^}]٘DOg}FBVxgyBֶs"8^Ix)PnW 2PRc(^ yo +Nfdm'˱z&߷7'" E^f.3KJ7D@ X^<';NZ>b@@=HY^~!I!d=8*|mɅ+)P|"#oY΢ _բ]((L׮O4^ YlL{lUƏϒh6xwvgxls25i/?Ĭ?܇oU{_-M|5\Lqi6l*di1ý& }RH %ቜqnS G7BζBVL2)Vx,&\jD2dWȢyx_?=5g E/*0bd,X–um*`rݬס yS&̇8wϋ-񱰧j[yqqԄwUEc>{^֯ OPew0mΑOgƜ/ca<⁳)F}1~u>zmzC>>.v?>;wsIüBޛyƩ-L1򵈽$`b0W/؅U,'T0i3JŽ$@PE/lX ~B_$r9n>V욹'%/YM~+qT +vQiua,1/"(Jf5ix7|_FPe[ZCa.o+t%d"x5e2! PLg//TO`5۶{νa9-KgqDL:dgٲSX!H3pV=׶opp{ϑ3{in[ko{Dkns]}fm&dyA T2J]d7)*95˵yBVv'DonjkY형UpNV.L"^Vų g@y,|kK~xE|f\m(l,MBOd~u+{'rݸf۶~4HuN5ϸkWc$aBj L :.cݳ6um^<f$ʶ5/D5wnƯ̊x*B }՜+!Jhe B2L\cd`4G2AX~P5v`g+/; K׌bU,[s-ZD@9V-d߼Q:4(_kO^6Qum0ⶬB=QӼGg#sa: ;=}2='/VqXlJFݶF;=f$ʶk/ݾ)sx[r]}ccX*X-Ri_jFi!-}bb=BXI錫:2PKEǒQk/Kｕ _3xl&g4f*nV8s56{0}DRM6Bbx+dÒ96'_.KnReyo`za|=wԺbؤ{W^w~k@̍a~ꘖ[^g1/ zݻm%%V\c@f43f~~|y0 ϝۊ{ND,Uv_L<6&7Ά*4"()baJKds3k) XnނO#ǖo\1IΕO" c<(46[m]ƽG-jH9Lz77nm9\m>lK_c{-)-t˖%8{/I6K?sv}~8Ѹvmʝ-mYBC8,r,^/@! i1XJO짐0Y,jAX:@ ,p@ ,p@ ,p@yt8.^<=.8dq>v8~΃͠9R:+/پcMz|zOs9x~>s19r!뭻 n7({0+?>m'ZwhljOq/>oS@Wyz.mɦB/=׀Ó1|]Óc'珆wJEۖb_bVNlax8b,dg^qyYqEuIk[vdEC FxĆIDDpT/"J 隢,gw$A6~,~9gj,wbEЄw.ܗqe w6k!!/uKxIhA}gk@YFA, ,Ղ+!TYBIA-hN>MABv6#sl;.]U 6 '" E3LU1o* k_b( * ?+P1݇9tlGx!'"y:Mkyhm 'qQ~umQ@.S?>ĪB֓'xgΑ S?`O-6~pDmЩl-ֽv=U#U{Mvy`Vcy.dU3ca{L_4wEv}dw\aΐ}/mH8su+_˩uۑJ~<3d}TJ<'z1&+sH?}m~62<.{((~a̷,o#4ve`nǺnYy1.ӏ{`#p~%?ƺ)YbV~4V IXOROVy.0xCYdjb|eaq8RdJ;Ʌ,-$+>f p%_w_e?y7OhA+ T+U14բYgq oيZ+W$W(T '>oU!dD,_.d{mJ/G1rkyW싼-u")ӵ*7/97T;(gov #ekKz?dUg=f 9<3JNZ.{]Y)g> OQ,ždq>%d;cP^/ )/:j{uHuF>X>{-}0X/͓q; Cq!n q& M>L[*C]Zfk9 UV|y{ $Y~Q^p}->ڛcOiOXuͦqgC}|gYX,d9ԑ/bZSCXTէŶ{0[뎫:2PKEǒQk vߥ[+cd"Mh @IYa֌"RV|"u67\ *-Mm-:,W %slH#ntR^|7渚Kτr-\)O!F*i'c~W-Dҽ }H#fb?bd%s[̣vJV}+{~/3E9|SPEˏb\u{i:':"{i|*c#υ5y#zNEgBD!3>xlWFܾW]s򘭰K,0d6G6ˍ|?Ξ򹍘 ÿue39?֙֗OsYY Y Y Y\6gñu;·T;8~yٳvUeѺ:+/c-d8 } YESGoFr ޽?nIy6o|~c{;vHO\oOq]']ĸ|) ƫ<.kڒM,_nmx{x/ϾzkSzbxr,$>~4Rc?Nɾhy_lw Yy<"kp儬?$}^@ ,=:B!&8#&* A'#,%ƍ!^7EvA\]'5YI|)Ńl>7<# H!h4,]g\[c)WKr߳_  Y>[BÇlLB ֧>`x1.(\u8iGA8b$B{OkՂ+!TYBIADЂDuyo _J$䧷Sc pr^MƉ}lu*sg-T,sc@dz4

aAOeWG_\/VPĦIpBh+ ;7E I(d,q>eF[6~u$r9cŮ{RBz/lTH N[,ΊB./fE Hǻqpc7**nJq]zԅ,'X,!īi, a`x3W4Jh?ay lBKU>J0χY\ske] &?r7]Z48=M6 Y9a ,D[_#í?]H䡯ƿ}Mno-=}c(En~Ol8nq- &dyA 2J]dk)*95*$4_H13•|a:];N,Ҿ,)F=`X["-5nGϛŵICr~1<9^,ïY.PWqOԅ,B?E]>{-Ѣ$4}6.J>ds8*FOۤ=k-3Y9a ,O=58𿎾x=G_~}DHUrvFxll;kI枋yZĥ@tk[7e O5!ˏ^ׇU,/ 9W3Adle"PB2L\cd`\!-|-Y6qyjbgWT_qͲ1b5K %hi Y7>}-d7( YR!кF{^`n&簱Cۺ",~9Oh<}ngz._영Á aoNBد6ۭ Zd \a!L+_}}ӣPOkXȲh6QP TDWzƊ$1K _K,zuO,?b! ŧH~%eر􉩪Oma':}l8O4EǒQk vWؗ~X,B.Y#0 kFYYdkVs-7o+[OP.Q*oG^ Y|MqrFI'1ˣBM$.ԴRmCk턬MBWKlnV{]XZ qY-qpB?]sžBa; D67o-c70߇o\1IΕO" cqqow ŘD{/ZW~¶WG!y+˘=)cm350w7 _5/kTq! ;>IpBִ &i)qUȲBד~# UȞЅҵXp#(mU" _}k( _Rq8;q$,5JR_m5.Bǯ] [@n;RBQ[7 ē>i`ӯ;<H+dyb۷b!|],?&iF>\O d5wMWA`W+?}smu,'’*d{ɷ],'E? JvYMzR0R%1iKJO;@ˏ5,OX널B(}G'~wU'c]W|*i A$ҿ+#~i|6IB-.yB7Y*`㵟}{mYI01DYDV{eEs)2+.MB7`oG_sOkY}=e49)5hN=OW駙Z\kE]Bw&~O믎%d[&d_=bPIۢUspww9N ,AC#YVL!l~I+dQx˜%2iz,o|[$i7+puep~q1\>{yz6\|q:[ɾ|8}Oj38~yɥ9o9yϙhĵ=u]W.}=em$0=b!+>עүm"dv/%y p5Y.duw8~wu;C~mѳvwoŶgvާS\CI1| *˚dS!Rx{x/ϾzkSzbxr,$>~4Rc?Nɾh{%s\~1J0SȊ8Qg>I+e!+B:B'uVV^!K2v}ڿZ,Hab0rh0 ;Rnb{Û_^iuu"]Se;pėR<{я9 >. k|oA~Rx55z h;o6^pVW`-!C6&mQY7ܽr>OpFep/ꉙ#Zˆ/WM}/dոx=sNjz ! DQ*<5DOy1f}1 2ܭz,[4㾽5=Z’jT>$ֺO+JUPR bE|囂 mFv\8l}{Sqbz!b9 \e62U!Kc$#6Pl8 O <Rݓ dzF6JPm,>u4s˧SџQ~u鍊$V|?UL9SGVoWat} ;3c-ih}k|8'T8_Қ;yx\..m{]*.ٸ v}FW!_*ro&^ƌ\6d''oZw{_xO6X#GuĬ$ԟĊE PID–GȪh=g Q:~ pTb{MKA,q%!=Y9Bg5pXUrB&1 $ZH/|$ċ.^Em!kw\aE BbfI K~;{#>>2tun+U3 YA,/eW]@ m#fᐏw\K~3ߨlbbF~ YNrOWYB ,"WXf/ޘrY(n ]/wT ~glmg6yLݿFfmזֺz:I\ P_9qq~o*?& ~}Py94Ӝc{=BT>f{ۍe}M‰=OQ]7H vXĮBfE) L|ʋ+uNjO"]ߗC"Noo-!d\|AH"c@ W~Uw>:+.R"Y@K}͂`h |e+b+)Zr%? O|,MBOd~uH~ѿ}~9m9O*Bu2uq=_oYzv{udinlV X;c>2sBObg6rq'}?}ucskNql@>6l1߄ 9bK?';#wm{WnĬe;XrA#-Ri_jFi!v,}bb=BXɦθj[ίMuѱwqs~ck@f̂`h抁  YlXBU0Z1\ Q$\&tYϋU~K ,g.DNpR˺(dD9}6_I(+I{kj\쌸NϊDfQϳrokXr~Zَ\p_p1<čB1ыkA AcJ9퇋aGi/y>s7C>6Uٳ}(ʉ-mհ fP F<"p‹`l?  KEh!& lbjq) Yڶ!舵قvBV ٦s!+%6l.U,g*5f|pB&]drB0D67o--}!-]/.߸b+uc{Ex1K.xnQhl9˺R~,!G ]iǓ|,cN69 }S|rZXhٓ<=wOa|okx$M b,n硎E/Oj?XGϭ_ uos,ouZKwvPL[Fuŵ ;C;+'ZW6j*dOWL@ "],WY@OAP,ӵdD,,Y+Wk,p@ ,p@ ,p@*B˿<}0Wu㿾d88N߳%N^_ w&_/6%p~AѧFnppl2ngÉնlpE.C̿ytquV,o؞W.7=9{wOǿ뇋;l쯯|4k/ÿwh[ʖw 1m:257W܏mcvʞgdVCZ&|U;v`W C% aox&(? w+.o6CdBg@ 7'|~?=9ui3m;۬3j}j"{<.O{f4Pk"ubm̷rw]Vy݈Ye]"^#~rl zcC_fTh>$ ;uw8~wU?o=#o>fm#oh'Rkkcyվc܎1qĸ}:7؞:ᬳΫ"\Ý; xnP!x-kY߫X[P<vy_gsPsD?x<s.X_Yd]BV}"@5W lϭ"F '!kd~\2? Bڄ?w֍`fs!˾:(ƃ>h͏O:PaNC Z?q_~@rw>OVBV+$N0,aD'bߒ;a(½qEJT ] X"Kz3ekdѺj(Mk2@15bV[1-t Ǜ{3]qfM^;(bB}}GBjҘ[ㄒ:Kxgۭόh \v&h FO|^6c|V,A?C1gZ]ҽsC9w& Y5j\c!kNk5v\TdR Y?϶^~ߙ+7BVrkܛ uuM0ްøy(VF4)0TἋtLpSϞç'{ LW?B+ #w~-Q62?oZ_N(* (h]EB izD,.(DLZ1kYo Wxy:pT=Fy{CmyN yǃ!r1qguߟ]=7Qßr/FE!d.&yW7<$x;Dȗ@=t%ئ00!^7w/>qM^kōCvQq1爢Ngl+/3h//Oz}9g#yn ? ՙ}Y<3Y{ksl?mZc=Dq"5J?7#'|_MV})몜)4ٯfۏ?oG#|A_Ŭ7|#=̯,}s̋Kk,NOz/Lm&gbYd6݇dLh|ypu5qZcFvzоS18lA쫙Z>{ѿٰ]/j#;~9XGq_*k[~86* 'MD5y_cޗ~tǣ˔ja>]QvCZC qn SUߨ|ת]kR{ےj7vHNS/kW1l1/kYbsUrn,frK1nĬ܏l]2~(퇋3zϳ?3 ?lދx|zOn|F2T>8OXſ+gÞPZ<w+fJ4Gl|):\AatM][FxL\Lt'cR.}pΝLt}YY9ϲ$;؇Ba5`]?HCyϜMguǀ2qNGy?csk>d6KEcn+3WXY}I5)03P g gu\O yUaP|6(~{?<-YL?{3>?Yݿu,u|g)t^WG-YЌu6ƪ|2;cYl<ᇓ~N0cSyM@HBں^ mNpD7[M[CVX+ YSq:UˊkK,P fo kl-uU#_d۷=ɶ̯PD2?W\gHH?0w_&a0ߘ{pKt}p6 k}#fo~x?y.B`퉱Nbz"D6!d.%Èe[ W\V|z-\ZүN~{~g|]gzϰθy&!K I1/~l76u(v/ٗ;@WY,5?= vXgy~d? dK[vY:K}_ykGum?mBtMײJ_z_2{hMLv\3]_d^/lm1my(vw>6ZbP^?⚩o~_{/{S&_{Nƈ((X?n1G#G0C~D(g,nʝԼ/ך;#Z }M;k~/iZ1q(C.ZZVWkbFkZǬMc]޽gڪ[egfޟg ܟdx=0[nك|]9do\+~kQi^wcb?q_~7cRnsm/d=h> ɖ?Lt=ͣ9g{+;_Os:}8DKjUfH Me3}I@rA¾EP7GW'F+Psh]p~}ЅkAcOq7bcV_+>ͭ}*t`DV z7?? d6 -gLe >SIreck YSkBSinz[dئ}lvٮ' ֈR8K=3w}~xv|Pδbna{^:j{$)GZI݇G~]LƸx9o9-7ָz9RgQ'Z^N ]luC}_w|$ ~ub#9fWQEs7Y9fNu4 svKZznFｚ 5m̵7 ק|ow{!Duy1t=?h|[84՛=7N/s@|G5OofH C7d!bDE֦W_fם}l*<68!G2q4]We?a;}ޞKyj1~Ĭox<ׇk"!G/OY[))ޡxë7[#X{o&lyGJoć G54\$G I%>)cP;hmn" T6g+c[V{9Y%^g=,3~^egxlbRPν~v^1{8n?s-:9밑,c"rU_~6sc6z ߿ GiۍϬ?}ǫ6b+7l?G/ 5vKyb&Nuk]Nm2~nb-|6u6ee1#*Ě6+N=mC9Q =Ba5gܱF"H5̶h_mѽj=㵧J2|\\|,so,f\YgeGvdYC8~{oJCg+{_?V}hrlrz@T" \Sg2y2{mӱ&r:oߵ|.`0ГFpu)dK 8 A#W)Wwz[@\#`t.zoޙ~Uq׼B%rIE}~1ےK؏K !k)@C>WLn UGqB΄E,**'d?MWXo}?,Kۏk)d]y=Ts Yӣ7 L:5 g{VD;/=5Ԉ_<=.8Cƭl86bM>N%z8".W7,[9|׮WsΓ~ ugw6:K=yZ;Ë/7g4wKa~Ln͋_ݻ 7g+ؑxV;zM7(ɾ~$C{:q) 풿k ޞE#Y4q[Z80ó !X9:΂Ђu$vG ,ce`d5ש"2*"22~0weVfdddVV\S32t%VH!+|xpڨBa,"y!~x"O>en߇?GS̗2Hj(#IeMMTݙ/*slH}1UƸ56a[}.mGL6*Ry_%(YC\/ _ZW%9$9p퟇"o2}wd}#o!F[66O<޲mTx듧|ˌiŽ߶$]koY&g"`mW6dx[S$gH{j}#r5/2Treple82U~x)sP鬳"BhV=nw3VnGFd,nY;+D6`N6P߫d8fJcX#d]<~UH!std۫(NX9J}cX|<$v5{X쾣9x>΢];S̚8\' H㡍gTOgïBDRBVPmvqM7wGb;Y_ ~OB)&h,»6qO?_ɶT$@,Px2h_ wN@ą-/^վ߇'_F66D츦i1*0meXPFZ>=-hmhG<-l l}0"0oALE}ɣPwc?դѷO. ~O|d'.jdOh8ƥBVF/#uA{}a ϙ9!2-{Zp/_Bhv|wc)ؽEjY͹F½.w8kr]OwD;W!+sޚ6 kNϕ =ߏn)ۑm;a|}ubV?;ĝ Y}]o ϛ޵3Ŭu$EAkg`{1,l<0N""wz/&XFa57PFfmk+%+=S#$+E"sӸ*J96)cP>jg* 8f˗͵y8`*Hqfzݩ}X8tnOQq܎7Q;|QYo&\喺^fY&$SOl) t'dw)q;BȢMv{l4 ֵ&vI~qʗ*ߓEc,ZW/0 aug+:v?Fm ]ɽ96}HSYiI}> c{)r:Yɛ[wMߴgVGxR0N۞Fg^cZ"ŸWFH\QųF9d0qU$:ؓ~r,P)]o+kT~ӽf 3|8}6~vdyvG_|ٕܶX:~NKӿrE)LlgqKɌ{ϘgtS>-gq5)wq?:{ ͱ r?+g4]ٞS"ilSI]+}^;Wsoynў}VѢ2?՝ۧX"ϗl,gG9cY_rnƾj<c䳮83kSʨy|6e*1UEXH^*{Bn^3`#}ʹm}=q"!+=t:=\#7Kwz Yc[+V Ya#)-4u' p1/Ŝ/)`ɓsb!&T@% [i9_3mO>+ql;qicDƇ#=q8m|~xC(Eԗ5<[ ۷[Y zT0-7 +oJNU/yzYַ/ [EݔږŒl$֘W*m/f|ݍضB_-~ĸK}~jyj,Y`Gni'p%U(K&j_d|U<pQ<r\7E||jeKm>q]z3YVu)$3¶ſ^)֬ՈvQ9D6jU%|oJWI6#'8ܷGf3R~&뾭B_ 8ĸ*ޘY?]g'[85mV3Gik~ m|PiN!>9N4|V%y?m*dMRB)d⑪eAza2}'"xS9SltLvࠉlaD 6 QЙj%QK XO ba&JdIy,>&=~+V4|>BXW9K:|>gd 8tG+΄m\6qC>H'7LCHgXZA@v;me;>`՗mXĦYLQ)F,XQ&D~aI%1 =Mm%譱}r5zJu{H>径ɿ'Xkb_~76yn ;&FK\ށx.{_s}wKNWg}:,ءڪ㌓q,iݍk?iCf=_-}[_;85yJy$R?1^{5-Ο/ZmΧ>4ɧ%uʌܧ?Ʊ\Y+[Om|~Qg]ж;댪߰>ّ˰PRT,W2 KVǖ;bBm|e4|րe{wҾTB{\nt{P^[ٻl?{~!ϡ8lW9ϑ?jLYiwχǿ,,w ev|Q]5S1I[<n igT׌Q U}R[®|W)gWBڬqߥZ>"B4N#dUtHKXU2@jB BVں^`Ʌ%Є&|op\s91%AV!vfo51{Ӹ1'_d[sۡ\ϑwMevM܍t85%qVDyI1nyH>6PGiIώ|Ly}8yi_,HО/!po+Lۖ4?aŒw`ڶ•݇^ WgsxE=R%bW]$s|ὔݟkoPk'Ds3F~Yv~/ߘko<҆<SڑAy~-4N/[xxSI(Hz3PT2c@_L)ƈڤҽQ683宿1#muzr)󑭋4'(G/9Yn7Cw߲=k}S"]]UBϽ,}LC)bm#c.sYjٝk|= _9b =p^~B{w'|{{udW}j\ ~YF1^{}V:>1~Gf%76rpFl+D[ gn H>ؐzbI7҇giCc>/(E>82rZ(_lo9Ӹ{6~8T`A?B֢=5P/V Y볧{A'^3\T@liIn'B5& BnNJd'E$x71y4O~tųUڝl}6r\žkcymvSk<ŦԵ!X#Q\~% R߲='4VvDZX v隽O^ƫm}d蛘ڥKg3]aޙs&yDymuÞQ+.7M;me}neo}*i/;i|>dU.0Y"˝8 ens_ű7sz});Kcoo(3Ov*r^{|,XY*vяhgOO YןM F8,nؖS?Am}eX#;Q`и^ [D */ d_KB 5& $B:@ ZcG1I!Ȓ՘$&VZ9aY}\kM-F3occWs"v]Ř鮏v!iw 'g9.sy͵elWaS֚ZFiُ݆8sݪ-+'R—Ke?pmĻ`'1SC46 o_MICQbϲ{m>"'>k}P=q`hl{\+Fz/{{{S}{.xsh}!=lD m!bT9cOBzpə߶b+>O_ ؎e}>>J׭YgϖX+#.o9zuR</dɃT|öBVvdSIm>4BE4@/`=3e>JhuKzIQf_j_vqg]LHdOY8 U>3ww*AsJ/q$O'ڮղ-6y祇SD^es`WayѶyQc凜,v2նw,?PEQ"E((_,$8_EJID[&ڝۊJٶde˹Wm;Be$f:xY<ڽ2 ZT|lŃ*B+;7OvL9v{_m ڷϘwKnQu#5iC~/]sx^"Ͷ8,cu~S5?6ױdbE3|X๘q.ǡm(ׇ߄gڣ#}yke26՞ݹ~>5jQwѾk%Ϻv>R[Oi^?{漋 ȼB-wy 6CkȲ^+j&{_ʟ,NG->:R=%I*ؒ{Zk{${=s{.'m~ֿ ijYڦs14lo27.='6>˞r|H^بǘ.R99M)}/]3b[fYgZun(YmNBY~F[BO̓)L}\Ngnc>,[֖IX"[gU8^նW Y{HFi<w\Gv'Wt{F'BWۯsq㾱e>q}v,t;xg;=΃?7 Y[#!+m"Iu:q̀3="{0ɟ[Xc>V! k83,๳y5}|'Q٘,l}8CxDjɰ"vOc%im"MhZgEY<HȲb8s"0!@`! Y,Y,Y5nW_{^v3\zש}p̔p0/_q^k)KQ}@p|mݻ\ 1w''gOyvJuٓdSO\cIq9%ۅw.o<|}[(*[??{'={y듧|k'~/IMdo>_'ۢؾ\g8CA?|I׈}_xG*x:J?=x RRφq-6eݻh1MȪ|;|_~>}_rX!+qxsi|qYx d}aE"X8A&A# 0B'& %] >AD'l{uJSYv K%Pϟ v M]|1ע4L*ϽJYǽ$PX,B> F$8}67W/8"(JjIn0EWG1VV䎸W&c(&rfu9잒Jێ#+D̗O-QϿb6QKgOE®(Xܣx-M^ϓ4rTV p݄,گSl&e6}-qe{Hh]K,:jY7yxE"ھBk"m}5М~:6q,kmS?ęG{}D_Qx"ȯ?ksӺoR3lŌk18Zc>ĝ}'P! !X J" <7  AB Y%*P3\{vlHtEHa> ɖH7i4)&pKʤL=ِNs"(&bٜ:tJ<הL6K`W!!2L $$u_cIpCu}ַm*Ŕ*Dkſde#8` G/&=L9I~$plt}2$fm42mD'jNIP_[%J Yt}pOP$d l=!6DC3yQ֣?u.%P^߹YbEG mh>i^my$">HHȒq9:^ 5k>>1qe{\6͇mO<}xvp,49gB_i@zK >v5A&A}Xqt͵|Wrk[oSo|,G$DRK ,"xNRK.rR-OpU׼Xf d$0]?ciHL_\%$LRWkg"ߟNBGB!k=q$<(-wLx}v+[Z1/K6j~M&ne VGr6|w)ǷΦsG/隵ɻiuwxL]u-Fsm/Nsmu"3Ewٮ_a2Vm3Nfc왏x\fπf!W.VO ~5wĎ_LUmvR"ӯm?]eS*K_mV= -fXHq@Y"E'i)6i&,ao;i[hZ$$Sɖ=aWk͂QȚ%²`52%UŶ^MugBViu&lczv9y;㮟 <)miB.ŵ>o%S}>$sV}U'Ԅ蛕O(^1zp#s[_94CP 3[b6֝㮾ohB[k/8}_|4܎]|Vhp۞`(^XРH"`>U'‹#vڝvɧn[/v.H\k5*{ 3%5-p+?!9"Q 9 S ImnE󃟝/rA+K2_-"+_CO8>Lmd U pmHIu$悈Bi _ (ǯ4Ar"}B@:9?7 |xP_~H\RHvｓ =muڼSO۵Oۓ_x엏$ڦmN,{|P#60C^0_o4?|.GbRs|c}Uv$A^<"'^^'wGnydR'8uHVe\\96^ ʮ/uIKuqO}~ _mgݷ'nЈ:>$ $BOtwUɘg/;Fޥss^yp !4ڮW2qڿsT3 BVZ ,o,]&\+sӜiO3Ͻ³KFB #B`B A8D(6L՟(wa𶭗`+4 WBϢ/Hx)|dpc$iٵU·QqJܥ|-$zCS$d q9M3cی֏u ;$.{y$4^HȆG/4%($@2 /6Y]EyNT)/񧏳L>k|B@SRl =%0oZz Lld1^Lg惮>6.{_Z'm8"ƍӎ'$5og^^˞t0OC-;ӋxH:/^F͇oq<8Y']7Xcշx/N\|fY_3*|I#g2Nݿq6יqtUiqz"_HlR0Rsį^sKSz ݕ3Cwg*airŵ,W)&βWUqVJ^D蕱č}g>m GIq-qchf:"[c"8x>Ix2rV߯zvm>7 '.˄GܳgV"ģV(탇9~P}qUk#~>\X۽>4d4c`jkU8)3NW 9ݶn_(#棣1:g:n,=Y:՞'퍝g왺^\r>seMKA_ D.xo#N1fu۾ oJBUE2a^],|9߉5z:Ħ"?wKka3oLmdchjbV0ᯡFfKmoUk>{B1s[QogάDq'iyQY'6UbCPB,+ ;|݊2AT#,%&%t1h Y?+E.,E:#9[ ݉"0C'\pXTì~݃hџ7-lCi[xЯmWwOK]b,`QeBaėzsō-%D?gږ1܇6v_P/7] ' KgڦRN/ue_+Eݗswe$]6̿Oo{~[{JJ- 㝒)@2&O ”$bHɌ4]SU݌9{x?;"g:v\=M2ǨZ%o\>߳S9<:ъKaX\6qwfu߁;ǭ3kOƖ9ǠC%sٱ@o׫YڟlBs}8w6W~MMHfd=/BߙLh;{|.>ն=j棸W~zk}ԟx~+Ŷd YAD$,H*u'!NS&뻢RI܈ܾ'4IM/dc\ JvwG 5\祐~.|fj#'\p1@^[ Q6m*H FɎ|.hYW{|Eke֏d:ՔV^{uܵ#n/`BaResWZ|] 3ٗm5qx_fXiacY%!s@pSr#[;KqJBz>sdNn),׻HX]?ms\FJƵgg4a¿TKy$gV,(#<2vO>ߥsD\Y彲}67TgmY1OƖZ ;Y̙|vH}.^xͱO#d3[]f <_pk?+rwq"]/칩 cִzbmT׮9(G `Q|{˴ER%mm;g $om"8]3;r]jø{@e 4;8p5u5;ٽ$,?~[:#KW֧ykKۖ1Gu'N1ke5l|nGw-x.rLcp=_=ʊ{-J儃Ӛ0  kƬwdwjs29%>;$:io} [Vczb, 䔞YSvkBսͺ}fqh8.yDWumHޑ4"^5yYb(X ?"VK4ʈ_LcQiѤ-ĠN6v_rw|L_nTܖG5񠄬xLxPsvh;Xm GP\B7By1+dKߟq &d_j|T_1> mkcG Y{Jmsou_F>”l x)}ϷJɾ#MkOu]QC)ݰ_}u(`s"wmoqvgkޢ/gAqQܗF3eQjbuA gimQ$S vieݹ롴6{>X)彤<3g!U\t;tprdˬ%)-J?vz;-aMmgAz-|/&'_]Za Čkmkw>طY,Npc5sNLgs>W[+3(dEĢu@D0qD {}Q< D.͠ڝvɧn[/%!˶:,B5H1dbEKg@4y8q2A0=Y{tx ô=n%űl Lh?^!bmxZ3Lj,^:vc8P{c,/[O|$D/K'*0뒯Kw*&|4_Y`wLewFLf51} BӳqV}i>A2{_dI_ 37OI{>QsisRl=ֵI@̮3f8KHD [ޮ<֬}8 طr16\6w> Pl5s[i\nY4a_Xs9ri]Y:FUASy^suUY^i2bvkL+k>+^~kQ}˱}o.}+̘n_f#'{d clC8WFnawu>{v75s΢=#vgj{'xsq>,v! p(佰 =I5/Y;S~]r{4ў3ENf,N*vٝ{8=1 N=~3$,ܳ3&! .X3O>Pwnv<)d=Unx KsMjyfc~ZWӷ^N6$3'Μ˗*B;>|xP_~H\RHvｓ =muڼSO۵Oۓ_x엏$ڦmN,{ngm}UT,V*d `xw߾~*|_>Xj_H/~;BBJ Y/>.SzxG| -3N>\A_kG^_]ǖvp|W^4 YgIkGzj7ӿj룍^mcQ7U6aEˆ X8BN&* A'A!,%] Ay2m%J"E(h=^J6?>X* MZ }f౔lmCKre$`%H'dsgƶE*ǀBĬ"9 Yg%`HdaLP*A"XmEHR>{mC2qid:;sB9RaM/BŻmfk}1ul⾌63 Yξ[eqfS;6;ӂ+%T) I_ePJޏ@B L,iţhwW pBUYp`} Rؠ8l(~;.˫7I%%)i5nQ a\F)閾HAJ:r먾 (Dc:XQ|fY_3*|H&dEJs$/jЃjۋTfr1EƒcԖ2~j׶ݱ6".%)Idf\jžt%W$DeŹ|5ޓʵY;_]T,e[P^#E2;8 j=!L1{!Јei,ݟ!95%#R}|Yc_jBֻ}ū~!ɿG!.^H Z[Y!R"2.*sJ\2ewm*RZ\3>Tn3UIDATji\fly?:q.8WTg D!IO)8^gb,1Yv?Ԏ#p4]IOis('Xk'/ Uj Ae'ykܯ]Vs8"&^ 1p7BV(eyc}԰kmC[z,޵J^Ymߊ1%ofmrċ[z|&햻ؘ q>]z5ڶq}3B B@2+nYZ&>O]UnŔ-ʬo;nU+ Y5Ŕb\u\' Y)~1nN!s\ken,KZ0P9/&lr""dR@'v (`Γ0v5~ԅ,*Oū^^&S̥/͍=[X)aM&y?:ھ}.^d6=v[tLUkƤvhi눹s-pO$c2 @Sw/󕶻 L?1:'7"X敗wg1LҀk+Uz 7CG}o=밸^k[)'̺Rk@(s \ؒ0Oإh^w_dأܳ)Yaٳ=Ξqh;QpM B&A~I~Yč U' d$^#jDamn!ʔ$pe/d:)qbjhey|N< 7і%V$<% ^[ Q6m*PI?8!(QbbHC?,jG^+z\1#qcbaݘc\mH\Um ^y۝&؝ڝ -ękf+ PG$JbO%ErtM%ڿ~x<Β|I vGAP\L؋&%]=g9g[Z Y|cQAopc\=ֽ{ufסGI6Mbؠ"+\߼R)/֢|-T=z]BݾQ6r,ؿxq,bT?Y^4#uz`,a[N67_|+Ga1[--.lk۹?8a=L,"BǸU/Hxײt.$9/0u5;ٽLtIZ{n!,{|.c̛{7&ɬ41bp>ϯ܀Ҝ4זCڧm2>6}3Γɔ,7m~ʾY޶OHh>Wo+~^Y 2w~Tᶌ-[1W315zlNk9NX>,u:}=#Pu]WQm[Gx#}~jL1kRDMC5mZ1(f!(b1V"FX_kMQmSU6#N6v_bߵx_6}Xe~۹bn")KZ$4p^̩ e>֟ !9%ܱHI,1R8!~8/#_G{=O٘@+$ڄuf:J-YqN[~{HYo\hoqo)Q6!R2/j6o^>$9U[-$Pke"N/|a4!bɾϗ:r] c5~FG{J{k7@{p8:~-6W/2bk]?:d8KkEkr$|U5b8c.=xl`bQ_:R"OI(fp +jSOFvm;Xj| Y > m,xBl*?EЋv-C%:K ^Kre>X~ZB8m)6 r} ?d+>!bmpߍ$%IXhH}įMpxI :JK7Æ}9|lڮnLf>dD[̥Xz>;|,Wu oJ*dB;&SJDc$O: sm;&o9. qm-[e;~d;c٢J7/uvXSkw _vXF߸\!Q/ʊhYbQm}DǨ_emUß#Wf1{ BG$a3ÌMsg5Rűq6Ro[D,`X%J@Ӌw+u\\ᯫTL! _@`SX#M'Y?#$yY܁$ĸzB,p@g ,p@g ,p@g ,g7ëׯ?]\.>jyf8˗8n\[稾 8Eػ|ѳ|xP_~H\RHvｓ =muڼSO۵Oۓ_x엏$ڦmN_ou7/} Y!IKgz3>T~t{ >B /.Z۠u$J>7!f烟ї~A -\bۦ߽s]$4eNLYZ=$PX,B> F$8}EAKԵo%d}<MbR!Sѧ+dw%(|x33ņ\zC0?9\e7!˼b8[ Dl_gz\ٞ1_}eZEVM'- ^HfGo:4g_}NcM-ZT6Ə|~8q^CTǵ<ܴ|n)#[x1~1Ze ;>c֘ d9q{mI8TȲB& BTōxOhV BѤ$d׷ Yv~4Ԇ/Ys~4bA퍣<}IJdKJ$D4K `q8wL^%eR&lH}9_lNQ?%BlkJn^yt:)D_a`4DAP2?OxXz}vzJ0;9)y:'E9&:mٸqτ׷۶݈1\yK[8Q峺ƚ5k,F/ny@ga-D|QY?`^?6ql3RѶ|Vo~:MEg˗b. #Wqi}\83s;1.6xt ߇%{&Åe%a[Qf` $P/m!k}vk_IȪE.,洄( Y徭ɄKL8z7$ dBId5'cSt'џ7-li[%8mWwO*PRȢ볨&!Kd+,n)YO$Vg؝Įv)b-:bnDOI;Znw|%iEDGrGB =A bY/+?i>}幫geu뼍"3\mC@Ol#‡µ?kѡ"B ڨ:I KLJKh!d$pFȒoz6ؘqcHb\~>QdHԒFky"E&iJV?^s8čVBvJ%R=I2FɎd;ÿ?+?g'!No)le_IɺgF?"mj0_?!ՋŤq]?^߹ʖV Јv>vN Y'하Bw93!kA b's[km;9nǶ)ׯ8/o`ú/֦%2\S8#-g|4CL݇g!+V`B#n)q& _,J-- &mm;g⚠*씡>~^ٞ$o%;HEڈHx7%RdIxEN NܘV?^baU?}_]@?BID RJ_yXIZ(d33'Eq⮟ neK+pfCOĭlw HΆ>=tBւ(0]6y"ϑhN~iMѻ[dq."Lfʸۢ8cfl=P!,dE, VY"lETն~1UisێQ"ӯm?aESvMKZ_d|ü#*")%{PHIgJ$M G"KjD;[E*&&8TeOXll`q/=,nɯ?5$%dmz|I#d-dglSmݙU{ @䘞:]Nθg;OJj[1/KqmO&[~8m{%U_d~I?5!+feǯ6t>JGrL(=+\hNy,7P6s1B̖u縫ڦ֚5Nj,@zy5A=3c(3ڶ>ܶXF!+%2:R "OI(f ݙپ m|궵TȲmT|!v}zչ΄%}7] #S /Wx Z(%lDַ8mdk$Խ庻b,L%Q!+D%kp YRoC婎B,DY`l߱~w͓nflSߺW/obmyR~[35_ݶ Yd>?,ҼΣK[7X#5H 1Yۺ6lvvFz*1ܶ;0lu߷Íg9M4N>gD# ҸuɈ>43ƚ1XnG"᩾l pmHIu$Bi _ (ǯ4Ar"}B@:9?7 嗟`/,ʎ$0e,qI:zx~xGN)ySd]&0=9aξGWC'a:8;?z7 ũSѧO~cU\Ai!qK!y6f]3_h)LiM,;ŋ>xhb8_9 |n!A'a,1GsgWe&tz?Tez/4a{Q"B,I#xņIdPp.L dT WH`),j;bx?OG;,Cˮ>ضw\ }] YA,`6VH;g$]&-ApJV~y!{B?{flQQaaa/BpA/a?QSy9P`(!>2TaLlt qOI NOg}5n`b#$dD":vgD'_8oL2ÿ{lE=edsӽ9`xlY&6u{ŐO(; {vstޟՈR۲ϮO AJXD(Ij_T1ZکX60RQMv0btŕkkytI͛SzyfCNCB-y [G#ūTq|yO/qN*gn\:%Xj[=Eci_+BַLb!_`U=i$:LƩ79:1*m}?Ny_ZK ^8 I%~ڜ7'|]~a{c{BC 8^ BM䴐ۋkY8[SLe3$#4爵,)chAߙO[m7{Rc\K|s|8&Ȗ-H5' {I;6O-~s8,rϺߊwYY4EFt,Hssd]=ݯgvCkׇ׽ޗ:~ ,]m 33]{ Ҷϙ+s[sL år3nM-D\{^7gs(_{V;KcVušBB7 ,*nx@c߮]tY. YmB ==BV\{HL_5;ּ8C#{HNtHPa{P/~O6S|(Xȗpȝc^VR};#[⹃5._pTL E8 6'^O*accvgF@">?hݛIsг=?#iXH_OeOƖZz ]kpvӼ3*k9y 4s̐ϝmMcnl-]; 4|gޞЉ}]ئld[~W|ٍ;?3SiWg>xL۸G]Uߟ!GqKԶB _fŲ׭0 P˄Y KJ bJַm*SJ׶?u)Ew]XJqv^p:(%{spM蝗~D|}A'v kVx;姥.d`_WyB?(^M2!0 -(&̋/[4v.dyse%k[pؑzBnKɒ!V,_ʾWƻ;}JN.㧶=_=Ns ZA(%30̔OI4]SU݌9y/'{8k3;.ޞ&lcTas7\lwt}dwdi2Km۾e)cJ1PDޝ}ά/Z?[fk݇>zbk .߮W?لPZۗő?pTc-?4m4[^;>U>>f<69_u[sZs=\9{Ҍg[B 1@u5gWW{>ɲx3ϰy[j{le7!#Pk֢eU@B"T$%vgBw +m Uj\Ȳ!KR1.m/l8141=}j!{㋳Af/jnþg/Lc!‹5]qw>{ά JE'clLyw~yY,L>;$ɾ\Hgo/nllh]ӚxMڗ㹳ir9kޑZYKP3a_\<0>ީ(8e.BV&GRL:-JP|ܟkLye{x#5߇0~AW-?˃ A7q^5;ٽ$>p۝Տw Ygd)4^ymi2pƎ,9c/J/&Zki+T>6[7)X.fN刹[FC@뷵XBYqbEi,i:J{IJp81]&{.jA.:$Lc\K|vHu:$>+GI`hv(mˋ&5뾃S YOƶY_8DWumHX/dY?kq X?Slu3TgOv t胅vDOk-h}LqPߋ#⢂ze&/kr?+]g>؋BVQbpa%†AZkj[ڴmG(}YWƶ}&)Y7{ umPԽ{R@!;4CbEڪʽ΋^!+XcX&6ٗGR$hp(ᮍ)FϞ aNݿE׈gQ2s1/^x/Gh/yb8|S4x]g|~ mb7Wyv]%$;9껶ڎиG5IїI38(K#ޙC2ǨB\u18IV>*I-קiKk4X0 ^SDޝ~JkO3k{9oƵim6ֽ~b9^u&vq6:Bx1Sk3zK}[ۚu=߸^e+bCT٣J'a/>N?Wy`  t)Aܧ$Q^_@Vxwhl߆K>uZ*dٶsE Ys׫u&d龽P/yzYШ:IG{p-8K^,c)~x&̔Kq%'\?8 YDlǓ^4|pȲ^*sÁо?se/)䑈c%brDe&/{ץ;FrXﯬyA;2s_P˭ӧ <Rv8!c8J8~"v'jN5|N*}7GҺP1Eq}g )RCa˼۵ǚg!f,^uމB,s3O"|_=޺Orˊg7͵šs̑{Nʺǂ$]0Y3"۫v \d<8K/W6|&W գ6c\s]|>2׉uψ9|>t8ȟmmrW/oslTL>5zvſ!/s{bLf?Of# ]`E`?Z/wLz :WS~y׉^7s/k;½qv'޼iqNOL.KXσXE*B֙xxP@gB "'̓;)*vqv7iqN9&ou|"M̓J|ś|VǽNBV7s?lg'卍3"D+~ WB8K dB8K dB8K dBwͳ럮^o׿ ujy5<31W?ͫvkl'ZowDsgWYv!띋?E>|y<|(/?yb.Vl$oOOIe:m)ߧڧɯ_p|yOSȓDYTp[v=<~cU^rcD{p|W^4 Y>[;,PXi-5`ז-O/"O?rXкgBVW qbvbE,X8F&* @'Q! %] Atz2m%J"Eгh=~J6?> -\bۦ߽s]$4v)dt3ZAA%2vo S$d q9M3cی֏uc@D!ynbr`3sr0$ eSЖSRDH2k{?Itv~Z:Gx:&.93)/ǺUiDk/GvK[0W˹wBVW H#>ol41?xhO AJKj_T1~Avvm&"LovCyS(b,d|8m]-a_eM7Uyٯ{C&s)^^ 7/O—(,9LIe^s+X6JIDB&RLIlmmĹ$\Q{k+PiXx]+%/:kҽv9I\$ M\iO]?ˉ"@OLw?_4ݫ"nʺOx׸_gqDLxBRZS= Yv=zxBVZ˳=֙l|4+k1[:ͬM#Wx1~}WτrSp/ocvٕ!'0Y$u+QD ` Ŕ-do (Qb_Մ2}ms]GъR~;#8+x (/Q&ϲ|Q \nB&'"~O&Dl< mWwOK]b,`QeBQ?\3U6Tmq\ۗHkS/ܣkEZUнfLnni?K1ײm>['Dr0%cb:$EQ&ҩ~)d1SP^cwD%5HϿƚwg1LҀk+Uz 7CG}o=밸^k[ YA8r*ZUc P{vrg6*wWz1~&u|brϦ_ng1mg϶:?g1 YiE9Thrླ‡µ?k* u'6NS&뻢AuѶRC*I.dYj%)Wsm1kX۟㢗x;N&D[Zc󤛗(P{m%DlWB%aDldGND~" !br/yqWƍm7STywc"q!qUm{ɟselw`wjwF&|VgCnBN+9z ?L"اDb=-y,I D}$8]~agз&$Pճ~Sxu %ϗ*:DV 7Ci g]gvydc߄ZJЇkÔOo CY? \!K%Ihmcӷ?Gټu`nűQgwx&O{cXԵmslNF g5 x}" .BV&GRL6z-JP|ܟkLye{x#5߫Efw-{IWIBR̾{BISW]JcǚKDHd;m!ϫ?Ȓ'i8Ƽwc"q I3*#&'( >+Ism=$}ڦ/#ya>s.0v):H\rw4~^Y 2w~Tᶌ-[1W315&$uL1q_$ǚV[_59 v4BV}dXm=)ưwMU|Va]y`.;X ?Kh֚;/6mn2J_duյ ,y_ {lJ_5|%v'"џ tIYJ'E<DŽ[bNmx/dw-aEHjf:d1tY\C?,}!xijKk.1mo1bYܳ^')YIe߀/D\m߸e^=͹M6v rmo wb}kvƲEG-~8n<_hW;A9יl߱>q繂G Ym,| W^Y1{lbJk;;).dp퟇"o<>嗟7!f烟ї~A &!bY0"$B,$k^XlZ[JJAˊd֯Ц t}u=<ɗ/(vp~? !!!yh]! i 0(5# 8:2QEcl3P}O 3So3mae_~K~`,dPr-_$Jm8Feo{?Bb&~'B?EI8-~(2 Y 8dldTvcuTo9GSRI1vq"ydi%>^&1jSE{reybC=!yJݲe^uJ1ل]ƿWb3=l K>2|_G-"+&WHDW|M$ͣp3/>[1&Ζxm*G>?8h!ZOGcmnZMj>7q攑-sq-Y2{|g|^kSԜ`=$*dYR8a/z9:+įƾ]Ai!dmMJB|}U _87 Y% عgjg~/B7 ɖH7i4)&pKʤL=ِNs"(&bٜ:tJ<הL6K`W!!8LC$u+n,D0 %At VI:_L)B^必E.[]INdnQr$ Y1F5 m}q6{|,G$p(^M{5r*YxIT7]dI'v (idV"N*i򓠾J",ji**I" !{ Bm8*vgG6]J&sp,}>n'~IDZ-d}ܑ%mOsuk&mg}} VC:g&d-:^ uc} |Mm V_9XlX5TDf~c[gl|ڜ1y"d!lbzf{˴ER%mm;g`0#>~^ٞk{|)y`&T@"&"^Mes*^rjy7浕p5X$DU gWW$OEb*D')deW^c?k,t? YC'L DQoAg;[Ҋx1\P6m2q+]B=k}wO8u6`>x1L׬M޵Hcsdk5S_o{i?%vk@mY)¼vuٴ2n{ƶ(Θv7c|46{}x6 YE"ly#vlbj{Y~umlŲ0x}o);^9xEöI=n(K$ 3%_b&܄#%5C"l-p"m mZ dd*ٲ',6}6_bY08 YPWXLbFƶv3 L*ͽ݄M rLOs.'og3'%-^å'djadsΪ/x㤟}}Wz[:%K9\O.v}qοs{k4?,ҼΣK[7X#5H 1Yۺ6lvvFz*1ܶ;0lu߷Íg9M4N>gD# ҸuɈ>43ƚ1XnGpi%Hd"h;\[&pRIg,p& bpuZ+sf܀p`%NAY,Y,Y,Y5nW_{^v3\zש}p̔p\O)vG6W?/_W~~;\yegƑq&n;!{멸,">ѭgq8p4rѩP11=r1Gr瘣H9 1<.ds1\~ȇ:E'T_e=*m7x;sv^[<]=~HO_m&Do}'v=4}X8Gs^ 1;| oH_ _]|_~>`/,ʎ$0e,qI:zx~xGN)ySd]&099NcS 1_^3xy_:e}tG<#ajю~!вkIZ$z]?Rbٹ Y{%2)}o S$d q9M3cی֏u ;$.{䗒 zi$ȆGBv^@ʩ͑[w߆$ e_Ogg*0&>qOI¿KyMk_o I}Hg|8\S}'7z&<ۚo1S<9uhv%PjUC>NAcާ5q s1{a;⹴9hp v*#$F_PDݏ@b%NĒYm GIq-qchf:"[c"8(4}F/'YE϶Ȯi2'_S YtX!94!C;vBk=ِN4cYK#_6!wkzYIr|*jBֻ}ū~!ɿxQNsbC۾Їزqh_dhs;Ǒʖ -; kmx*iCu>ϣ- ' <1n=./:\w/Z{ ,gM3]oMuMڦlT}{T%$'CR>E>;$7qsLgk|1w5zK&;lk{G|~к7sgo{8~(F<0Ǒ<-kKұa^=͋9ʿi0Os{v|Vt߻||L~={[~)]K/,+],0 %Ix׭1 PD! $P/m!k}QFo D) :u':{FEɑ$d /H>SBةxxQ;@+ ykA'_Dlmu\iS~ZB Vu'dth/#b¼EjB?_Ve ᮍ'ۻ~!+o;j s]ٗ"qx7E5Ī;}JN&In@SZInQMBK/s9vR?9'{vTmU7cN҄$Rw1E>;$tx{eQ1K`sYHqSG<0t-5'8^"|w[gJ-kC%ر@o׫YڟlBs}8w6W~MMHfd3uiݭދ?|U|HmϜ=iƳF!ߞrگ*6޳~dwiW3|opZvX0UfD %*$R|W5FT?&뻢#1& G|k Y;E+\$%1á8bZ~8M& l.Jn3VDIQ~36iYW{|Eke֏buͫ)QkcGˏ]';^* {/Q6w_e 2}V/w#Rm#Xc'"όub:rSr#[;Kέ7.0'NȜ]$.쟍U݌9 .c#%[Zj x{eQ_*XH*=1NѶ6{5/3/sgm~s{1#յҏh2Hϰ/Jdz+GpĊK x|+T೭mL\d`ԇ+ۓxm,Oa˪sL\[BY֍70HCGEtqWyfXW8BVQbHe~%eDcj[ڴ!EV_]~fZ_SŲ%/^7_y"Yg4mҏ8wv^6^j!8{(8dpȞ~^,ܱHB[{AB7By1+dKߟq &__Np6ܵ#d_m lehCSoQ.¬5"Yf{L K},̓Gh/qh e.(!~dEد>{K{Vߵ}v=ڝaNL>>1Gq_–9F⪵qV>=YX &mEޝ~JkO3k{9/Jڴ6R~uaמ oǺvO.%幘?:Wܧۡ{n#}TF_fU.ILh,W!Ӌx^ ϋ>zWיg3)٘ L}-~ q.mmk|{їZC|fʽ쫞g|{ Ǔ3{%(dEĢ ]G $A:I԰ƱKlKD3vɧn[/v.lۦG|L:gT\QtuM}aʇ2:4~oAq,%=Ϥr} ?d+>!bmxKfY;p cyzbw|1Qc%brDe&/{ץ;FrXﯬyA;2޻EϙN1} ‹!b_p8#dgRI^go2H{>QsisRl=ֵ@3f8KHD [ޮ<֬}8DI@&*>Ss1bvٺgRgSx$4mu; }[V?#Mi@֜csmW}" s |{{V|fm˛&-}Ml٧3M}Ik½ő i`_rvyd/3qu8.[x=gҿ6s.ܫ8ہ{gw͛g겄<["d|7 <_,p& bpR<@ ܹI^L!AplgwÛg_ h Y=M!! d}+N0;| oH_ _]|_~>`/,$ŗK!k!%\U,}}zQ/Epu>`.kޏ~x$?Yu?XpMsݟ9OeDzp~'kvrÓO?aA YnRzۭgB B`H#A|ī((L՟(Twd𶭗`+4 BϢ/H*|dp_2Zs^ $]?RƱX8xIL[ Y?B\6_E"QHv7X=g! D IC,DKTm'W&GԶ[6Y YHymd991> Bta׏o u')4NȪp +Q8b6<xhO A UF( Ij{1ZکX6.RAP%?OPUx>6.>EԶ#Xi΄,J7KyjMyD daJZ/[u?B$QJ/'2qfN*gn\:7#)(N+VTo?ĪBW̿ _; Yѿ'˨Z,b!8cl$!+eb ڵmwH5 kDg_پ']v:5 ~gq._qrm7}c!*K7M7 +.-* QUy'gWk Yfc:{۞5'~YqX_?S٫7񞴿zjc DM-(fG\ [ ۩YˆhZF| Zkwf^I-bcĬjgG/l۶Nilei X!~ ơB+7 ,QAWOkN!kue!kMȪIԆ/nJvsiنlLw*dŗM/Δ/޵<ۄ;& l_Ɇwj_cB#:tJx<הHe} YE,'7x#%hmgXHuȸyJ+qxnSLܵH jqPIϨB{&q>mmĹ$\Q{k+Pb%ec[YEf+SyjY|ۖkڷλ~^Ymߊ1%ofmrċ[z|&햻,a8kmLOZ!" &$ TVv7f>PD `Ŕ-do"L[(1Qb_E.D6IY",%!+H|t(? b-P^MeI+*9%܄MND|L Dna%LyFۮł] Y"x5+˄`4ѱgb =%ld4G'B۷/ Ց tצ _G#nɵ{͘T6-y1be|+d 'SP'Y DMm& JޞbJN8)I8\$)|g9"cgVݫnzaqN׶::zCº-l⺛65Vێ5 i^w_dأܳ)YaٳٲLBVg|xvMbڟi2#+u'6FT?&뻢%~4R۝䢌YquјSNAigQ} Y/V<ѝxj'n-KZHyKlیU+0~"pBQ'"EF~1|uYԎVfbG+ƶթB¼1Ǹې@92;M;;#| Z>+͉3TW!7AXH^g=xI(BϔH&BR:KRzmgdOӅm- R:}gWYNٖց|6H<_XTv[1Wu/vu(Q}a t\qWǾ'BĵfӾ9e*"FeӟU<K;RaQ׮ͱ9qcc2֦8k=X .BV+/}O~oT೭mL\,I@}X= SAWyA۟d ]^U/➿sRU׼Xf2'jN[ϻs?$nIZ1oݘc\'BҌʷLjI !<:ksJs\[r hiH^GD8\ԕ kU7%Km;L)kCJB^&19ɩZ/8 }KnRlÍqQ[:Y,Oc<$nB\K /t=Cij4ߢ8lV[e1l]Se}`?p>ͱh{ų f!(b1VO2WЦg=s)ZgϢŽDUh 9%xj/?s܎_Np6)nLD_F ϓ}/uSpk\\S5y]6?9E DS=wo#ui.UO4Xfb]ӵc[ ti-}hMb)GXI.(X@H$`>U'AC;3۷E,)m+$dٶsXe6g>RLcElzѮ%cb0DgI;7Ӌ}I̧k/QKHR-0%FA[/A\RǕs\X'dA̲8) -v\o c6/)A'P<~ 6z}YB/ǜm^ەuٍi̇l}_g➥XdB&cB0D`L4N=_o[Kd"&oYhHrr#R\Ƹ[G3-?jqR'o׍5F{}ϸޙ"ɼumga;l G:Pqbܮ=j_ IY;uҞǨ?}u߽^Ř)& PƂ}Ϙz 36~.׬r Gę/ڮ=xxd /0"C$C8WE"F譮-,p& b1@ab<F:"y4BEId!B8w dp? 1;! %Y! %Y! %Y! OW~y/׷nK:ϼnr+k|-W?~=zy9o{w678_)>W^Y1{lbJk;;).dp퟇"o<>嗟G_^߹')'NJ7Hy}G{B6 Y$1\0Đ(6L ՟(wdƈRH`q)$-j;bx?OG; | k)>;¢R-ǩ'wz(xB֚ ERhge겐$q{Ib!k!n)!+-+Q[^OB%ڦ۶.QzB߇'_PȢڽ/~7;䡝v,kvW3J2NMHB+ T#3So3me_~sG7ag>}vYa~3Bⰻ= ٛEȫWD%?{$Hjī#+d?+XHprG+IuR9u3:Q}ǿQvOI%Qmu˧I|kJxx1^ĨBֳOaW ,JQ<&gg \a*+ ~rwnBy)pfv2_پ=c$.%uh|hl\8[㵶lp̣>/sky<Wi7ęSFbbƵdYw}{1LARs΃>pe MKAebEط+(9BHz.. BmBV&&|23%˨>&[Y@/7}=i%}'&A(A-)d9n,%hRM1y՗ I{!DPL|Ų9)Fu Iy<){yh5E,Ab\T>B?HJb!11)Ȓ2>fR긛^ej]?^nv#[tp/?oO8D pk ??UM6ZGe-zÎaX/- HFYE쫶67a{/_ފ,̏_ťgpgǸ|$m&|ۗB>&XVv2qe,$3p BGShWo l>p>Qe\D()̛R_i<C$p(^M{5r*YxIT7]dI'v (idV"N*i򓠾J",ji**I" !{ Bm8*vgG6]J&sp,}>n'~IDZ-d}ܑ%mOsuk&mg}}Sqh-'^t'DR4*\)2I$Ui&nlیU+I6Jv d $,]<Y9?+< q~Ka+JJ֥?S5BĴYneV <^l/&]VbF l!EesoN=lĬΙ Y b=qݘke_Sl1q;hN~|y}3} x6//٤沟!m?6hyb>p<YA8pGJ_2mTo u!k[&X2A߸w"'*Yc`pI/&DL9I/I<ÉSW]JcǚK,Lq^ttdz+Y"1I~qH2AJ^+5BT: YY tơ|& N3ۭliB.vq6.\[5Gy>\@:QZ0Ek&Z1yU92uյ)ׯ^;͵i6z},ae~]lZ=c[gl;ݛg>q}=>pS ImnE󃟝/rA+K2_-"+_CO8>Lmd ,Xm p*Z :xזo(TG! .X dwJc/7 bG dX ssCB8K dB8K dB8K dBwͳ럮^o׿ ujy5<3E)&pSݑmrp{Wg0;!UW^ٙqdu[xqz*z? (btY<1<ΛY<1ߏ3*v7% g}pR Y\ y,CQ~#Usg/Je#y ":N*{4\i'O>m>mO~_>k[nSSW&N\qٷOey3)8TȺnǶO߼;| oH_ _]|_~>`/,ʎļGN;ҋq/,Ó;|#<R'8uxK! 3_x96^ r.{wPV s>xhac۳Rb;8k]N %DV}B|ŭg$uƳS6 Y$a#)dC0 T .nSL }ewEFo9N}X>עox7 8%+AB=!=3(jWQ(N5b^X8pA/a?KHy9$B^@F?HIo]yz,@$j[d3ׂim70*"S;|U _8oL2/%t๗횹O撽<9`xlY&_ñ\Űi+,~,'_xxBV|N䫳<o;?о$%+t{&`$!R!~mکX60R#˛B >3>g0eLWy7o](m'$5o?YU2f~X[b١ 'OlϡOm|ʗ"!rfu_[1vq6IU|fY_3*|I#g2Nݿq6יqtUiqz x؏/ YrbR3֡ 6 9!2 ݕ3COw#zq^yl'Eu#^Z=YJ1<*L2I6Y1"&iL9ZwV ~g:f&ߨ3!e9&R3Ig|V'^s9TȲb& VyĄZx^c߮d~x]V]ۄ,;?V)s+c*=S3Yҵ)0CY:Vww|Ճk7%n+PX}wj!+;0/ဧ^ÿ{^F|^u龄9WJqdwwb;7;9nY;(_~2ֺ7}K1B)&l|O{UBw77]5ܱcj2QgY g{SLM3{Mԣq6~R3?BٿkWTض1:s׸p%ƪjQo۵’Z٠'{g(í2bo{\BlL8"5#Z_Vq.EՑˆgcd}WT*I @jY\!Kbcq) Y4q\NFwu :7}uq`!9;8{8{m%DlW"$(Qlf~m/."ڑʬW0͵|i1C{ᥥm:p^"$w* {/Q6w_ne 2}.}i=f[/1%;!3^m{ļg?]+=݇=vY] "8#V_E/ZH~oTmL\d`G|J>}&uJȆl^6 y10χ%5;ٽsZ{n!,_Y{ז-cwm숟#cHH4\k-m%41gssP8zkER=0/r%#֞b e=K |G\<7fS)z.jŅL4ƵgY>Oa˪sLQR #'L̰=M_qušͺ}fDZddl[u)Ű]Gvi[vqr6'q7:FԷ^>φ6uJY3L_OTu>3C3kоvH{B.~bؖUlUZm虣U=ۏ³˶"r_`3X ?"VK4ʈ_L=cQiѤ-s&W`ob;2a]̻`<\f&Pc! ߵ w,Ҷ0XxC:/za>rmھxpJ}E#G wmDxYyə(b} {=&`>_ '\Vyt'&ʎ37=El(׏uq$~د>{K{NV㭶#4l sL>>1Gq_–9F⪵qV>*Id%ބ ޖ^{wnz(^x>m,uYCK1}4MkWָ0i'dWmf5bn4]yS;a?(J9݈>3}W?nGTY9yJk.='^{'\_%oYҞL=w^78z ؆bll fB̈́Jg}XY1fJgwN]"s6 YQ0(AבbD}N5D1+`~fPmhSBmۊ:}>g*Do~ݷ̏SZ., Lܓ7NE:f퀚_"[PKɶ>6r} ?d+>!bmxX3Lj,/r;g1쾰l&)#/KR:L>/ 9_y,Wּ ؝S]^R\ӧn ⌐VSJ%bx}iyd5'ٚN>'v#i]{IH}\m1YBDԦ&P2v-f+řULN|?Oڗe H1]0GD<Źs4.F|\{/94ڮz XcIRZ #wH;bϫbܧhfֈLעo췶eI;gq<Ԗ}Nsh]=~O~3$ˉ؇of,C8nBm&Ǿe>k{lCLVUc$KޞۇxS`X:WSd/E/'67'RzAhϙǢxoz)>U=;M3pzbt zb%sǙ3QgH@gB "'̓otȝ-f Y{g;8޴8'd YG;; bo2gԥD!gkܛ1$dq#}83vqvR8 &cEd8N @pB dB8K dB8K dB8K dp<^~=/vxp]W3SsӚؼnǘvb&pG==p\|5U8l޹.YmÇG/Ϟ_ oFDtxThxn;O}ڮ}ڞc|$/ֶܦ4&%Mعno of&-Spud_o4?|.GbRs|c}Uv )^^uY )'?g鳿 _wȗ4_?<}CѧgO>mWxO{⃟CY6n+;?_!+/yxlʙ'%dْQ~~uτ.r=<q2QvbEˆX8BAN&1$ @'#,%] BA1⇄m+Ҹ.X\i Iڎ>#jю~!`6]vGXTjԙ k*|-ˤ۽%NIJ//uO7qόm3Z?U9)d繉YyEsJ4B-268ќl}oD;HrY YHZ/ʄ,3'˱nE ]%ڒ4f60q.a*:i*4#$_P"Dޏ@{>";C;6Kf&]U7xdyS(bgl!F/ Pv\ʫWo˓%J KSj|܊)&R-}90uR9u3:Q}ǿQvO@QmuZ"d}$V-gVUڑLȊ>8I6_Fbն cd%YY-KeԮmcmNE1\X%J>S͸\=JlI>s2k'Օkhw6 Q?'>% \'RЖ)d]dgWk^Mڳ~Z,31BE=mz,pϸ t{կٟ˛xO_d=1ݦw!)ZrDza(Xy_V,k}Hm #&f57V;<{qf۶uJHg/}1\WӞlVR} YV$*O_ UOkӏ˂|}e'* zeL%gJvQ}F;KqVv7BV|َ/LI ]^˺Mxc;OL/E>lH}?&4bYKCGHp1}MɈT+_ؗo_D_Ȣ{:|qQȹ8>R֖qVT DyW|6d]׌t/kr6[ަNK99w9Q8qN,Sr1$:\^HR:oJ,v'ϫ4N2IJkb+b tA3ݽ*Rָ:7/|0{G@Ąދ'd٘uhMdޮUEX}>4=7 l[:Z nelVĿ|-x3kHU'^ }z3oܥ#+t?Rm8F|IpUȲ$&Y$q"0 PgJW:_L)B\D.u_M8 EIt T1|ߝZȊ/"%Y/!as^MHDDɤOԛQ2'ajNi Y,XU%,"WӽL; K_{&j0S& M6N~t+}r]p[@wm r{=r\ ׌I>ms)Zg  D`$ />" mM+5~߿~/2)~<_P]mEܽ*PƸzZytm+! iE(XVqmZZ[!fu?]^[^RWz1~&u|brϦ_ng1mg϶:?g1)g||vMbT-/R|HaD31+*č$Lx_ 5I.Yj%11ɇ޸, ]f8r.X#zo,/݉v20&ڲՊ۞'ݼDk+!fX* ''d%; p"R[LliYW{|EkeV+&p2nlYZ,/̻y j Kc᱋DG pĊPK-JPܟkLHOg ԠoNQBِFaq9}@»$t!)f_=礀ym%ܱc%d"N۝Տw YdIؓscܻ1ǸNfoCy~u\椹>mӌq{_s$L;q/9MKG(Z&(ZXIyyg9N,xR2opc\=tֳ8zvO}ԑ{LZíquP\3rMunw\J8[mIz{J5= Y/gmWRWZ_WZs?߫WcuWLV5Ǯo2ӥX٢59bޛ:xv(dEĢ]GA:I԰Ŭ^b]"A;3۷Oݶ Ym+v[DYu 2?N!hыv-C%:K ^Kre>X~ZB8m)6 r} ?d+>!bmpߍ$%IXhH}įMpxI :JK7Æ}9|lڮnLf>dD[̥Xz>;|,Wu$cMdW Be˦{ 46y;%5q Br 2'Մ+dٶ{;GSZ\t%Ѻ?Grb̍^YH?({cþgL=cZ?fbke3+q6Y3uB,vS~5o 1'&"W_W-7"Doum d3!}U B8 K6IFx͛>O.4<(FH& 9! IqY,Y,Y,Y5nW_{^v3\zש}p̔p0/_q^k)KQ}@p|mݻ\ 1w''o]8|3%Fچl:0>Px脓,G`%.7 rHߙTuLUuyfgW~{{zY)w>\^Y1[|bJk;;)·dp߇:Ǣ{/粧ΆwJe#>? iy6|~mc{/#i|Um;/Ҙe*ztvBɗ6_'y\M5<3 _Gït_' 6BOgq_}~}˺7b%oUѫ>/߉{>/?r{_!+xsqby+]@J! !Xp$ nLbH&OBBJ߻@ 2`xKrEtMAf볳! >  G?B b1|4&5Q:f#I}NW,bTO4cښW'F QWX$"Q(eO(d}MBU YٯPO fG'%O(D1T<ږwLxm[F c[ru>[cV8d |5![^e{!+0"Bk>VEr[>l?l2p}oؖ*mO'~c[jݷ,{Zea~*. .qfvc\Ym>h  ۗlB>&1̊e%a'(Zܘ(_ qW#䴅mWDi B0nb̝Sovٷ{#8` G/&=L9I~$plt}2$fo42UJw(%,>ZZߧ okHvvD"nuơݙ<(um ,Fk1ܢ#>KtC'y$",dqGrCB =A bY/Wooi>}幫eu뼍0*NEf>ۊ8aׁ$ џl :=,lfBV@0k˪  ~:I KLvg1FwE%9AKhY%۝䢌 BsPG#fB*{3ӝLIZҨp-O$TkNl%DDj'I&m(@>vhIX:{4_Y8E}%%SC,o_dS 0NQZ0Ek'Z1yU92uյ)ׯ^N NsmFoř"̻Ȩ㶧o⌙}{>G?fπ؟BVQbHe~%eįOm2/:Ia) P>k A_Zƭ;f}1TI/&DN=S%&)mM8YR3$¦Y/ҷ`Z$$Sɗ-a+5f(d_@{Xܒ_a?5$%dt3 )[7&d^n&9wLxmIIK#p)uV2^hsI9/x㤟@<&θ(^1zp#s[_94lo}mx}Q-1sqW7O!kY{b샾{>gc(3 S,cFבC}N5?=hٿ5bIaL][|\JBm1mI9&tڮC0[HşJxzċMՒF)e%ž| $[#-fa*J,Y!*_CVJO_ؖ IBVlK:# Yr8e9Nbfcd<'3+1ܠ/Mߦu_\m٘u\"lWΗ`w5v^¸eqo5R~`vsv7S[ Yz*3b\;0luÕf9OFc:ΈF6VKH%#ڐS(& R67 _dNVN&xkDN# pۀNUD,nKAwJ9x;H_n@@p  b ,p@' ,p@' ,p@' ,n'W7o7//^.7] uϼr xbwd9~{3\_zes  {ÅWvbgrݾ~qֹ1BmB,ʨ]{w8s"71wmmfϷq'l;s >W8*녬Ά><<ֹ?<S~9=}v6S*m7TvoxLyG|kӸ~y/I?.ַܧԧ2mۉ~~e:0+n]z}_o3~>|sM}|1d_y)%&6x!^X.?9|=Ӱ]]bXÄ?n Y}&h`|">s qcMGz6h/BB4`1AAܘĐ(L՟(,dAȨ{u"]SYd;lxHOi<я%6t%>7IYT#tn+k iF4x 8%+ABO<=!?=3f~H n/,y5C㥁|nC$n؏B"goQȪBxI^*|mh\[4 דa΃Ƹ>;/v,TY6~Q 8#dl1<=wOj>3{ݖω$1[cګ=޴dv}гb6ӂ+%T1E8_` Jގ@{>"?,Ec Z~(/` Y>엙 c;L]2(ΈYU%_\/c.:K,;a~|QQ:X+_Ӌٯ/j;| l+c:n"d}$V-wVUڒF#dg;\g?E6q x؏/ @LuhBپ>!;T~a{{BihĂyA% Mq! YqK뷾8_)gs$e6K9Zv6- vג18t Lc8&ȗ5H5jLZcF[_cʳͱ͇ߨ;խ})rϺ_wYY.Z?fN,2Ҋagy̑9V1e?44'oLKzkaOHxHq 賹׆x&-`3X8vg ߲϶zFQ9k~{s9q6,6Dm.Gjа]͇nۋAsǰ}XTY;ȮBF7 ,KBZJojC\ѥ_m:!+yg{H2d0!ˊqqlZ Wt`,ҁ;tx ͇{!8 cYKCC}3mMT+wG&d}0Q !ߍKl$u 9o%2bơʄmGz*s_X6u߻jƫ"w(TӴ|G[S/lBnjo(~1;o=Oqӳ&5k&9A!ݶ>z.~Rz.qxޘ//8Dq&"@kBvn4wj3sgb$ƹ*p{FUoqJucs"Ao Uqjŵf%k湞U>45~i|ѬѶ+&;pk~{X_aS! _fŲE-.2aJO\!NHh Yۖo $߱cU~MkE$dJqq#t`CYvpS/=y/c#~OFE{ƶJaSł] Y"x5+eBn+gq#c+\Y&22pƆK`abo=4UŗX s]Z=%x0% "sͼ(ސSrI,ANbGSbT)nQo\dƜ$ƔO 4lsNmU7cN^4ON4hv3;.ޞ&|cTasxlhY1wq^ęn8$71)X̝}TV["bݹ`wqA{NvkݛLkb{G7sxm}˴Ge"^h^g1M35Q5QlҳR9fj9&_A\xwFVJ8sÕEݶ[%AuA/jOP^߇[{e~ N{&qb3!+ ZUeU@B"T$%&h!z^/BVv'e {);9bC8Cws١?Cyg+!fM_x%R{=/OFɏl7& !_dyj泹/2ty^Zm{kcC8q&H/Q[mVmAGܕg|1LsuB#D5v$) -qSr#[7%brbdN@GJ&.#}mφmU7cNذIRَY5=M2Ǩ_*Xrw>{ά J=ֺOFfVl?3uOjُkw)|7sI=ǡϙٿR:̭GԯSy~Lc\^o'9חCi].7y#_}gϮNzbXO_=Tl8z;b>ضwds>ُ~WdWŽYArʿBC 2mTo u!kmL\d`FJcVLwۋ?޼>м.0bpa?FC`V<[ /~_{ ]5{߼n!,5+--cwml'{ܤQo+ I5ZKk/\ܺ/5o7EQr=]F^m㞯֞b e=uz [j/[O%[䢃L4dv}[—1=8 ƘyWϐוۙcdJiԾu߁`/cX6>Rag%ۙ7VDT=ݗl9w#bOj[E{/jgj~VF>ӿW{}9rʻg~5LP.̱_Auz?ҞPg{= e_Ubm7n>z =P܇+}{x炳fV YE"~Eh֚;/> KҗEj|(k qdc(-CV=e:a9l /9H}Q~By>+dKߟk3Ń̖W+16 6ܵ!kk`%f%gCYЋYkDl*@;y~, :h2ٷw'3olZ{&P_?^\a۴ZSHvsvmoَPG5 oіI7fNqQܗF3eQjbUƘ䔗]{H5`!cc΍W?_ ϧ3kh{9ojm6}=568dlD[پPaC嘒kVsP,ލk3R +(X ڍC}N5?='4]SVI* YֶZ%%e/c; [syD_QD,BA~>a:{2{txPC=lh%ž| iЬ-ݖ .JO߹?8 YDI`4#,o~Ѭ9@ϳ²x$D\.K'*0y%_nULci<J vt2Ώc^*ѧ}!k=ٯTҡ/\ޘ9ִ|N*m7m#i]{IE_1c6ɦ&eZcW3N1 Tc|VUJ,h,&zc_=޺Orˊgilk/C1{9KLu=[q?Wev/qfH(R{dq@X,)22Z%m_}Ι{ٲωh>vt=\ixF|vjDlA3Z!lUa1ɾ&km7ٽwe} Œho Yl/ۑdh Hq?'Uq0ndܚ-I$;N4m?CGB8]8*'}@ܤ+Ԓou'Uqm3pDNQ޹cb"Y6sI]JM&-I>?qk8Qyk xɰ?:)ANY$NY$NY$Mjx_淫ܻNS sؼǘ6b61{{9z>pX/dw6i(?*˹鳳RHn沈{gv^;=_؞~Hq>>%|iN,;+83݅p v3|0| Cp?χouﾉϿ~2/쫲I%xܯ{C@ZHI>4W91t8"_].,yQ*Tֺrx0OMy{>Th[<}=\xe{\+d~Q,j/%qWHD:OdknU6gDvdz ;J! ! nLbHg&OBMBJ ߻@0`xKrEtMAf볳! /  G?B b1|4&eQҹ|xs\7t) X ~幸 Y)6?D"QHvX=e!D IC,dGRn;$my9þw. NZ:ER>99_ y1]a!Wf=hCcuB֒0q).i(HʈULAI*kly;7 v6Klx.Uj7jy(%X2P6f_H%1-ԅ,2a (~=\?p$|ȒÔ_[uԟb!llnˉLٯ/j;u߶_+*B_ Mb!_`U Ʉ8r9Nx/jЃB*qz>"cI1BVeb Ov펵9pa (/o*INxe{MB/Y_=\1~g˒1=% )؟WMIk|[ۦ$gڳ~ Yfc:{mz,pO t{ծٟxO_d=>v D\+8qFKuۑɞ4OO>B@:>dmX11knߋ3k)=7#1;4 3bvKc*dYaR$dٯX$~5!҃B|} WRIVmW?r23]T1 :M8?i{rb5ϗvH5[꠲޼5n׮+sL?"&^<NJKŹ+JNILBmE)-EkkZF"u kIǛYG":b8o;@1햻,aCӞ/ 68F\ c%61eV,+ ;QD,@,%3BPnD-q]rAhOcU~MkE⋵H@y27y%zH䜗rR69}2) fo 0$])?.u!+ĿEDjW˄siƗFǞZ,Lɦ쓍Ӽ _/l#MA>~\-:LUk~1m"#R̵l{ݘ-&S2) (198 H1-vL< ͘Lj0%Eiv`J.Jnwg1LҀk+Uz 7CG}o=밸^kkk!]k*Sgc푴:h^?~۪ʏҋ0|{4ו{6r?1lc?{Unce&!+>gg|l*=l&dA T q H] dwcd}WT*QX{_ 5I.dYh 1>i ~ mowx̲:<Ot'h˒V$<% (6}PI?8!m(Hn1Q!d?9,#9_1#qcfujЃj0nL16$vȾs&_֘ękf+ PH^`z@)mڻV& `^2Mmk-%3\]=g9g[Z Y|cQAbѯJ^=:PΣ$*<'\q֏P %24߻~E9Uw/EVmg,H=E]7.c45PKl)<l"d#+**~Sf&L[*[B]Zg۹?%ĽҘ2b o+% ZЅ}9J}KDHdۜ_t YodIؓscܻ1ǸNfLJY+pǬ4'͵%6x1>mo;f.QE g/1h%GL Y9ɩf{] u?/}y,lj?_-K~+ꡣu&ѳ>0U5Qk.=Brۦ5;Qm[G2[YO>? `Pæ5UV1(hvdU+RY_jFk#6*>m ’eS=),Eo FD%VqsZ'DRIh~O/1ᖽ e>֟ !9%ܾHI,1R؇!<~⿇o-^(8F[5<\2fckb]D+o̊s@K1r>,[[JT,6yPr<μD6'He3'/DrO_UDo\X'dA̲vwǻ8)y-Zdv\m c6/)A'P<~ 6Fzǰ1f m_9ۼ+ċM\c➥/$$u,1mkE)͹MPv~$(wb}kH;c٢J7/urn5ڻ{_|qNuC;v ظ\A~T*s_홅BHu,i{c!+17SLxexD¿A ~=c1i<7,?d]c8R3|1Tzvp`X$>%|հ8`|6j9w-\ͮBVH }9$Fϓ|}T~h/=d?/ԽFh,;ݾ{-&Q Y5o=>.)'7Hy}g,Zb!+Q,-HR$$Dqf^$8D$ S?4wm^)W~َ>;0Txc~M!-sʾ?ׇql|!x4EAKԵo%d}2<MbR!#Ѧ+d7%(|x33ņ\zC0?9\k7!}NW,bTO4cښW'F QWX$"Q(eO(d}MBU YٯPO fG'%O(D1T<ږwLxm[F c[ru>[cV8d |5![^e{!+0"Bk>VEr[>l?l2p}oؖ*mO'~c[jݷ,{Zea~*. .qfvc\Ym>h  ۗlB>&AEXVvY"bxD\P/m!pwU#\'1wƪFl8LL,=<+BVLP%&dk2!$U ЩnhOԛQV"N*iU򣠾J",ji**I"B{ B}8*vgGmb׵]J&sp,}c;ctE Y2.41\Njaf}~:ܿ|G_Y(ׯۦFƗM׽F6:.8qo+l$]<6F5Ld8 Y,*\/Q|X$,1ٝYXr(BVv'e+{)%&1CcNy'dYJ•'t'DR4*\)2I$Ui&n<[ Q6m*+II%? OZ y9{V$,?g'!No)le_IɺT!ĴY2Y)yRh/&_VbF |!EesЏNט{RVY+\m!dy{Wp/{1ʿcMm V_9u ,R}3+} nx6//沟b3BBmƴ9D?SC+G`8}'L[*[ l;g 6T@9@PP}Mh܈sӇHq@Y"E&𜤊\Zčyn_f d . _?"?,W$:I!+9Db_&!#ku8q$<(-wLxm~+_Z1/K>M&n݅k+"9߇{ ) Y 棈t]4: B̖u縫ڧ`5Nj,@~q1A=3۱kmv )RȊE E&Su/m\FVxD4;3viL][K,k :}c Yֶ)cV/ h!+%4*!^6'.(bdQJvvmA/%:H*{u7ŦYJ BVJPӿmoC婎B,DY`lX?Yyj;OJ 7aӷmWo~q[6fj"!in-ȷ%X.hyG0nYd[ch"%\g6~vr?Gvַ޸njmr6׭3[{p2Y{ј3" RuɈ63=!},6%h%Hd"hwL$xK: YDH_tA!븄/xǯ4D,,!8n"@' ,p@' ,p@' ,p@yr5~fx/zxp]')`/J1)vGV:7W=/^@oW~~=\xe'ƞq&n.{뱸,">ѵgq8!{sv8of|;qB϶=Ǽ-`8s%^zl8mcQ~=Usgg;v KeϴNw>{i=4>b}}J}JaKqnmKsnƭg7ήBp=>Vv/7O~?9&>b_ZļGҋ~/,Ã>[GnGN&^oRF%>z5?>oƫA5ե.?8;xRoȣ?uꔹӽj?ξmgݶ'l7hx_~g $^?j̝}WTGv>_5~8W",h$d=_6GL <&s-!cOlPO'y!z,H=O=#eVR"aD,"H"'řIxEp.L <޵{\uJSHZd;lxHKi<я_6ȳpI,*B"TdD_Ʊ(j˱4Jzo!wApJV~y.{Bֿ{f(jQg/5b^X8j8KKH.oUz_Jf)1LIo^urqv bƄjQ8J O{J2kA۴Fu Lld1^w|U1\ĵ@Ln L/d4l7N19MsmɘlJ9|f-Ico_ 3v>항}7ll3J2EMD&KDݎ@{>"?,Ec Z~(o5,T㟍Y>#JXCIJ5ޏԼeEe2:̏/O:ϡO6J\>{{zuR9u52Qm/mtu߶_MRUĪB7ο _[HuęSl8L8&}?Nya*_󗚉U97'|XT/wrp|qVH< b0? MqE+v)&βWᙃI$A čmgcڲ oGI~-L46cb|YsTLƤ5f4U99Gv.Vw,rϺ_wYY4lT"#c[*=.cz椽)sIc~- ~)d}60Ϥl ǎLa_Xx[Oو]_~u\h.y\v2d;ڒfv0¿I`)5"Zx^cۮt$IIȚ_1v1Gf1"DS/j׊q޵=xàwse@\[XЗ$[m>ߓdPR_B?틘ik:`^s0?5!?UE__cqn^b#iSlȹs,[6T&l|n8S²CWG4^Cix=_<@Jl%t:/:٧k̎[kOSC̎f_R&[úymS2CGwl؞I?$fZ4fYFxcFsk\ƙ@1 ?N0υ[؋4\|닃guo&AϺ_pPyca#}=ybAY6[>]Ʉ,룺FTGcۅX];w;9vͳL1\vL=*N`_1KyZWf9נc5sZ?5l\O{I*\њF4lghχh}Wyam}M,+|L $DE !ET#,%3p:_L)Bᶣ_5~%!FOcU4 o.q YzVjCPp!ߧ$!I>m)1n*f (wJd[OR %30%1h}sR"%<9۶U݌9){x?;ע1%tx{eQ1Kౙ5faygza(`&w$޲ػs֙RE'cl{V a瘌Nm^,uxTeN-sxm}˴Ge"^h^g1M35Q5QlҳR9fj9&_A\xwFVJ8sÕEݶ[%AuA/jOP(N\i~{о)ٿJkr/g/>{vm-p?g|Jb߉#nwe~{Lv9f!+V(!ܛ O~oT#v5A&Am$4fE1Cc7}k=/8a4?z fuͳp5p5pZ9m!ͫNR+{/R2pƆ%$$\k-%4>gssPػ jI3N^ix>dHbqokOpR9)ǏcąN>P#qOtR]eǶL4dv}[—1=D KH5?7{)*BFoNƾ?dw>6ߋX~2֭,r=Wo|YDܸhS|R2.{Q<lT5g˱Tu>9fb=r9g kc:]8tŰ-k4u3GYK[Wz}s<mfU+RY_jFk#bj=BX鋬q5>r{R™Z_45b(U/rҏ8yztìs$5֟_r]{p"}|:,|WȊ?oc/fǙ-W4b8mlkcC%;JKDч0׈f%T1/wJ J"x%OLgشM<~g!q:H Uަ]:Bkxv=aNxLޯ1cv349/s=,Uk]*4&dUzlW?}bi+/wvc΍W?_ ϧ3kh{9ojm6Xce3 ׬XhMyiʱRDy7csء{5u2wp=Reگ֘e3梭9;Uv%yc4iQڇ{QaP#ۯXۙom{Lhf%>s}nHg?`bQB#E TKW0Ŭ^.͠]SR!ڶNߘBm&ńn7)ZVq6EA~>a:{2{txPC=lh%ž| iЬ-ݖ .JO߹?8 YDI`4#,o~Ѭ9@ϳ²x$D\.K'*0y%_nULci<J 1OI8?Axy|"tU_lgW*ÇǙ7/I|DIusRilIKĤv mjy%tLxH65q /^orkֿR"S1S$ŽYuHB}2P6٦c_=޺Orˊgilk/C1{9KLE=#6my+VsL5+8(FjF%wĞW"bMafֈL,omEvΤcȖ}NDsh]ݧ~Of3|öS;/'b КQfݯ یM}|0YhCM}W} Œtٿ&4lLyZ_w{4js/l~"fĕx,zV8}pk޶8'&IDNߓr8X`&  YDH_tAਜxRNsSKB֝V n-~8EB^xӎ#Xb d̉'u)4$ƭFBV7ۻs;lgG孍3"o9fWԟ'9GB8I dB8I dB8I dB7͓7Û~y/Û߮s:g^WOL97Cbbc+ۈ "9ap߇:Ǣ{/粧ΆwJe#>? iy6|~mc{/#i|UΗ1&pm`xW{*d!g`?Lߜ}1d/_ -We;KŹ_h}hscp?D\X>ãTu`x5WT#83u%G|$; .i}{,G5#5ہF$!gO=[&duBxnbEˆ X8BN&1$3B'&,%ͅ"~2*+p~َ>;8Sxc~M!-sʾB@>c YŶM?YZ,$M&n-ApJV~y.{Bֿ{f(jQg'8H&f--iOY:A(8'CP& 9Y>%ѫIF`m- NiLN%TrNZ:ERk#T/>, y1] Ymї3oS]w\cuB֒0r>Xnc?-QB/XD(3ILT T#3ةL,Qv ِIv-6$>#|-VZkp.uGH6+=#i"8FLknvyڶuJHsz,a9vKc*dYaRJ"cbض+(A_m:!+yht\!kfEJ*Mk6A$~ 68SR¾({צnt/Ӌm~'R>DŽF,K} u ι?){R>(^ YtO?x#%hmgXHuȸyJ+qxnSLܵH jq͌&cF xMNl͖iW'%􌙸wv*1m̜Xˍ tF B>RIڤ7V_ IMuo5\ sw&c4g91VKf|{Uí~uPY o^ka[9z/dşPgi]mB KIkRd(ѱWbp-k:ͬM#Wx1~}VϘv]ư xiضqV.eI(/bYI؉"fj0 1ͽydVHQ1 8Ay|?o\wJs\[r 1mӌc, M[&ĠML/}pQ&㽤~H|INݎ羼~Ă/UؖhÍ~Q[:Y9]+pYp`QS!/dYȇ޸?G}oQUle>G>y,Cb >zTYen̖ǰOs5l]AV YE"~ly {oT~1UimOX,W#'E" k}*k %dmaO,i"^c-{1'|?'B"r]K}5YbN-YBy>+dKߟ \X'dA̲vwǻ8)y-Zdv\m c6/)A'P<~ 6Fzǰ1f m_9ۼ+ċM\c➥/$'ltʛYPe^B;͹M6~>$(wb}kH;c٢J7/urn5ڻ{_|qNuC;v ظ\pD>*NUui=CkE&)yڵIkHؕP]oݻs$,xO1UH eo,q0c|ShuYH1k6~[.YlBK.B H_`{brz!nm *u"y%BVזiPB8WU` d0} Y#}tLk׼ibN! "3BYȁNYNB-D,@' ,p@' ,p@' ,p@yr5~fx/zxp]')`g_e7nZwc|cNИ7={ʝ$ÅWVbs̖$RN7> .~8`|11V_k<]ArI?OOS٣WG QRͧx︞_jtͳD&d||j?ws>WJ+ޜ"qk}ފuWD?ධR"aD,I 'xIxEp.!L TWI:cq)$-}}v6<$4dكb/a2]bHoǠR AD{<עmxB! $ς N _e겐$$TtR[JJAˊddIhBVm*}w.?>ξsxcnvnؤ?O F@ UFA]8!"Dގ@b%`n36 pXvK&1jhSE{reybC=!yJe:l®?Qf_Wb5f_ٞ1UZEVĝ3#}GaW!)!hoX bUU/zOMmtr5__'dl]!Ke*_!+&Hb%%l"MMJ ;&2)~O>}NW,bTO4cښW'F QWX$"Q(eO(d}MBU YٯPO fG'%O(D1T<ږwLxm[F c[ru>[cV8d |5![^e{!+0"Bk>VEr[>l?l2p}oؖ*mO'~c[jݷ,{Zea~*. .qfvc\Ym>h  ۗlB B@2+ )I FY K2$BuRm!5~%!F-X!/j7oc5sw-d%XbK qO&BNR%9 k|L>Do%m(⤒])? k$BI!Ϣoۚ,! d dq;ч[qbw&O z&v]ۥjk;;Z ?'k=f|3 /H YDܐ%}Osuk'g}x`[rm}_yjk|tk:o2SOj."F~u ci'[DN#۱D-P%!D}~YD U' d,T76&jY+\!B?[Bnm+1^LyNڽN&J-iT'RdId5'LxlUW"ד$6J~ d $,=/sHX OBRʾu鿩!B7i/\eRf'8z^L K'3ᵝ|iplC41 VB:'&d-:^ uc ƚbAFsXsfVnmbm__"Ie?žgیis>F!>g!+V@b#n)#~>F{˴ER%ԅu3qM}Pe xuMJq1~n)dHq@Y"E&𜤊\Zčyn_f d . _?"?,W$:I!+9Db_&!#ku8q$<(-wLxm~+_Z1/K>M&n݅k+"9߇{ ) Y 棈t]4: _c1Y#wJ LɗI37HdI͐~gm ܾH߂bBkaL%_Ŧï%O5%aqK~e`52Gb_T/lݘU{ @䘞:.'og3ᵝ'%/^åyL&[1 ~Z{%會*~jBV8jcXoK磄~z)TҳŮwo}Ұ:>Ggm;]}>[9^gh9؎]|l+ضcnOBVL,Jlu0D0qD {}Q X% ۥ1umvq) YֶuTʂ/d5cVYXŀn_X~C #S /_x Z(%lDֶؗodk$Խ庛b,L%Q!+D%kp Y߶R ! ZZȊmTGD!KWG,0I6Yw<'vdf%ŰԶnX[^Ը-35_] Yہm~,ҼΣK,nFj4nc3nc?[;fja;[o\cF̶u]}-׽nzRq,ǽChVj|_dD~řd>XfF1N+AZ'@oõe"[ 'Ցtm@'B "Y%|;~q/7 bF d8YGss]B8I dB8I dB8I dB7͓7Û~y/Û߮s:g^WOL9{Q\L;W/9|܅+;13n_8ww!6[Mg!qeT=;9ޛy3Gq8z9mI+B{gFhw˞>;)|@hx^*7<|muڼ#Oi\[SS(跎wcUo**ztv.o~h~.͹7Oג}U'8r^\{a?/8r7BB̍&0:bWaξGWC'a:8;xRoȣ?uꔹӽj?ξmq۶'l7Pp.]?ױZٗAhŵoY9' Z; Y}p=aĽ窪 -!cOlPO'y!z,H{}=#՞uVR"aD,I!'xIxE.L TWI:cq)$-}}v6<$4dكb/a2]bHoǠR غ+x 8%+ABO<=!?=3f~H n/,y5C㥁|nC$؏g}гb6ӂ%T" ul Yy;7շ_ ?,Ec M~(o 7,1#9eEdk+!EWb5WO~q_0/PFZ[G9F`q|yO/N*g\:%XޢIwX@XUkK8qgsGD>/,a?4pb%I֡ 6gKSJB! ]ۢ/"qo%A{! YqK뷾8_)g3 tMVDWslL[=ѯ%cqFqL /k1j ԘƌƷ8w-?~v-s2v^Ykq3+?+]YdDwlמYl>.Ms^PṤ1t>{mg6wcG{|,Yo-llĮh:.űJFaO)_7SgRjArޯ }||趽؎m4i O.߇~V}70~{]dW!)! oX bUU"1 +/rƶ]AI :Ku9vYȚx|%[Ȳqa/cOz.Wl3ā)0CY:VwAMuwIGz*|ołSֺOg2!]pۢl=3_d{箝;LJLY&qC;\p~~v[0vܯR<+3Fok[CUܱ9q-Y wyl{h{_u3_4Cm g{|g>>]z778M,00 eV,+ ;|RD `eH$_L)Bᶣ_5~%!Fq~v!;_8ǸaR qsʲü< zA{}0*fot4Urwԅ,*Oū^a/v#]8[ЯBb7ٖ1܇66&X4] o=4UŗX s]Z=%_ZK!ٟ&6u;֞㦒HipBM@Jf atN8y`smwM5ls|^4Oq^ע1%tx{eQ1Kౙ5fay {P#R ܅`Ȇcr۹53bݹ`wqAfmֽipJֆq`}Tvk1-xy{p75Di?D=W)KώKݧ昩#8wVq^m3G[+a_ Wbگuۮo:=y;Cy}nga͟s oNzS?N}Cl&dC TIQ_R_V.ABIYb ]XrD(BVv' d+{)y/ IVlcycpHD!L!;{}_Tá߳eߦr'iGIk||/ȎV~5\˗Vs< /-={uܵ!q$嗨 J*-(/2ɿ|n~X=?XcG+XbmqSr#[;(|brbdN@GJ&.#}mφmU7cNذIRَY5=M2Ǩ_*XyՙЇԷؿ➓a>rw>{ά JE'cldoĉafw1wl >j"^K׺l=@F#r.893W`Z5^~y=\}*ڏim3gr(f=8okOٵu_O k)mGo|'V۶Nn$L_Kl8ا)|9CTc̼[PYrel{N?!^ǒnm}d;Ŷ9]s"n4ߍ)>m} 6uJY3L^*:3C3vVյK~vH{B .vbؖU|UZ]虣{ >,mE~X}3,8mfU+RY!–(%XDQmSUVJLV},\=BķПR<_c1Y2{ .?.alxX~ywE-?*k}^!+{X}h\/gR_шQᮍ _[+6{Eœ͂^̺/^#b՗POBq{(Ƃ{;F{}e{ƙ76=E|(řM6MucpHݗjZczxҚD;V gjh;>k>\_G=CilbmgIV3Ǜ +GaMVxJgγAV YQ0(AבH}N5D1+`hKD3(3+lԵǥ$dYF0R*pێYalRLd}_c@o2ӱv uq`@eg;7gj:$vmA/%p84{u%K߸w}B,kwX3Lj,Ǜ_4+vc8쾰&)"/KO:Lc^|9uץGrXﯬyA;hǂ_BUrΏc^/g#!c8J:q&7&jNұIsm{$k/ځ<\W{̘,c"Cc(|zXE1&jZgP%{Ծ._Lc_=޺Orˊgilk/C1s$p=Ʋ8uvGN5+8(FjF%wĞW"bMafֈL,omEvΤcȖ}NDsh]ݧ~Of3|öS;/'b КQfݯ یM}|0YhC}c?knUm-m3p|b0At?=I*f@qϐNED,ʉ'u;7J<)d nUmfx S. dw8X>,B͜xR@ӟxKkܚ>o$d~#;36qvT8 6cvEy2ON @pD dB8I dB8I dB8I dp<^y3yyry=j8}pĔp\<$6/1/jmM`!ž8q_Ϫx~ Y }xysx,?r.{lxT6nq{𙶝ig>6q_>Ƨ\oOOû:3ELWR6=U:B/7O~?9&>b_Zv$ŗs !k!%\U$?|u|G߇u?Ľ5}^%^@?ÅW7uBֽ{Q,j/T|{8/-HmhۗÃ?G*@ nU6XxvpgttIENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_1to8_train_output.png000066400000000000000000000315571471400134000345360ustar00rootroot00000000000000PNG  IHDRCnsRGBgAMA a pHYsod3IDATx^5Ybbp[<6ōq!jo Bm(U]\.5B̸k˴żr͈#D8?~?p97"22NȬ<_SA5P,~ӟ=|,!GCxjP A1z=_^rz#v:җ;pSC'a[tǥB~`ow5-Tm·}8ޔO,Zz|[^dܸ7W]ϫ92j1}PH#Tm_͹։zR2iM OX$0Ц!OS']'=*wiߴAy˸ZtC_@Mۘ3鿾)#U^?FZ|:5|t|ݓΊe`"rqέUb~vԹ<ڶr)KQN!fZz.gA>URqqy!Pgd;K{ڗhc_>PROb^޴/ƻoo*$"l&,} 0*VqeHm}-O}iC%|'>ؖNJ:x7p_E b" Ϻޯ7%'ƭD,#3q!d_o´9ۙs_o-}Q EHVIoZX ZL!H_;}mfd\lcc]cK0w۷b%oOD+U 6&YΊ5uݬ od[\H3\h[5JsM21|f3mk*C/l璎˗@ΧDNisHy~nD\cm:G>|8'?9#34Fߑ -Y\U[zŸWK\l'}[>[N{kyq_+d(t>鱉=c)?N?ǣl/['kbY2||xf]4l}Xn"$tu6 qs ! R!D4C1rb;-O&/n:{6|1ɋN=7N,|w?l?K-&Oa3{|y]_^&oO= Xă!=Gykϱ]68>=]O'1o*x4vNoH/ZMs'or c+㑬)iKι޿[Z#lP !z.}QPT*Cq65<*Ȯ8w憂zj0< (ԠSbO !<5(T!4444444Goo~@ 1n A1@H>m N u[`>n!@iqzL0q{P Ocyۃb|@|C4`^$6-0 |^Ǘ777r:kN/) i\|lz΋T>)&:?Œ7:g_/pLcM2M)NWy Os:BQٻ{;ŗSq&f1r8^]%OJ'^,ĸӶuNi!!e8ևi&.4|M[\I|Ly=oֿ}w-@6dqTҊW^r㇓_ r1Xsv>Cs|Ov=q?n&h(C3qLcг|%S 1}z.|S9?sX{i<1+D""$b,9d }?>ƿ-q&-^8B 'm֣&eR󔓃 s|L :?'Ea{[ %$y`$N:d39V4.>8QdjkD?)E5 }&ךsq1DX!'AM8O)D89L疕PB"i<“Og˸$u,rM૔%t\^ Ѳp=>ކ?)m!bo] ͱ(vUoGnD&Dd&)-bsI)܁%hE1uw! u$swݱ u֣ܶ~kmkvHi^7~M%V̬=PLk>(./2ʖ/ z!u&jŐ5C'\^@n#MÞҼ.GbB8B4*e_ph H;IH X e̮glq<\T'|RunmG"F[~nOrh| GzΉ"*cJb埔]cU].l*l`kԒϽwkZq-V/DQ zt 'n|h>&mb\ @|C4`^$6-0 8=&m=(hɧ 1n A1@H>m N u[`>n!<5Ê{whk2Wh?V _p[Ct?hָl$pC"⟝^6{Z1DO{ #-#=XA .tV Q&ڠSj1n7mC ɖcp>5*~{:A12(pZ܃aw___H +!<5(Ԍ+~|>6m`C|v>ɲ}ϸb 7O ĸb(W[@/h|9_}k} CENÛ77k_\Oo/ǰu$?2v_/nh b#FFL/rn G( P yX1aq=˙sB+%/a_[m^p&vW QpS"wK1pIɹƹDwJQ> %_u Muf=J?hƳگv8*eγkίEeeb' ٦i}o`P/}`g-ꄴ.&u!yZknPd~.$ B(^gsw#|-ӁUي/U&R8  Ɠ8>vd(98P J9,;JT}eZ_gfNޝLwUT_L瓀r3w'g9%oڡx7mxX7\`{MgGүb!X%X>Zm.֕XuȿߦByTm=p\@ٓ8€eu&ƒ31$|gG}+(b%$˶,)lufȿb.mVV])l@6үW㬗 ;yp/dڦFُ>k~}C4£+$.p~'_g<ݡERX݁nW~.妤k=^~z_;mvkVsL؈X_؅l>~qлENK#] ]c\1akCwCWɻ4O.'d`<! bO !<5㊡ߟ߿M=+|e /0q>6OO ĸb(W[&5>&N#.FB/DYKI~jvso.<.]}1;_p&P U #9Og޽ؔ.a[x-$v$y Oѭzfw[`>]1D9N -ń'ED'\zx[uӠZiKƓHi[uy;_wy\@l~kmS=:w+_y<DfF\Al ##Jr4Nş( z t1c75얞 E 뿪ߪKr!|Q 2Fj[e9>m%_~_a{Mǥq*A'q|Prp>Ar*Xv5W@ɬ~៑;yw2UVr>R}3k+,Z1Y=sBnIУOrwK޴C }#m7|o^oϘ_a.CrA'Jz|l@\%Nfhִ535NT05Oʺuk|8Hs }^q a:~w^c}1ޜΠ2:65Vj ;yp/dڦFُ>k~}C4£+$.'ۧ@z-QsϭE76 <ݡ1{MNf(t)7%E廠3ߋO𥉉ͮ|Mjݓ K g{]0ݯ0zyii y 1FÃǸbì?|HtHCwiF#\O0y>C!!<#32۴ȸb/?YW QgԾP0N+rEPO/a^^0M*A 1ד|חxwIl]~ydG>Ku/[ ݌]x?%'1]8 cd< L ntbR55Y/| n G+(8)p\\"K*n %'*rӠl֟ҏ4ڥqKiߪʾSd{ [ksrȿ]㡰L$57 /g{]0W  *8M&. zŌQ,_H:n[z*)~.Ʌ^=BD*<˿k>n1bV|}5I1a'CJ0Qʩ` Ԃ:zRm'JoNޝLwUT_L瓀r39,,|M;1v Uf l H] qP|B?+y4b+)\=z6?535NT}wRnJ:wAgW+N]\'26"'v![`><_a\1|HBbyW }~ih. h #Ǹb`SbO͸boF"㊡o>;dپ g\1DOSrC;1A=t @'{ͳz}|4bz^i~ \g%MS3en=yOR]#&~/rn G(JU7W|Gb]8 cȷKӮ?K!oX|g ~l3-0MDb擢ss .J-Ӕ덛G(bԺ2UX.ֵDMU}T@<ڣk'5qCa 1HdkoE4^`>2 * 8Tq ,%HVy@D@m]-=,=SozX۫ l.nx8^s:жY"[פz\ 'bx. %*D)uYaCrZRm'JkoNޝLwUT_L瓀r39,,|M;1v Uf l H] qP|B?+yv\%Nf\ %uYO.Ѥnw+hB>{򵾳r'BlNݶXr/DoE$A%rRYC59-t;dپ g\1DOSrC;1A=t @'{ͳz}|4b~#.FA/DYKI~jvso.]1D9N -ń'ED'\zx[>'G](AG4(άGGҸ%RZ5V]W}k9~kmS=:w+_y<DfF\Al ##Jr4Nş88]\C\O1 ZmvKO"_o% o:'<˿k>n1bV|}5I1a'CJ0Qʩ` TT}/Jf ɻӗ鮲j<ꋝ|PY.ҵbǓޡGdiA"Fn޴a,s5a1ICjw%W4(mp8ŚJ:K:S!X~p?W}tHn44y`4Ʌ c\1AP A1f\1iXq7_~l}3'}i`W 加v :_G>`>@1TDbh4Y3&/~ͽٺɎ}1;_xw[`>@1T^M8Ogr7tQ\,!B]iWAr⥐7,ktk^?p&vW QpS"wK1pIɹƹD^%^}҃b Q13ZEFxm>M[uy;_wy\@l~kmS=:w+_y<DfF\Al ##Jr4NşPRk=: kQ_}Oi6Bžg|R/ !`{d_ߍ5s|NV1Kd+TTKA$TO員|B(TZ& $+ȵۢO;}*v9cؙ'"]+&˻gn9sYYNnɛv(/bM֫2^+~V.(DI/XPH]]/_3SO.qSnw+hB>{򵾳r'BlNݶXr/DoE7%i򓉑 ea#^s'+z<1X2=YH~.Q=JB_W[a0Tw!^ɴM~} .&(iGe_WXAI]@{4N'jn";T1f˼2߁Vo&n_m]ką]J~z__ߕi[͵{2!c#bIzb~al ãC>9-t;dپ g\1DOSrC;1A=t @'{ͳz}|4bxfk;Jz.iⷼܛ/숼ѧe9\| C%‹ ztw/6zb!t t< L ntbR55Y/| n G+(8)"%&n>) :98rS~Pp)ӠliKӧ~.o+N9 o9xʵG"Ntk22bl߈6hwu|d_1T:@*pK(G4A>XYKP E}_=Q>얞 ErNI{,Uy6}7R|b</9h[,R kR=.P1b"25Oʺuk|8Hs }^q a:~w^c}1ޒҕ eaC,$@^s'+z<1X2=YH~.Q=JB_W[a0Tw!^ɴM~} .&(iGe_WXAI]@T}qdj\Ztcs`[$HnX׵!BզߥK)1d/f?V65mvO&dlD,IO/Bwu|xt¸b"..v F`10_5!ݻ+$D>0a<|+6!<5(Ԍ+~|>6m@+2O/xCT@1%km3b^lJ*s:|:~ w]Ei]⥐7,ktk^?p&vW QpS"wK1pIɹƹD^%i}7n"G= &@QF-iߪʾSd{ 7\{t~-DW&8.x(,;!lͶh ~^GFC h&?k PQ{t֢/(Zmvb3UV]I>|Q 2Fj[e9>m%_~_a{Mǥq*A'q|Prp>Ar*Xv5`N%FktWYu/vI%/\kdy-g4|<zIYNnɛv(/bM֫2^+~V.(DI/uT[ASY$*B~ ruzM:K4_[Ƈ㊴8گЧϞ|ɸ,ۮw5|0!;;[A]P&&eZZ659-t0a<|+6!<5C?'t5}kܗп;wm?\1 "c "bS5*B#Y/sb.yGC&6W 5'¥Oܕ "?;97d0-%]$?5oy7[_k1&g]Kdzu9.|EFNċt?A7"Ok1^9[H+0*H**ɖlb~|4beJ/pr㶽7(|PSwYBsq!oBj?3%4}xX9Z/&\jl_ߛǞc+N-Qp(.8߹xXN8n:ԓl 8)H]>sҼ[uy;ߵ1\Nzv9Ǐ|9yy ])e b!mAY_w[={[gg<3LSHa$Kc2F>øFB03xbV|}5I1a'CBF ]&e &}Dߝfodk`ɹ Hݢ/d Z,T>OD-c_kё=3'Jqw/]e.wX/vfͫr1Y>>fgOF㖼iELizYk>c>~+ruثor%P%,D#ARWۡ~Gd^" Cd~'X㏎>nu _x:joM^ЧOWrq! X]nk`,9*d,"(*J]O%g[,'ŝu;(6k}f?Ec7y8|;YAQ`%v9Dr^VV]7&k^W1Xzne3i6f?l ãS3.ZsJ>QrbO$$'dC gf>g'K5|>gC,qFe#mHl YK UQ#G>nylʫo#ye.[bV.]^rtdEdb_XdNl)s8Jִڨg{]0ݯb.~tN&ѱ-bkii!Y;>t5z2b|r?7;&Om4<Ë!?ۦFC iZ l CxjP R%оՔIENDB`llama7b_autotrans_1to8_train_strategy.png000066400000000000000000000374701471400134000347610ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_CkptPNG  IHDREsRGBgAMA a pHYsod>IDATx^ͫusȟ88?pN6w$8I#܁If?6Yk&ՑA_HHr9vUZU|?YݫWg//A (M8p(2((((E^_^^o//iNoo[>+˗tz~+K/-R !Us4-6ǔ_|z|v{xO==x_cEQ~PmYҖg']p /A_nNy*J?ܢ\ۮ>n>#^[_zdYz硢 =Q9«[NU/Jɿ`h}Qz:fl)$ (6^x)>b/aꟷN?]ʾ.UGEu"R_m˲՞ȶv}IO$,WUz^<~U:_>O1XWmwg,euZ>νû~d/ᡢ~2xr:~C^dtL|979+,/i[Ζ|lIwDz/ Sw}ek*"Fn}N_kV9 ux=~+g &_ bwol0rTsvS͗(b={>#X(ƉJ4uNrW|b΂lfVt~OWgbm֢;lsfi^| J^aƳU \kb@9fdOm#@,S;CWCksn&՘{pЯK|VkV~R5E21WH$ԑmqU+Ӳ$I~a{͞~fZb)D>i:隰yH7d_,=쳋"gf.՘AVMc=|J+}+u,sUst-2Irr6l^>sQɖl(f_;lU|dRP12z9<_rl_&[ϱċ*91 1AMgx_yk>ivwJKFA9Oυ?5RY "b(:(yVB<DZ뭮c11F}%6l_"R-󖱁Oati/Y^^+:5ؕdl}e6\keyoIBn<"5͐e<9I{fZce_^K׹n~%J斗"M7ȅQ "VEA tQ@(·qơbU]6] ci"~ɷ((((( E8pW?$fQ?tZOq=ǃ ƀ z,E ?]X "&@~&@<>EL$4tMcx|<(Ih xP05A\(`зח~%]~_W:Y{}{yy W_T<}'cx|<.^ByBft=%DZ_zP^빹/?Ifwy 湿rq9;ݐC؃U(z^ī %pNZI<4}Wy ӹ]3gKEɰo!l{7W~W}3Gkg߷:N$*7gs\ X>{_| xh=|ُ򝞬 2OB7_|8E ?Vkb"^U8yp/8K/S_=(믷g͜]?X^x*3"rU~Ͽ~ϫ>{^y e6sGM@>ZI)' !=6eB['L|ZII"-/2WE5Oddoq&V鯴9*e >1Et[}-$?)cfZNMn XI'>S#Wa+Xֹ"&= =S~EV()Tdy1TÓlԌkICjz~ֹGoqɝV\OCXr;o/y[z>|w!xx>(_=J<:6|6|ˍ؇U r_RScUaE5Ϸb35%^_F1k//6OM+uFȂLn{ TXR}]r_P율( oMnU[i\y;Blxʽ̤ROU4w6YvsZ[@ =}{EEL5A\(`$1k AQHBcOq=ǃ ƀ z,E ?]X "&@~&@<>E͋??}?'E~WůN۳yQW~?C'-EoY^͋[:xEWG:h,3ؼ(8"(E߾}{޾{H#nEѿ?ﷷ?qآQ`;]0V( |,iB_֑lO kr{9 ESѾ8~rPO͂AߋKk_c~,y _c=޽v=%Mgӽ-\D_\s 7ӽ/!?i"r\o줔 <9!uZei|KԯH]b.-G4_ ;MFtM<|b'B} ۟O{hDFmKThx5D\j!VnIB/zE/Yŷҵ.PqڢΙ*9f'fNpbJk1+Zz-d aj/p C97_Uk7QsN4m+ٸ퍰(kLjO?O#{Hb&_W]/&oR]棹ڏخdlkߓwOڵ|ՑqY{6#Uõw3Dbr P;ϑۖU#6Om.KNWtQhߚg%h@jqxB:~ /zzDwYz'XەWG[<3{Z 'p}NKΐk2YZ۲SœiȱzWqTk`TqIس˓#I_>XkV6m/ض+^o/\~"Y/،=eV9U7 =UW7v#AkZhpH~۬nrZȔwz,O(QO/Ak!_Ws5?V?7[jz@UgsR+g^n/v<'n7sewQh6'gUr,8{辰h\cygS~rH@r9QU}d]ݵsHvE#L:Yg_S+ tNrx(btuwdWZ}Pح(3C>T KQp4N~[r/gF،EqETcq,[q޾jNB<8lQ$Eqcϴ)v{fC|waNi? ɇ9[m/z5#߫-jwcE9ţcϴ)<$#~_I-RS˳|#yOԽRn8u7^-WƷS@G״+gwEQ -zPS_H*T8wU!Q_M 5$S٘#Q1"m]\5 _R/.o|xZ^˨Eޛ|Ϧ{{[S`,\g-]o={_C~Ejcp9ayr]q-C C2 Ӝb5ب_=Ũ]Ziu͚g%h@jqxBIp_(qw'vC[q?1w]MgrQ'';Č;6ֽ'9\Ӓ3d0L,A0g8rCU9g876hjF$a.ON?4K_Y51`Za_^Oy. SMF%f}If5ַ[csM;aV9U7 =UW7v#AkZhpH~۬nrZȔwzXHE=y+c|ͺ^ͩPTX(Z@UggRpf5 s[<%n_l<{ߖUr,z"ۃy!owޚc'T吀hO":7sPɺk=t -Ft!'(ϾW9b@Q4<YWt[ھ\(VNsxV}d*BE>>_My-٭(j}~O?0f(GF؎CE(,(C$cʟUs1a"y<(z{M?Vr`jlJh!IWl՘RsRWr$*!]r_HkW%CnOoK^M'+_g:{qqm c~,y _c={ն{torK`*엥?e{ t apڢW-49 .' >O"+yH]{(VcYt3_1R<KQ?͗qW6Y;vOlr*?}ۉUsֻDWcKA5: 8諽9-Y"^be1?+_T[vCtץ~jAW~8:QX^/Rd=yWƋk#:vmF%OjOB7KD8ߊĞ0Em+c)-NN1vOwl={OƓs> %Iig`5X,m)aδqX=jNsθq50olԌ:Hž]NCf'yD3|P8~xsi^oO7W/1HVKbt56#ek ӷr͹HVYg'tncN  pZBCfvCBӓBC*\>h-5kjN"gDq]?F9߹<'n@h6~[b>W:Vɱ` TmCkl=V 5?[~,$\DfN>q`*YWwg}A<E8$d q<\,(]c' v*oK7 E@ʉ`{݊"=0tCE EG%L2ן~Fa8lQݏ(.P`+YAQ>I>Dz?7Y$cExJQG?:LrVF_eܞ &||ӱ ceOCp zޛ_w{[Cq{Yx (5xt6嬭R"!۳|@-A)Y '^Gn8u\ [ R5bo1꧀*5iWΔ$g1ĉTZyTCq%B$4:XkHR[5|۳1'{%G?b%EH]*xr#fzs ۼWmVE7OrUd,x~K6z㍱t?<^kն{torK`*엥?e{ t apڢW-49 .' >O"+yH]{(VcYt3_1R<KQ?͗&yC{ze5#nGIka'7Xſ%*4^w[ Xy_A_!VnIB/D/Yҵ.PqڢΙ*9f'fNpbJ8k1+Zz-d aj/@}nzNWdڇ~4mȸZYc>~ZvD ׊|^evr|z{jI569FcM{H1mSk]=n/֮뮎2U;%\-J8r P;ϑۖU@meɉʟEY==Z~R{ =q\]"‰V'yO.`mWnߙ\ɉ11鎍qu xrg$) L&E-;%̙6Pm~i7N QIس˓#IM<HOY5i6n{ح8~xsiSzDr_㭱y-\NkmE:B:?W]t:h71{Onw8i!mY!h!SIPb~!RM5z5ZCyS`38kal.oUɟIe߹-};(|s4Su-1+XE!56+vi?{MU T.s3'J8ѳ{N  pޢhIg|2RkqΉ`_.EN1U~Hͅ hD=WnE_:H`]"qwؒC~~O?0f(GF؎CE(,(C$cʟUs1a"y<(z{M[;,FngCt, ߱2!p_ g뵾)%tA{s뱒^ӡ8v@<@QS<:LA[)6ieܞ~2,Hu/~#7W P FYvLNrV{ C(_AUEJw I%<}^"$JB)$_Ucʷ=sRWr$*!]rx8XԵ!7b7Wu+3Z|%۟OY:{qixcy^kն{torK`*엥?ʇY)߰{N -qB#r(Zh5eN9jN/Q z#usQ|y^.t( >6:%qW6Y;vOlr*?}ۉUsֻDWcΫB5j/drKjz'X~Ywlǖн'u)~Et$GT1;)5wSY]_%\9SlI%eh}U{ssuo"n$4mȸ퍰(kLjO?Oގ6Z\V_«̮^Wy{O7/+5"HKYNݓwкiX֛:E}5BAD/"FWxG Ȫ] kBQr"؞+c~? $PQ0.E8;Ee|l!?SDQ`3[}# #lǡ"F؊CEtPalO kr{9 E*9z{tLo]Bcuǫ Wk^b,rOg/*َ-]ߡ{ORP? 5 +??-xIcvRj'fJrؒKV &Z|EV}gIӦި5^eoGtP~XW/UfW/gNjɼƛTcׯFcM{H1mSk]=n/֮뮎2U;%\-J8r P;ϑۖU@meɉʟ8kܣqa '[{~u%" o}bOmWnߙ\ɉ11鎍qu xrg$) L&E-;%̙6Pm~i7N QIس˓CI??ej<^mw\חWS|sޟKÔz}zDr_㭱y-\NkmE:B:?;yMvsHv~К6>2坞%RQO/Ak!_Ws5?V?#F06P՟ٚTlT~rO>\ٝl<{ߖUr,z"Uo{qkִ=&.Br9Q@ā}d]ݵsHvE#L:Yg_S+ tNrx(btuwd@V-mh.EA+'r3UrNJĺl;*pXKO%#̣UeȷMBI׽Z9qz.šg%7#mV~D\W~$55j%U/&oR]棻5"HKYNݓwкiX֛:b/}P'G%1:X߲3)ӷr͹yѿwrb$r Aَ9{x$ty?hM oUMA NO 'relԐY׫9ʃY c#vyOlN _0^xNvBmY>|_NzOtX~r]k<=; qXωbO!ўEtnDuu:zv#[0CBOfQj}M=7s9⁢i>ޑyjׯZ}Pح(3C>T KQp4N~[r/gF،EqETcq,[q޾jNB<8lQ$Eqcϴ)'mux QT/+C? ɇo=77+|qF{MjEQjN3mwJ̖q{6bXnT~{Ooͨ[{@o1꧀*μΔ$g1ĉTZ!,ߩ/$\9,}wIht~56%א+jLgcN JDe8ļKvUr9FҰޫm?NA~)Q?e,A=hŮ7Kcc)yo=Un L"slOwSa}A< N[㪅&GQ?tŵ4+pj,4Ns՜^b~Gv%g*Ϋ'sOkk%*4^w[ Xѹ!VnIB/D/Yҵ.PqڢΙ*9f'fNpb]MJrؒKV &Z|EVskl܎@yYD?3"+X~|^ev=[y{O7Ʈ_ yz ƚ}b%,ɻzh4^]] ױeн'l3Q;\( }wJhZX9#'p:(l9Vv#-Y=.Z%$S۲˒?hbs<+G/fNds1ez~k }7ĉV'ız'Xەw&urb@xcci{x0Y8-IJ;CbdnN ssPm~i7N QIس˓S ӪOmvšG1qۉo7^dc(x#[~okoYǙ[V<_;91]gU9pulǜ=A<ͪ&g񇠅Ly'A~ԓK26}Zj׬՜j OEψ⬅L T'6'J(Ŏm'TܖU'DG'Wg qXbO!ўEtnDuu:zv#[0CBOfQj}M=7s9⁢i>ޑyjׯZ}Pح(3C>T KQp4N~[r/gF،EqETl> U1a"y|=ѱo–y*,7+_w{w ɇ9[M/:7^ 狻klߑޟjwcE`͎yφxtƛoDyPJ7 .[~1)17WAr^ajUo1꧀*:kߓޟC-^^Bx &$=QMs9/o枴K>eywlOwGa}A< [eg>@809pq)ƻ1U@*[|@ĵ+pj,46͙/Vs> KWl=h];ZMYgvb姜D YfLo]B#Fjl)hbcFr35ݣdE?5 9GNa'1V[vCtץ~jAW~87;%l\g"M\V_«̮^wK_=^LU7Ʈ_ yElݓw?h֛:=GqxV}d7y]"   Ó^.}IENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_ckpt.png000066400000000000000000000133421471400134000344440ustar00rootroot00000000000000PNG  IHDRiGsRGBgAMA a pHYsodwIDATx^4IR?ă^<,^ד/aQd=="uƁO,mvWQ/23, eF䏊짻~t?TfEFFFfFTO@{{ޮļDb"QPPPPPP6Z$f"@"v 9cD p:4'W[05܀}E?Ppq>ٖ?v1*FCբ/~| p)7=v~܋ƼY*%Iq[kI[n5ΧH2@C->] ]S;\$uSWo^Kqn⫗_|8{K]5(S9P?'[v1?|h2yJ$~(=v?:yN$zDfS}VUNnҼOe=2ISjH}-'{":3_n z-gҼܜ=ROm4PU>򕰞b?1,F6x~Z>ZxI/n`cnO3+m߄5Z쎋(b5u ~cvk3R%X tX7K\{mM_El.Bg0A& 5?K|m7_u}-aa\/M|e!$/3| Ovud6uzclh21o=~^0 /N*Pa#K]Ϳ4nM;d9ɪ|ݷl){$rk3 `% )_~(b̄jl᪻t*LC̍t6aoJoOB+뙰*ۄRGZ{b &EiٓA(;w _ҷe"Ԗ|Sҥư|$x{7yleS.'|2kûZF?ˍcmWVv1VfKڤ"HZY=BavfK?i~-6Ղb |2<nsI-k WIҿ'Br`ي`qe_9}[֋ Gk"pFKK^5[(%s+݉iܤt=\|ǢL mo'#jBI`=vʂOA8ҵI|1bLNOzf=sAUcX_Ys>ilvmZ/U}ǹe|'{-;GqAv!?gShlkINi#&Rw ̧ly7JJE[/}`>& M}n~@)tJw<9LYGZ{i\qq$#ln]Ϯ'i8GϿOc}_m)L)7];,Hg#= gCWi`Kd^KℙN䏅:,݁{X(ג8F~ W'CX쭱,DǚkU* ;1HA"v 9c9 6e^z@ >n$r:@ k~Oڀ6A"6M >n$r:@ k~Oڀ6A"}S-b/ ^7Z;&3$mDN_+}(L5{=|:$r'8ynt^ xu{OzοXݕc't/ޟSQ9Ж|OkqY5fg>`/~#~6ԟ +v Ÿb7$-^&HuxK02ڧlD-%aqu4y`GEW&<9õ'/?w \ۓ~9NZ?C_u>}E~?qԏ|$}6wFDNͿ8%qicœmn!S*/}M @$E ރ!4I"qj/X'ĉOҵGu]jk)mJYBw剜@F>:'=>~pȉ?\PGГ7Csl}Jײ hJL`ٞ8}#}7ʼݣd[Bt|tC%pZ~kݏꥤP/]~nںV?ZKn->Rw3wwkpD~ t{X0;Ɗ{N,21} 8gǑ= ,@w"W|kH$/R"RO:>&CںN#B闗c䒸{;Ok*X?8;JҿGFqe֮C?.j,e6tx4oEb{ ΀ԓ%:ITyysK1!8|pϹut Ql\#QΩ+\#sbsz@uX?8Nl@v+Ʒ PoKmn$r:X/do-h!6A"6M >n$r:@ k~Oڀ6A"6M >n$r`,?~sRK I_ heDŷ8|o|Qyw'9ƤX FOï`dDNoٔcedꉜ+I~ν`dDND옔<;9}Nױd07r*e)[49T'uL"B>^V@"Oo$9 ?t>P_,p=D>y3/ 7ni heDNI~\3dd,K \1^F"o<'.q+<= S<9kwh̲apzDh; jO!}g ١H AoHtb9K3٦6=׿b$HIWLVAR \>Hq,?|kLvƖY_Wk+㓺41uelGD?Hi98k3D->J ~8p,93MMMl:MlA6ΕcNZoq:έ80'mwkVkХ:P?> dgӽ'[ PRg/IoC}o~2eo*X>j ;fDïo_M|ϲ`dDC|W#Nd'r0DNy,$K}VIņ*}|&ϋp2g5e2Aou2EkjwE \0T@}3FS_c),Ϗr\VNL=['Q/RAmM[̕Y"5D'*GwȮx;'4nk9y}cDp7Xboh=8t% 娝tpxX;|ƾJ:X&8 (ِHuҘL v5YǟFM_%-B#i wf1[n[{OTK_M$cCHIW_LW >Tsq" Go3[f~j"dMJL͉\;.87@R.aD`NY.R]eIDx:eLoME!{l r/> ŤղR^3ޓoiǶ~QH۵ DӷH.WþuS[z}SG2^Z(w'rvx<4p1R߹E76XKZۈvK.R{ynTQN디SbʺBuv}#m0t\-'J5+gOoZïWֽ '_y?aq$?aSݒs?_PORoج*u wb>̓UjmlϠ6w=S}%Ӥz>09wNOGѦVP]=@rs_:WĒX%Vۙs]Bؿpn4nۄ.f;t/Ū4%RhA߲ eg(QOr@:+E6&mr-as5l7W* [S߳v?7_4Kojb2䞎?S^U*?5߿w~/%瘝PkX'}cDN^_Ψǃ[wW"OL?~KڼGn|?Uj|kLe!YSR"XR"`W ;1HA"v 9c|"lSsIENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_log1.png000066400000000000000000003171041471400134000343500ustar00rootroot00000000000000PNG  IHDRU_MsRGBgAMA a pHYsodIDATx^4DZ 1 f[G0&ƣy@Kr@(1Cr$_ @oF~TEfEFDG'̌U>1N"/zB'+zxBh++g#|lqz٧(^޾^_.__ϧO -:^.ƾzyi W:7-se~SoE6cIϠcwտ:aB92Zz>/'~*~; %Oa-|hkI窸LLocmљnc?;Tte{^(E&>b>Ztfz>_)~wgT>lѯٷ3}~Ep^nʪ'PQrtUEp~E:bIh3Hn'5 w.:g{/Ipz >"|lJh a|1/SZgzR=*r#A~,=]šhEo>SH+y__\6ЧĞT s+R7d[KߺT;G$#/4)^,o9>oWmwM^}Ӿ,Zp蓼QtYհM>ֿ$o*FF{7K;ob?$ꗊ SA, 'B|PD`a۩Ev@WZu+;g=c[?Mk/~wG4?%;ve:i~yLǟHu!樗%_'؟-zw? {/}/${L~y6 WѹNM3첛Y p{_>&9:4W1>HO [z/={1D77jܼ{pb5` D78 oQtVEE^[+Vz+N]cO :Lx?}Y> gfXYk<b;]+2.Tc|(:?;?s\{/]gB<>_ MOXmQNf<ܴoغgevxkwv(CStUnb{qsqn/NZMa@?Ë j[[H7 ZOY_ Ɓ1^s[/~k{mQ"xsOD_~]O#`c0ڽFo w]fsI47PN?uYާA#rQn9j[P1gCi 'a<ꮥ[TmOҳ|OmTܪo٦NQdz>ufO~H#[IO/Jr E~O|HR}'Tng${*@߀~Ϲ/}멐íR{Kba6(j.'=~G~l/]v}痍bљW\?$qwܴx:@ܑ^{89]O?<'(: +i;UOgxw6oמ,~h=@x7;szyb}}br}F_.__+"w{=Anr 7n pwyiolOunon| -a2,Oz[C?Wd:ɖwGg}2=EQN79\wן/_vwSǂB*o1 oΔSO\t7m*ҕ\D7\T x*X/S{3ށ:ar ![ku7~<~s}=z52N/'Т)O--\3~-: } Bsgpw|y^,)poܯרoX"ebݼ%_un$2n Ri|X4>8)y:sZgzR=6r#A+z;5[)?`%-Њ^?|dmCE篯F܇yI__l9i|=mh67D~4gѯOl/1B47΢sAx妒 7f|蓼Qts^հM>ֿqT8h '.-ؑS'w:EYCTVO^K|Tā7i|[y]*YOzG7A/:](③3OIcF0L|Cgooor_}_KFW*xE??yWѹaL7mt3ƊI b|&Іp8͵@*-膜n4yTqUh&_r1|(ltOEh텴>K{ } }Ƕ~b0ιWX[tujCi,ů7Or_=݉<zRэ1i<f9 k{3oJ Fq|$.o`ͬ"Oտ^b],z'Ai>.$ZDX]>?whﶯSr'Vb;]+2.4vKȆTl·>鼇}buљ {o^u74sYSsb;;:c<TԢs`aW0r霿ܔE/M6[T[twxNLE΂$O;džk#ݭh#_SrL(/EcqsxӰLm8EL?ﵿy[Ns{]ӗcqw:EF&0&Mez}t\-gaj[[H7 L#?kA2dRQ)$g+Jk *z ,GΟ^3EO}LHy'AWuܠ Y"V]?Ae_@ҏ[Ngw-~5&y\t.7e M^s[翎t#T?hB?Snl> /QHտ=j#OݵE I~n*˹ևGͺEԧx֧._ˢM#GmR J('G;pStNO.>E8\&S={6/i=7ck"/8?-59'I9粣 p7 y&,"W{v{EgpG 4z [j&L}<|4@ܑt&?1F_& `7Ptஜ//Oމz:_/v O<9{ROߗVnNә8р}~wg=`Sܿ7)v61.6>uٹ(x].j6{{oo-=F}}~v=]Ϫ Eg?1~q}`ESH2ӷcom?];C{sb0ǟ^o06qSٌtcW'0S["ee"`y:ċQN79\wן/_vwSނ?a=EDܿ;\О8ߐt%}XvUY<7sS.Nr[I֭o CWDJ`K=!<˚Lc_gY?ET.AO<7_I0vs<]"G;ށ:7zտ&YwSy|O|>ψxQ|䏛G>M~sU\&?7|1 ͝y'6_c5|.JW\5;#5x~}8o_ϽO [j,|uu<6^z}g _qo?gY?_Q(Ag}gWmp;<;luM'^'ɤ't'Ymg<%> EOIǠCm>^LJca]8Z?k苃iOOHL2:>ʓZůg 5>tw kR~`IcЊ^?|dmCE篯FzWJ|Im,wTט[h[Y#U|;'mƟKXod|kҭ Jc2B.#ߔo?Vbw;x7awƳoe'ǧqIs~ }_pςg|=n_{߳/׮vOKl/1B4 -{,:7:YN#]n\'7my'$Nl d6v5>n>D<%p\y@XfCG݉oZdUn-ȟ&^t~Q #Egǟ;p#p[u4}\h^llM|7kS d9쏼_J{Do](ΕluBZ&v?A8Gm}W?`ǧjc~*w^E(eĜ#*_:6ɗۢ?]-V 3[YMjl{dcqĦn?#{Ώsq$.~>㽵g/_;o~?sogi("g <3s X * =nh ,)9%.h1_ J4yTg?|2 /tIcBZBiFd~+snU>w9g>~^t ;><s'AmvL =ѐj{8;GoFkk~% ޮ78}]O3lK؍I烮O't,->w_9Wc<\gx;;s(2#s4]2V;DZ/ٸ[߄ֲƇ/Đ?~&Q_<+7O(m>M@\><[G_?̘E},M@G [oBRb"|5~տ^oS~}7u*7e?l~T6xҫsw|"Ƞhذ~V9R'dqQ\5XK?ZZ==7⛾ {s^?שOK+YfeigѻD9_,Ч#IO7>fnbD> T/ Dz/SƯͿV{垦w<U=N{1^~-e“ ZW?Wu}9?>q#gGǾȏ mĈ}k8+pv4S,eh}R|d"~iQ eWE͌낾 eGzcM^`DSRI}Y OLtBLKI//=>>mM KXEǁ,1k'c7}y{9?!nʷGՐ4^K'Im,k~%zE m.j2> o?SԷ>}|Vcӗc+t)>X9::ZN_^~ώiQw?vߚ}3oҹ;BCU?R{CQ_aIhgFIoo!_l܅IOi|;߫$cH7Ē-lqot MN>*YSM_&UpW+^hMg9|ң?7ߥgΟ^?^;f}^s?e}q}úr5]NY|[m?U6ٿ|kU@ze^ ʍvI:(ImSc}W[oο'4W}Y~Ϗ_UsYۙv1JL+<9sI:q[|x{u/qM }vC?Y>7w<BO~$۠ySڭWq_*v__[?K5"q}Ox@CjqlCxjs :hP|F}Pϔ1oؙ:1տ].t7=$;ߺ'}I~n9#=sql%o֥sQQl|$ܾy* 7Er3Pts|j=oC_Sqdx e|/>w)ruoɷW>ir)imR~cG1˾`k_]_oͿZ<~>s&baK p! ~|47}kQ@=߂Ui}RR+,y >=Ogm.>qV_X?3# =-.jlniDHO:}}!~\FyMOѳ/q=-~@EgpGN콲5?)q17pU+G'G]? 6_K?=πvEgrk]s7-/xoޑP<7IK?oo)ϟ-N/qp 'g E!~p<7- 4~n9/K(5Y=xv{<+xsl[t>m9HJ{b<];eB{s4ǟ^ox6f 4 oNz|["ne"`y:ċQN79\wן/_vwSނQa=J7B'nt.+Of^/~M_3֯or4{/u[#>1,z<Q;e{sOt}f N("/"!vs<М"G;ށ:7r ![ku7~<~qv=q<#Ea?nzB4גUq ^?ڢ30~*4w9|aў3cnn2~}?pW`|8G#G~_۽u,5 `_Q$S_[tZE?fpx3z.$2.\VɯYt-݋[f]!|=EFٷxuIh J0>ql/uJ?=A2(O)FTleɊtǠ~چ__?TO="&Yo2ٯ1%%F%vO`fI{CV_ee/F_GK mwg+`B?+o{oZVlp~xΟΛ˾^2]WW_/c ~؄?-D,U hf3iͲSӸj9:7B_%?1\Y _ϟ;/,:7'l$M N:nmM~[o/j£!dvlƎ?-g|Gxq|c. A@ l(c;] zw@4-=v}[~l 'Ǡ?_BHљ{#>(P3Mq}x+7Z3[/u|_? tZ![>=rNﲿ Pspx7;s9m~{}}&9G?UtlB/E}K*[~VoƇH6ɖWO_?kڅr+g߆}3ϮM'ypƿ}*{kkUqY._x Wѹ>vRNtB m) 0MCVkA7\tu9ϣo_|>L &59Kq:1Z{wA/)mHVoQtڢ.ǯS݋΁p#xsZ^c|ay]q@Go?k ۿqL:gch?V;/+1u|?s9kyWO9ܮ~ٗl\oBk\CtbHcMd}@YtӞTGߢ|;E|OE^[`IwTOY.0LoUvG7>̋>_ZxkEmlxAE/"o*:SY*:B G_5kum5ҾzL#_#IOwĈ|"[a^[$_:e_l?__ ʣM_x:cS{Н cKz[˄' v~LO?;ƍ4}:;8k~>ܿXfuSOj=|t wMV&|? `EE8n}Vc]EzBi&E~v,s@tY1q훾[蟏q?=[.+O2:7.i,=v77'ڿu|Ls/ 9[ۣM_^OBucl?Đoh;]K|yE<+ώiQw?vߚ}3oҹ>;BCU?R{CQ=c`StD^~\T;K1>/GΨIC?#] WB:8/KzJ3݋>{}6:}?E<.jn^}tEO?w>moח,s97'*T6ߟt"}Dp̟)':% ᦐRd[H6n^Iv|>[q\U0[c;}6pIO"5|SׯimPN<~'$w2OE蜞\}qXMmk>,׷oNqi8Y<(:[^) >0'g?@xzE%weяȴ<17-se~SoE6cI(I?T+lN>+>[C,:85r~^[1շrOnﮟ"ھvx?on/>?x񊞠\T x*XBG;ށ:`z5ɺP<^88p}*PVt|ӓak __oe)J/nVt#k*:}P=0Ի"v_lM,6u#}KlA1_kۯ7}VF6Qɷm~HG>-A,:žv^\˯׿S:{nM!p|_ʨѽ9nhOίDMR񠚫a|b{E^pHE)]$[#q$'ͦY8."B:0N,ZNH~׺%żpˏmA7VgR():8tQHڿNwM0_)$^|M|eXk?_9쏼_J{Do/iȉ\Ɇ^^οq)47E_pQ ׯ?u,:B2S2G5 |<Ut.7ӱxO7fqOnY<4qvxٮ}i4yTg?\!PB?OEh텴>K=>Ӿcz_AssnU>w9g>~ڞ$DžIzA?رm߰eHAv5j=蝣WQιME__~=rǢnE'psnIi]6E^[+Vzl+=}ؓx*ޓ?l_;EgxҫS4w|"ȸҫ hR 7a,}y}f{/ kzźGl߈o"0y^?-e? CKQs7E P&uW  oh^aaM~-:;F''oA?Սg5ޭ,_O>O:֤oڎQt}>|(:?;9]^cc|8Gӗcԯլ_h__=tLjpg/: +tVq|;d7}S.z?]>wNv81rZVEab_SQ>{}6:L.:7:0}T^Y_&UpW+^hMg9|]t|^.:zczFjs?ɿ>?#čNg޷ڭ-V跋J|U[j#,o{UIdZѹ}s?<妾1=_o1>:X̟)7q L}ZV>BSy0?U+t;鹜ێg+q}.[([l|>vYc/ n~x^K&\g؏_%ɸLkU<1aoGh%Z]XԵlTb-N/o6>U%rA܈ɏ(:m˓ \f.Q9}"\hsb`хêIed󺬢3|w43~Dbnnr^ 7NܸO/Fxex#Rd3o-} uGUn&[Fo+>[C,J8r~^[1շrOnﮟ"ھM~sU\&?7|1 ͝y)/V*|7G߱=^Yc|)-<Q%;fTkp8PlXtf'cg<|Sfh9/YtN7iA增io<+-OίDMe5W6Z$o߶h"٢7@[ ?{/Vx~M7a sl w'$IkݒϫSnwWzǶ қ?k| z_E)iJw(^_y&GN/y&?ز? }}c}ʯ}Ηs#+K7gљi|ikq.͈r/^ Yh 7(C,?KMMVkA7tu9ϣo_|>L &b|y,ǧh텴>K=>Ӿcz_AssnU>w9g>~ڞ$DžIQA?رU툤d?ΏVio7be3`;'_G짾|v̊g}O)\oozӥ>E籂s.'= .:׺oǢQ:5 ެ$w_>ӨL'{k̘E':*tn -ʓDӺlE?Ƥ+VzR.=}ؓf*'8ƇNQ5,b;]+2.4vKȆTl·>>Z3k=xS5y+V?+f/J9)}54c__ Zw<xvJ1Eg6ha Ze^Zt][osaO)Pӭe'u z4٭;+Nu)./[V?//ofUp.to*$|JXE&0jʉ馒s `cWoz޻U\>>mM KXEǁәu1>Wns4}y{9Fw?ko?sc|?[6k1@mؾ[?5?zPt#LЏ/˱Z;GQ7'nTtnkjbIzcE8ZlĭǕ'o{4jƈ}c~tXƈ㳯;𔛜^~[7fog}fZbwq뽸}/֖˟  E~W|HR}'Tng${*@Z[Zƿ^5k/Sl"~~$<wiW?si>'L}J?i&?c-y:_<_ go\Gy񔦗FrzH~a.v.ڙ£gY-za9!''JA3oUQ'rn4b\me/yhvJ_~ˢx5Z\?$qiF**ݍRD~t nڿo\Ⱥ[QAMnVt#rOԱ.HOEö_WE4'/BJyڢe<ϟ*funQД RA@뇈g= 9.[u5 =EOaKm E{w(- ~nC.BYLvų>v2"syJ/޸6zfP̭SЌ}oB[?Šf_B^Řg_#@EgpGNדgE+ _jyrMَ[Rti2Hin9y},umvA3k?g=[ZkȠ [MsQ뷪(e}΃EOYZK f wo2uO胢3wt}y9m:CIT٥ ޝ 6+J)H +`n㘢܆/EgGѻ*jY4ol^O8ҾijZψ6謯m'Z#@Eg>E6>~OΏୂ}#z/H. uQtv]RxtKxŞ S'Xł&YR\U<]۾zaϭΏFM| 7Yӻ+zۛ/se~BKo yE6cIĦ(ա+ dH!ZﯯO)γ_//Lo^Q$?8yo5_&?cs3˿}wǯP!3Ub~Si(ŷ} ϋN."2EIQH8TzxqY?D[O2o\f}Vui=JljL/VސJ$$k/Zx}|P[.xc^ T`pƾLxOWzQտ&YwSz= s_~])7}NX|=r8L?xG_K*>Web*({ksgܙ_^W>~ ,H)!O''K"[Y3b+XYA-T[LcṆy5_Q|\XTn[ 7gGsUk7v?I& hZi/X!o)gax')0im>^>ǂIn^`+T)_Μ1 ~^TA+z;^)?`%-Њ^?|dmCE篯F܇yI__l9uT޶ߑ0v>Š[b;x:ӭtEt;\7f|蓼)xΤjm7mQ mEEˏŚF>++eRب$ꗾ4u XB|h,9Snib);ȏmA7}tgR():8t;IC~aEN~‹*{.VM+OE3u?K=>Ӿc[^_VE|-:Or|:4{Wǿk{^П/,|ݒ?> ?̢sf˼iǷ(ObO/]>K4vE$3Ǟ_!z/U yƇ׏5VtVs)JƓ^Že_lxAƮyR}MXEo*6KECt>`L_t7O`dNo's\Jn/A畛} OZt7RQUp.tnd|rջ&_;<'n靝+K7B OX#~;n+kkЧ`'ش@0} |8Pt;uEgj}e~k7 ov_O?s~Ǥr{tOto~2>w&3V9ܠտt|QXgڢZYߩ@t^ۑ"G^ce}Y"mM #X¿]tޙp[ %nmeߵV1,!E}yj5ѻn뽷:r9t3p6gt#W?Sn> /Qտ= Ёܿ'ZX) tYI47CԏߛY_Oцdz>uуf"\lo?-E)Zz}WN|'$w2OE蜞\}qXM-9n7h/tl_G<{oS,Gx{v3F?Q{G糷:8)nJVtzL(:;"_.щeo-Nu[z 2;|G׏bU^w^E_(EgVl:BnOP'\)d)>BdOվ3 =^1W^΁SRmT ボk\`MQ,ʦ"뢚r*FW..7hH;QoWIW簦^?q-__N26Gۂ _n/֧KOOw\)ĩ_eeh΢_^tc,KO:7O/T+J\ȢY۞bT{\$ͪ1h E繀"ajE*n~[ ϱ T_M] Û#1=5H_$(7Y|@2_Fg}пO/_p'5w:O /+ziIE8ƥc,Y +<$,ͅPn4f&% .{EMſ]T ]Ӻ5`} fƊΖ}~DLر zҙ?99__L^.E+^VlkJ_p~k d/SB(J\E^QS/-\b+./=:Eg|<7VtVw vwgyc<8̞(F,V)\s)Kep~%GTH\R. ȟrBQ0,.OAOj ~gYG~Yϣ;a/^IDZ!X.zb_NZi}DS~<-̟mt|pӸ/yN^@(-ֿ38e?_m,_u1EۯO[t}'ٱO[c"Nz9O_tsatN^Sa*-Y˜ZA{kAeN]m'T[ok##َ)@׏|/Hn@;^_t#mQxT1ss'ќޝ¬i_~vpr=EOPr}j"|ţArfTntk,^냢38}΋: L?Hw6މ  |Ģz\ L췖]T+}ݵG*Hm`WĢBqqqo(EgCcT\ۭC9/;{t=bۑUtOk]\;'plr_RdnQ6wWMhX_t&ٗ9KfJN~:bs=Qɟ_iG>8"$'w_uOjoڃ&X; /<Ɋo yTkSјůTpD. Vtn&D?,sSʅ\X6j/ד*WttwxQП/,|ݒ?> דi׹JOyt)YEN+Il-cuyr++.$:Egx6no's\Jn/A{c8~x8ܾ~ TdDhh8RYLw:~ o]yS܏sB~ͷ]Wn !X.ʕbgۑsLт^S~<9mt\.X_t "wv(s^ZI9N` @_tn sz5Fz-A>OE9GQ~q$TKF^?,o惊^Zyxy~ѹOXAn?A?o[^ XhO Oެ% pq:4j7(: vEgrj~qQNKADvtf?k~wON痵<nlA?/O/m뷉np;E:st񎃢-@)?Xv[t>i好q8]ϗDShon, a2.7A8hAVꄱo ɖj}ÍeeA@t}zyYȷb%6˿|\wן/_vwS'{b5俶HE{~(oU- ^b[zGzuI?X s(P\Nj恞`㩠zoƟI7LKm_x_ 7?ћ K~])߇r}pB|=3Xo>dN!V|hkI窸LLocmљnc?;wJ,9Ι٭([9J;Gy??[~5*ͱȧXQxpULh.{uMʼn^'ɤpb4lZ!o)j?:=N fMo<|1B~lBiLOIN2:>ӓa-5&Ak|2bz@+:{ ~zEj(_j#b{>Tnd\#x?m~bcj%>s#Wɦ2$ߩՈe_l/cڗ,ǭ?3΢sx墕 f|蓼)xΤjm7I|@7fmюEEˏEF(@(ehM?TN7Ŋ_q|?c|T謓niSnwW_ۂIo\&E~Pq|_ m_Œ_үߔ?OZGy~?u~-:5"[?EO~~GO.4^(n @am|CwA{p^UvZ dpiƛ I~SQ+7Mgv}hʹ~ߤ{.MMa)19i^HMxOoؖOlF9*kӻ3Nm(:Sϔ?㓈mS޼=2#%x䇆} '?=+gOPfS>;eBa+v\>J֧>opйǷOjrᎰ/; |[%KѵȢ(kt_n6<]_vK??|~6Xy:7Wٲ/se nIR 7a,}ysҌ^o?BA3mUF7K?Ng8?[M_x:a'墪>FxFh/տt|qc/ٻ to.N7I煽-~VDd}\ҩ)&H7OپKk *z ,G뱋Ο^3EO}LHy'Aλuܠtl\:^rik򏭿/C;wooٯ{lg,ٿ<:;mYEg>caֿsI~@;x/ErP_ҍknk1 I;3¼=#\01j[Г Z膣 lltqBw#ine7s/! g}BZMc#iemz/ #IESQ):'Csc.V}Kr[ Ϗ,c\<4VPJisԦ[OXi3͒}l'&5cUk/m]kΪ}^p .LqQ 4hn(O?./ܻ'|)i xg7{Kw?xC `7Ptஜ//=55#ޝ <y EHv=[[<>{&{v.<8Z(VOnU\3?#@|'=^/ 6텊t7q]'(/D*0vs<Tח_/0g*ؽ^.AqoIԇhrߕBo Z}b5><.F[ {su7tQY 5*A\beH8]gGswuMe8$ V~NjV|H[ʯ&%>NO`üٷxr v'y}Ob :ImjCI~A_Z f=]BLhEo>SH're{[hۋl75~~leΟ'<dK?R@s[oרu\U+wT|5cr~Q8]?K~9ˋqAULj?-~,b{`aa_#}e7d-_7!~\k}00}d|7o9Z/s3*ϥsΧsM9"i/t;0cϚ'?%so39j_%/O`q24^Y|h2y7>eyhowJ#K#;dx~/=dx#Ckǥ1c16tFߢ|R̊E^[`f_/7SE)>|웬QorsZfL.Ƈ;VtþN5t,b;]+2. JclX Mf|3Tqt_eV|Xk/"0dJc#F|N簯_INx+)>6DYߨ_3v1XKGct#ii_qC/IVCW,8ekKbEcanlϫԯw\åW>ag݅S7[]r$]4 Ka%y9C_Z'SǗr c0ck~ѷM_^_uXď˿}^~ gh58.k{vuOkci~FFն}s1]{X 6//Y3ްK~87#dTm~XF?.s1]{xoN9G ~' "Jr…4m߉3ݴgbqlNIoo!_$IOi|FMNkg}k~yaoG~'yc|Z>j?菬/KI'v!!41#W¿믟nAIZ?[w=3 !~q̯ۗj][0>ߢ;g|$#~&l%OKUms/#Kzh¯/z~=CUs L:(ۤ?/Ӆ3_uL yCԉl}foyxg %6IG66οX$0*inOe';1x֧>d˿E?SN:.hmk믟N<'$w2OE蜞\}qXMmk릵s_w}]-]ߞZ_˿:n,Tx<3࿎}~?@~:[5sq?lؚ;|-:M_E;ć?N1X[Oo` o>2Mi?b n=*g뻾ɭ]X+CkC|,(hsv}FW?ǪM!A#$'CHE 7C`Wv~[ }[HOIKmxwПN=*>9[F=_U<l3#iOZi|~'Fe3<-x ڀtޑVււ3ݰ'pE&~ɹroN0AO;׊~+9:?|~A~uH'c4~јEgz޵Ei32 6\/):C~xڔ_-۱dK #͎{חGsc E]jC wzP_|*q8}:7볤3[gó1Zy8/OCq_b`}\'f7=MkqܨL` v/&W:)!>)yMĦ㣰MAYM מ !/~SOdv~>@%kwr}|{{N9],Oюݴхl tJOu p=?\Ƕ|S.sm.J5HEPqn"dz ߡE"0dcM:7Ɩv%>L6~܌aU~ɶ:n '6A六KES݀U.["APAOm=Պπ'qVҿ_oq379&}i}-m?U#~7ϵ?_5? ?t\XҮɧ6i-^S yA s/Xx1;w qEK zQӼb|m"V?B>kErVKZ[^k$r 훑p׵/Z^4ח ǽݖ-Ȧ8ٿ\?`qpY/E~OJ ɐt#d6^0 s|C ,IkS[]Z|X򵱌y7b$}olZ73] EFȞ3BbHa=?:p|w9emB2<)5ʡeRKOQ\Dh=?mC}9L?*VoW6o:޳!5"c*>hr,,YOs}kqtck4_p~ ڦ/?Ln7?]$_ mQ_1/vYSZbG˒1g?fc(j?bncox+6dCrKuxљqd Eg؇;uErQ۳rQݻPٷԝ*?*c gU9ڷb\XJ>7 j2bl8^#O(\7}ֳk[V׿]HG_ZNݒi[O߽<ۯ/97 ~b.s`gs~7M_^Is]M1N NV}~X|LSm|@m$T_ mP uԴO2Ks So.zviQ(+ 50O6Ư_ z:_QW}gHo]8emW;Q&\EEVw)wqo_y)󕛘e7i^y]ѹ2}wџON U@[ XnN2CHYWS[SKzM: |Km VHKvK>Slѯ>^|Cr_qAϷ߼P;u7oڙ'w=eX۠$agnAskWg/>mG~<_?P|zTJrmS ŮX;U ¡œVa6,I[ q8ŶsYG̟U!ڝij:׊w|=Z&]r,e:;La#bA:ﺘrk# 7$ʛv-;Hǣ|\) ^. _$=X|30 o|W3eg][Ks>/oo~?S@/{Ih)>3c^*qo>ɾ({v4J|-<:Ow.׹D.mbo]}K>Oca ߦ3-,INn|=?> ^RlW,:ϩ8'xNbZtr}_- {2'*fJ,_4YIr3k]f > a:MmЋDp/#`0nRGb=%sDlLb^#%eaϗv[k.f;67Dh{h+QEg)耰^3sҳY{ـ3t]UAI9*:6Em.H] 8Y) {%f?,tKĢ+w*–"<4q@O,:~|}zDYo/~Iw 7Eg)槣!qN x b\0?~\^ׅuEBKg),KT?^R[PcC#鎢3@r N/a hA&p7]-\[t~_[(E#ȅ~==io+{Jүp˂<@GoȢxKLnu g"8hl;[y}EۋW!5;> ǏS_OT3P݂k!(i VPP#mH. %ZJI^lk =BѹmP[ EDp,F>o=meb>ۜvkEU*EXL[Pvtx?ZT9]gGsUk7S{ƾ?BO>at'(,S;5=eMfo7Xi0>alNi}˓:>ӓ7u=7lIъ^?|dmCE篯FzWJ|Q WbSW?| \ߥ$=&wzi6q_ןD'OYOC?3_O7o[_vU?G;̛"#'K=F*NY!UkEE1 ŏv\^|U|%O"o[[N+C]^'=ZM蜊XSϺ_Ќo}upmT ͵ͅ$,Ra BO>vt7|# #UQf8>w&Ο16.+x^XX۩@Eىi_IkżpˏmA7?kыΟw/JxLSҽcG!ӻ>֟#̟G7~_ώ_k)a_.mX.??3YQQt_s={l a󿮟plg>czT/LhO(FE\/ZGrxJA_Ksu4o mSz/cu5lNQZ(q tlf7x*Xp;ݡSq]^ ݔs_ ziz:Z:ȦI ?V}v04o.ǧtIcBBiF-~+snU>w9g>~Dٱ><sSjDݺQ􈬐__Doq*ԩ"`+>eQWL+:3_6V-V +q*>g>'״_o&`sVL6'>az-֓͟}^_AOɯFzE~xY0sa7)- }#tn#oo\Ζ|?a-S"uhE/ 1v+: O:7/P7L_Zm(:>>{-룮wc)kvL$="~Nݢ]?mGG /KEݟtV v-`_k*Y2y%ڢfZ⬻VTQgzy/UsO\ X(<)>VǶ\~3y$CZ[Qش;d^ޣ{*7ɥn/ӫs~'@QP?#'QnWB:(,KzVDG~⟂Zܑ}Fnwџ&V J:KE&BSTJeYWt׺ zU Y.c?7ߥgΟ^?^;EOo?Z'rEzC_>/< w2"^m+çO"NN:vK?SB ܆\H4 r& Vt.\IӬ[^\t-e[1/(gѹ|amBsG.:(Ec|*t7eL1oH_^-yzx)'$w2OE蜞\}qXM 1&^6>1sA1?osCQ}u".ֿK=ɆrzX>ߛпzE,~oE;?ρk[@q]P zWxMSt1+? ^Qq nq_$ =GOYKmgEgpGNm^t>ѓs-Ne]k4~ܵAcxN @;C7{Pz!}NspMr:_m?VtzyٔXW9(ۃcC?{7` P!\w;No{G-{?Ͼ>[q`f-c߷?UO}ֹx{Eg90?/tǿIN[`_-q_~t퇹 :ӷؗXhK{ҍY?\]^pS)LycA;tO tQX_O6ٱ=tx{dۘ+}cUHN l(VnM|Y, k.a/W[l]S{o_ձ/i#âk}Asɡ#B|:o| ĽyG>ÖEh)&ϊ,]=ӹ: C[߻kmw{?yܕUEg*V̯_ (믦~Bv[4ER|^  Eo/ &^} itw&]/nCb_8.ɡ/&Yx?N$F?$ &҉֏d2~b4D>׾߷__ |!k}cϕ7Ɵx#ho>'bۣӮߖ?G_K߼mHVdݕx n-)El[?qw3(gޭ5v．>Vy1^߷ȶe]v7kN<7[o/ws-~yܕUEEq9,'ۢ3 Tn *;o3nOEtQnc!n>l}|8@ݔ/ ŧC5&==K}K/_p/k|CLҵv>c~@>Pa,|]r>Fh~'Vn&R{3w?CŰ؇]`I.k։&ī1a}*JknW[m{=z~R]V>\?揱'oUk~V =K]߆'n+jc{dПǽϔBL>Hc2>H/ƓŁ|SHQϟ~-jpԯΨǘSK,*'hy[-mJ|_KI.XA[ae1_5x0Vϓ?b䯕/o,h${ne,om}4#JuG[k ;϶̆aaٿU>xjVqzB9;nm8'NOJq^tO`Eo/DJE'zLI%$x/ P$tn˛6}/ik+O&ͷ.bi)?)DEo-KM6j3VAo]s+BprNt}O?OxA$#s y.r"n$/cOYy@klޟՎoOap~kn\ k[$h]ٿl]?Xo-D"&6/l_ҧO9L%dE~T\GBy~7W[e91ƛ_]B>6>O?9O%WuxgtII{J|[Y+י/{S'^q:Ov(8$~vu o5M?ٰ?,z%<%S7uTm M[, bw:ғJ:b{㛢s,g=䧟_HL }K.6YEJr *ٌ6tn,5_ɲ`zB,l*iwZbquD_;ud*wk'.|v~Xi~#'cm&4>_WX̋V9/>鬽rClSQ9~X6) W)IP1D8>3Y~  ^B\2xC=>ӟ^:u?ݢo]V/O~S_KΉ| /i E;=?-vOo|wkmտ^%`XN$˱n9Xo.7tZ{+S'rL9[=&#O+ :8g~u} +jpOkw$VWus";1՟ͻo:#olo|Hw{q72o*jS 1g^8O3w6E_(V+R3cl%iÿ@SRdy=kݤ`6|)I"/l 7U>^֟ZWn_ NkHno m~XBY7f>tkY?珌co~jM׏c5[BǿA[3~=0Vto|mGЧjբ[x~v.?}I~מZh6c~z)]hﳯzGd, }ߖ~Ԯů>3K/w6DW|5~\oxۺ>5wXK>xJVS!w*'˻](B謷OXUtǧ'Cc᱊I6l-9= ֽ)KW|.}eLOk;Pq?SN_+w?_{g;Lf0Ur1~,1 ߮տ-k}# e?#$kOC7X^9 @?_ioWOFȏJ7'#罌_mO'[;#O/Ho9i~#>eg;z)'#qiot>d_7_xGGu}"4[M/į4^G_|uL1#űw~ v(8󓦟MgtoZK{ۺ>xE :aѳ/)YYtO2WO_H}+wڣE{j+Eou?;>by)jt_,$N⤥lDsQ:Q}ʆ7<|oJۓG~3zs5Ȯg̰H'jO?-⿰.$N)iozfK!h!w."[,y|zX3gex?etZV>zom|7sQ?[Y? sk:2l-b;~/v_5dW~=jSQ˨cMao?~a?YDw2D_$xD́8=*"]XidίԆa!ڿ||.:/ pK{8\ fM{O|&ݔܘa GEo_p+ (:o(wWxhWo?]tri|o|^ܴmrs^?={W?,JzBcjПmi C_[q ,(: }ݱ{{ߴ?rXG7!.[r|MuG~9:oPxWgo~f~&ٝ~]h;]?$#7~?p(C5`=sˋqZlN˟汣Qt>{qt?o=V^|O6oj/~wE[蘈ot{O܄~*?L8[[sha<ߏ{ςiߝ汣|eY+{W`GQ=?!_?c9z_ .>ucIuh|gV׍얿,}+b79z%QϽ{|>nu<%Jt}>_AOꝛʲ)CPw+&R8N3'6Ԟ^j6!'kLԗϵNACj'<{-znOzv~m{EFrF_2v;>>\?)|r._/i 'ϯϭaϰb{+ț|o"W_V?e,V<7ڴ?-MOzGv+G?G9#[8ɞڷg~ ױq7ο6'b]׷%ߊ !EF,v*ķ\c/Eʿx}F|l6pu|Z4~y}Zt>]?8__|ݴ寯TߴEО.+]O/XOoao{kWy~sMmXFc+ݢ%I Ĕ _ϣrP΁NJ!b'^†=OAM6e}Q "Y[?k&J)rH$ZNQ셓$#[ *gld8sqo"ϯws|3]M|4&):WoQW_8^o]{_c-Wœ 5=/Ƕ0so/ײƾE|<]vb8]m]A+Fև::w0X1LNφ7Or x97ώI"CS8ڿt\[߭Q?3="~͸>e,7)㽐VG:o?m֟и#W >Af5񷚿[tBw1񽢳6~~FmO~Sߕu/xET߇irQ~O^~%Rab|D9Gc$;vaOa/+76 ? ="~}R_vlZuA:aDz~S/39&ɷϽPS0 b^NwmQgzE{O!8.7kg\{c]<|`Ai3 Ŧ>F̽/[5?xMo?0YEoRuss6Slϣψ``t%oϭܟx~+.Stqݵo#wFjkfl6G,Tdp 0Se37|PoMo}%Uw6_rf~:7ֳk[V׿],~ft6וkꭑwT>!6YeH5݇cw^|y0^CjVT/!uvt5cT;{Ng:sh܈#h]Kkޡ#HO 7;y6l۲M4~dp~[uIg'=׶hyersы bWŎϛޟ`C?JƇ1&.{ Sy8>WIfaG!7ov|PylW|(Yܟy]K"2ni}v?Y* 'Te_O̶[}22O[u!5~ߔo}3o_P[_}Z|Y n~>m;sѷީ^Ut'mk|n9Gz㿷x+-<3WKtp&y gu%OhXGſQggZ~[۟oʷsC0]x?o}Qڟ|C}ng?j<P88W^GI^1_翙}o ~Y}]?_3Z\`|%?m?Zr׿JlY߉%hk<-J@HlHjK]P.zeEF|zƃT׿ @aWϧ#!PT|'߈<":͓x{tz~R/x`Pto+p_k5ҟHtEBO4CҾ>Ay83O 3Ћ'zҨ u޻yNח6xRv~)?❕+yӸG_gXOl?~8]N 䯱]u'>ɩH&y{.c (Eg3rKÝNWj7iju@jۿw/I!\w;);_eiΆrҲ'toܼ{ysKX=V [W7(Eg uo{uYGt?d#XAщ{}=+9(5kk;?_A短rx#*:S*f~\J@UЍE]5$ۢqG(BX Kh/z~V~a'&<.__I3o}.]hM7nnl/8' vp2MZH6]l^/(MEvOk_zۯ/V.|!k}cϕ7Ɵx#ho>'bۣӮߖ?G_K߼mHVdݕx n-)?lgv݌tmͿs^m+Ażpv[@e_͛_ _[?t}Vn?VȲ\|^oά/o#7\tS) ^dzBXqLsLH0bV ]׬M,WcTuVo=Ij?).l+dڟ[X{*5[?+Y텞%oCǿo~R~׿4G'?Sf@W+?_baŁ|SHQϟ~-jpԯΨØSK,*'hy[-mJ|_KI:R69W5~$J2JfPׁ6^?K.~\BQ[DZ[4[sb795}m|c,rhKZC=? IBE9}pű׸"7u}:ܾ+~VZobC6p2?x,XG8?/ŏ>NE;{eblOƏk<UE穀 *6OWݦ-w1;KIhp1νM9ѳϊ/$dJɈN?(*ItNlFJcۛRJzӍFI̒tžlHIĠoddSOq:Ao 74ccW7ue]v?%f"^W>c+Uic AsşWeA5mݿ5nVOa-wS|9oOFrhKgNm18^ں{J_;ud*wk'.|v~Xi~#'cm&t|$kmg~>3 32_]BMC#LJk?} svF|҅je~c[kp9?gH_sߤ~x/BT$nSQeR,E ݎt^BL=x{)h+ U+TKɺc<# p|fз t13*NxQ!̿e\]33yE-JUљ 5E 9x7KpbP9߃Ӳa@? o`qM⻍K׉dw9?#fܿJټcEgE}O7!?_2x"AOszL(G&M7^V%;tq|GGVn zXa_>UYc ko['7#ŏ$gm7b&p_?HvuGKoo$sKD:{3~ʱ-I􎏬˾fx1α@ޙLOSјxEazs4syg\Պi?qo\/*:3+yR wpMLI un]ӷ|)K"/l FU>^֟ZW.F_ .:א*o|Tֲb Ue6[k,5 ?|b°%-l݊Bi-OTmpZZvkOeG#rs=]qf_>%>1U}0^_]kI.>7c}\}Ttc6:׏ o/y|ӹ>]˱Js}2]abx\دb?g~!+Ω;v.py!xQtۧ_*:ӓءO1XEM` $ i93wKps>p~Ǔ+jMrg}&')b}לF#}'\C;?8.e7]ƇѯœeeODˈ Or9Ӛf}joH"Gh gbX˂dw|ZV:7{r\]yٿaC6hqX^K: @lCY''ȏJ7'#LԳ76nStHS?+%a?ݷ_]߂4Kǥѥ,ۓ}Y0S/dp2OW1ٍ./oq?!ž#$oe>>wsY.ý~^M&U_'پ^XmeO=8[ge9?\=9}F"ޯi2mSԁćo~q擦]}sH6 $rɈ'2VK\>tD?ϬK'0=l>9CE*,򗵿/}~{lx±xc.:/ J\?opRI f{O|&ޘnIxvAAXÁ1ڂ-ߋG[{@< p \y4$C-,@ҥX g㉯}SEE6M99y+ PpON ~M=}#6 Y6X/;[`(: |' B[t}y [l'e뗝# g??߳:=]h=Qxo0v_;1w@巻\?7NgeB_}L;⯬nA˦*e{pGso,տS9pIt;uZj{+@7ouQw/AVX˗4]3_e:Ʈw+ȣlz$+?:qoѿOǠE?_ _[([OvM7VnkgxUEg*V̯_ (믦~Bv[4ER|^  Eo/ ^dz}}Ek8Rp&חR p^8z=v(2N/gC B?$ dOQro#ty4F?}}陿o,[[>W}ݿҟmN[_u~k-I~![kuWYo7%;SN;մ!k]oN9ϿG b^c -/mlMp>-NNqqLa{o=m&pAXUt^#pyyѿ-:|pYOE2xѓ aF71qe󞈥X |>>ܬVnʗӡod[_ڷϞEQ7ēco]Ϙ_:u?_{ܿO ܉]P1,a$Ie|:Ăx5&O_s{^$e}Wϭcfp~kx[fuBO}׷_7[Yoo ~Dqg)3Ly GyJx*${*8POOcJ)?3ٯ_-VP|U|y~ji_E´d;M>o+4_߽MQk#V矔c1?kYkH~d(DX͑?EƯm]=rd[ =%H9tђ7CY^FR KmZ_5)s:Os<ߩOBt}O?OxA$Ks y.`r!> __7rԟX>8?G"Ɵ!ݚ'~I^7텕2 ٺw>ʸ_ 9q|ퟒJ*?6TOߎ_G=u~*)͌1 I?}u\}$ ork~u} Z?X<іz~օr/cqEOotڹ}U{OW*(?ņme(~$s~sѮ2'#=>7\ tJ?m,׷;;b?S?,h滭b-v_ ٿv s?gOed{:8~0-ڳY~ֶo/mݵCrFo']'/!㴋!䮝5VIx7XI)ɯpۙ_fLÌoWߣЧjPsgeq1>js{D;Oc|*Cs2X~[5;vuE\7^K;Tf}窽yx*:'Wn(QOE1CJ=oJ:wq7IuxF̠odbf?]_0\]33yE-JǃE 9x6KpџFRPԲ9߃Ӳa@? o`qM⻍K}7c:.Ǻg!0>#~(~LbЧ/[j_/ٯlIg>Mi~gұEGKDYy韞x|+g¶ux]ÎmO_[3^i?Oʢs*Npy3 \^^$vSt 6_+w?_{g;Lf0Ur1~ёxO˿2>A_ҥ?*ʸZҹ$g0?Re}_Of"?*ތj|ʟ2QmO'[;#O‘Fu߂s~u} F|:/v,RO$jrF lOe}uNZu>v_dg7~C]OGZ[<ۗ?>"uwG/`_C<5tu2!g>KߚX'tYYtO2WO_H}+wڣE{j+Eou?;>by)jt_,$E"6C2≫}<s3kR¯sГhX#NK{83ft'+jO?-⿰&$N)iozfK!Ȳ'B.Z#۵Ynog4;61"ke} ooqg_+砣~ Im?:4??YWCklYE?,_u%W˟\FkC;1h cɲ'j]ח!ħg"9x}$bA'?G\O: v~- ̯ܿu~_ԮtH۶AhYCXo/<޹k3Y>/Llv|mx‘?x0V%O{8ޏ tzK='~nJoL$< Xڂ-ߋG[{9opzO6F1囖mJcNMvxS/]%}ۏ/x67^@Qm^ߔAg^7 3xg?p?;N@~& }@):=9]O(x7qZgR;x`EeI};\>3 퇂3oR PA9J!=8֛@pKg]E/:IÂy9]_^IewïONOZO?~8]N 䯱]u'>)׾x{.cC_S?O6"?}tqɾzk9.+:ӟ/_,zۅgm?%uœ^^6xW9(ۃcC?{7`ş) Vz3 @{ʏ(:O[}},V5kkl"G .r Ҷ7]t@Y)Ey?ۿ?׿~~$'˭dćp8/?V%O+&pOioXCqOd}y-c_Me0]<7KLGzEvl=AImL>);_?yiΆrҲ'tokc;ğ]_,ٺ§ߘ?E߽,:o[c/_~8t}G??.kA7&In>t?d#XAщ{}=+9(5kk;G/wPt~g}k?gݡEd^ѹT~3|nWhTPTXWSOb!-W}q)>/tqWh/zb2}c5\Qq&]/Kp,ZSq o;_MeDc|rNɞC_*\'F|Kr>׾߷__J|!k}cϕ7Ɵx#ho>'bۣӮߖ?G_K߼mHVdݕx n-):ړX_w77][o'qs}1}%6  _[?qK%%Vn?;l8ຢszz mCx`$O&s4p= LHWV1Kt B&[+*/[Q΋rdY.O>/Eg֗.:kU/zr2=S!,9&WM'etS qt!OH\O&{(OjMzzu-iR?{T"G`]d}ɜ|XڻE}$NЭ M$fb=+ I.k։&ī1a}* N[zȓK ه1^38D<-J~:_V{'ɾ/ě߭Yڷwg)ELy GyJ#%>5d4?Zg_ѝQ1秖V-XU'LO=B[$]ms^?ZkHd(DX͑?EƯm]=r(ŇA^߲1vHF4ry.'ӿSx~sQzҙ-tʈEv]B$$t I }D'ٔGsv]B]D|9YICLJM ]mNK&}:v7 ).X vk?WBOzJ+y 't!E NG=C4]AI}5@l)rE|!wLI +^q퓍s4-b) o?ҭa|b{u^X- -7#]/][D~Q8ֿҧO9q@2Qi/ʿ q3cBn_o#y-91ƛ_]B>6>O?9O%u!k\>vn_ x+~O!oj~T\[Y_>gQHݸ~~-Ig?ڧc~kHG5q'ӼkG><~1zz{3{r6O}֭.5(^usJA_}@QS7uTm M[, bw:ғJ:b{㛢s,g=䧟OS2dD'tOtxb{HB$:`'J6#]PM?%FMqfISHb_N $xbз|'8 ZCC?Oٺ?GԟWoOVpQ/+б*g|^Z{Ck SOeh_El}}+d'u0k9wS|9o3 32_]BM#AE5Ai>?y~s9U4f7 $n\?~˱-5k8㜟3GFx-\ܑ?A?,xXۡȼ,<5u޿蜋&o)r"qS¬/\b1/OXEv `J({HGM^Ip.2_ZJeOP3m始Pqŋ a/ĵ/7}Cσ^/ƟnQ7T+DW'e ov%_jv>/ۄDܟ ;o|wkmտ^%`XN$˱n9wGD;k牿Ч/[j<שe 1Oy~P7Fx[!Ox\БO=[X>Vk|Zk%¾ i}F\?ה߶OlobGIx3~woL~T\[Ko֥k}<#['b4OkƎ:rqq߽ ?ʕ] ,=\5̢\eVα=ؙ&~):mzg2=ELEcz=́sW+J秕ƽqsT`>*:yR wpMLI uf˷|)K"/l FU>^֟ZW.F_ .:א*o|Tֲb Ue6[k,5 ?$5;gw|.nEg![(_/:z1fG_sAO2G{׳OO0GsLm}/WZAbXWC&43~آͣӗ藬[)>9VpO!l_  5c[Sǖv" M1yYOS#S7V*:cB^t.{zzzϊɏStNܩ.vv >bUY}*:n!^OcA0_]dyt|׫;\IVm%?3.?I!wC'G5?W>JYNiq/Y7>~,-{"\F(OxķΩ$77V{C!ܟE>?FO@?_$YߴOtﶾA~>ޓ"G~_-Z6ג>'91/_֯7+E@3~vQoF5>POG{5s۴H}OtwF絟Iy#t_XȟH/8..gٞ$'z Ѕu}:\= nvi}y8y)#aG$Zm>ȧ+%dW'7aE4_9>QN\Go?n;_h.:ӿ~A-1*˞_jlg㻵jذ?+P(gc/'~k÷ǝ}'>|Zn6_l-b;~/_5d4fwv-ozm2X?D_F8_@O=Qꝿ QG%>=<^qϑo;}F|>\V}gc!Đw~ efK?[5pIFH8Eg\ RT+j~]`=];SVzpYN\/(qO}޻IݳhooͽoMg5^<o^=zjOps+h;hμн ^=+u_:5XlqEv,ys܇{d蛼şH_^ #=-/x67^T@Qm^ߔAg^[?ȯ- vuMѹWL]A)VY~[,O;P gK|u^}5k9.:7K_WodHy):=9]O(x7qڀgR;x`uih/oq~ok# Ģ3 j| *]bx.dJA ͶSF*: X R}q0[ lt>ѓs-N_ɳY.(pp䯝_cdx]OIͯx+bTxEgƗ8˿>vf/w:9_Y0M'UNP Xr ႓wye뷽#x=cg_kbMo={^n-薶=\̯3E[b \'ߧ?6.&9Yn%7'>c 涚/D(Aзh}Q1c8J{ ҍC^d}y-c_Me0]<7JLhGzEvl=AImL>)a;_?yiΆrұ'tokc;ğ]_,ٺ§ߘ?E߽,:o[c/_~8t}G?G59`<(ħ  NGw1t_E?_ _[8Z|݀߷w_?8[ǃ(oEgUEg*V̯_ (믦~Bv[4ER|^  Eo/ ^dn_1t.(q8eF.חR p^8z=E٤_qz9 O4b(!_9Gɽ/.X#%LEv)ϵ/=e+gߚenߘsߍk`/)koQߒ7o~տ&Yw%vo_C?e+crikğ:n[پt/;ư{}"%//mlM#_ӹt\x[_[*AdUyQ\, E?qEgm<BEONg*4>JQnc!.IKfc{(OoiMzzm-iR?{TzXɱݮg/:şK_'O DAjo.y2fhmbA֧ү܀jտm[OROZ2Wϭcfp~kx[fuBO}׷_7[oܑȑ|Y{Eshsɤ?dH f*#D"_(TԅJ @,m+::̻zzJ>Cc7pȇ9RK,#r?~EwD3?Ϲ?Qc?~'IGq[?m|w|K2?GGH绔k6󣵿X#C!o/7:hn}ya^/.ܿgiC1=&i%GS/U>/Źbs~K~MvqzB9?nϴ`͓XÓEы΢||-z nA QgH*p*`LrP ?d x9>&MC"=`4&dD4xeC2 1|`s+AOn}{&TBpv>OUѷAI{X@/l)1$|#o;z?ʲv;ecl^S08t@72a`evh| Q\?l2bluE;~|럒=M~mTOwQi/sϕ*s7ܟ8 l>rϭLdW[s(~ ֖C=0qql?0v~z~S#cvIUPk? ~}CQe0/iީ߮K<7Tw/ .>k[g֗158c/ʮs-Ƣ.> ᦰKbQx{3'Ihp1Β<):"zփypc 8mCiHGL!,In494ub_NP(-?Tt ??  ߐ.bq+:mzg2u}#AvS!vpq!+:μ|z;St 6\WY~~2㥾Kq<*/&W|}eOR;b;.>a83xz: 70|Br/U߬Wf"oaEc,N1$>] i0yH>}4?}Lֿ!~O+B7K/s;?5yP^u|~)+$\~{|vVoD#h'{16~nwݿ}^>ksj| t/R$y\ډ. pOߞ8q_ "ؿȯx=ڱ~M{Nqv.?}GHX0[C>3_0ׯq!n.w^~Nᳵ>#I7#ȏK>먟YtO27O˯H{GמڶJ=Eǧ򉏟ߥby{R.p[tɂ%U1GܸxX}8S뙭})0chZ?lK{83~fp ?5e0ze5~/ 1hBb~!v(8M+:~{ӑ-qY"˞_;QSZ|vֵۿz:s|WBJ݄m(؎= Ŷ?ɿjL?\;د_-62>Xu o/gd;X+"X_| Y^_y  ݀q!lkl`E~m?7E>A>2|7h#Gn/:/?cZ_㳸>I7Ʒlk}\ޞ~gc?ߜrcSP7Ӈua<ť|G'TbqM/ͥǗ 7­(01Xk/n{.z5zw9 Žpc%q|uos1C1~w\sI˵4/IO⼿X.'.Kibqk_o(;)J4;fݳ >eP|d9w#JyX,.a<a})_,d#V!Z8׾oV/a ~ݳC|xSǔ?9W|`ѹ_ \_EbX,bX,b:΢sK;XB Eg[tb3)^E;ry+/'tNm[9bG Eg5ϧgJ*(IDAT 4p??8ʫ?~~{+6ӗoNw'g$|V6}MpG%wqkk} ~ ߧd?8k8=ٗR(LqcA; b(Ș4>X`׷Kw6vb@I(^SOW,vk}{A~ `2x矈2{}Ygc/_~8k|G+!`oz~YٙGto|urAP|σE<"?_ _藟|g־\{?fVhآra+nOZ芞iQbw [-:ylkUt.lO:o^ ߦ+MA7ߝ~ ٴhK! C{++0x% 8E|ODz.l}as~Nj}~'FY>rPU?~p-`_.g|~oP[o?}n;9|~r|үm'`7o~ֿ&Ywe>_ _?%v1߶Af ~Ϳg_?:g/ aw|_4zlO׷þ]{?س1xK)>ڽȢVe /G- }z@|[wt'=a)8;YIi̧* =lmsDgl}7rG}eYO6E)SZ:0?׿RRb2 ;e4޿(.z6V|G#-hsg#ӏM~mЧrTˏJ{{W̼0?o?`G~.uc'95߂`Ccp. 1nMЎk%]WA[47 mG4{#ſX8>ꟛ?j|I[;k[?O[Gֿ%'tqS@Q5~ƋΧo})fQeVPStnɛhq_t~vk7uP%O7]ۻ>qOB+QtI9ѳϊ/8j2dbPF'$Ԑ6X@d~HzuɽlO(i2IW/dl %A2M?AǺAPg!_vb9>+L{9U$MY{XIikd|8Nҭσ6TOחk޿]l8< |?,s+Ώ= I+?F4ݿpKqQ[סO.xcT;KH8wKvu]gd0.y+~1[˿xmey 0&գs('^609;;(/_oaG|v |}ZXq찯h~Ub́Cz6N7~Fߥ1gBko.&:x}̟vfgO6c7t}$>.5A }3(W贊|+:s`Gy]tѢ|V.H1ՊΖEXMLh O<0ǹ}9@ټ}t~Z+*KrS+0*:n8;*5)>{[ݸ@aA>w-¶_C<>v XlB+Q>֡A/ V1Qed۟?2ejY&aŷ 7柡C_waEgfiw'{gi`^氳ha֜k^m.I)? s}kjV'F_5 پ7#>v>MDy9Zџy~1/Y86?xr?/~?ڌ}G^iUuI\苜L)|3T=i_):{@frs*n|rڙ..wEgbWљOOb{*:K_!>ïI((.}/׫;\IVm֧.?I#H{' +T}~b_1eOoWyo;S;$tj'!)>,1:3qX|Yu?i.o7cr|mؠͻ܆Ą/:|6K;ODˏJ;74LSC?D;@ >pW0\9{o971?}g;z+5~vKӷ'a|9N|$/+^j|v_GS/ngpcpsߑ?}%!ȓu5y}/N}0qtO E'?"qK߈ /ASNs+n^N*nO}mctR d$g{ NGԶ;zbw-rB-"si*[tO27O˯H{GמڶJ=Eǧ򉏟ߥby{R.x[tɆ%e1GX}8S뙭}ihaƀ'тm۵X!wy 8>?|l\Fo!]H?S oEg?bu쉀?خŧoO74;&gcx,kewp o龽ca]˭砣sktrTn۳O;2H؎= E?HUOd긝x~sZȀhc 0ӿ涟`zǗ`uT'~!dy}}$b灔ϊc!,x}eĴ}ǷkBb)Lw_?j%ӟ_ދUq󇠿Td%A[s ga+f}{Z.ƥ|K'o?1ar&KRʈ_==|gB\a Z>[EY)ZMZOٽϭ 6b~M_ukQҍe7.^t.?Ƽ6Km2ׂCk#=-~X7|~[/*}oy~_7-5韕_ۦ):3WE)kԐP.q|yX,.a<a}_,d#6 Z9׾oVudvkZ,n):[4h.bX,bX,bX(7}I~/w1omX,nC/n4|j}zY燳qH:56{f:˿#_~Hg?t,|1w:_6~>=O__} mNNAǩcV^}{pVͥwQ_SNxz'~`-t.5A緣Ҵ[ߟf^?[:Uٹqs,ܳ^ӶUt~|#XEa7o?= ӗoNw'g$|V6}MpG%wqkk} ~\?h8+_KR)˸džw[ MOdLl,}p %lo;1 _$Mj'c?W;[uO??OD߽,w@ձ/i5#kPp Ge'eA31~ܩ~u|kķR֗eu徕Bo]oL X {yH"ʶ];4.x矌:>z߼mYdݕB|).iG͏ܶ׿=^~iͿsǟe~ɾ.8G_͝_ _r6* :v4ƿg׬}=3+X\ ]q9˓߰pY"7) zr2|BXsLdbލR qp=|6X~tۤ'ٶ_>TEK7vf1>8?_H_\_pO"=t+_*]a8ރ tĂx#WmjO[yI'Mz هlkoZ,>J^96o~gG^^jpf?ϐB>ʱכ~X %>jLˏzLkW˟ 0?M|Ϲ?Q;[=Nz= dk[89?JGҭ=ߥ^6YOp_,c7GPțC4yu?GΏw/L~w9t`?K"Q1Is(>y}.#?k6Xmv}\ntƲ.r^tm[G lPp?$5H g?%BHg:^NO(IӐo{ĤH<ͷ.;CLM䩝?H/W6)]߰ vk?WBW1lO `'#?T}$mb`rO7C,k^69:< [Ot#cW_O_f{aƗŅY/5~3q~EM/܍c@r~?Ҟ:}en2fSݾG.ڒbg>.. cǸ!75>B;iw_EޚVm7 M;_?柪egǭοiߚ_[eWѹpcQ^\pS%m($REgIc=?&CH&SAolI!j1!ktJK)$)\JzL}9Y*@Izbз |`SOб&x ?0P0kCN3GqEi|w?GJb֗$öI!kS}á 5i2m~, AyPWywע_Ƿ\Wޚ{[G?؞aӏD~_GkM6>83:}xC{wrGNʱ6}o]ɯF)v-nϐ 3"OWџi{f<4H|{/wdf_Ƣ}LߴY蜋_>?׷/Bn-p- {禽qѸb<  (%,AWC~WTgWH%Vx8=6 3C҈ѯ?SYg[׃E1q/V@1FBS1߃Ӳa? o`|wg7.?N851ø~@+:kϞGVrbI(ܟ*n׿ ~ :zW[%bֳ7V;k3_s`^5'4ys*wo̙ЇY~ܛ/= =>g_DX{%\5SǍ#]>";"??8!.i;}x>xS 1g\8O3;VO+EEcI`s3(JƦOI)UI93,})(,>b\Ex_+chG|p՟[ڛXMh}E6h ko~˴5io|Fٱ>珌#7o}Yhֿ 9~k|- }gh`,(b]_+:3O_b枦M EQۭϸ?k;fif0e z'٧Oo$A}[gy~5֜ /dߌ69a|k]GQ`hg(-$ryg%br/]¶u8.jAfSL|(O(_Gg_Bٞ~@ .vv ]YoUtӓអcẊҗ@H-s~/]_QyW/IjMT')_ܰrdiY?<=v!u}EoO?a+3f0" 1Or{sjv.X4W<$\E>?F>~&k}Y@߂ow?߂ο'ץ~\H{˽w[6 &:~c ͿlC_i'q ]~TY|ɟ>e'Atnpyy~-pwK鿒?Pqi'4=}{/ƗNZ~xD_zP#c? \|q;+뇋/ I>c9޿g/`_C8B d]p Dhgn9AE.? HETl 8{soްc)Z{e8?n.#0lUtr$,:''g$=B{kOmx荞SR<ʿ=} )jt_tI-:dA*I#n\EVK<>Ph Ty~)־{1I`v-6FHȥ=Nڿ?3?8Dkݲ?l4!1?b;Na&ܿH?zdKC֡Ȳ'D`oj?jXch(gc/\'_|u/(W'6skveP,b;G~o3'u?Yp\?_?\sv}WZ]E:*HzG"@qHc7~}bZ$_fMѯ_Ю:by{8ͭo;}}:뼼3k3Y0>ςg!AĴ}R5?,y?< X'v` ñ\ET<=o9ԃIԗa,tl;o&]t/ bq7.k~Q}m.= ߸/Xn]E_{qKw־;aW-G.L@IJ[&E >芴 3{!}Eg x[usI?M9$'q_Y,C%k5nv/^XEߔnt%߿n[8SLj>Sљ-TIU!ȻPs@N.byJyX,.a<a})_,d#V!Z8׾oV/a ~5qГXEgF;^tNEY5Zљ/f4c>!>.oVUt^,bX,bX,7.VrLu>̿?._&rtd{Xt|^t~' LۿX=ǰٶbQ9׻wrӸ=@Kg! 확{o>/?]d}̞_I)o[(LN+Ҿfѹ+B_<Μғ6;/?'/>lsÅ6WR' 3+pƾ=8? Xׯ)+yp?W:7E:Wy?n=>nUٓfƽϭD{Ci۝ .jIg ss?W"k>q7la? =LtTbxGG?Fљ_|s_2|sGt߿?;?oO?>#Oo'>k?-Qӗ[[ H0S8> 5H<ő\aȾ/Ba ylxEGƤwlOܞm}\wuLBٴn2?fyssX {^[w^;DBX?_+Y;_)U?,so,&gegUxk{C=/P~\'Ks2E{?س1gdSbs+nӝE S {}E2\{z gf5#7Pm ߾ߟ~@A9tcQz/M=LѸ;]@=_ L_N// _^! 3_tz,.la6l+/ >'e*A6g\́ꗾx'F| g|~o [o?}n;9|~r|үm'`7o~ֿ&Ywe>_ _?%vavC̞wEt_a9q2~td_ #%hV~\A.ۓf=G}+f{gcfψT Z5AB͵<'K~ӧ Ϛ^82$.ضdz7Utˑ\|Egt/n冋Κ<I!\eГ$cq%tnܗX\7oM?~I~tۤ'Y_\T !m36sk I[TI>HdbaTצ&FlkW[lT[OOʢ{; هlkoZ,>J^96o~g%o]zz >C(._oc6 h'2.?3ٯ_-6s8WtG4˿i?ܓ#Z@ֿ6;Qdyy}$rN{-pz~K~$r}(DX͑?fPM^]?ϭ󣇡=Ƿ2ЭY?}(6DclL_jR!\6>ֿ%=b0l rAsWtQ`\dW^;qWBX/]aͮV NO( sQy˖vxR>"zYO`Eo,!M0$EH4lDB7Q3/''iIאo{Ĥmo]vvP#?4^ِ⦎vC?[}vr?ɭpcA  NG31H>i[(68ۃ- Gtl cQC_Yn'l쓍st~Qy ;Ƿ֟F"ԟA// .o/ϻ?(.7sg#Oo+䧜߶y}*'Oʿj|̛ csO tzo8Ac}t9%Pό}\\?~~a7DGh5[׊E/I|o<,|C FtdZO>fև1~8o_> E{Λnr~ҋ|p>ڮ~/l{79QUtXԅg@<=vI[, obw:= #FY'EXDzO?+liC$PCDkhcmFU_!M&i<$]]ؗL'}6Aǟ poHCډ}(3hW^d6?dayo8Tz&_O<9Z;;H>S?J^_ nyZtk6Okʼ#ly|0'毮={N޸^5 ߺ}*wk?'"F]F'r c_ۻrGc eoF|3$Ì{giڴcdOAߌ?iwdfOƢ}Lϴrm> !gۯt RuIgǓŘF^mKA_g,:aK16+Z9m|NoO_>1Z$&ZFM{)q*:SyIg/8 kQ Kp]$$.2^I5+(Aax;<졂@ 1ey_3~z˳Aj(Vc_~kϵcb;? z6Pocaקe?:"wKvu]gd߽?(+:+yZk+9S`LGE8TQ7Otm`npɛWKc΄>G]~s7>o?sfD3?~ʵq#qGdGooA5B玢 \5Rqt]";RtuEd OH4g_9h;)b(ϸ( qnfw6oE_(Vƒ\ g*:#jQ̕l7B\s=n\UP0} ʋsI}aۯWuo;wV,ojofcu6(h JikdFg+ߘэc}2UGo|b5r0[RC3gW{Xљ]k{67Em?9?}/ƚIH1ufd2?0m}rg9Xs:(ྐ[~3jw@umG/Y86?xr?/~?ڌ}G^i3+ï'bQ/]_QyWwpW0\a@Aނs|5n|c~:vz)Wj<.DoOe}r8|Iʯ^_WX=j8C;_uo#$|?ܷ<^bH|!`;)>^;;DԋO~:Oۗǭ/}|#>˒O=:[X<\ ~^,>]Eߑ3us[+:7v͓3~e=ѵmRFOQ|wXߞ} /:$C!tI$r 7"%.V"8S뙭}ihaƀ'тm۵X!wy 8>?|l\Fo!]H?) oEg?bu쉀?خŧoO74;&gcx,kewp o龽ca]˭砣8~`D@7!GE6_ʠXv,i죯gROd]h=62>ڹ 0ӿ涟`zǗ`uT'~!dy}}$b灐#y_ `_ Зd/k߬) Q_ p绹m@ojźXG_g?g`w| hkJL'/}|{̮Kp3Q=k.rZ!>9RtNxu' wT3g ??$.y_,.)/2ar&KRʈ_==|gB\hƅhiYo=O:߳?xsIe~sYd'yݯ-?..GzZ`k5nv/^TXEߔnt%߿n[k?+.Ut<VsX,~֗BƱ>kүsf_Ga~xb*:/bX,bX,0Ͷ-ōr [ߟYOZO^?p6_1oϬBgwogwNgϧg 4p)?8qʫnUٓfƽϭKks>zMVyEg~A~ۧoO/ߟ>?x{m~rM~S>1\lߝ|Zd_@EA(/ "T 2.,hVlBK߱=l&+_;q\п79r+e{?'3Qܳ{_f̬bq%*:C)f~J|@SЍEw}B6-70ER|tq^ :|J/ zb2 _$_68C$g(XѰͻ/ZCqԏrO|C HU?HTTOb0/}Bj3lܷr׷>7vk}{]~9>WDٶkށcQg_O7 ?_2vW//;mаs{'}>vr깹藦+?9n~Y׏Ιys{0~My=kv^zW^!ޓ{=;`55kkO/W®sW\sw?-:{q7,7\tM * (&y+Yw#澔B|.?1C#_/>ݥ6Im}~b' ,yB[}f >P06|"~q}= =t+_*]@1,ރ tĂx#WmjO[yIRIYtog[!pc-[œ2o_)~V R=Ɨ |l1G:.2ϐBcM?ECp~,PHAw(#r?hl~EwD3?Ϲ?Qe?~'IGq[m|w|K2?GGH绔k6󣵿X#C!o/7:hn}= _赙]rA6DcP}4_5)\+F~0ldWy+'Sx{L ƹ(iK(68ۃ- Fo cQC_Yn'l쓍st~Qy ;Ƿ֟F"ԟA// **Owqt}ΝL?QrOr~>]~Ts_5d͆19N?}}\s뤃8Y[c}t9%Pό}\\%/ㆨؾ]5ykZOs_6p~{O/튿DsOWdZ_Yeo/N~dXŭ\ π.yz)쒶X t{Z)\G$Oα~V|Q!$)q 7$ҐI5B:P%YVݔir.I %=&iž, $Pz=1['X<(!yj'㣸4矣]z1Ka$婾P~F?Mkj <(hOeh+y}Ykv?wCBGqAʼ#ly|0ʏQf"?M/ܣu&XAuT>FNO7XO'Wn(QGE1.%f x'֦:+B\-?9`F~v>=/?ݢO_Wl,Xw'~zo֯?׎x /~5ژCՎ];ox׵M?_/헼vu]gdpcqS2VK>>go8vg4*1!j~Ol's?\?#hUߘ3bL~l?93"/rm>nH|\3S}*xBQ"sWt&q+,^Ut&@63"1<iygVՊi寨h,Ly|3blm\—cϰ|WtpL>b\Ex_+chG|p՟[ڛXMh}E6h ko~˴5io|Fٱ>珌#7o}Yhֿ 9~k|- }gh`!S33u'{gi`^氳hak&!ZƠq1}O\ڷ}Wc頀BC&<pk@=):,} d R?Efw#| G^$6?S}`wv]|`sq?ʑg=gtn?`څ _Y?1ćѯ̘E޲'7+_ukA5>B;֯ip)pݿ~1 >"_m _}f_0ׯq!n.w^~Nᳵ>#I7#ȏK>먟QPEb(>kވHҭ蜊m_@ybdn_H+ =m㕢7z: O?K(/]}!$K$cqY-@1Sq"3[Roaƀ'тm۵!!p8i:g̰*j'ƃav>k _bфCPqHpVt# A-qY"˞_;QSZ|vֵۿz:oY%OQ 5C؎= پ0?C:n1@@C>`%7+o~}v}}d+#/nn};bWwG֏wƟñ~-}w/ GY\Zik6X#&GqnC{s)tKK1WЧOΥ|G'Mu1Ew\, E7|~㾔2bFu&C~-qEϻF/Y0]γ>w-kh,Wk~ sQt.?ii-B&?bqWȿ8/^ŭqK)*j7p+urL)Ok'©0Fz255!`?Nړv*:t@Fvf3<VsX,~֗BƱ>kүsfRk[\HD mI_T0%6> dW`sGtHcg`bX,bX,yA? |_ Sds|FUsqSh@L 4-Ɩy%ޤ ӎSt~' LۿX=ǰٶbQ9׻wrӸ=@Kg! 확{o>/?]d}̞_S S0 z9=pk4oL7~Nkd+jy_!ϑE:Hpzw:_6~>=O__} mNAǩgV^}{pVͥwQ_uSVxz'~`-tn:t~;*?>n}#{iz|,f'΍{[7e]zM'j42W$ }NJY)7>+:סk`6cp3S緧bzO}D=rϿj?ߦt)DN_>nm-/X Ob vBGvs "T 2.,h_lBK߱=*p{1oq 1.N 1 eӲ+}ja/|9/{lS{OQw/ auog?ȟYXLQ7K*L>s}*EPlss=-"Eg(6o_O?h ([XȦEh\ϝ.X ڋ^A/_iCOL&pHL8=˦ qEk(9|yžAD,PE9(f _~` IfE=}n}___[)uON?_瀮l+l۵Ccwɨ3X'͛߆I]z+ėOhP#=Ч=^~Ϳsǟe~ɾ.8GJ/{K7^A.ۓf\ y{5Ǟ?\4N\h (*XB!^Q*]:OFޢSt(EbnۀHѹ+.Gpy򹻟ѽ.:kP&pCONOP_<<|ǕlAER qp=|/ʗ^`_|Kmӓ{z/ŎcÒ'X,]J柯]$//'|ڞ /HN.qA N~~m:AlbA i+_vxN$,}k}揱o]?+YcK >wk~6L~qae3䁐rb!8?y(${(, r?h~EwD3?Ϲ?Qg?~'IGq[m|w|K2?GGH#ZK~h/H~P#( _~z[#G y5-s񏂹=_iCP:$v*~e^KY_(zpmgZS@WT,gSz.rqbьm{tЊvc׫qzB9?nϴ`͓XÓEы΢||-zȾ`)($EH4lDB7Q3/''iIאo{Ĥmo]v_F4xeC: @n3Ywt'=a)8;YIi̧* =lmsDgl}ұ7rG}eYO6E)SZ:0?׿RRb2 ;e4޿(.z ڗOg|_'ʷ)䧜6O%|2\W27yan_{_#y-c'k5߂`Ccp.籶d`c״K毂"o_+inΏ,(u ſX8>ꟛ?j| v&~1k>ɏKO*EG󇷧߹c}r"/WȬSx;;P'#u؊(isWמt=Z['o\/mqQ[סO.xcT;KH8#Ո1ێ7#>aFj=34mZ_h[314NƢ}Lϴrm> !gۯ1*POn܊,owKvu]gd+n4?eQ[Qc?G?GoYWF@a I(ܟ*n׿ ~ :zW[%bֳ7V;k3_s`^5'4ys*wo̙ЇH)_W.X\\xWB!Y{SǍ#]>";"+'gǞMh_,h("E֍rϸ.=Ή߈Qo|):7OMP*/iNX}xWtdx3. {ۧ[W+J秕$W0ΈFZ?Rbcoup(a+sI}aۯWuo;wV,ojofcu6(h JikdFg+ߘэc}2UGo|b5r0[R_X+ۂ=?M.5yO=M Ջ[qv> ח~ךt-c?>e~z'arM>stP}!۷fGɁ(;/\<3>0Ϗpr2藬[錏<~OFt}egtm>u}# i SOsEQ\rM/vvS!vpq!+:μ|z;St 6\WY~5~NOJ_jRߥ<|zu~Ǔ+jMT')_rdiY?<=v!u}EoO?a+3f0" 'K{N1$>] i0yH>}4?}Lֿ!~O+B7K/sǟ͟6h.Ƥ̏4žNwQigF4&{zJch:{[ ?/[pƷoOgbَ^/Jy\ډ. pOߞ8q_ "ؿȯx=ڱ~M{Nqv.?}GHXeFɺG#seIع& Dhgnt>/ŏ[_F|y=@%zu>^%Ax's!X\E$sD_Yh}tmSy|*]*Go!E!El]R&}čj}Q6.| [Y.$BlCڄ3~u쉀?خŧoO74;&gcx,kewp o龽ca]˭砣m*AD@7!GE6=jP۱<ľ'W?`Ȁh>Y'L"~FO_/QEǠ9PB~p?>n1 v~-@_ ؿȯm~/hGFB|1=ַ?}ĈQźXG_g?g`w| hkJL'/}|{̮Kp>%A\qO*Zj%]t/ qbqw'M&k~m.= ߸/Xn]E_{qKw־+O{aы \~Rs??C&?~-hH?tvq>^ŭqK*7p+ur_YuX,v<VsX,~֗BƱ>kүsf_Ga~xb*:/bX,bX,0Ͷ-ōr [ߟYOZO^?p6_1oϬBgwogwNgϧg 4p)?8qʫnUٓfƽϭߟĿSܳ^ӶUt^,"Fљ_|s_A~=}zw"ޞ~|Fr_koߔO| ~[w/, h}Q1k8J;#aȾ/Ba ylxDGƤwl ܞm}\vuLBtn2?fyssX {^[w^;DBX?_+Y;_)-Z2`#B|: 7tM>V`v:o(Aؿ"EʯW{Խv޻=ߓ?0ŕ ߦ+MA7ߝ~ ٴhK! C{++0x0||di8Rc)F.l}/~7xx|bhb(E:~HPĬE_*\'F| K!5}n}___[)uON?_瀮l+l۵Ccwɨ3X'͛߆I]z+ėOhP3k|r깹藦+?9n~Y׏Ιys{@~My=kv^zWΕ ;zvkW3qk־׌_\?N_cWѹ+.Gpy򹻟ѽ.:kP&pCONOP_<|Õ,AER qp=|8,?1/~ۤ'ٶ_\$ qslmks1@/Bۓ8A|0=@ϯM'M,72!>~ ک$Evw128X<-}g/ sl| 'nφiP]~m==!|Uȱכ~}8X䡐ۃPX@'2.?3ٯ_-6Ef~s0s,*pO^hY?6NGe~Ətkw)gײ l^ ֟}~G+ׇBC_ ou:9?Jzm&p׻:Prfo F}9d{Mſ?e}b_sQ??oNt9oOwm۠+\\Û ryUt T^>ӂq. 7O:cOJ9GD/: #l QgH*p*`LrP ?d x9>&MC"=`4&d5vh!?4^P⦌v*x حG;:\ ~Г[_ǰI?Ճ $Sgc|9Pmp!3[A> wN{'v) o?ЍE_)~?1텝2__f=׾̻_:[ľ}F(rOr~>]~Ts_5d͆19N?}}\s먃8Y{c}t9%Pό}\\&/ㆨؾ]5ykZOs_6p~dA1u3L4jT~qoW%bHN[{k[g֗158cG.vERtyB7˄a #آs~8^!8Z;F-f'{ zWѹpcQ^\pS%m($REgIc=?&CH&SAolI!j1!ktJK?$)\JzL}9Y*@Izbз |`SOб&x ?0P0kCN3GqEi|w?GJb֗$öI!kS}á 5i2m~, AyPWywע_Ƿ\;o~ʼ#ly|0ʏQf"?M/ܣu&XAuT>FNO){;lb_VStv<"wҎv}DvDLTp\ .:to蜞Eg;Kg+8α@ޙ OCxEaxs4sy+jEWT4 S1Tt6} NJJMjo쏱gXՍ+: oá%mZm??^y|p׽Y?雉?لWl#16}aWZL[& 0:+XGn등oȨ:R|֗gㇱ߂ЗP33u'{gi`^氳ha֜k^m.I)? s}kjV'F_5 پ7#>v>MDy9Zџy~a(-$ryg%br/]¶u8.jAfSL|:ɕ$5ߦ~{_Yg}؎ln?NG9GxN}?_\K7'Ɵ0[D~q'99CLb;OovB+LcϢnA?灵,ooA__ka R?.v??Y6h.19Ct|N/įy5{.?*ވF>Ot2j|s[]O?ܽD-lqy~X-pwK鿒=~vKӷ'a|9N|$/+^j|v_GS/ngpcpsߑ?}1όc|;5?#MᎿ@ԋc)Of>)F|c'p'Wr b42eg9?<9,>#Wc]{j+Eou@'>~~Q_HQCHm9$ BTI"@q*ZaBcЧDLy߯gÌ? mac\븞3C4O)(-|_G!Cq !m[kA9d,{"@DOkS= ͎??2~Z:v~=\u=y[o7XXnr96~hS'0-@7!GE6`veP,b;G~gQOd]~=jJ`o?Tpm?#']iu/OO"cЏ\H")y_ `_ Зd/k߬) Q_ p绹ma1bus8֯e;>+Y31mc|{~Kp]DhO:"ws5t֮G4Utf~6. H<ƿX?aeS]5Aȿ6_oܗWF,p`}ȯ%y?k+yG,Zнձm30'-i5C&?bqWȿ8/^ŭqK)*j7p+urWN*ƾ9Unga_1S~-bX\*x.RX8GzBp}q߬_ wo&\$4W-Oo/FW3cTtɯ)XEbX,bX,'-&o^Z-:m*B°>O}}o~ ;O{g{Na}mbqr~9(?wq{hӁ:?CA 䯱3+'k|_~=vۂj,Β.W,0џz=<՜u4r_\ew:_6~>=O__} mNAǩgV^}{pVͥwQ_uSVxz'~`-t.5A緣*:=ϭb6?햿_=kvnK*:9$EXLHs*{yAo{Jl_'xc7o?=O_O_9#ߟޝ󟈷\nZ'7õ(ǭE $)~VL$NCn0Rd_—J0]e<6M|#c`c;'nO}\6f>.;?x؉:&lZV{EO?_<`9=es/;|j"e!|MgG{,&gegUxk{C=B~L7ɯplOgc=cʿ [At+fѹu"dO vB?~Qq)*"4*:C)f~J|@SЍEw}B6-70ER|tq^ :|J/ zb2 _T_68C$g(X6]8/Z6~7xx|bh(E:~ ÝIfRHM[_zƗV %>ovݾ1ƿ.`/|97/!(v;p2 I~淡gkuWn ;CKGƷ8vEt_a9q2~td_ #z~oM;c _r8?0kk=3+vrT*"w9{ ²UtuvT@nm`foKOq?0#}=6Im}~g,yub5|alE{I[TI>`(qAzN~~m:AlbA i+_vxN$,}k}揱o]?+YcK >aώo~m==!|Uȱכ~pȇ9RK,#'2.?3ٯ_-6G&ܟ(ܵ˿i?ܓ#Z@֏6;Qdyy}$]l״f[}(DX͑?fPM^]?ϭcO<ЭY?}(6DclL_jR!\6>ֿ%]P!8?+=}EbsʗcAX):rnu̅ZEy?n]E@PNO3-p3-t}D(n:/XB2N`Hh3$%g?=؉oHg:^NOh`!IU5&du#H!Gi!M~ G;:\ ~Г[_ǰ?Ճ@$Sgc|9Pmp"3[A> wN{'v) o?ЍE_)~?1텝2__'aH\C7|_382Dy%?}*'O1p _o=:oϹ?5pk}䲟[G|8sƷ ؇GP-)z^c2^l?0v~z~S#cvIUPk? ~}C#gH|o<,|C FtdZO>fև1~8o_k F`+>RwY[ѹw)"r#zC|JYz]I/>e~*:n,3Kn -w1;}➄V ,ɓs,g=_pd 6t$PCDkhcmFU6$E&i<$]]ؗL'}6Aǟ poHCډ}(3hW^d6?dayo8Tz&_O<9Z;;H>S?J^_ nyZtk6;?~~WO_];%{N޸^i3@?C]穈?w|,7ῑqxkWoAu>S/[~33ݞ!1fDߣ?sOӦ1Ѷp&>?y|3??1?~˵H 7cmZs+JSh>9< -b._f l!:4?u*E`_9m|NoO_>1Z$&ZFM{)q*:SyIg/8 kQ Kp]$$.2^I5+(Aax+T<졂@ 1ey_3~z˳Aj(7Eg8 ;}\7 kv<~mLj`®O~?tE7xZ}&&1X넳\O\V PNJΊ}sDQo-uh|!0&գs('^609;;(/_oaG|v <*cűþ.>#s@1Ul|b;;A7Ɯ }ϏE/ÿ]~s7>o?sfD3?~ʵq#qGdG)D}y`~٢z.LBuW'xVkOa8α@ޙ OCxEaxs4sy+jEWT4 SQbd;Ԥ<{[ݸC`A>w-¶_C<>v XlB+Q>֡A/ V1Qed۟?2ejY&aŷ 7? k~(zXљ__b枦M EQۭϸ?k;fi?ggk^m.I)? s}kjV78Xs:(ྐ[~3jw@umGCkK-MtG\kp'#O>q_6c>fvڿ8CwTrvS!vpq!+:μ|z;St 6\WY~5~ 6mfw#| G^JoS>wI ~GlG6GټF#}OO'>F柯]p]_Q.O|ʌY-{"L8b̓e~!&ħ7V;!&I1g?ǠZ_з įI{Eu濊<=?܆XN'7hMDy/G?D;@ >pn \5{o971?}g;z+5~vKӷ'a:isA5>B;֯ip)pݿ~1 >"_3k e>y}d}kɾsLKD$q})~7#ȋ:,D~{gg9?<9,>#Wc]{j+Eou@'>~~Q_HQCHm9$BI"@q+ZbBcЧDLy߯g D mac^_B?3?8Dqkݲ?lt!1?b;`&ܿH?Ȗ8CeOv->};~վ1>cQXO//_/O~Kf _v=#o# 9 7`veP,b;G~HUOd]v_ϿZdmd}?_?\sv}WZ]E:*HzG"q9nL`~>@_ ؿȯm~/hGFB|1=ַ?>AźXG_g?g`w| ]Wb>q}gv[c/wť|K')_0X\ܗKRʈ_\-]k s^X{(:?!_juX ȿ8/iCָۥzQaEw|Sӕ|in9NX,P΋bqINX,?Kb!Xʹ}#?]bXbbX,bX,bqz4,0nbq<fF9rP~\ɭO֧ ' /u~8䯃_cgV~;巋O;x3e3gm~yx|8fUطg\ڿU9pw:w BA緣Ҵ[ߟf^?ku|_s֥YgmXD3 S緧zO}D=rϿj?ߦt)DN_>nm-/X AOb qvB GyK} _*t?n+6鏌Iネ=q}ۘ7mc'똄X>5d~ujtϽ'w(חֿ:V7wĿR\̽yZOG N\ <Wij(M~+e{?C[f=ߓ?0ŕ ߦ+MA7ߝ~ ٴhK! C{++0x0|Q}di8Rc)F.l}6SJNj}~'FY>!rPd _~ܩ~-`_._֗eu徕Bo]oL X {yH"ʶ];4.x矌:>z߼mYdݕB|).Puz`L{޽4<͏?9}1/\p{%_ɗ6|\'K,ti\ĽY^3{fV~vr/'i݋ۿaᢳ&EnRWwN{'v) o?ЍE_)~?1텝2__'72bl}{fG(sɏ.mCJS.?*eoGo=:oϹ?5pk}?޿LdW[s(~ <֖C=1qqiF_; Q?=ڱ}Mk*(omؑƳI4jT~qoW%bHN&:ڧ/c|k~q#ǒ_,n]EZE]x}pMaŢ.fxOܓJ:b%yRtE !OY=$tH}ĄHҁ*MVݔir.I %=&iž, $Pz=1['X<(!yj'㣸4矣]z1Ka$婾P~F?Mkj <(hOeh+y}YkvW?o$yW rl6~`ڳ}\_GkM:>83:}xC{wrGNʱ6}oO~9s>S/[~33ݞ!1fDߣ?sOӦ1٠ǐw}O%bX;;k=}]sSȭEbREat/ܴb1.Ws?gt^Dp%OJz*㕟T[<" s}cзt 搂a?BsH759<{<3:vr?]_z_`8&%WXV\;&3ؾ?HhcvV;{v}Z#/G /^S|7_% gw埸>G2p߱Gp?߲<R1ƤzTsOED7Fc?gg?=-숏nGaw8vg|(1!j~Ol's?\?#hUߘ3ͽ] sfE__?ڌ}8##  1g\8O3;VO+EEcI`s3(JƦOI)UI-|y1 ˷qWA(/%mZm??^y|p׽Y?雉?لWl#16}aWZL[& 0:+XGn등oȨ:R|֗gㇱ߂З?$,[Fϵ3sf~e=دؿ&韹i|zQvk3O>{هr!ZƠq1}O\ڷ}Wc頀BC&<pk@k?o+O]~Qu (G6o5 Ok/_W ~fF2cy˞/"$g߾w:vIl'NHsCR|Yct1g<֗-vw-k^?,|]_f"?w,06rkiLΡHgKJ;O*G^&)| ? o?`Gw~[w^{o971?}g;z+5~vKӷ'a|9N|ꤝ^_WX=j8C;_uo#$|Ϭbl=lwkG ;Dԋ܏)|gd>)F|c'pX\;;IatO}S6^)zT>T,oO>BBn!YJ1W| Ty~)־{1I`v-6FHȥ=Nڿ?3?8Dkݲ?l4!1?b;`&ܿH?Ȗ8CeOv->};~վ1>cQXO//_/O~Kf _v=#?, Ktro(9AXbG_O?qSv_ϿZdmd}sM`o?Tpm?#']iu/OO"B~H(X\ l|$_fMѯ_Ю:by{8ͭo;}#OtwƟñ~-}w/ GY\Zik6X#,Wť|G'^0X\ܗKRʈ_=->y?k+ĿO%%IENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_log2.png000066400000000000000000002435201471400134000343510ustar00rootroot00000000000000PNG  IHDR/sRGBgAMA a pHYsodIDATx^=4;w cx-رb'I6><)wqF3pZ_,VuWu7þbqq}lvw777p2nwGr>xpm|;y>?l;lqذ<5?2np [~ol[?w{v}QQƏ[dWtx*Gj&W'|[mm_B>v_K9_ew?n~4ƟGCb?Qj Ļ-7⳥~kFײsNlo4~؟~mG? ^|OԧoB ֿ?x9rHg*c,!UG n7yڛ?'o%Կ&YwQmYף_.yH>{b};GJ:d^"m}=\n?Ӂ0QEm|h#7gڜ)wxygK֌_Ake瞝bh^khz G}W} QquM4=ԧSa|!Kc1{yȿL߾sM&;s6}aZ4W _o|V>'Sw:̛mȾZS?A{_Ad^oR["!1N~#[O7bQ~d+OںbѻG]ͧpO=rϿkom}()v/sjQ9FgꯐϟJY8GIm/8C߰jh_9WY${ k W5C}&TDgB/ֿ y$.7i#C"Esy]cM~[_O¦<$e6՟;1~ip?`g˓SVq(}[QBhww7gW~4o>?*qJ~w97:EW+[[{9 ס+Q~u$pӎ,du/?KOצIJz%V3i m!Q?M2[6cT-n?i2t-vۿV`_7-/zO0iUrRgO枂iߠFǶ p9k̟[j;rx◯qE|0UdGf| zR?kc=~O% ulK=16n>v_M_}cygY iT,ڝ+}Wم ,>뻞p;G}noHӿ{\@:}d6ICHN IIөxE_ݧ(Ǩj%׉nY#~"`W?oC8\v'ٲ/׾xAI_i[ޜ߄uM\bƯ+~)?CWLB=~?'}5K}ް #oֿ~r?y헨[k8zƑ{g?BI+B7֯kY)przQ,ZB_])o?9?GQ_ٍm.O\_J?韀_K?<> NNZ+GEQu2s{B*X_A<7{PuxE>W_Aԓwe93'u~M[x3L<>W7ăOS8=sIsQ{<|y">mO&ۢ}urm1>MuSŽG?.Kk/?=55vƷv߿'n,\a?⟌?'j<3>VKrAKntGA=~OFt*Z^[yg| ʔxq-mNH/{>^]џ@}8}YSw)S \p9-HZGnd,FHw~&P⩏ከ7o!]_ʿgj_I~&׭?(>yjR&-7yX/4{ʏU-?hgl8gQ$M$jg mn1[/ vr{G%__O;)m#kW!uwo@}>sqwі| \a$;SB,B?S c.ܙem?:Ypdw>VnCjezw?y,asUlW, M+L7u~q 8Qlkp[߰A~{g&Or$rp9|(M@1[W&%{ZMy<_,p4궔lmq~-?oE]'!lk?K7$;"?mOvvƏn(:Cߨ2b1-6&+[?o_|d(07:hN,P_Y_vTs>gpwgn[)pa'ؓz`/x_8 ol:wB\j1KX@}?x^wS\6WLO!GscA}'>u>}W~o> pb]UoGOz=Bor\ c@} 'A}| t9ՃݜwpFA}εc>A}c?y^˷Omۙkbϫb U9~P_I 6/שz>Q:5__'yF7&(y_YվyUnyQ_%~K%2T_{>uP2gUw+P_:/ڋO/A}DF;}p 1y~o߁_¥?6ŐƧ/\>y]\cΎ_ 5 2Z_P:O7}8b[A} WԧӀh:v!XsgлwzeNxZgЯb.3ȮYn?jWX׺qu:̝Y'gd fWA;K{MOG/w}x9/SNK[Ec']~s%9%eAfCهJ|Z}5~iaN1{"O6^XIWu*^__kXa꓍'[4&EC?py~UrO &ۘp4ĭ79WこO(4aD)vO(aj ]LvIzi]o6KDU.T]^EY}$h|}dgGbo@}Q˅3wbW/ֿbKG}Q˅I2>y>o<\ #O:)rTgV7}K1=3,ȗxHVOGݜć%Q~gS =ڿ]3g 1Y;G}F(jUv>XP3Roo>c9u/Ls~c{*P_R_)^z7}VTO4vr~_j㜇MFsӢ&/'OhwwbvM^ |f9g*O@/w}K%,oCI&Lme73M}Ư.~(?גV~},P_R_J6 IHEme73M}Fz?TjQ_V_J6 IHEme73?>/oY%OFϼ8\>AurUI5M^\.$AOSVOj@`|6/J<縖6$v#Q_3]>Ek6惯:ح1@}k6^^_ '[/'bgʵUG[sDqVOg|DP_UG/ ֿkw-֗3~ڪͣ? 3'-oqVOg|D>p9`N^1l>$S_ $jgP<:|_X.dK}ibhG8R1FϬ?+m@Ϫ* &.PU?;_} ZwpQWQ_9+a8c}bl>G!JΪ]>Ᾱ/!؇*rVl#~GW!sk|B;ɭ ezEEcF>Cg9}k03at*s& -SK"fkL^v|>;yi#{#ƧwƟ#k>SgiA}~e}Źؿ֭|? Y&+hGOu?s p}c}??!gB};Z?3Yfs Y\@1bu_`XVr4턶/>x럷P/_ԧ&.KX@}? 9?oy!P:'ϥ lT >[<ԧF@uJP F@}'>u>}#7c`uuE=Vл\>#lP_ lP:Xw6OXbKpՃݜwpFA}εc>A}c?yp{ m{pkb_ϫX%A}>  |Q~B}/y/X>cY|G˯u 7E}>D;ޡ~K%0X_{͠>uP*#ؗꓞ=m ?Kcs}Ňiv{ӌ׿??.WO^w ӯ&5CA5O] Oԧ@:z;ר/`}9@}>EW\g/XP_/ `sA}sW_pȇ˷c~v,[(݋J]@-?pƏGYu?왯Y'&||ҽ.j~8 k/ _kF~},rp'R_9%eAfCهJ|Z}5~iaN1{"O6^ONr|^____kXa鞈P?!ȏ>[ 9^sl \p'}J8*䐄Ccuф;RGQ%z KA(^O^<1^,8yvWObRvJca<Eg$]eC+=#KOv3kۨ}Wم `C}noHӿ{\dMK~ezACęhSJ3&O]UQN7)jo|G,7Ż]D_AŗG;o*L$r׷_’6nj+Qͥu>ژScTM`K2G(j+QͥF{F}6'H_);F}6gM>p^%D_&]E_5M^\a.;&bgʵUGb~HAqLgB_3Q_Ogu ucz:#<3>xy}e%P\n+Vm'3>DqVOg|DP_UG/ ֿkw-֗3~ڪͣ?,\@qlZ(!5!9WV䬓msp|l*zӋd3}GgL2Nr+9v}z l\laڗ?޿6_ gO?>R VrVbtOymil$m!y3W!sk|B;ɭ)zx7ymilֿN^ٿc 1XP[#ſIn%g.?X3Mhvpubi жK V%%w H~J ڬ襝 'Bit_ ?4h'W_pGh_nGOu?GOׂQgY ?!_mJRGk pp}9ڽS];ρe>=YD??'/<?r?b||g9PLg3ߵAT+ޏ؂u}'zq==O>>u6u!P_:->/SA}]ԗαy .N~LhЧv60NXоe6cA}l DP_ "O`)A}\.߄1-|WI F: zB 1>ud ]> S.P lpS|϶WLC:A}/o۶G˯蘼&F~s\/y/xqB}/y/XP_SE}>Ue*hGG+0yGWQ>}q@}>en!vϹ6F|kԗKfK>uP2|>~B}^| ԧ\.c{$=`ymW4@}IR~"LF}x+S)sڏ?}/XP_/ `sA} .~[e;vL9G?'/_-gVzq+tߋ'Ӣ)S{(_.'/-~#'/5~Oߟї';G/9^8ljr_"G}}ęs_|*r|K,3ʂnG!EV__kX+ ٵ_B ?UݿʟWӿo;Ƙ>{_>%铚T)aRb՟\ؠ[er))?SQ!ߔ`~_ݣ+ɓMT.\??Ec&Ť63']/#IzrneXH宓z/WQQGiW 0>^1$>kYYQA}>XҽԎ >tJ_/?+I#ߋ .pkNMdרwfMދSO^Y.a*P'jxgY9E|f99;dć%Q~g)^z7}W!_Ћd՛ 1Y;G}F(jUv>XP3Roo>ow.&jUv>XP3Roo>cbdڰHzK69=qB2O=WBPA2G(j+򏫽Qoo?Z+~Z7r.0Cn*8[/aɏ{O2G(j+Qͥt~w5լզeֿ¿?"~mnX̑;ngnsd}b2ڵB{G}X}E*&%#wݨG#Kԅ!&gx 4>mzSƈsJ׻8AuWvMW+Xǃrmџ񁀘&˵xq-mNH/;/>^]џ@w}rv*ޛucz:#Ǡ\[p'v rmџ+/voȹbgʵUGɰFqVOg|DP_UG/ ֿkw-֗3~ڪͣ?=3){;$w-ALJ591;3kBeMH8/y};'~]U֋']'JΪooq_h?֛}fcl}[lsUU{o"9YϿgk)ӟ]DP_? 1XP[#ſIn%g.?θ?ؔ\&-=I 5c5>R VrVb} x_P,7O51z#k&hzym4A6P1pȐhO^4x_h)2M.IBoů 珥2y|W&1>u3Y&3IɿIBZ7 Q@}2Q_ο/G?b||g9PLgڊaSh=\Z7u3Y&3 rcE{zhQؔ?2j߱y 9P_sB>uNׅ@}럷^ .Cؾe&/zop1>A:%OA}`#N DP`;m)!WлLn! lP:Xw6O`׻@}'es.w䇁 R=mMu?@P{~~.~~~nI L޾]p}b9/-39rt}-{ GgO7_\]~Џw{w{ǯZ~N :_t~uPZz߰~P:^|Q:_>oFЏ~ ԟۋO7/K`<$I,zqn{w`J6i`qGk+DaDdO_ԯRg\A)X~;\/Ir Fזs|'0O c,&Ld6d&C'}~m)>J˅K}.O?o>k7dA"N!]_ftq䏥=.{g]=5#>ϑY>C&Tv|ׅ ?WݿʟOO{ w"1.Q?w }^?Oo#GU믑^9cLD|َ ?7Z~LW!''%"-:'3KBVi4J44q .nIcJb?L{+~.zP֟r \ L:77?,E$;sD_GFɧ\}.ô~KjghŠ=X?ӽb|Pz|s%~]?co3 UQv/O.ӵfrh'UqLZ-vҏSh9'&oWǣ?sϢ͑.HVc ,󤯺9w}_(j|/=񛾗V;Co]eK/=#KOvus"rc|*P_R\8'ÄL_ XM*e!~If ? _?x<+)F_ =r[᪓W{;t߀Vqo\a懋/R5o~ԯAjg(a|/H$so\~W[Z~~__ҿ}gmWv|.3[|Ȳ&[_-d ^#+]bɁ?VnnA_j@@.뱻/S=ǽ9'BzUv~KGo5uGZ;2EPUG kw-֟3~ڪͣ?wVQgDPUG /G :_HR VrVbzdg'Q//B=o_p}eJ>v(YS'w5HkU_-o"&qo~VHhhۥ G|@~rXwb?*'ߗo?Gb|g~p~gp߳ׯ^ګ{D^e~=O\1~m[b6?_Ms^en_puh'9V, TϮ\B~8P7B~8P7=|fB}E2;8٘?འ~A;O0~A;O0~uqe[л8nޯj. lPc`# ؠ~ F@/>\=m g9.k?uyA^kr<1|`ϭϯX3{v+PpD7_5pƣ߰~P&G^|UW~?3Op06I,=_U~ŝrRs T{G~ -D>X?Q?o/>_ԯTC_toX?Q?o/>Hp{R@@,t&zFaq9%V4N@~ ʳ7?ԟ́U7m@EOO `F9\WHI .&c~6 \n?n4鶇A`kiߟA@$ -L?މ s_-&m^3t G?7ʙ⣕K[u}ݎ ?O}OZp52kF}͟#?$|/هM\~i# 9X?e ?WݿʟOϩfQ?u72ڞQkdμ~@<\,cBP"DS%Ϣ]*mr }*h(>=.h?MP0.&F;ɥ!=Y*D=nkT?|=~NJJaQoa/",ߨ#υvoiEV{?`LAQυ7Se[&X?_ / >p9Ok3N.+%{mT,xHocZ6L.'svد oG͢_͏WR|&X&oWǣ?sϢX.I_p'~-!w_ޞ不;~jKNF1?7Q %ޞ不;~j[s![MԾB}I.~?\'bm&?ݝvݒ&]3Rd< 4cO.;*žgjRqٯς[;XdQWvOW{;t߀?J6 -!aɏ{C2-߿nons-g<>ybPj+Qͥ/R_Ǯ3{7ݨ˰<2(a|/HęO |r1iHEOӢU4[@ڪͣ?i̴8}'VM,+;:/2ˈwuVvM;[+XQ3(^ڜ]_w$~^]џ3&isp[k3~ PsVvJG kw-֟3~ڪͣ?뗡p9ukpOW_[yg| p\vN\>m$Մv[Yz(ro/\,ˣ\Bqѽs_ir#ho#/șX7=ǯ>}+~};OQ,t ÷20#hcOڑms!~Q[<;귥W~7=I+DcǸbb~~G!JΪ]]'hk㛞a$eW >pQ 3$}J ͯfBljNƉ뤮߂>:=-|K\юߧٗz҄5,,?!_-zWDkSȋkV|%fh2!8Yd8'ȗs BLol9ڽWww\B֜~P 6A>O'?_hO_u,bB֜~nSV>O'?_Ms^>pUpE[N/wJՋ 891P; 1P; 16Np|$aCZ,,+ȵqP "1~P "1~P=\N_]"/M+] -7|UPg lPc`# ؠ~ F@GO.8.l{E|NnN;η0pAP_~5:`~M_*V&/H>/>kWO^msn/"N(:O\iGaeE/ԟ o_&X?_ BΊrkFUNJE{N{HZF˩'We<3ڞ9prYю-Iߣ?sϢ͑.HVcc=ˏ֓J_޹_ȀEӿǛ]eK/$}O]ď%?$Q3Roo^~kMԾB}I~~g{ߎ^Ҟֿ7t_V#?7Q wgE},") л1y4xG⩍Z@E $sx{\,x7(\kK 0F>yڔWg'77?Ϡҟk/ __;ćӃ뷦[>\_" IwQ_ٍm.YIY$F{GYeJX/竰t^ee4p[<+VmӅ'Qԭc@W_[yg| p˔xq/mNI/{~^]џ[_`puGZ;gCVm2X?_aXNk6v__{QgDk eP\;W<3>?pV6>s9'NyG}L{'YeUX&-o t$2 jB;ɭF6?֛}bUMzַGeMO8+hK)ꏰc\ 1kܭ6xW>ݜ_t_ZF7ozo_!'wk|B;ɭ72g'hkӷ'~X?;zfQZj = Nζ]@_dwiO.D{RAjE'^7 O1h |*դP!W_Ker Cofj=(2ZE/r_d)fp?Gb|g~p~gp߳o])A>O'?_RhO_u,{ 1l\!ȏiWX+pe/XN?߅@q71N?߅@q71N?߅WS#Lh_:%Aw `%Aw `o8!C{p@G_IVлwVP;O0ﬠ~ w `?\>p9>'fz7 l{h?w {~~@P'ogpssM/2_8Ccs}b4ߐ?G/ gĠ~Qo 'wi_ [oX??<\.a&0)<7zvKļ>9ӂG9W`q?o>g/}AT{G~EnX>EQ>8A~E%(7Ur ./uҡI'g~ `q[!t!O5{8X?~uP<mKE;|(jUG{MaߙI99q Z~9(ws;{Q~iXMK/O }O ? ;,.=,EoeY'Cݐ/OwίR= ok{G}ķ՟!w"1k7Q?w fo_%?O[{ wb_vCC^~xdsg:;`q&rXr9 { ).ir7{z{KH09.ҙɅ&ibd_?L{w|{Q[6O bҝ>9L:Ce?8~%u$pӎ+'ltԯ)^SD7G}4bT@ԯ)^] hN6bv?p};\NC,;Mti49ssq ⚮YWS2Zi'WefDdҟk 3?\p^ka žЏڔWg+6/oIhQmX?L)Ime77MƯzF{F6H?g);F6oG}X?Cm1\p J2NI9ј4iT׸4{ÄV7JC =D]Bʮ9d{C-;W<3>Åϗ iukpOW_[yg| p`L~vb9;W<3>Z X?}]DP5|/귺w:nN P>p9$e9\UcѯrCVޑb4GE\_OJT`Oϋ2?5-L"$E`N.gp_$jǟdmab>Y n;&/ozO%_GE[稿W)Nr+9v1wMNk}g~%?pN^ٿ_V{߭_$jǟWCA~X=Ik)I`.*I@`J[XIh۽5M-}m8$wou?MI5ϵ_ oۥ ֑Glqcrgr0n7$ҟt_~hI !7 W^'~PL_ן?1~u3?8PLo7'XaM~{e+?@M|e9ڽW];;D{8~s|6g?puh'Ľ9Z Yco ho{!Pc ho{!Pc?x2XqɅ{*_ckQ@gP $_A$_Ap4L_Ehw.4܄[1i=v#lPg lPc`# ؠ~ 8\Ѝpz07^S3h0~C|[1 pAP_~5:`;p._C} u~C]?߰~PEswܯ>9SNh]Wo_tP"^|#_ԯYg~9A~E8\vqv|/xSW;JPU_*4bP]X?_=>JP~Y#u*P_R)^z7}X8s WX?D.ߗ_'w{{FM߫/]tc#'Ӣ'ѻa)M1&@x>ŷZmŇY Ho K~ɷS_=-.NNZ/_}թզe? .n?߷_ҿ}g9GQ_ٍm.Qq*>eosdQWv~sK7o-?ib߻O |rQZq{U,k< b*^MkLv#Q3b|]Bʮ9Ln}wꛡ[k3~ Pڪͣ?_g˵;ωrmџQO[k3~ Pڪͣ?_g˵;ωrmџ3'gQgDo`g.Sc賂k5VrVl\:|_X.n\1{ ٗ\_R!K36xW>ݜqr97ƧD6WYM⃅W#ſIn%g.?W7y?8~%m!y+_!'wk|B;ɭ귑?,7yɍ?DP 1X?Q[#ſIn%g.?m%7VOO˔brDϓLis;NlӊD 봀99²W&@;>`_,ܷb韹0gZ;,^7z-}uɒGlq\w~T&4WkS@2Qο\vU@2Qο'YLJcAfv?Gbj@2]ߚs!k&'W!N,Xk@0AkǃC]G C]G  $Lh_2;D~wC~A;O0~A;|r8>}\4ᔯSwa/߄1PcCYAX?61|'_E`|">fz7` A^4A} X?? _p1N~|ض=8Z~E01|Xn???s}oxqC]?߰~PEwe*5y,G&+09-9F?j iށ~:/Q:?Co/>ԯWij{ٗnGaD$`GP:_Fp~o v=h.h'ta}qPBauM'gԯǬtw[!Ëc `tAK[`{s{?C]P:S9P&M1?X&^K߉0tOMC/ }OI?[dO99#gT?YIκx~}1L C&T[9%bAņ]?7هK|[5~kaN1{"_6^]cL1D5׽hQ2kX?a鞈P_!ȏ~\p9)$Fb'$}*x*t+"$ Hh"6VzпU0hυW89vWObҘyu懛v"how]owW~HDUoԑ|Z|?L;귴]HƷM ~*?7JX?_=>JPtfAtAz̾7i4GZyX?_=>R(_puTӵfYh=$O-ԓC֫k'g=s䰢[&tGE#?,uM3=/;C"=#KOv귆RHUjrOpsد oG͢_3_GQ %ޞ不;~Ruۨ}Wم `DnoHӿ_ꨏX$U|zA濾I`IǓBm'_HHEme49Xq7#jM GoonR3g& ߊoY[%Ϧw=Tױ+ Ho K~F6o_;žgjS_7[/R>3ϣF6o_65}X?Cme7/pE*X?L)Ime77yf')YhQ}t׶I׻A*Ɂ[Vv698W<3>@|a3&ҧP=ǽ9'(B6v_{ԭ??_}mџ/3vD_j@`e쓨?ጟ?꯾j@t@rs"wƯ\[9f3swg„+!-<)'¨'y.E{zJ>+k5VrVl#~rqbѾ薟0v!ENKo#?S'KnqSNQ9/aǸbb~~G!JΪ]]qKM^rcg'Q/u1X{߭_$jǟ\گ}ު6ur+1{dhq^?9cֽOl$ט3a)MNŻ* BXZ H/?b|rߊ2dOnO_u,bB֜~n] M~{S@2Qο\vU@2Qοg_#M>Oofs{'Ưww p/p \>-@x c_|v_xo{!Pc ho{!Pc ho{!>~2C8aCZ8%_A?A]/~ F@]/~.QCH_gH_Hv: z~ǐ661~Pg lP?\>p9s&?\l{E|nj;X?? @} ?p~l-cqzh0}oX?QKG~@O|?~y-L!zzK[wrŝgFaD$`G~q.~oaExSE$= yo& (;OY/V kA~uPoJ?bk{d8װ~N}'Vk7Q?e뗳{U 9X? A~?꯺?^MϩDc'q$ w1QD_86D,Ss4d[>\[IJ44q 4/: zɓf(cc2,E$sכݕQ~u$pӎ-o_&X?_ 돑gG?J_G}4bT@/4Am!>J!rmez0Y޹䊇 Dױ!oY;{ɡ]e0MG3M ߎGEۉ/\2z]ߒQ~g)^z7}W/X}[X?D.ߗ_'w{{FMko{vZo]eK/ >ˁP'}e>,DGꀻh z=WVJioYSOp/Zɇ S߳Nn77?-\_" -!aɏ{C2G(j+Qͥ;7>˼d(a|/H$s$?\~Q~<A؇=¿"mX?̑@ar=eK[ kYM,+;6¶/2h|tTv)cw X5-\nc;&k6NW9?Eg~HDߨWe귴yg| {rv2ޛuv8'Ϡ\[p/v rmџ/3voȹbgʵUG/1uGZ;gCVm2X?_l;\%ȠwW¤LTA~ӏ+^~M=ZNNqJ9ٷN^(N0 Q,I~g^#O7ⷸ/\cǸbb}_ts~D4Ewjǟ׫ls6}&/zo_!'wk|B;ɭ}>ҍ'Kna$eWyO.կQrE[AkK4'0i 26=![dcX?KX?KImižJB\hry/A;+_`'wVP;2n^w\=Xm g9.k?uyɍ>V`upybzL4ض=8Z~z~Q~~zcfK_}DSDp|U¦,x>|kW~ǔj~̟WۋO?AʘKP|oX?Q?#o/>n:Km[OyJ 3>?dU/Q<_iauM4?ԯWxϓ~7?`}a})}N _0q~6*r6'vs~Qܜ RZ7y;_O2xҽ.K>My}ӧ?h9/tAņ]?7ʙ⣕K[?)mU|z5~kѮO!~>oJK}Or(u!GU^9cLD#|$2mħFzf$nhh6k7_]n. 06,[_l ONP[b35p^m~?kZ[SWRoN?/MZXHw(:O\iGwu?`L٭~r7 զ6Li7/a>YlF3EOvo,=3,ȗaȶH/^a3&ݫM-E_>qد oG͢w]/pҗw}ߙQ~g)^z7}/U hN&jUvX?_YbcᲵrӦ,a|/H$so[?،p4>6\歾j@ Xw9o|e3~ ˵}eDJ׻n2UGLJDߨWe귴yg| 8}]Sԭ)SW_[yg| p`L[Ġ6.aIpcϢ=p>KDjB;ɭt=luFp=ܜzcp_w+mrO/E?m'~TU~eJbo_!ƞ~-#ʧC o?G~ y7 !,맇bY- _U{oٹpI6.ǯݩ>)_S7 㧯vc&o)߂>:m@289qn,Pظцq`?'}mQt_y'O!kV|%f7> W֟?1~lg~p~g.o7]!+Xvq-._vþ/[ξ/wJ/OOX/nG C]G_H|ЦIZ#lMOa. DPc`#.%7g. euu4k7 ) 8y@/A;+TKp pF,)0ηlnS'{~)X5db'g:Y[I.d=|),˿u^?<$?e5߿o=p;P[ 9:kG×&_^y lmL6}Th99ŃCsF^v0 78NXg^/[/΁C=cy/X?ƀ^IoF痑8?ve>!?_?յ~?UoP~ym.dT -_%/Xh!9 }wn+ݞ =!u2 $"@nBۤ3xH-t~l@vUޡ[ 6NS `h{9YsQس(}b0vo+u/?TCAu5;R6<~cmiE{MZ ݋UHutH :_7t.׀$.c[;VBk凿 PWPnVF>{&rqיZbޝ;osA;pH. ؟9Kcgvi[̦!rd}[>ɼ=\}[8IcqPTk}ɏ[\Z3p3dCx-[￴o_~xHMv"5ݓpI(EТL2OpG>۟ )}$Ϳx-oVneho'}3Ɏe^,_k%G- &KWQf{Af4?xsk4)~o+.>MHF>tomG;ן 66a۟|_Ν.rQb_`ξ5EHo~te-sEkGaBN{/g%:}I53 is|O?EvEgO?z˝um$sČ?Km+ȢU???1ơ3 vZ,)ZH}}z{[/ uG/H񿫽fmg?['׻oUO ]YB+pw_ N1H~\4u+twb|bkƯ;?ZL|W~o~G qZE?}$3?C\mZ7u~$-^tT\Z>=aO+0;;w_.>u̶Q⣌h2f29?`GFNJ߅E5})5WD? Qu-O>mU>iꌕ9>}Egտ?QJ{"+nPɾfpό2+Ϳob4ft<[z6^6mw~V2=sdr@1r<,zf,nҨ'sAZLg9)M?huZԷ=-ۥ>=/raZ!:Uy}7?bMif1wkl}l·?+ر#}:o|"bjs79cgud+Ol?2?C!ǿclx:^Cpx:\n\Y\Ai$NM;VzQ+r_\C}Q[k6طտf,}8=--Ʒ'|r$5E(W)?_/*m^?WƘ|_>9]m_eWXA'=!/uUOt5V;X#4+?2R#&B~Qw©k|B7B?U_5~\MO~²#c?Ui}OoWſo"&ֽ Z\sN4Tw~?w|ŭkR 0E}=u,> i DR7bW6CkkIg7)&>}5s- F| Z=^?Uo|K1MhˌjɷBPJnˌ475sgY idVK|Iӿ.EF?N~mdk~̘ (pW}otנ~{_%ai|S?u`W8t'|(}zll-~~Fht#WqG#{Y?5bjtm7"Wh6KnMxlIejgj>=b2?}:St 6uLbpM*KxH>3QƼY<9;7ea"YD?%8gK`G9db"@^x\W.+{U|h Xj_gI-qWyMכ1F|1kH)?Z?AgHdl?߂Ie^&Y[?dsr۟ߒ]Onl&G37/n{g3-#=&H!6m4C(_g|پxUF!*9(?Ҟ(or1%4z |Q?3p5%7OvSܗUCcl?2'{HL?ڣU{j+էmğ/ӡx^:#nhLϏk_ysȷc@;}h+ͩlI=7g`\xzqKWG_S>>acߎr s`ql%_//n%m*epgns{nޙE??Mf`c9?޼>b2?6W\~u@`>?˿>s}C|8\.`s7O LC^-n?qpQt|gYkWc?6? O7y6~=Xb<|Rpt.S"MQ7l/H [#fzi?Wƍ~̯&G=fdOo鿺Q1>=n)|;@?(@V) >:Z9#' 3?l(Ui}nO?5ߞrҋ'-~OA8_0sz"@"ϯZ/?Oߟi3ۿ@:>E'_~P6~}I@:&e(?WkK}??ıZs_|ך_ڋW,g}ճap}|;Ok:hMݟ8\?rn/tvBh򷝟X/_swĻ֧vܼ?X/i!>/@wXH_+޴?Ojy '^dL:qɹO/~xX(>B]e|g3ɟZ {b/ڛX, 'h$[σ"?ƷҫXZ>\ܷS7G5U[o?5AFz q⴮e=׊e,3Bo,I>>M6O1o-jg\k!.zc6OWeG_J{y?_oɺ%!Iylj=ڕ?EȆ)KcI!͏WOh 럾|pq0[Տ:n_?F5~ʷ94^ ԟV1χ+ރ|tiw=K?(AG$+lO>м =@%\]v=1պb?,V}o!f[czMQﲿZibŮŎcgEOu}s7@>+s_?l';JBD[lR3O~_$ޟ mҖtXTȡos8nUJfc&D;~{uQƷܬ0? [~@y=67+A`7ۧǖ_GX SQZ6j%rAzRF@\z(kMcFOsꩯO֠//Kioy{-DU"Ys#R~~yKY8 2ߖ/Ҍ\\+h>~%Z!c工Xf8bZu\_k`?1 d7V'.;z^_/GznrCtV_rck3-+Zmw`?笏t!_v? ߸<#zG`]WD~jW>KŢoq5v?\^3~aQ'O8!yVដe1/Q104^&},$[_ O{C4/ZwS@ [Vwx$n Ռ1Z ]10)V5ql?B2~l?~jWO>o S6o6NI~GMݾ GZR__ǭvNߛt/wЏQ^?u~cg?a~mƔ濪_R> ދVφN &i'ߵC hП_u˵_p9O 'Z|l04}&z ?S殉_h'{8KOojǯS]Q_]c2^~^`ϱh h^[=&8{O5\/Ĵv%-U_og~`t㷀d7̸$WoO%?_|cwggk\;wiWc 53O^+MnYo''u7w޹įf _Ez8]P$qzL{BY=M7~_^]g/S?i|CYF i~wtg 2>˯x/+yތݎ;*:WZ{k忧8؍Zppyii^ f~?^3no@kVY~UFֺqZ5<58;>3M()L/k[&Ŧ5bgd$w]UYc|WK??SlԦM~c'xk[~Vv?v?fg)t]eVe{o&,ug0r%o "5*ovo}k`٦٫%vIrR-\U=3cC}L0c{wMv.ܵ|M57nr&a/j^)G_k bL/[]_b'e҇ b`gon̅6t8;\Yֿ֯] O縬X3k,F|v+~f~O~5h\SZ,{n[SGn I6ϒhg)ROU?D=AN`N_;ǍtdltD?] _1~Hv/p}maū"+? lկ&鴚t"+q-Zԓ_5Q?< &%v* ި<ZbM{Oƒm4^_ ]ۣ>4+Yۇm? 6>W 2ynZPTm,u6uq_@v>M8xA2_@嫢CNk~l~;~`dj/+w%uM?~[c/r4W.bʯ}GXcjkaͯFqJ3v%AVHֽ0X8G_*m_ES߶~ ͏"=7߿ӵ_{f1~e]oW_˶|H;75!# {X?/[.Y}?[ԁkƲ ]}½w\lU7~*W 5 >p| ˯cGB Jw8G؟;?b L<:|7O?zwTd/MzF88)ĚW&=/pJ:!nW Cx\88 E˟~np p9>0wu;gQ>ns{fo?B6Xsس@Css~o>ْ>Gx?x>NP_^:\F p.sgLV~>_P}e2~pl ^\܂?wٿ =ܣ_}Ogk}pc~6,sߋ϶u0Pyg>ߋk|K|8qʷPe_lk,6Շ-. .iqY_OKC8(ծ~K66.ڋ8Y}?[?V+=b;0P;;S'qkti. ؞'S! Oߨ[ȇ76XصŇhC}|/JWOｼ0/xto_ߖ2IU~i/AKþ I׸À"_?KxTۻUAG;U |GG 䇩_s?ҏ ?н\3ת_'[¿9oGp9^/&[xW/]O说׸q-'ׄCI?~Ú-T2YPKϫP?wt]˷VNO|W>˿a{':5kwW9>|\V^\/FB c4^zq?C\&D9w,"UV:X1.pkq.O"NJrqg?]-{ZJ7.~DEJ.:K/^ o/矢7w'>oO~_`B[^P3'wbcT~&ryko˟_8{{^K{я}n|߲]n(U#;pgsHN|X?u3GUayJ$^uUyG23]-&ψOdb4F|5.Pm<5>Nxun3]sC ,?{rg|>>|r 8-_]O|Ws`BeݿP^8.T/- +嚅"?/QAsn9<_I\Ⓢz؍|vȖyW2Ls=+W_w*_Ͽá8~$ZmD뫁z9c~сr#|2a- ?ay6Pp Hw8O-0e&שױvU 'AΏw=/OOte6iꇻoG3|É!>|Gv?p N0'` o̯OY?{i>7ZUg~oϟ5[^u"p - cLh3{m ?xŋТ{<O:@ k֍s:G槧~vO,Y't>:kѾϮbrc.S p9}MQ! fuyGoȮ.|̦@w?_wmagxqdRm>ˠg{־U /~p_K9_ew?n~4ƟWr]>Z-ݭǐ:{r|+=CWrVwgQ=7'zo?x9rHO$]NRI\,tPYäW@41O8 KM>mE({$"~>Qe~_ vז[:\V!okrw9qћ)⼤͝tUc?׻٘gV<{}\'0jBma@1A\u?Kq7{wM֏=\q?cy8G`WpsXdm?xImEr;I}3IǷoQT.xs9|4}/|WQ C?FSR'&q L q\>K1O7bQ~`O.\u.?->0G5? O9YU =_1|ezH]ٶ C݇.+k.vM~%/ސ^w:~{RU %N\'kV*dYGjcwV~a]Ӷ7GfןX)ma =]O\S>ۯa+u}!`š?5]1X&˿_Hf<埡_o..+}[EKbt~Wy7' :f"ZhŘ|bדy{dX~i2މ=wr$/tO/d78_=!DPPR<_bv} YOzw%B?\?~s99\~k;?Hu0.?}5CHV~݂%F=Qs5ڏ~>;k_Ib> |9{sq'ǂۍ2?lXoyC{o2;'}ʁ⛁Ԯ\P-M? %~Mpm,\mYf%ŒA?GY?AfL}F%?וX˿oGc:}sHӿg6ߌ,<™JD^OS-LD]?A;M5kAMm&ߠ$MH&'E.<>ԓs5) ɯؿ9>Z{!um?sOoP~lg_q_pyzr-pوO_ս]X姑^j+ C] |ׁ+~Z۸av>\1hk4_#9p~u"O/ޓ}7>G]LEݟ˛|m?$KTc끓oٷ.lOp$7;&y~fo`K"L|8}&N?.rkS3-3C}v|MQ'`.Ǐ{VHjVE3_/w6m]nY;7[sUpſe7o~0W.+#l.\I/a2^DATINOMXt.%7]X[.Sϫ?w_͐} ¸5~-$DU  lAo7?7Ao(Oxg_i"O~L:\vůʿ#>,]-?ohw駴26x g#%~e _σg~ob깅dt~uޠ'?$mO#^ԇdk|+'ʷؿ]?:VŞs?XRo ugLLc/ϙ ԓsnL~#m.<{ }M+ ӛL۱8=s۟lߵ׸w'?gV՟fȾm^?cLJDz.*6f3?mjZȯiaƮ{Ba9~,7d7Sfж@}-_ 3FU2/Y?~=(VnֵjׄkOѩK7"wͿܠ_g#ć/g┿u_%~>D=1>${dG°1?3ԇkwt6e\2J1Gnb,Fw~& gbR(&Ie}<=!oYMR} Y>_E2S.@7TMɦEA>ht=GO?Qþ,G_o}/<Ip.9=&# TR~I(߅_5?s/2d_ԿV[7x7coW?K[P}p`O1G5j| 1;?(ĸlw7u'wAfqHz޴hy3 o٧cF_͈RCs G<⇐X"g9.[_]eC߶Oɛ5N\$O}$a߃ [>p9&]zò@|_{ӄS߳x0 eLIT`ErB,Կ=htcp\NU;!oh}³؜?@/T-:bOlW,'wM=ď}I_\Z,.(2/N:OO:m<6ណ8Ɵ4ɞ_*'/!E1eL#~r%*߅_V~ԿU0cBmCOud[G]_9kΟC2!G׏D{b"OJ|'OZ :68ynQkןhQ[_Fo}2z 'Σ߄-b*^vX_u̟#鑟WxC߲O篒?ٝ_F}W>WWτSq&L.| |C/n8XS |8\oS\p C7O\mt}2_?F?]uk~4G!?5A.p m˂epYCW/Gq_> $=N -A^p S.ȟd9'vd|zb Go&jaQȺ{x/|f `S38:>WϮ94`S3p_:kѾϮbr.SClt=dv.ѾϮbr8˒ڱWDEo"SExtdI??*~S}g{quYc?>i~#wpcsVvLli|H=eܶ7xtߴy#wpˋ[XNTc_\|%חsvH%G8+@@y21 ߳3E_K-3ou޳}ߒ_j:ccCi[ؤhn.in?MUV&ist ?"V|J[Ec)CC>Ϳ. ']nS>ufM3Ga:ϬwMW~*s  @}#h|AVI=F7, J}sytY*]5Sv>Jֿت=aO?+֟?2.ԇ~>g?貒7]/Uk'Fw)pz}1vUz:\ ȉ?'%JH⾜GL.e8>-4q rճ8~p6nnwć߾˘b{ЧLｊOӆ(f0N?a~,Njl=pm,T:7?m=tb_ߠ,F o>#钟߁P1j龹}$ {3R}oFψZ_| srsr+>=bT_ԎOogOa~G˿oGc:}s;?%?#j}Yu/y c UߺI{8ʮWE?\f4',Nybph҈ >r2Il_ŴG>39,о~isyOy&7v=CL58}貟F41nO~i~y~fo`K"L|8}&Nb_.~tQ{jF7>}E8b~f8jWɁ-귁vY$.g'?Ma ć/g┏$õk:Ur`KmS`.&75~m[or[w0y,`]) #lG~](#,;M}4Zx7?B2Jc/#]dg1dWŕ׿!Ɂ%F.1pk-9G?o|Wo7 9ݥ>ʨ-d}lЏ?2]~VNbc9-_ω .Qv*rLft'AK=qk]fAKGuv3>_b oN9YŴ=l[Z[׵}Mt#~GҿONlЯ3֌?aɗ3q狫 >D=1>${dG°1?3ԇkwt6egяźwCǛZ[WZ~8}&N>$]#)?~,ֽ>|W sMn_|i PyK$XR`?@Lzܝ_ˉ3j&#>K`Ym.OzOA?k;q(?0)D?i߲O8ӻs G?reпy"g9./{qC߶Oɛ5N\$O}$a߃ [?|=N][??OA[XW3g$il]oۧD? G= }}`cڵ['cpyfI%~Խb$D]c6cU4Dz~?/&0um$%wL. 8~ra'do\FnuU/W>} ?Zcd[a5`?q@|ìˋU:ψ{ϑOsб/R[o}[1>GOzvQߕ3!ߔo3kڷ3}&X]Y淳8|}8gYTi [Ł5f_d۹ɭ&.QV˵m_F.7ލ|_|W3&J16\L?HK~*s  ߁pٯG{ȩgO;1gg3Jo..+}[EKbt~bz̋+%~kgY3%N|>Jֿت=m]?HavOE~B6\}te%o^t(wORLLc(Zb? _WG<\gdP&^%OSBS%9Pr uGƪOJJm$.$Yu`ofVA/7'yA"wo2tu~AQ@}(t>=\R}oFψZ_| srsr6߆%#__#ڳ~GR}Q;?Nm? 9.ԇbmQHe?!7'1(W'Sד*;'KEd8}#.ȉ颞<\ %R<1{V<~8}貟F41.>'Z@87u+2[suaɗ3qϋW^rkS3-3C}v|MQlJl/Aw/WɁ-귁vY$.g'?Ma ć/g┏$õk:Ur`Km`x|ݢo.lO? c%a ؿM\fr hykV,0.Q qAɫt?߁wo@#_5^7굿mKP])߂OF8YmY`#߁wo$?7Oie?\2|>N`VW(RAϧY&(\*Y5bOѩK7"wɉ 9žA[38a2K8UX{ZJQw =2y#a捏õk:R2n|i?ꥍ#ߺwCǛ~i?~vdq%LI}D%H6?|MGjS X{7A}ո[u%99߫g=L>@/8~HԧQ a;ӱkǯf?)Z&n8YvAxw8De3"W >mH_#?ɯ?/ϕMtG= }} R㪯1.~zRv1w3bZ_fb}o(?0)>˿6zjFD? >mHV!#?H?$lzRv1w:cX=ӱkǯf?)Uia~mF:KC0J{KtS&p=MfuFm_=~ '/UP5:=F=lW>d-Eb{*O}?ٝ_F}W>WWτS7/]~!jpԗ+_CdQmgl -g^fF}@~([_*]@3G~ْJ~Die}Ps^fF}@~([_*]y\p7p-gxq_0~צׁ\` e0-x \/,-?p摾8=S.}g!D}\/,`pl"}nw@ ^ԄUu!گ_[m*$\/Xp@7ϭ;/$>&#a/%*"0}R7y?7pz:Y9\2Qeq= z캀N _\M 6:gϞzYB~-k#裳틫o,6] 79 gO,Y s/Et|*cŠ )2HFz_?g|quY=Nu4n;Kq9 d+?;Wu gYT\pe'owO c_\)_T!յ-e~;+חsvH%G8+|\G,?|.cz׽Iw##K6'bkj–~ղυ"?O* لws3LpefA?~g~yw#鶜T;'Z-?b zխ_f~[}[EK՟QRLyqoum,riߒgCi[-"<_ԇ:EFÊOic,ehcg0Omkؤ_? qw>\k4>?A|zѭ%_?1:{H1k3kjs?4`]j`#@}#h?8>-А\&69u~|D\>Q?Bb6}\Tѵƥ}iYNsr! tZ8{6L跟O.ȾV,<"HdP=uې9{+kdC{W?]/j7_>gįsYVfgd ݸX7? ;P}4F#7cP3֗%_ܜ0Ơ\?z O_O.cY`Sؠ,Q@}(t>吼ybY'\\~)hr_.{(ZFh>枚&6ϊz|?H>\K^F6?ŋl]Ur`Km`x|ݢo=CL58}貟F41Z?Hn*N -:0K8c3p.{o{Ҿ~iA~""L|8}&NL?tx\_'X~OB\edrW_I|=Ln,ߘ\o^-?,ʷؿ]8Umۨ]Oie?\_JH'NAfX?&B?%A?NVtw>$][9)(j?|>'6SG׿ _? qG˿!8='=\1誄0ɗ3qRAaRKl%?Lx#a捏õk:R2n|i?ꥍ#ߺwCǛ~a?~_,zPd݂85ZjStҍy~H?9A[3 G'_)G]dct6egяźwCǛZ[WZ~8}&N>8$796?[>Iđsɥ!W"o\r|]ɵo4pӿ'bp?Ldd(?'_G~ҟrby_Ar|qU_1>b\8vn/0w j}z濡Xħ w|-t,Os߼ z?OcC~>%o&R|U>?Aɭ[Ԯ]L?Xħ w|-t,O3??46PSf"gK8\#$AlU+r0靵%  [k WL.A_iL.y&65?\IO=+;|毘 |kFh&O]~!jlZ@DxKc@ί5j F}{Odu{>aѢķo}`{w ӿ&z gArs:ܫtu̟#鑟 䯒??Q{vY+׺Q7-J|1֗ Fw~0ko1u̟#鑟 2Pp2 j/@ܜlxX?A ?H`Wp8nxOv;G,5fS  N pe`pe [H_H_^W9htC_㿶uUH2_$@29@yXakNܔmƿA7֗F˿Wif `Sqt|>] asi|6:gϞzYB~k#裳틫o,6] 7igMuY^Ўυ9WߢYl@n`s9ZN,OCWIOq~gYT։x>VvMI@b/\n_j[$M2ݓJa`ra#`um tߴy#wpˋw ?_ ?~gQ=??lgg>t.OE]W}KR|wNuu~\nͯ2sT'4]_lU?&ߒ G/SM}j?mևVXǶ@ #~CI}rF]/>c~%3 kzt"?`!?.ԇ~>Ɨߑ`cORLLc(Zb? lׯZȸ;P.5J>wtVQ=~\͇?a2O"%Z[|uP"U?icreº܃>UxԇkwO"_?: 7pI r4h/yis`K{W?]/j7_YPL?a~,I'߶B+~3s/I7. 9b2f32./( ]~>֑GǒaCnNcP=uOҧ'UvOlڱpm,)lQ(tw>mhZGon'gD/K99aA[ծp {,F]\1) ^&W݂_sEq~/GM.Ʉ'Aw/WɁ-귁vY$2,.{d #jl=p-6e??ib\6 8Hn*N -:0K83@]G!ﱏf$9Mx#g[/gp.{o{Ҿ~iA~""L|8}&NLa>\K^>t.埐x!q#%95)x[rL.qk6j5~|ғqbCK?}Qp[F?Q稥_JH'NAfͿMlO= a Gd8YmY ok~yq@}HƷrr   Ocwv[P])߂OF8YmY`#߁wo$?7`2%MU d&Sx0&CW)P=ojSݍ&gʻm6~| f~Poo!a0G =lYg~=(Vnum_ůf?E.݈o̿5pzDp%LB~}=-GHmʸϢuv3?7Cw dq%LI}D%H6?|MGjSݍK0pL3sG GigD%MG?@?" }} N.Ɖ1~/㢏)k!O`خWYXA?k;q(?0)>˿6zjFL:1ECel/^Og@~UcC~>%o&R|8{+c-_Sj.nOg~,`S>}:mՌ'BW}()y3O}BG~~~2Hnu_`c1nkHq0;\oJ؟"~4}x~}\cP1Lm|}r/L*>og6M2 Pc@ί5j M.{?d6o?Z7 >:ȏ%oۧcD߂MBo~3J&W?W?#?G#?AǾJm!ogGO] 5)n`ԇ= Gz%L[0 D=H~z9-GOpeA?p&#W{Z| aװ`^@2߳&8\vkG,5U;t:/!3 ?`W 8."}僞оN7Dk[7|Uu\;g ?h`W 8.f2~~nʧ#F:Go~,%3*^JUE`l.^>^T~*A?`s9 Ԍ*0~+?\d'F1틫o,6] 7iݿc|p ^j9˒ڱ0G>[;M N?S/KVh_ >?>hC?ɜ?~gQ=!x>ֳS^ϟIzJΪY||G,?4?_ /GsvH%G8{QdǾzS~w* 9C`k[8vV/;Kq9 d+?;W8|}dp~wV|3~ Ү/wA_XLhn.in?vnrk67OXYX)ma =l 407w)n⳾16WO?a:~{T,d߁pٯG4>-zo3Y.貒7]/Uk'Fw)ǼR뷺6`.QG$m"?`!?.ԇ~>g?貒7]/Uk'Fw)pz}1vUzzpvS2ĄDo&QzԞ d"< ڳ~sAP,w_e?=S&^m<~u&Zh9„:#PG:ɷP6,_:|IqoQpuY7Stu~AQ@}(t>=c7gD/K99aA[UoC̯ Y#_ev '߶gBC߶ƨuvJ~F_؟GϿu]a)pۥv8\ə?%^aXg-Ʌ&0yc%ďdɷ@D6qx|/K- 3[suaɗ3qϋW^rkS3-3C}v|MQlJl/5^%g-f}|貟F41NdS>?l8jWɁ-귁vY$S?UrŲMN%"]) k_6I^-8zrkykVѱM~#Ky~6,F_l5?w>$][99?W%(jA/zoF/Ćv~J(`ky'A?NVtw>$][9)(j?|>'6SG׿pr7i0>5~`?~vڔUB-ߴ =gY~=(Vnum_ůf?E.݈7>l)̿5pzDp%LB~}=-0lG ]5Mw7zq_T6hKGuv3?7Cw dq%LI}D%H6?|MGjS X{7A}ոx3)ￏy IK@!zo៹m.OzOA?k{]Y0)D?i߲O8ӻ̓ G?reпy"{o>#W >mH_#?ɯ?/ϕMtG= }} R㪯1.~zRv1w:cX=ӱkǯf?)H$`C߶Oɛk=/{Arǖ)kkO3?0) >b/}"R\(+BrOrV1%QmR&_'V @L.Y׺*P[+pmuwBo=MfuFm_=~  gr~1_M^~F=">BZ=?=_*>ևle؞J~gw~]\_1?M={ڤ};g"S_.{Z7s}0ÞEo=FRM-?s$?=ϖUw'O{.{Z7s}0ÞEo=Fζ\`5uO+\ ?H`W{vy/pE{|#_̚.]3g@ O~p `pe`9r(0s?֯p t)ov    :x,vлKoЏ{ I?c/%*"0}R^/B|\*?fhx.A?`s9 Ԍ*A? ?\D~vw. z캀N _\M 6:gϞzYB~mk#裳틫o,6] 79 gO,Y s/EtC;sqH}rӃ*=/.ljC^Y: o_pvH%G8竁||G,:jpeQ=_} ^֏}q.~QUs0V׶p?>__.#wpcsVvqrqI?8\}L 9?]ӻݿzk~ UL.7ލ|_|VM3&JQ16 Zl»ww>\k4r/zoF\k4>?A|zѭ%_?1:{H1k3kjs?4`]js>>LE&] T@Tw!x~p&$JĪxгh2/uWpghAږA@Ez}~y AGɥv%r- ]~>֑GPNe/7wd_+_ψZ_| srsr6߆%#__#ڳ~GR}QYZ}? %~s'߶gBC߶ƨuvJ~F_؟GϿu]a)pۥv2|>  sԟ)?xL'0"OnM.mt/ - ['FV'&>|>/G!ﱏf$9Mx#g[/gp.{ٔD/7_t]g_t[wo3u>H6%ďdɷ@D61? \Dp%Ldc@}v|MQ`÷нJliU ϴ[ te??ibW&>qr $#K|1$ңw 6coIͱqrYl\bј GdZqzw&~w>$][99~]cx[rnm~k.B}Nlhw駴2H{ү %~e _? qG˿!IL9G?o|Wo7 9ݥ>ʨ-d}lЏ?w\AH&mz +z}CN݅?DϜbڠp?NM|8}&N|qU!ևh>枖G]d+a 6o|P%_ӑڔqwWwwK۠Q/mֽ>CHa[g׃"k5l]ia3_K8cKltԦ,Xnh7x1Yq`OKt$JH}%"mr 2˦w3hA}i/pԂTN&n8YvA]& wKOL<;e\_⪱!wՇbm7)>k?=NvM6}bE{g _ )'$W\c`_ E/_Sj.n߷vvOlW+,[5 &>{o٧cF_͈RH!wՇbm7)>*D{g _ V-_Sj.nOg~,`S>}:mՌ''4;.8"ѻ2-r}ɝ]2jT u[j/U@7<{-3/F5e<_*>ևle(Ul_%?\IO=˨joʷֿlf gO}i{_!_oSI6lc]nu3>o ?Zc/|.aD߂`c ?G#?Al_%?z첾W9Oiu3>o ?Zc/|.aD߂`c ?pYp{v j/@lxX?A ?H`C»/,-?p摾8=,}TBD}\/,` .o$}"}k H_٢= ~mmW!8  p@7?\/;g+q,z|9E-tNs ꟫st|>] 9/`s9 Ԍ*A?`s;ezYK= 6:>^8L 6:gϞzYB~k#裳틫o,6] 79 gO,Y s/Et$`vF?\;4ADURn8rYߝK, :o_pvH%G8竁||G,?48\;e_}irO0am?~gQ=??lggJ/;Kq9 d+1O˘a 6M* ل7KR-6?I:ou?a6.5{?˻I9~+y3\nͯ2swtVQh~xc^\)[]?ˢ)qZPڵV59hK+O7S*Kzhuao?C/> c~%3 kzt"?`!?.ԇ~>g?貒7]/Uk'Fw)pz}1vUzWO?d\}te%| }rC !$zYP\sAS"_@BAMRN~۾4,'}:}&4!DW˄ukr|⒯a/D*o]Wm ٟKG>W5ڳ~GR}QYZ}? %~s'߶B+~33~n\,sԟŨ#\dgd ?]s]a5?| srsr+>=bT_ԎOogOa~G˿oGc:}s;?%?9p9oN\CG>!M.4e{ci KG϶'ߚ\8jFƅ[Eu6hE*9}W60<>nQ7kߞ!YBH[|˾ tOdw$7KDs\Dp%Ldc@}v|MQ`÷нJliU ϴ[ te??ibW&>|>|&Oeor M.cCI|=ʃK7 pe :k 2_5^7굿mKP])2RqBdi"X?&B?%A?NV#/H6yۚnGscwv[P])߂OF8YmY`#߁wo$?pi.eoq(C)1a>$VM ?H5P%_ӑڔqwWwwK۠Q/mֽ>C a[g׃"k5l]] W3SnDC7ɉ uߚQg8="L|8}&N>$]#)?~,ֽ>g Ej-غbӒgɗ3q'u _F?\7 D|?%q$o)A_bmcrɓF]>3Ngj%&D>1=/ [+?1~/㢏)k O`خWYXA?kʏL| b}߲O8I?(;ؠ_ןθ?ρt'Z# d>qJ-̏L| b}߲O8c#OcC~>%o&R|OA?\ԿI&ůIż%? w-5ڀJlegrEMl~AϺlӷJ?>noʷֿL3i QvoɈԗ~_j} h??d6o?Z7 >o ?gzW0l3N Ё'u4&Wh81= ?`")5E?R7}X*RZQRM[0b*V\OgQw9iBJ|c691gj/u3QRM[0?s$>='~hb 6⸸19 7tx83=:> $pg ~*(.J;s'y^#F?9;{翉ߡ߇gI}~.&}} sJF?򟂽?a@S25o%+m\qDu>޾8~GA?Ok~Kq]'!x FR/KVhGsb}}t{;M' ?8˒OCi;eT}Jz}'X:Ӯ7qtH%GV0%_?1{HskgCiO_AT o?u qw`}#h|?A|I.CS?\>JUr8enK^DUjD\Ӊf1Nԑ{/a8[_ZQ3/K9=aA[+?I-񯲁bOoggC?@߁oGc:qv?%>#_؟GB\\'-aڸo6[fBǗ͠Q/|:qXo_3es}|˰cl%T'߲&tBr/OWq:wg=ua懓/g┟}W^rG9ևh>昒(76"|1>3ԇkwt Ub}S:i(#i~""p%L̲0P%_q~/:VMg3?l5NKuۨM;6LK?}Qpk[L C2x ÿ&_L:5Fdn?wNtkF0ɗ3qU>D=1L}D%H 6>\K#)nE?؎v3>7Zۃ+򟚬g|>|?Ft6e@bnp2 bLa%̠N. [N gD₹~/E/_nm.Uss\ ۃ>3+Xo*c3?n>˿6'3}+ܺǐ~(;/~?q'ρtUeCZ}J̤i_#>o?/υMt8=/{Ar@c)k.f1>03!ӱk_Ɉ<#HASևbm73i~>?A-k-_Ԯop|r|$%8Aoo6Z˷Y6iq]Tf1z6e06NBc@^bq E ks bB;V~"H1~&Y] l^'Qo9b}ȷ1k߂ēdk|3~9YV\OgQw9i*>~>_%>Zҳ9]\_1>M,Y#Zl| $fK>o >j-h;0_} gO!_%>Z?Q?1 dΌY\<g"M8ߣ @pfGgt #\.K ~ɗ~1?*}>G4IDAT֌>[oǝ/,0/,Aq06z]96m]?ӥ82_$83_$32f).sZ.2{ߛߡ߇?,%}#󲊷|Lz߳7g^*>?f`9~t;C1@s?{$P\ޛ}qv_]b b~NC*`nt-d;k#裣'g(6 6?R/KVhGsb}XI(dI?=ȿ7OhW7]z7=?~GQ=v?@"?_ WuI?~GQ=v\\]w^\})߂<Vp?>v?G(?GsV?+|~~sZ\mmu{u\-vnsK{_!O<~}_ \huaߐReRM⳾1⯜iS˛߇Lߛħ"?B6M]X.5G޼c$~3<W]\)뛋J>w[|!>WJqV ؟ePڵj{PO+le}A#!'m< لwqw`}#h|?A|I.CS?\>JUrBX\A3-H $CFj!^H տ{Ps;:!72fgd >]r_PL4w`}(tܫ}$~ {ߒ\6ߌu}Yu/q c Uz]6d/yh*'Kڑd8}&:s? ? MX}4F#jGP3/K9=aA[+?I-񯲁bOogq!s`EnQwm.Y̓үe6;g'j8~A'N?"p%LI_n3S2#f[/gp.{IJlSt|4X%z7կv>H>&g'j0p '_),]5Go/ciTg- hpjD9\1_/p=$I^-33),0.Q8YuYBɫ_o|w`}HƷbr ~ \|K^+ߕی.ZFo/ϙv~J( 5< tɪLևdk|+&r_f sݥ>ʨ52|>Gq;^:-gԒ jX0ͫ_o.|"2Y{n.k, Po8I&݈z6P|{D~fqg懓/gɋq}{cK;?m3C}v|MGjS݌.7ݕ;qj?ꥍ":݌?c(V`&'_)O#.A5Mw3,XvA}g Ejl]OScS5/Cn=syfK3xu{Џz^|~~m0Xd?i[&&,;/~^gs"xY7>#ʆߵ>mIF|8_8^ޟ D8qE{g _ ǵ1xqǵY?Sj]L?Oc|,`gB\-o٧cFyF#!w~>%nf )?+}$a߃ [[>O];w17?񱀙 q}Ͽe_}JSc[?zdr_-ty\cR+2/&T :-(tOmn}:M[0g >kObת_NAFub~^'Uj}]}f*O}?_.|Ư |do_p/M&5c}[?d6Ư?R7`{e~1/|.aDAc ?G#?AzWOO}&5LnX75c_ ]> x,Fܚ Q3=:Hv>j V_qAY·y2Aux; ~`\@aq~\g}pdpg 1˗ޟ#ts pQ.9wtC??A pn|y^oڿ`*JU|#0}R;y=pxlYs|x?{ߡ?Àd ){~ξ]Av垍('ף@?Ok~Cg]'!x FR/KVhۂ6>:}}qv_]b bA@s<-dv?'fo_G׿Gt@B/.Sw)Oo~`{ے#^\)_'v=uߴptH%Gيg#|5o_{8$z?Ԡ'\K|A؋O?wE@/>X}QrrIa [l ?~GQ=v?`?r 7߿o>ePzU>.>#]aqOZ|z7N=omUaVy or2KG"tٯG-#ȧ7ߌ^szIեz/3~y>貒7>V0%_?1{HskgCiO_AT o?u qwOˍ [#=T+kAFaq^É6<-ԶesF% >qϛbɿg!ǥ}is[nگ%pӿ%ӆ~^+Qח%_ܞ0Ơ\?zUoCo쯑ߞ񯲁b\ ,kx8}_k^ndͬ@|u'k?| sr{r^W>1Z_evi}Q; '߶D~gGɿp|Q-Kr/=z\*Ytɻe'g?x+m.4asZwh>昒(76"|1>3ԇkwtKo.q!|:ciTg- H0$S[|˾g'j0p '_),]5Go/ciTg- hH9F3?|>? g:])_31gׄ2e _33Ac"կ7>#Aǖ/ϙv~J( 5< tɪLwZo (1$ft͉ 9~ͅƤEW|NH܌9}&N\ bpLj{cKl%>LxG°;?b|fHmʸŕso_OmG[[vA}"a0Ggׂ"k5n.bׄkW2I&݈g|>|?Ft6e@bn?1Yq{uESC/g̝EH&01xd! fLHmӢ~&Vx~L<8e\߶T6wID[\mIF|8_8VkI_G~ҟbby\ArRZ_q| y)k'~fߜxlخ`YP|,`gB\-o٧cFyF#!w~>%nf )?+}$a߃ [[>O];w17?񱀙 q}Ͽe_}JFDʆl}3G)X\SJuBƌo¤o֚7{X؅Xj6P~>TJ||z6Weշb|&D- >?_ Wy\|5c}[?d6Ư?R7`{e~1/|.aDAc ?G#?AzWOO}&5LnX75c_ ]> @ݳħG~ڃb-/n\ ~2qqbrIx83=:> $pp  7{$3f?A?i;J걻FlS%y |iTN0VY|6yxGQT-WXy<|%/';J걻 ?o~]4қw_ Mq\aZaqvO w!?˥z/3~y\k4>J>w[|!)]_i ]?٪!AS_)gG߁_]V' &k'FwiNQ\N\gQǀ ,7WDP⸆6oՋRN[6??mX-G7 'E6A[x?Ǘ<~tc=gl./j7׿>g!XPqYu`ߋ>?H7n.;s?QGuY7>#钟וbɿC߶(u^J|Fe?!'1(Wu'3%U6Ol#pm,LtX;>mhRG:Վ'gD]_| sr{{g7cW]SR-z7Sekyh-}st"y˓@Ou0<>nQ7ɧ 'Wv-C#z[ud??NTst X'ګ8;3㿃:0ɗ3q燾+CcsLHrΏDmNõk:^B*1Ӿ~ ϴ[A4d??NTs[f~8}&Nf8jppeY2\g~/am ^W+;]\_o$dlhw駴2C2xmG\b2ftq圿[׷S۠Q/mֱf|Pow`?~$?-ZPdum__ɀ&t#~{;':k5pzDK8@\cj>$._ӑڔq7ϢulGPdOM3NdS>G]d#k:R2f|N?cHYz掱cҷ&meOro.8VkIO.=/[\$8E8w>ڵOoN6OlWWYZYoq}hP|,`gB\-o٧cFyn?$%Qvѡ_\_:O$tʆߵ>mI@G~~2Hnq\l  _W%kϵtu#鑟}/R[C_1>M,Y#Zl| $fK>o >j-h;0_} gO!_%>Z?QC?ഘY5 lqqD{AY 83ߣrf;~:.qϬJ4YX ? NJt;>`I Us!?ۺ4w83_$83_~6,sp7=[\E;blꟳp8}8_7gK6MܞS@S2=?gߏ~ 9zYBɵGQ틳wNu7^\`nt-dv?'fo_G׿Gt@sgK,Y|PB??~GQ=vzQ 5oZd.NׯoQ: o_{8$z?lE݊0.㛒I[..|t🦒?SO|Q51>c9$z?lE1rظ-sV]okȲn{Woɿ>MNДvdk892 7m#ڲunk5EjUp)}_@|D KzhuaߐReRM⳾1O]p`?%MsMxև~>Ƨ=dtHf<埡_Y\]V' &k'FwiNRl, &QG?qr&sMxև~>G3tY玛?t+=9Eõ3k'[/ ?8#钟וbɿC߶(u^#KN?ַf˒aCnOcP= Og:_Kl./jGY ϱ,(4w`}(tܫO@ψ,81*o$}c8Ŀp~:98rEufcWzi[ӱ{l.[LEߵf67v-C#z[ud??NTs\8;3㿃:0ɗ3qIEŕ׿g!ﱏ9d$~9MxGͶ'_ ]5G]T8cg+:i(#i~""p%L̲0P%_q~/:VM`x|ݢoq~G_q к-'Ek&'cLGK?}Qpkprgby/O~M(G:Yv5D60(J~K$ԝ E=~s,'>_g!5.D^| o̝lJ<^^c7XD6ӱk_Ɉ<ӷo@K.?/CtH?%mW k}()q3=~pӿ%bp?6q>?A-kcJk}9G?񱀙 q}Ͽe_}JFDʆߵ>mI@G~~2Hnq\ld-rdUu'=X߅3!ߔo߳kF}&< Oygl 5g^faQo9F%L`? D=H|z=OO*Ѻ##^򟩽Z8Q˞Y ߈X\ETP=jP7qE3۽>e?]rq9 [)kAey8z{oVXea*|xaqyeG1  :"qj羻6_Ő| ^Ur*L{Y΃_WїRq9(HO_4H+&2/}10zs?UʮWO~bq9= 1?q=c+,P\>ٻz5Z _(Yϧ쬋1v J:)ؑ#"P IֹOOM.W\~J[ySzyz.Nj_UB 񶒧ৢe胊Vu!(8Kp; X\\^G#URqQ)E.ڊ]t5W).kxcq |<m&}y$S[}epKcqs9썟cI?u\C. s']8n}r_+k{;lTL}sqӶl#a]EALE?']lB-֤szrmƊV@Fqve8%^d'V/.sr"z<(DEIbldWӞ E~N~Y\|c7(./>)r' Zu'\Q7P|}>ng7xGe÷o\X%+.7'0'}^Id[ \Xgo]@. km^C}P\!/½~ T>!ګr}QQƏI$tz`o Կ:ۉچkѿK}6K~]V})3`sy?[?W4/_}ܞŋU9}9{~{`|WlE1.,xmE*- g} erzE{4.53_âzmw'+.i5^#Vqei|{^xXz/ LP\||q9$w*]^S,R*v?No^w~g*= N$.S!yU ,O;Afqޞ?t[q|{%Edb.򳺸\]_B,st.l&C%E\\0ϼ _?4AAY|_!kN| [h.y oҹ¨V؜"T\UE1٭ͷVܵۥO_;q$pmjx ~,Ʋ8GV\@b]*ǏIx楱K>TT]tP#ONtG"}~ۊ 槉<=yNwG/o MaS`yJ?߱=(13|q=A_ c..aN:})ښ|~qs(Fo2ۧ)2qwǗ/#''M}cZOa;g3 oL&+gKGr(3kο~\1:?󧯏h}}koү8 8&[hOA+\T$|a˯Xk)Tz1Gx[]~T#3uVNJERW1JXZ#œP5zZgq9]>.iۙM띹㫅gtJ P G"\aS[A}U_Er.|~ 8v _g+//8B4a;Ӆ8ͷ_K//sþ[~l g_{OTn).SgQ߻?H>n]heŧ^J ]k<+/s;w=C<~)Z]ȷtL,_^k_I\A㬏oưO=c'_3ZV 낰T d8g1Bz,J$=ZC>XKbmyGl:_ }YFF*.9i-b%QT m!nzpBb놿8Y=R.^:3,qta'/E.|~)/7||?\\wJrR}"vO ӾA-?.rn/Lm/.S/^~ƭ{+~G廰KO#sǕ 7A2 M:W/Ik-kp7(k_:7snoo}13~[#yC'َHe6P\^Q˪i\`:!}#k gda9lkcpL}'[MX ]?IZ6um1^qJ,1_/c4|A%?K&};s&\_e|u me~ⲫ?%\[w|"ȸ?'Է˧X**s]\(>'z>[ +~Ke1by+oddh{cba7/U߲׿~tW[/8S'1å G]ˉ.:绾P~^/.{ hj6dˢbXд<{ 5;yI@+ kmk yi epNW@Sa)y?1M^#a3}o93ab;(/ 7+.3O޹\B_1y틣/ ?s٘oBG.Uy+o¶etӘ낱33wE|;9kgƿ>FEVŒ]o>Kwc=*nQkTݡWp2KrvL8 ,8W|qy\%XK9$=j՘6 E(.g)tlOt10Cyg)}LLnߋr!-ԿU/Ԭ1!ȯY]$%d?,4}Y=% m*.MU2ұ?QȡS|dLg_9[PQL=vq7^˿y.#u9-ŗ;wW@O3oesK? wvϵ%O׽(}+v_큁ǭ|?m6<?gk=hڸ'' ˱'{ǸpfpϿT۰%6"QuwsǬ Y&*JE΢)ZN^ U\.~n]c*lLjqVB'X_\I_H+^ ~lOO sy'.3lB&%Ǔ~GQН+.81B.Hp I~E}㲏"1Y]ַ_(%}>HLqf=Xt qY^…>-߶\ߟT쭟<r9P\c${.4 c$~/yzK.ӌ?q﷯ctXv|bMƧqS]ɗ^?i~WQs4Z]'?MH_F.mQE4*I@O-; 1b\lϺdlQZ'ҡ(.tfN|qT݌iy^vy/n]UNlqrBqڎgipp/#~ޱ˗p%G/]/ozf_h.J2~ d4߹9=_@u zlXq}ogsdτTo6'"cm_0=志r럗O?/?Sn1~_ > 1?za юty3y7ͷ8|qr{brzZ.֕E*#{Ben :W3 ssq$Կ&YwQ={{oI3Q\c~,Ʋ8G}筸@"*Ǐ6:ޭFc]7_| EEџQ&!ONtG>mEQUDࣗٷ ~Mh C0ӝ?SÙV\mM?>9`aǯ+_꒨},נ|..R( ˟ڲ Ș-ȝ$ ]4Oř8v _g+/ 2T ~(la [EQh3aB~-?dMЋ˿a*_y׳ ~C߷~ fx,?c"[p BhT.dW30E"]\_/Gׂ T|X_,Mh?W'i,Đorш\X&bGkHӟ9f´oP~l϶ Feu]cY~ ҅Iߤ3l_~C1~[qYOz ˾^.Od6~OEe[X( qz>K8ձ\hGG\pW׺[y_ hF-.˅xh*,؝O̘Q&|/[0W̙xp_ENϙo1B͊L|w.WPW_^0m(.c~KCq\>6x׊/~ Nst~z_kG(.g>=7lU2_n;?sR)~gblVP?߉/-Y'5"m_BOb8o2qMEQ&+Uq'Gϴ~V|uHþK.cKz셻~IH8 ˸vWbſo88^}_.b]7/JEC2]6qz*,^B?3]֟N~e[cV~Z\dV$.Ƹ#HLzr~?6DM..(Bl/,O47aǚcڿ?.hn7Is*.;1 8/(MB;{*nXV߳~3?_Ǵ@h[ys~vz.Pqq `X\.ת(]sv72 t^Aq|ޕ8*(.8 _>{\ ]܊ގz ɶ5zab&ŗx6+(p}v/P\z1 |;q k(./N`V: W,m#Am~r 7KʃIENDB`llama7b_autotrans_8to2_predict_distribute_ckpt.png000066400000000000000000000162421471400134000366260ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/Transform_CkptPNG  IHDR]yw?sRGBgAMA a pHYsod7IDATx^=U_B7YƕlFE,BD0$`D;",810 Ņ( ntP2!0CflTWU]޽}>]UΩ:uߝ{zzk_D"H$I]p@.t4`E؝t>uéϵiur,Wχ9cXl?r-F>fcYUz+'&T692װ~%,ʰpuꃮ/djXX%eOuxäwJ݇c_"F} b껜\^75բp]=gyېoVLtK>67˥: 'ۨitI\RG u}Z3X.}\c8vpkpANP)ɗ[AW)݂yprtЕ`-/8ɚc]+r[Z*eq^I}=RAQWz"3hx{Լq/ud .2:^^ۮ'խ2Wǘ'okȶum4ѽe_Pt-I8a+/ϯ oMo}&C}-/K97xݵ8,R7kU㯓o*Id&L{m >1ƨ_lY.w& } :_2m%Z/UʘCҿ.O7qKm{1e|lcHhҏ1ҽ +|ҽZCEz(g/;'GIX705q4y5t26r֡뗮<}48g$+(3sJCF;7/C5x_-ҕ߲O$QgĚ.Sc2JS}O\t5)|=uR.09(/.VӶ^))t\vkDo )W]3)w, Y?%XcJʕ}I|+ҽ曾dD]z[IvI+07PԁfOr;NWnْX95n}l~xHig؝yYOIK:{j̛]Z\N@z՝U5tmk%H5{- sY\vM98E.k|Re\BUЕ8 |K׮]p"o+wEh"׋';xhlV}tlX-=w+9( &c (]rO7c#SܞM~,Czpr+te!1| m%k:jE\kǓBrrdLƮ/ɗ'gK  -9q2^k笇vCN&}Oq, _(G__9$qNضg2KTO%)jJfq~>^~\NY|YMϣtʽ8_髆?Pmt=N1?Szb{eƯ}ۍELeyk2W],Df8=,,ɑvrՍ9&˾\B5p UA"sJŕn .ˡx2rp5=:U pwt4 ] hA@f.D"H$&]<{=s O.Nt`}= AՁ/>!8:}]']'WzO B@O vct\\>{. {NxJН}7}[k8:;ߏ_ϻ_%IsqH)RExjqNP'opa惮¡fCrkf7UAk~//1?암)9^?i٣Tƙ_tuˢF::I'wj{%%:VefBȻq%Ǔtw?K]`soxh_@R];qjYɜ윐@r&s'\]AWBGapx@e(_Sg[f]F]ޏ-NtuG/'En+<ϋo\_hHmD]/xx6.9sBWpXvtIS8<~WfaNuN"T/]˧s+AlaݻYt_J=Ӓ}kCl0xw>rhp06^ȥk]pE.?l98D&AoHsH ;>g{i0Jқ./]$_{vܪwuϋޖ.دK,-Y#>UtZt=~>7X9wJ^^d4+ !5G-~/F6k!'dڷQ99 BάjݪC\y?dh9IPaklUgqpxI :T:)ɓgή;>5ֱ˪ZTXq^nt8 B[/bݴ> .UܵOOa'ѭMAAVv v] Bpԁ/>!8:}]']'WzO B@O vct\\>{. ='u_`}BЀ&A}LKI|?$}oM볯}I{~K]]Iu:NLw^}nM.y{%WHutu6 oO\%Kon.ltuɟ:R][gKR^N/GO؝}v)]$Ȓ?1Tz!I~wH&i p4 >ŇOJitt@Ѐ6AGwO^K{oO?5M?pߴ $_)?3ht5NݣFܪ?s{4{&>;Ž5cU>\x:mC*0nPMVy`yc_CKǭ{ .qV9j:LcwCPq$1Q?-N_Qϥ)318|U0GCC~_`fT%=$7w[M9it Mnaܮdo3H ݠ)+>䃲KJgq1̵Ȝo.H,Ǽy4-U{˿;9w `_>5~=Ss1܀%{ԏ{-K-G WCi!1QtTeAϝl0wԴY`ƺ$m#$&&}[ [=itp@Ѐ6AGwO^K|_ &OMӏ+7m. J?!'3]`kMU|hA"x `>]ퟪ}2,KEFY== [أ]3Dy `ꟳaCۗ!cOÿ Sn:1Uǜc?J!/G3n*ܯ%:&40sL8~M{?*I+JR^/W;9s_%{|6=2@5]-\̸Q߈ $\ >8gqATLjp~k1|b{G4GxTFk,:sGr70q?C,+CʥҬTy!ܛD9^D#zml F@'yAC(,m;3|*cN/qc钤ܪoʑyK{\nov vu {KܘGo]Nʙhe?4c[A.L(H>sɞ\,wFW|JS' o򹅼|q,C~ܫeo pXm]*@%o)U9RFx~+0:p{}?l>(*l0wԴ^Y_/9jpe&|ѯUrc<_aʖqq%Óp^^ᾥԤ|0R K.J$ tسCD[{,잨IC,5GWP!%ztʯuMCB]:ۂ=`ƺ$m#$&&}[ [=itp$".M׵O_JW_3׿nnA_t>/W n] gЕ^n.Ix$u&Aq# ~vƭ{_أ t*]M,:^-_olV#zҿn诪~̧/eWwc(I<`1+ߣ?l>[أ̓~<7]-,}C:^)_~h/% Z-Pyt t03zYE])318֤U`QƱs>|6:%G36yӥXJ֛.[7N3 OQ7PoH 'D);Df_;wmv]LƐ/$o](KƬ>sxw_Sxvx %N'xȹI?y=S7H ĚuPZOYH\~Js˝kX$=*s[KDpz˃_q-أP̓7~_b W՟BMɁ.7zR.2\}uymݢ)tyoTH\1$bRu̕gsYsާFs`<ga^hu0mpҖorZyo($|6O\cyЙ?d} cY3do2\*A'zBK+c):ʕM48>)sHa ,L ȟ+/jX7{lL¸“Ur굠zf9==]XםgXkrZ?8]syWIRn7ȼ}.77ex\q:O%n#7`7]7=$0[t9%+%J_jd&j 쒬2ܪ(_wyo gxLMHLIK`R}<`ZtTePڥf;jך{%g}s81\nPܷ+ ןw[l:[=^b~XC.Փ#l˧˳Ocه˳WN>V? ;Sߜ}{\F6yR<+`ňqTqۋ\/3?-^gTEYh˯ϗ-EvW#Y31mWLN,e) &AOBc^-NĢ.zQΙ1c{|}dyns{ϸ\Kf3,Ħgug^vĉ~}ÞkwMF B߰:\r~V >WqL Y{xJ7x !8\=.Vm'XZ 8'HvE`>%mէ_5֓}O.u\vܼ\~+o^F]E X/ef!Qi֞|YeR.ϱ/fqDO&(&ёnֆ2KRL͢Cr50X/jK4lɫc|WA&[̍-nk̵5#P[KnMu7= @(T X\HK!T3EQoԮt?V/|sb]mR,n7ό 0Wk>[F֏c;)n].YwZ[coIU홦me])nVcS6d)KAv\؞sIdtwvlѾM~US#޸㦋#}g~0.V{Jaⶡ"FX>Xl-I ۪@KREc2 nS7˻Qq;vپP.DOWCس/pL[<\x譏D&+\5m2/n{9ڂ3++=G7y#6s9r0qL{wĄDEI!>uq֮N| p{o9~;6cmcʔbxhÞkڎ-ֹ@lͳ&$1&I^!ၑJaztBHh:Jo97OO_[LBofl|Lvx[= k<@:c\#Vv/ybzWe{Wvȇs;.#6}7RT@ AW, x6\Vn6͟=FxX?K̋; nw@p q6v}zjО;r(T\kvy{!/k3b-z٦ ΍9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\LoOq|ssd_IQ(ȃ#wQm/]πgˋ_{ɻןQw?oe^ϗw/>_>eNkOR(M[t6u\je}~HݘԾ?ָ}cBvb)HIDE$46 ~j vB tD֋rΌߎ+'3'WvcL;-s,WO~wZ2;g!6=ӷ"ċQԎ8qݯosMqHAV'f޼\~jㆸ ?KoO!GQpк*x+חCk<R܎v'd-s/z؅{Y sb;e:6$![.D׉UPl; w[7OI_XnpEn]<-~>#rdXYX+|O*h{IkѠٮ\ֽikm}Ψ"V%V.X9m)`k'B*\F6- Haֿ~"4a%Bܖ}Ήssb} "K MPE(qQ I E.[XC;'!-6,q#E&Wb}|B0;8-qo=.n6-o~giy֎ DhP6v7kk{^ke~=.cBfǺ-v˶n66LPL#ݬ eKϥEi.jda ^Ԗ"h|W;6i-M[֘kkFqo{&|1LHT1ܘ=M gՍϨX^D*ayHĪE4~m&nvA;e*qRxkD] ] .jM6Ct+ј!^>]?Ү.0㫩ڱַ7=6^Wo`{-{HEq^_IUn:WߖA#ns )fo]zԏ9޷Gф$v#PC6LP\WX"8SFo(+S+k wnڥ#n OUX\HQę~x)$∺3VD>֪cDf ֲwp6χmA9VgL5gCqLq֕7ׇ[¹uڳv˶{WwydJ&Ѽ,`)Ȏ b. "nY=[ێ`\>7ɯJ}j|vvtq|/qƱ D!&uR4!JB,ꛡmU㉴;M?.6yk> S/V7m#Vi{Ӹ~jr=܆&ʊìfZk3 uhO~^ͯ{K}1[mz>%nlxTxe+s7>c{DEI!>uq֮N| p{o9~;6cmcʔbxhÞkڎ-ֹ@lͳ&|1A* $^h"ၑēJbθH†!$GaBfc|$mB>\f6fOc ɠ\XyD-yrqt`mVq fMBQoMm\U_}e7oy\=)<{wX>9񃿾{]qpC,ydO읽 /n# 2FQx(xD?GB_H|>?3 -)}3%ɏ#W9ht @%@Ulk9{Ju͋Du,Xbo]IvVsqi1+̤x] \f&"pv9&>V_G?y1s7[U嶅fez-b*!.f xZJ~,k$S49bpkqZ)U\G\we?>PI➷CC6v}zjО;r(`+1m%u\{}zʺ\,W./7c19{B)#HB [IrBCl ,p{[maTbTѿϾ4=?v_&#IDݕEH< *>,5lrykom8[Cy98,}'=񿻵%l ly_缮bNjkU&9ĺrn?0g&9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\LoOq|ssd_IQ(ȃ#wQz=*=n>[^K޽ʿٽ~("xme(/̉s.ð :Gq.&,/n߳O\ό_Ns#  *$ ŕx}=Ѷq8>RH!`GĜǰ&űq?e<!vYy{ʨ\Ra*ލ|:..X1r!9}kDg$zbN)ˈ~>8yG>qُ$&cBvb)HIDE$46 ~j vB tD֋rΌߎ+'3'WvcL;-s,WO~wZ2;g!6=ӷG*^vĉ~}ÞkwMF B߰:OWa?j{˸YN@>>'q[+{MPq8W;z(jiY X{=,Ŷіqics߭sCsE1 *y%ʹ"otc_#}icټvM~wmG=#EP=ֺ/7>;~ߞ-k! ^Cn}+lu?7Ul=i o-V/?x-f# zXO~v[_b)WdS /\9an]4ib 3F^Fȃo-9ѼN,'X^:EߍoIcur\gOI_Xnɮ۶te9VN!%b}'4y=N.兮6=`65,fΗl3+מ;4o=KϷ*ieVLk?b΅ڿ?IJ:5^{_ўYak?-/~|/e \k$ǔBﵟo3k``\ †v !\CmU^-ĉT/e'q\Ps>'qC}z4//<$?*Rb_iHR039>ߵKk|>rYqG?-Fa2_}H&nK^;U2BmD usU-⶜GsNܞkĭmS=O81=mZ }ˢ|}GtG(cu\(W:Z][gA{\vm[,?ڟ;ǜն̣ x9\ r:2}Yn_X{l3:]qe6o2Y^ pLQvj%|4.vs!Qc֡;-!^k|/_q={Y]9ξ _}+~ȗޗ:?|JBa_`Y ’%X,>\ĆfqEYk9qEG.m+N+i0^x$m^+QᘳEH|ȁٜiiY}c;'p:3wOqXχVmm6,Z-sreto3{c`M'H7kC%s)fQڡ YBqu6UM+zZv |B-5fښ%n7:q`J5qFl,ឋ=E5~3ܡ\.D Ha$_)?-mqDl7ANmJ?ssHTZRnpQuT+fնe6eݕc˽z+c(7e_Qb>CS;p÷%2Dv҅ͳ=Rm/;,?}،ϩ 4a|jԳ7"4:ezܴ \<5ƽ_9039khl?l dyyXs\) byk<ٿزy{ؘF1-n{qQqI|VBbmںM- bm[@n{mmz,P;Ciqh,?+vuqDgC:}U[GO`вwp6χ{۽9݃e=yR<\3zgp\_vk[Oy"sg"*GLX6z`ل2-xk%%.a*ٶ5]9s,)B~iguLmu}?TOH5|R#z4,%Dm'̾iwww㜹i4oc)q[a1zg0>ͩ#5;X4j'C~lb`|⶛s;]s&:źsߗ]CfJ&Ѽ,`)Ȏ b. "nY=[ێ`\>7ɯJ}j|vvtq|/apv(DUh޵I T_烳!Z #*L \$XqYލuGr匀ȉHC|Xf[Qxyl#w"_ێ% ۭ)ys^ Ut [^o~/9%BNJ\7l7co8pdyW"h\!fmw wWݞbmW=11Q"ٷCF)x*zvC/+E={L%(HBJ4Jd^cPsvcIm06Y[՞ueӢWsivwm?=D I/{2Gq#?8z߇^wZ~^W__Y q;B|,p ]Q' AH16 rvlm4*Ɣ)c#^<gm=j׹c[sؚg' ҂^Pۯ`w?Kh uT}3̱XyJl/8iS#k{2(xjBJ[A%#gy$}c e9:)GiWݷ:'mZ+ =\/U?WGN 'v||Ž\[;c\9m*X0v̏"7oej5xiBu\ZY #i>rD!ccC#\-g>y'}z5@7cwm~5u06.fn?i.O=8rZ>7'8IX1DZF"9ЪOe.;mD8 Xy?9l;\#sysP[Oh3Rd6}C>γʖ}1`7$8}Qq;Ǵ K8un\GDZ ADaQyq\h1⬣F:cGZx0n e/m_h wk=aip7v(wh~x\~ z+'|>&3CE?mLhr<5{\Sif,}~/ykى~19h^|FԷţ"Aj vTW?3B^υ^ w,&7F˥c \9Gv[1>ϣ<5Vsye^s;z>aͻ6=fZkV^yd&^o/zgVX 8KuFħ$Z(WKIplx<=s됇x e!x"N'|?b׿c \sѩWǵF8s90[{cW_GY:K1.ڌc["0je_\jS|6 xd>0h렘:FfN;#$0?(d8ZbͳQvh%pϩ=F# oȺPH\ˣcw㹭9ъےRu=>q@AbPD8X6{m^>B{7ע<;Ǵ}Vj|<4}<{Uڮ봱oqco s1>ٷZed}'EY@'r.Qz=*=n>[^K޽ʿٽ~("xme(/̉s.ð :Gq.&,/n߳O\CۇzO.sKƗd 3!n; $&sЋIh m4]$s} QRbY/B793~;fl,Ϝ \ٍ1}Bbϱr_=]Zߙk|_ؔXq}(QEq_߰皸k67=ӱĕc>f2c}m0>?m쁲S܎rm/;~ߞ-k! ^Cn}+lu?7{ҶfZ\_~ [F$Kq;fuϿS둾w*q 8W|t? D!աd5:A4֖!׉zcy}vwXiřS'?7bymvGu?mvX Mo^=Kqy+}nM6%pʵ'?[u,[CSbZv.ĬWvw7!٠bG\my#|9k$ǔﵿo3k`xLܖ&v!enrfk*Tw;lCr~9q{n Z}ݧ; Ӂۋ),ڳ&š]}\޷,ɇwO ~8_ǵ~O}z塽%n|MmצsyQm< Wå W#(6èw[[ks}}-c{* G_ny̴ulg恐QAcbh7r5fsݲbl핻ZΗuQسW5ؕþ٘&OR秃CcSS.௖僿S O}gM'H7kC%s)fQڡ YB6quM+zZv |B-5fښ%n[}_ZSv(Y}e4B;ҁ>K*ՍϨ?XX7ʅB FIĦ[ؚ~SԖ٫3/mG.JZGE}No챝bF[m[[kS]}1y߼1ױ2"γBh }OMT}r3Hw[)eYvuK;e]ccRe̋Ů͐ jŽ2F%~7ohUs8sUΏ>!QZ>H#|/gjjp'"<$nbc|k[oó(fF,L""v:WߖA#ns )fo]zߦms _@>\C8;j*XTY~< f+ʲb<^߀(Xذ0 Q[ŵKGܦ?,yl\{0lk9XNݮZGf #e[ ږRuKHϋvDC(/ֱ1uZ}[y#&T3S F sݲ˱s]'X,̋ɮM;fdvC/T1QbދHf=`n}>q>,?땶1_el$|9o48A:8y/ԙW_whȹgk $x1G9w_jm/.*!ϊ]^[a]m -~o9 q@04و=}o*S17cC8;>C"Zh_LG41x:DcZu -{/noh|Xܾݛ=Xs'ų0s~ 5aiEܔ7gh-FPLĸ9[eɵ_봷QF/:Pe=s 3Q7vb+c6Xo3-POu9>^'u$q\<3KK8~(a9'8A<'c7-?eY ODy1(h>p=d'˂՟GS#c?{T yȹr6O1qi<9]_+{!}_21%nhބdDži17"nY=[ێ`\>7ɯ&S#޸㦋#}G\پy١TT:ȖD왣y^XoQaRJ”Rvmfy7*n.}[|*r""Vٖy!^zHWǶc f 9.Gvkmvq^Ĝ{C"֯כq~Q>Xl+)Z~/d^HJ>)lqǘ9Y\q!n;0&Ť= ,MqF}F3IS-ZBs^7^ox}7-6KWgVN-b.@se,GXc~y\2#1@1C<{Aik3;Z{#>~NK?S6C6kcAsy#v"4A.ELdn0o@~pC )mӷ/g wM!?M|jX|) MAIo1v1{{bX!O$|~c3މsX7OD39Fi܊[l.oy?9뼅%Xs}!qpD!íq*i m=χeUZIWH__V}1ƹ<텬Y1#DGTmsBlj[~| }/5rfmKGlӊ!s"1cmTNcv7|#}،L9r.kC,斜B,xYI$4"k}.nK=4 n΂|u[~xs KDCbzKa! } _G|vZNmɳmYE'p3:>c_ZyA${Db$!PHǘ{ZޙmY8 =l%a48-zmݧxhYubFD} +b&|f_5.k*< +QdϲO)%fmU1y67ʲRyM!c彆37&k y>i7kfyRA'i J́=\-Fc؛VGLk1/Wg}Q1W7`N,!_Gȴ2oCR5Nسd1ZqD+i|:vEzy)i͑P9~{*DBž/qE˰F_KO91m}#q??(u6s\vrB-=![yY-"93ַ<1+>?[[\q(zeG/bęٖ4'Pr1e$qo'[~>㈖\^Np`:- OtI%incyݫwߙXȄBp$46 ~oh(){1!3WWOgNƘ>!1wZX~篞._̵dv>BlJq{oM>u,qAE(jG7&n|d t N!h_,WgsqxGz?x̤zhs-Zs9FG<4oIFV_;}[aTZʨO(31b^_iEV?-_s. U㋶FD-Lf*fʞ'ɧ_wr^-5._V82o /cr6ڼDg{Š_QQ<9V[Ayt3FsadLܖ&v!enrfk*Tw;lCr~9q{n Z}ݧoHBkڳ&,R^A/x߲_'b>GA eq,k#/񈾒kw)ϖ_ ԷY~C(͋۴nZ%ۅ*`k]-_=Ǒ.Yevw{0k6wEԯϚ1uWʹCںoЁvH3'fkpcl|8D-Lf*GVܘF2d#ޡr/Sn;ON!&j0r^I㉖Y ~7$c3h?r3}~1/on;m=DGY,K15]¬b)EЈֶwfm]["o1713<@m-qb6(R X̞&hW!d4qhOhc7'4 ۅ()+姥-nM5q#ک-Wt"] &(e?xV=7BKs6jɿe㿽T /n{D+bcۋY9| #}frɈiUŖ1ᯅ?s|yr%0qze\vGcP#ibT~f[X\GI {-Zst/[ ?=.1ۃ'-%suJ⼏ȓICC_skJ,jP$nbc|k[oó(fF,L""v:WߖA#ns )fo]zߦms _@b{(MDQJm)|+~xEb'6,? HlbIGܦ?,yl\ Ae`w}$:bAsIRUQ})QǠj{̖+w!zs&o֦u- ݶJgT\g# Lc܌qⶨ[[rMVkm2/ngl3ΡmiδqΔb}51=y|c6=qoݱ 6aB:\p8kWԉB|>AqkW~OVhnیieD.clTK-wVvsv[1e:<+qE IDam*{ulw\:A $.O&do97OO:BLE[asLM?5/pK'ra\w'eo}2On_簃qs_z }.b#)_km\V%H:q[0c뾌K򻳧ȹ~+򩱟 ̷/q̵yLۻf6`?( nH? RLQ3K6kpzB 7s[`4ft;ڸ9 @Ol@ЈYٛџvp nw@p q6{m^>B{7עp\,Wu{aW\.k _je@ɜ_})||/(\'GVJm×OY/gƲgJu6m1?|dw.&7ۧ89\˹l/( .'_ۢS:M}# ya'Ӟd앪O'&nK^;U2BmD s;N[!m9?眸=7f-S۷^8 "Rˤh=K bіC;'(h,q&9CJZgA{\vm[,?ڃ;(I_lK8O' ت-ַ_+>:[a|5cbsb^E.="yZU)}:,n.m=DGY,K15]¬b)EЈֶwfm]["o1713<@m-qbvX2%nb J >P,T7>v\bzomJߋkIi$b-lMfvjU8@[>/hm}%w.3_Lu3k#|m3я^||![޺>ԾCbMv7z m߾ ϢIv(B3Dl_{~[96Avq~ٷρ|= b`X a*w"nsk"LOH!nÀ=o-%qsm uTRY_9ꆸ-[)}kUtpg3(rOJx>^jRWP$>+w!zs&o֦u- ݶJgT\g# Lc܌!JuUAcU Mx)bvKxԱc:Fm^-{/noh|Xon+ʞ[6"ڔJ3p[`~Խ$b;˳oOYAT{2} aJ&Ѽ +`)Ȏ bn5S}E~z6}on_Mv٧_gGqoMG}2TA8UJa$D1(CDP??B4k AXoޖ%,_€Mu,FeBrF@T' 9 !U"2dN>Py˯[Z{/6>7:&8[{aQWM~c2r,-hY.r2!nQQRe.8za+DS>+`'+vlm42n" 6;+n;vPέ۲m `z"n{F`S ulw\:C $lZEdo97OO_ç/Lv7 -7~jd-^vOM">1Z mWE8^F'S&6]=nI9Ne\+v \GҜxg[12&r|ήml% 'JU]h>(Z-o^C݊3 > q6&P~T3m ; nw@p q6v}zjО;r(x{,\]n^ח/^/ZYrrR+{X`j>ggv^3ĝק]nθdόDWo<~>!=ws?CS4F9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\LoOq|ssd_IQy㟜c?EG'lůZ~Ϩ7ˋO/;_/[#S*瀲cRTb{Zl8ၿU}"j¶,oOyOviCqwz|;X{Ls]󞐉߃cp͞q1U's(lcBvb! IL¡$hI £!'^|osfv^!__=Y9Acic垿z|3n3ג > ))?:7qPhog\"t#N~\\cm2f9{8}Nܕ!dz>%X$K{4滴~=cWUgsݬխq5׻~qcMnX|l^s~ξEZ>{&l&{*hO9۳?KoO!GQpEɺoi=i o-V/?x-f# zXO~v[_b)WHILw;%Wt@voܗfo7x zcwof2B>'֖8xXN~]屼 uC~q !n~ qŏM~@Z99uy !+epq=vZ}|p? ~umO8 RYn#ߴ֞u=++;,s6ҿ65uxXcUĦ5.?u/s*ocn[昍1!ϊ1׮kҝmN㲠\dيb[ai3!礁W߶W?"b,3cڞ?ͷK}1[6͸,l`0] !X.#tK0_?W4a%Bܖ}Ήss]`b>7ޘ[+ڳŇux3`ou!Ӈ"ű:}?X_+g:9-qo=.n6-o~g폆o+|>Jʻr8`M+)lԷᗧa{ YUd;e|i1[ oS+v2τ/8Mx8=%ឳة:|-<}9?xr1硙 }9ί,9O~]k1r,!cT*:VLָ-}:1kεoۗi3;:Z"46c?dg5~5ZwO9e=N\e簈aKWNϽON{\f癳<~|m73wzN3{2/{!٣|yV%TEn;%%b`O{.qcٙq=Vo$k8l>fܷ27WL)XnК/* }~sLcur,nɄ$:Pf\YvBFfM)F\u3{l"߄ymajkV#Û{suoBM\{V!3{n7΢n|F؇,f6g~WOK[&nak6MmGS[fǏ }Y?ا|2hV1?Q ۽P/|^A>שŠKWo#Ǜ;{ȏr*?jz>-Z8F%:q氆[o͉ ?.|KRg3ٶkɵM k:˳-꧈kpqQbB' 8gE.[/v°.ֶ}_}8_JVlĞھ7)cr옛1:X]zC֌77^yS-i"XVm3VE??7iP1tr! ?>lĬY.C/Q7lMߟzWrǠZWWˎ7{ڨb篶]2wo>X+yG;"rKQlnK\v^_66מY>Զrwþsgcuҕ9nŸ7m ĬSG˩:?Ng\2IouMi7dP-ZrW퇖j+p+i(5Y'?1hy]StʝM~mw\&s}:g#9\m9{o)|>Hqs||(ǯĤ> #bJ&Ѽ +`)Ȏ bn5S}E~z6}on_Mv٧_gGqoMG}2{bo<ڛ^Yo!7aFlh7Lgfcy[懠lIΛݨ~_l_qCp5J;\3V6݉Yۖ*Fo꟢95רEs|!q׬}մVE~q1Ige2*_X,3j }7磙vͿui} ƕJd-#ڔg͹^14Vڪ|c #l1y= mտv?|q Tޙ62VYlwy!;9ٚk͎m1+-rޏW<#.ܹ8G6|㢹eLq=˄DEI!>uq֮N|֮c۱ݶh˸\&ب["טּA:cl˶u.yVꝋ?7C^v&*(I|ӕ:X,\f6fOc )337Td./pXDi_#m ۽x9/|ӺeemUR͇یEo,Vò/e.W:l+orLÓ'b7;ѯ*&"7֘91swmۑ8e #}D;Ų"&r/ľeڐceUic8f+\:sbu.p?\shD{[' lW14,c* W\~ߺesW;J۹N{>qsh3շ5mR}wʝj\F&`PYiΣb~F9#؏ygP!y^8{Vj2NuwZ(r4Pĥe7d^cESVds$e<,#I1}[9m|M璈Uz^ڮcq`7$ޟJ|/-&Y}?6}*NvȦgCqC夏a\:vAͽͳp{y- 76k>ELT>"܁# oK!Pjz6{~q[b]ϔ9?>80 mw >cD{fiIs Ny`spLbX+7G]?w249.Wq6`; nw@p q6v}zjО;r(x{q\v\ vLr\?=l:9!Qcrcp=˥Vvp|LO7.;_ Q~~Ӟ}S=sE>@y;?9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\LoOq|ssd_IQy㟜csTqBůZ~Ϩ7ˋO/;_/[#s,q~qDm:1;~E]-LpL\Qҳ-cв]2gG]'xO|Dp9.;ڢ]k߽H?'$bl'yTw- 'p򦈿$n~rO~™x=k{&ώ>w?#5vG&n7KV=+o7}4|tvֹ;~ߞ-k! ^Cn}+lu?7{ҶfZ\_~ [F$Kq;fuϿS둾w*q+w!Q 6vL}]Et822B~-&:d?@:cy}{".N[VG-uZ!!n~ qŏM~@Z99uy gf+?߮[+ܖFv)>YXYY?4gJ[A-X{qd}:\kݞB)N1~m>ܾrOq+ĨNߌz>$>vo!X$m/_e5j>Xe D]5~~ٷol1ܲC,ϵ>?>}Fs̡9j+ev1Sf-աc˖oԿa1+QG(9}9+ƸZkV]2_o׎16g\y,}tRNt5`Umb[@.¬m\*{8|q[Ώ#9'nvYuo~A} ^b0p^! ^ՇH: x!Ӈ9ű:?D h۟-~qq۵oi;k4||ћ!2] ?LR.x6S؊Ưlc-k+Z:qmx\sڼfc-!Csqgcׂf ;~8?~|or~ʼ{h*ےl~:H \Ę0m[Wmu,rŐ|s!.ƬlF<[}zfQ 97"2F>W|9]f0鹌{?fy θ: զt~Z9Ho}n2־x"ϼdS!{jlaХ>xӇbJii$b-lMfvjU+O|2hV1JsK׏Ic,֔)c_X{ʍh^ܶm(}b\ζS1c}÷y: eo e9׵^ŧ#\2G7Twm-Yֈ_quƨ+ c.*(~3W~t#o6Ozj4z~WĘKu g˹LˏgY+&[%COSGUEވYQi؈cQ&qF6os\"0}E13fQWfA3پ"lrqmH1{48[6moz!D{V?I^aeڇu'W$9܆凁[ŵKGܦ?,yl\~:ㅯ=ԩր&;dm#9RQ٩R>ϲ:e|mf5iסcZp||#?>x>WU1uBq|JZ_GδEE0$Y ֋#7y60mE=}$Wb>:<goMeʘ;fCH@{V`'>*C|aY>֪c0D[N_ܦ}@1|M:r|ȑaMlǞu֏Hzk̕sVF27|y}@ס %m=FY'R0Ww?3~o5moSgǶ9|z$|yofOlί測:cTyǜgڷA?'?j?YiŌ,} L7vy߬l錋^%ڸ~c0M6bPШ7ʮ̈{sYWjc=r"Fh-޸X,|='<2sԷxej-01(k= q;B|,p ]Q' ŭ])=Xcm3іqMQ/mDYqaϵunmؖm\ ;qەCKu~p>?(~U s,ot | Tܽ\f6fOc )zp$r  /yz)Neo>:Þk>kk(3rcl:e[sOmoa1kpgc&Z}pB3 gXVgl.s,85^D9few7=b·Qk}1h4\|(# ,g9<:ܷ"x.]ޱ/uJO%> S.pgh|:gG¯Y\- 5!6k`֛ 4 nV nc n/"#85g)nP Pd `p@p q6{m8%__-7K|oW˅'nE9w=gÇR+1m%u\X~]_je}..7o/Բ z/Eik^o1'ndxW_:/_o?,?!}URS٧٫v]~D}ٝ)oξ=r.#<)2)ms&$#aBvb! IL¡$hI £!'^|osfv^!__=Y9Acic垿z|3n3ג > ))?:7qPh|b6E(jG7&n|d t s0 4vE'k_:sȳ-z9|:>Lq[-{C\~ 0Ǝ%g Zt[u8dݏM}+חCk<R܎v`'d-s/z$JbnCKOcM[?ux!}\8i nܵ_94Xo$bo;aMY]jv)V`GwKgJ:kPj#׶^F X9:Lܖ&v!enrfk*Tw;lCr~9q{n Z}ݧ׃;ą;|,G]VurZgA{\vm[,? p];[jI8BHFT7SurC}~yrgm5kͽ.J~1[ |یn[Ɵ^չa炷%4Xs8<ߚV|2_9ofgSïm;ĘVt9Ŕv.cEdWǘĶq~xNR9\%ܦg;Pke m=DGY,K15]¬b)EЈֶwfm]["o1713<@m-qbv88;Czpdm3f/XVU?/p!^rg [siնryLy.c׉u3hsxOn[qr3cq|Il~ou8Kqscb'j*>r΁:? 7+ǵ"bP$nbc|k[oó(fF,L""v:WߖA#ns )fo]zߦms _@>C;Hj/@V>E9W3Svf+ʲb<^۰0p>~ʉMX7?Pf5ŗA menZРwrs=ccN:S ] XrӍ Ύm/ ;{6JΥ9[E::&99c9۶d3ĶzqtPID4oJX DT_@߬{mvq{0MWu]oq#l߼찢Ճcu)-tx"gC;T|KÕ:W&VΛݨ~_l_q d8h9Cshc΅7!V6Ო-cjuN[J~. ֿF[$|2ƿ6͇\鵘{+{^v,'W}sʔ uȸ-46n N4;ve;rl=pCVn+S}?._j먃c?c^+yZ 2!nQQRe.8za+DS>+`'+vlm42n" 6;+n;vPέ۲m `z"nn@Sšh!Fa\=Uo97O:Hutp+ufl|Lvx[=6xRr>̜\m?s*"G} N˵U>9amjzXb%ck\lĘv]Du\. )ڈ֗q=L|KKSfVushĬAe[;˶n2Gk]we;\W]9\Ǥ ?\/!b33idbn5ј N{2d?( nH? 8_zcDV}N8/1mx:;s'h69xEz` o^C8 n[yELDLq9aB<7 nD nc n;'ƈAl[sډxdP)~đ .oyUk6@6{m; nw@p )j]n_壼^nZ.>q\}-#.9а!8ލrNrBs\,W./YiKm7~a#eNru>^b~XC.Փ#l˧˳Ocه˳WN>V? ;Sߜ}{\F6yRdy'Uϖiw?~v,/.?//D_~}|n/ʎ ApLÑ1ECq|!𠶏)*^w\jeڸ{/`?*sl% PHb2 &AOm 1׷%E}/?]>"{3c •'$N /嫵qgYM)nMqa׹ۇBp߀q_߰皸k67qd'75xǰ}:s{{ v,=[B<EV%~oW_ۧm'XZ 8'Hvc>%mէ_5#}'1ݍTⶻv\7o/n۴aމL/Nu+W B$dq:axXN~]屼 u~XiES'?7bIw/#qNjUZ[G\~];Vx.|p5rw?mk^e,\i[E˽۲:{ 7._؎>j׆k} ~ ksc܏olW7E+s{{{6mych߽^ڜrE/˶7 pLܖ&v!enrfk*Tw;lCr~9q{n Z}ݧï;@~1pО%q\@^<'au!a?CYsQ'R_I8r6?[>&kSfwHP.>w,r U"FT)lE-k3cq\Vm˵'_usݛ#04p=Q 9.Ceԯ$/mRb߹Ni(s~~\=/%_Janω~1#Y.oۗi33Ʋ/z]楃1#mEhl|,sdM'H7kC%s)fQڡ YB6quM+zZv |B-5fښ%n[}_Zj&iB*T" UL9Nv\`zo; ۅ{1~m&nvA;e*qHs]4^FEoWNRà]u)U{9׽]cǬ1׽=ERQە>8[zL"~ Uu]7Ŕd\6jS~1c.c_}ؗ<}Ƕe-0$nbc|k[oó(fF,L""v:WߖA#ns )fo]zߦms _@0vPfBvVJoEgA\=+~xEb#aa|VqoK6*l*$Q|МDܴA[ x/S Pv8 <nW=\cvwb֘]6Zz,>8h67zU}#ɽ*m\s$3<\qղ|wj}aZ"ޅ`Αu[Zڶܢv۞۾+1Q Cs3P۷2eLs3;(3A;hkjqC&gA<3VD>֪cRp}4Maq~sS/7vɷ ԕHbY5f#Xsmҳ7;ح_ۥW5Wߺ_VMMDzWM3U.Sjue{m=|8@,ST|[k@ VǦljsVm͛R츰=-Q3W7yk۱]kG6Ud}j|vvtq|+7/㰃2*TDy&&΃R C!T}3Աp߼j26ҷyt|}\lWtlq1\C_x\{Ttn\~Q }zYq{[M lhgG ZLgo[U n; ޼ p_qaFS0%֋ oHg1P"lx䖟nQmx&i pH nc Bs!n?VH-on#s={s;E߾˛ۭ1źx@6,Ȣl}YZ#s [3H -wVI96`7b=#nKxo6,"&Զ~ zY[w.tÜ-?Vl͘]ߴn6, \7-a'8-nپς+q'l>o-V[ sP_I.H5X{9Q .D Ja9 F-#\# ^}9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\Loҧr1']~G#2ڴH淟y8G/K |O/{{fyqQEx}|y's~QNJzoo/~ npopk{Gn!ǟp~"uQq;mmfXY[Rn2>V]˶Mq[ eBvb) ILS$ȺI «!¤է'^|osfv^!__=Y91ciSq?%M]_\pOGAcwy1ʡ&ؐp~>QԎ8qݯosMqHAV4/Ch=onD-qElXႯC\m՝eD܎B57Vv,QnBpGRlS#W&qm'XZeV$0ϋ䗬7w=QaGO~L ݚ^ ['\? xkU42B 6tC9^'_Wy,/Coc<.N;#:S'?7b88 韻kT-=G]:|}9rVnWEqbޯ>n=뾌w(CJwds·oG{aWγ$ QdHs^ab~]|[%ڷ(r/t YPcYَ?q_(hLܖ&v!S%˛M,kߪȬ34a[3m9?m7Ł kx7&1\DܾQ^X&Yk ul*!'D,[rC;'A:F,q$9Wc}\^%n|Mmצ=,9r._lKJy_ ~qXb}5O7o<l1Qf18874q; $n-Q^% ;+~Zve{/7#YֿPceQ/V6RL){YYvBFEZ:7%ߙ=wSn-MHic~4ǥ!7/mw"n)ў S1`#ERqeoNh Q,bJii$b-lMfvjU8hZ>/Xm}%TD`! {%([~ QՍZ*d}gX ź6ٶ|l)2V<&e|c.el?mg^' C{CMCm%wT9ABzf`/m/,bbe[hQ'd7*4[g}M0SL=ذn@wP.c(zm^>:]5Am$LQ\W{V 0\<lZPPlEXY[+Ӹ ﷊kMX7?Pf@_Gu,%Qݮn۲yM:k3!n;l6XdWIΞ}(sioNkw Y=x:"n;fY(IvVL^LTDC$>-\Smmj Rl -~o9Tp}Fܮ)_VLr!L~ˆy\ugCcZulZN_ܦ>W,YY_ Z+EnCy3][嗯[U>S@or=fe /}Ӷ5Ȩ!=Z9cJ6mB R ęI17Mt9`\;>R-q-[otV 4My;m&A$P)" W~|J)cy[4)IΛݨ~_l_qT(X(xc>ĸ\UQIGN| }5_b\cP_kYnYy-們?+^g>wǴ)vQ% ZbYN: b!۱ݶhS%nKB蕶 uq|ؖeW5'>_XD'0!OAL$ױrD)paWQ:X,q{8E 8בb{MpۼSSSEg16wǴ)AIvS1 n; m; nw@p qLR\oo߮ ۛvmR/ĸxK18x q hYL\yĹK࿗'ԙFM)%n;S_^orRvd}s ´=/Nqs /?/zX괸ݢ/n,xBe@&~ۉgkS={<,Ow~ܦB,_ޟ:釋{눱8ˌ:j;6Bň⽋#){}3c˸~27ط*^C| k|$x+i^c\pl(Ż\[*B!8e?/O y/Y(zΈvΗ bFADZ>fZ+=GUE8>xM&* Ǿ}-X`meXI:ӴN}^u1 s)eh[읏ֺඵzBs8Ovv_sBb!oEwI[~99 s7C|-S{M.hbjݮmG` !xh 3:Sۼ;ָ̊v·>ވ^?=c(  d\gu* nިh^vB=O3J_(j82Ι*~\plY)ΑW Xg\p]]&D?GJpr.WHtRPEL֮ v zKX1󸶵 M6*|ү݉棲]mq3J4Y*?nn'1^yX#CboQv~,DB?8YT9CrNuThenDd=|"/6C{lY1/c$cP٢g\Bs a[yseBSĢDv\q!lq; p{ja+a iQ \ucqJsEsO3>=+zqٶxoAZ""j C -P̷+,n7cøVCZmr>~ľ!@6xL^Q}I;qsvnZcQ ,`@ؾCr> nm; nw@p q&rrB)]^^/k؀9z'eBB_QW_*pq??(uQ8!r^Xp npLYaDޝ??^^>pW #qse ^N7n'/ޤVڊvď/%{ryYSY KtZ.HdVƖb27&o[P}@Khvܯ$9K}{\ݼ cɓߔ/ᵣ3nG.@e26b>r^Qm9O]).s!ы;yDuM91 8!npm[|b A QT*!M7M?MRމM< (Z.d/27a%ET7>S}1X9qۮ,:2c !@pV퍊%Mm't˛ۻvK "raq[uCZk3.n4nۮo[uy 1&\B\$RT ! 1kq%]uܖ?UrxsSB-Kaw@f1¯Y!(K n3e V _ A9Q,iJ5=_wAIH++C]veNDl3Nb6Gv[f CDvY``błƷUܖ>»*7}kGϳF{DϗZ}8-;-k10CHsaWQmMX"r5/Ƹ\/WM}ƶ%_( av8eK"Vx^ʶ=6o랻~ np~Lyq6{m; nw@p qSrs{ܾGyyv\hϝ}fZoo^wag />1/r\R8`T{rksOX|+âr,Ye+_>eX}>\*mu~MĸZѷ_ݹHߤO\bN4>Gd.i"VG0-/~%^F^Y^\z^_<_މ|ܲ_E)Vb . s}MnNĄR&XI dd]$4@,{a!3WWOgN@Ƙ>!wZT~9yOIoSQF9ڡ"CŋQԎ8qݯosMqHAV< nC>{z{.n6?K"DGEԥφQ0HA*xD+חCk<̊syqcp><*Uh|j>vng^/W&K OЭps:tKyo@r*eS_ٓpN+4ϗ=!nyF_, \N\?| bd[[{\.uw?k>{_>oV-؇/wvyq3f̵̙-3L뒕c>4֋kg|V:-߄26 IjTno;7 [5qt;lk"qf-gP8Sx7&1'D݄ê;`QF(jϒ`aWu!sA.ű:}<{ GĜX_+g:9-qo=.n6-o~g#!k 1ֲkWZWոey?yf%uR'O/jR[|p[dG5벯^s/vw|6=i4LrviGb,|3ܖV?ce_a/mkZ,nK2$62K g)fQڡ YBUk1lɫc|7ꖸ"߄yyV>7Os\rO!xu~پyUhQϔv0.S;~fzoFm/p}/ƯM"6mڎ h^%GE 1,Bk$[Kڮl){$zZ{Jņ̂+;^ELmѷ+}%N53Hmsm3ߧZ4iG?;k{η:bιv{cgRTL"!^l^-DvεFJ?m^?εp`8<m/&*"( ])f6P)x}_}_*>#nϔ/+&pZyf $O?ī!}T ø<<[\`w5٦>.:boCe[x xLۦMHX }83)憛"nY=[ێ`\;>R@L)ns[MHݎ[%\6MnI;DlxP:]t J%JꛡmP=d@ܦ:owv}}Q\o:(xqc5c+}Ƞ>SG^O'p6EΞ"ױ~fs  oo[$ىY=y$mo-fOqX=Ik>Mn\殎i2RY^:?kwwME7qLۻuLw3DTI!uq֮N| p{@[vlm4*Ɣ)ccҶm]=81ehU *;.C?Me/C;HCpp"do97O:DQO+^m of3AZn6Z>&n;- ʅ5w qӶ5h/1b^-S^J>q<+j~D99&}aIHs)(˷mL˕3o^5Nb\G|[-{yVrk7Xm?z}|b^y_|sm16 _I6)O5v㘶w똶2f$na/@A@z_.7!lx@~L@| ۏ ; nw@p q6{mNWrR/rr=wkQ7V/˵ݻZg3q/]n^urrR+{@!KΉrqTm\oomulO5fO3wCݟlΗ\?{@uOH`g>r.N叾W}r}x:av8#'}/sOX|+âr,Ye+_>eX}>\*mu~MĸZѷ_ݹHߤO\bN4>Gd.i"VG7>[^K޽ʿٽ~("xme(;֡=Fռ#JkkǁZȍ_r̵|8|Hr֣<[aXiy${oćGٻ<\ qaT>{~nyyY{)5K6\w6.~$ oJ˄R&XI dd]$4@,{a!3WWOgN@Ƙ>!wZT~9yOIoSQF9ڦ\q_߰皸k671>,E~`s{0|X?XNr`8~vTg9Xiy备xs#{;^ܾ3q.E?{}qA?|/\iBl׸|gY!:*.x6(7٦G %Vm'XZeV$0ϋ䗬7w=QaG;S \ʿqw.6I،:YF7ϸ:cy}oIbur\ _?_~JvS ?=0r8x>7gϷo.72r[GZ1]?lq/n1.a[/#|+c6v1EjiǬ3+C15|y֊Yo\rL\cV9>C*0۲aT?囜VԶؤo=I\1܋239磬+گT"Wd.YDv.YzzX1rXPsBaѕ͘|- /<ڔ~sqG\ϚUýW,9~.lq֥NѿEy+ں܉gAGytZɹˌuhyVWDg}(ELZkӷ+֋g={J'_\7/v5?g/\>Z4V"Gkg!rtj}eFrEyc"&ʺ?-MzDWCrK$7۹Y]ߪ34a[3m9?m7!ǟΧ6ų05p>!&lnC 8nrU7"7>uؔ*6'F_'b>ma eyCsu\oPoTl'%n|Mmצȹ<8`Qn7j}/CY%yzU2f|}+?L߅|X1/c@Y_o_\=k<~{52AsMqɘE~21ʘyGuש6moz̥{B}y}+yfy۳EoNK|!*HB|7D3RF͇苌aDk:~xNkÝ8aWsO%Wy&m P0sA; bU["_~>&%v*m~^c}Q\ 5<ɟy= 7g_z-`q;f22:>ke\g5N.hsmSy+zh~s[YܖeOIl eR͢Cr5X/b|W;6n -qE )2| o64XB%e}ma1k1{ڢgfՍϨtX^lH.Jߋ+姥-nM5q#ک-WG">5Wȋ*ox)碝`X6oZG֨[fӏ%֫UlvW\9xzys(i$1o_3xC=\nsXy 9U:>L=r|Leh>: h/kSHҎC-nY_f҇c6?U.(B]Ko\}=U=G16:?ߪiY;f9gfNo^Wc߾x.m>+m/V9=|F_U}Usǵ>/F>Tvqkw-n.sdb>X9gؤq笽bFbTǞ7aM9vkhTA۫l~(n (j&)oswq*9!m -gV1smQbe| ڹ^q[sl̥ѶX#roxXIP"zۋm E(G,L":n*wlrqӱѼm]| Ǿ]QνN}{u;k\T^`Z$ mło.HCwS"6Koي+W$7s'o.qv!nnq^cJ s^*RY3fwAޜ12ɫz/nv[e_^GխfCkUq5w-ߵ"_s3f/ɐ\<t;lbR1-n{1QAITB pM7SV/J- ݶRqq~|Y1˅3K&}BX {e M='ܔ3kug:}U[GfևްZN_ܦ}M)şu|Γm }Yqkx< =^ىY=sy${Wg\-_j|-F2HXReʳ*\?.hb>Kٙ2_2U[>?u/T'vUhTɑ(؍XOd_ț8_J|**&zi[o;}䑉FyˎK| ̳$n!Beޗ/(meSW>Qٵ?/O/LvMYlvyNFbf@1mCh1| r{?FVNYvQ% ZbYN: bl[m۱ݶhS%nKBD uq|ؖeW5'N7&Bá͎6jq2{ul.j1M,~Ơc| ؘkufl|Lvx[=:l~7n)sonTwMH+r8?EjێT9}_EUgǬiqЇTV\A}S7/w6՘Q_gOq9ܣg]"aOsa l|5 mc_އ|͸dd@?k}̭l`>eoې:cY=63(VV+o3\qTī{ej\UkdԶ㽽UbYr x&=g9JeUqkͲxwvFos1e_b}sJ_]Iʱg$o ٮbR"-Q~*߻smMg\.|Y='zu̬\y66WhUyg$CWJbC[ pϓAv|+bc'岑_f1Neo%Mu pA4C`8o'l [@,ޓ4P ,4fMq;Xswmpě_.80;Cq`M^Ц|B,a=2~vg@%ZYqྃO-?t| QH2N~we8W nw@p q6{m^>B{7ע7#/cr^͋귻r}5Erx̉۟>Y.^a>՗WO~e߿z|dԶ;|byi,py]ic6jG~Avb"}>sE9lb缈͈qTqۋ\)-/~%^F^Y^\z^_<_މ|ܲ_(Zs|#n;qZ6M n}AΉ qۉL$a^TLv&'1Hm'bQߋOOd(혱BzW?UqZƏ3|%˃@.=DE=Xau/✠/mWOM+%1^T1v- _U{:~CXT!_7oeVI41X[~s< N,'X^:Eh;u[BlS'?7b+]ʿH̕\9(4?^=/";oT}mJOb~c +sIFD=<Y[Yr+<<&nK^;xPK $kߪ\s4a-kpvm;Qܦ8/7\`_}{N+nz)/NPnjϒXEe.ioYhD \8_ǵ–xD_IЌr)z7A]7OGrAOe2C_Goz(f(ߊ.h^ڒ9;3soY7l_L˘s5eӛ0o=ŘlI4dcIx eKϥEiǸ m/zQ=]?Sc-nMțL+Ѳ]֙c%n_f4mqm .Lx%Upbm3.x "ꘂg|FXc7'$ ۅ(FBS{1~m&nvA;e*qHs}@80j:zs]ԓ9VUV (BsFZ'm:U~\2D^'EO hu[BϊKeq<p*)Dz50u )CzIovfLbL26-m Y6I91͹ 8A&_ 섫$T3^v˥g0y#uT}3̱XyJ1#<m煽mxl&HͦYmcӍ'/w :ŗ.3s͕J侯}|49:`ߢ3 ܩA9ʽNB4tkɎaI=&"ٮ5R<·_KIVҞRG9C9=$!3q7m6 nw@p q6{m?X~^~Jg0fZ+X\]nvoojjuRElR/rr34R+;yX#[mw}vG!'GLH,Lԏ/#x??kCF;x.rGaC˓(q 1s Eߚ"vs1ʯ^{|w!ʹya+f&n+CX/îG6)}dxW_/_<<{~C>l[xdH0(|+ډ1%|flU* Ǥg;)rEy=yV=>1wmsegS)9 q{ӓ/ȡ1Ix=Ww~F:*-n#)?BKM‰"Hf_ q;B"| K/7*"Mwܤ{-^r<íCo3,>D7 m[Qظdqh]^?q18oVVŔctPi ""+e_Sr}{|LcsJJb[/qvjS?Z8> :A $"$H^||ڻYª)x []:}KH,=8S R\KoP\vWM:ŸF.chڮ!kXڎs}ٖڸ|S-'??odcVfbTX̵nɳCv!̃[a{_j]gbϑu~T=vBp~!Ϭ==9+'c0*VDGѾ7./klOmW9P|Oe22|8b\61 [3%f[|,\9YܑsJ/WX9iG/u7#RX)џ9yxqv]]Wl[-l|otFm&?B?g @}%c`|ظ8R#e|јd`m;qr{ۻ꼿̮M}>8M$wa{/e{L,^Q[o)H}8,iQ딯Oוk7uCLs-4`zՉ=vZ1sk^?u̪\\=؎ n7ܿad.7(g:yH|Z3j/77@KаѤw8# \  QRڢ?}"}f@oݪ"$?9ڳN V6YEYAxh1!/9!ɏ_a+'c$cƭ̭u75Gx~ƪ k1Prs 4ŴE}a+|C-7 zSh-M/EEC6 Zn Hɐ7읽_@$MP+'`PJc~EqRi7.Bw#Mgů2.ߙyڇ)^OggzU9Xp>F{ {G3/2|)3"n*@%pbnz9^q01%as8:bu}Ki,3}+f.ϗJs@h;m߳y1Kzgw9 ֣ucO HzzWmhES\w'Q!2+rql^GaV`:3|1YBCm;$ ݴOolOJﶭo-{|gV}!>3Nwky+z5[!]63rrM;5bmg{srʾ疻}yVر8<+n:1\q:اS4lo~LO/9+j _3|!M湃{aκN@qٲ3{E#C}MƇ"LS8(ZڡL 6nGe!+(6يhـ~`ѿS>< y(r&ZD&؃\׾=-j{q} +.0=!Hv{=cZ8k"llneuoS&zlD {?aw^% ^=s}(i[)|PCc9Zl#/bc$T"g,؄J+:}\TѶg3HDy(n(4b˲|ƨo@g s->hm;2Ŭ?C:狉}Ι=)bH 6'=xdc-FNlbӌJ١ اvT?ڽ>gy>_ÍSIG2k@WŃ9(ϰzᵜѽy}֊z)~Bƿb1GwVٷY9.Ya?|y2 &C~mtˁ9*6ρ?'?cJ[NUH<_6!S; OEAWF/}KHuaz{mdѮ}-߯Bw͗x\ƿJ4 Uث"F|L{?C֏9>`,Q2`;M(#v;rD/5}|nW≭3щQA:\^ -kgcw =wK{.+y)6Ϻ/]߼G-މ{uP{=O-+"n&l;~z|Lo{몽-q׸[ƕyX<#㾏s#xYb|6ZW['zg&ñasgrŬg!萱2I@!Nx~" nY~Ƣhr'2Ei,{c+¤\fP, >#ڹ}| 3rl:a^3udն2:XQWYUK7M /ᾴ%ck}V52 @]OFUdk!*!.gc\ؾ|q{>`b#?*k_a셵x1|`2(qvp[{rF etu~]žol}q{96Eq{;Xɥ/;S;ʍf=і`ئNDԮKvkoo4?zvNv,p'>+8\X}:08agl+{+\ZN{ߤ%XSMžg3طxžomS1DB>(ϏuY&ﳒ,j狉.>8f.2ƱScP Ŝ8wus$ٲcژ+ibz<#l3bk=_pO>C؄}a}՗>DE잃o7 e,-C|PfvL.g\Lp恫mHl #A xP`'r{n?:i!/b1}+/.8oEc>'m]ͮLf \ϼ@ ~>c(wȃ<#^֧߿ ph0ڶ(XrhUیK|bWN1HhEFaaƲ Ǝ5H׶bLP6㱭zvv^ZϞ2Xkh^ʝ6r;@={^}Dc㶎uz~8Dgag=_&v3פp2/wy^;e;B=g|g j=kߎBPbNo]~LqG$kOJG<n,cFqxːXv{-2\fZ~,|ļZ;}S<տ`x]w7ڹjdm/cL|wd|<-mg{SkunvsL\qcu`4a.3QoXGvs8ML,E3sm3kܦ![Zw3]}B_ʇCI9+|W.N[#Dr!/I\9r_dH$߷jw(3wqɅkr) y$Ir8_Lq$n-_y)l_|QMFIrX~o{2䢤}_0!9$In玕'^p7Q/I\8DeIrAq~G1)s]f$RMnSԹk0ۛnDeIxϹw*_޾z%IrO>~)9K"LjWo;lVsqvQ/ߙez˂yVp. P$]wGd?=?3{X˞~h˟d{_>}ϑʿl}tPvfN;1͙?֧Ͷ}\ ><=U>"ϙ*1ٔٸCq~ |pǭwT_/y.S/"G6_?α7~@7ܙ^g_e 6~ݩ=c+8٪Pq$eU܎KDO/@$uio'CQ͠-Elض7Hp6/UopA6,afvpbK|~'\Yu_)Ity$P{|ZJIb2'rueΧ8v/d~D(U/9ñ 8wVҞhZ<#-N| H"1i/Ȕ^qi)D5 ͇=\7ծi2ۂb u;j~鏭m{!D8U;'ؾZ ֳ̙}aZk練NƿKKt|%fpܻ)/kR܆ϥc :wSl.^r/WRܾܾ;1ygsXv$3۽ *ֳvhf{H®ޗږS_Ia "âJ(돈1"]kp^̃X ~,cB8=XMv(QW%eNhcg~]|%jlH2EmגH5O@o<6Rϑ_a7&=IA%L|&@`_zߩ_2`"PϪw'W?QGٮe?ōVn6%עEH/0{lc}[PL| bPC>ۧ>rȿҧ1%m2k߇ΏqkY[? 1p=Ӈ>9 cB.>+Uhϔߣ䜟}dڥl .Ve?ΝXܟ >e0Gcy)v:m炈Sk]hcCg;@>gbY1g Uz6+QeV[gtnm;cL߽s5IwKALfXغ5 Ee_Ϥ]3*X$n %h@͇B}{ _%6:UKU _&r_p2ᓜr Dz%W`hs4[($5T{hgb _JXvj7DsP>>Ӓ;/^~Tq2?#'HtT,k"mt2Ś`~U}%lžzI̫pb!`[(W"}!^H)_96^1X= ]h3ZvsEF938>b,q}Ǟ;}Wc {({=8n^092uSqXXJc x]1si-uxT:vwBi͋_"{=;Ye8{dGr3LܼnUF{/bҾ,9a;W\q/kD-9#}^su츼o}he0P8og-*yal뽎ovs/>vz8q#P#W'U2ɳ _|O3hOj}5r-862ю:ط)1OFԤgOpdF-Aƿ&Q>&lr,_$(Lf;Zrh[ABc(&:"6FB%ŠMBGNCm{:D"N#,gzF@;s+nٶ#cZ۸WxW㬊tm_^{ yG8bix.^I&8͸Q?7׶JU:ŌQצkU"ICZ?Zk[G4t=99c,8l?O c4'%IDAT 9 #bk`igc$w~}w XbQHf!Z~M?1dg(;IƟ|嶵uvC 219'x8hNl2MbdS $\G& \p@vD4($ܟ,J<˼]ֹI~Lcx2e,<{5}֧%2~n P {*6˫"*6ρ?'?_inQ;\A R މ@LP- E26}A[Bbݳ%%nK&cpX֘gv%Nk~M~?kRΫ9xUIg;#Uث"F|L{?C֏9>(B{C흂ʉ&yTI9g"ҩsxbEt׭7s!DπON Ƈ`_Csj5;yL1"hͩ#9~Gl5qZŇ0Q5 ]ovD~1ȹDYY볁xCu`M:B{;cINv Dz)4'+{n68qӶ)㹘|9&Ir8rO|mh׾nHA~.}IZ57'$fQ۔o~8I:\Pb$=AI ?Lj0ugvظќw`^Bqofn ^6s\2;7~_A}G'ƜEPV}%KBS8WKc!vdW!nE`1d߽XY}{Ea=m "I1g[wO!bO+╌AA2o.xb)GƻY1ο.|_Qô<$pvTW,aEu'Y??eb.\q^/?SEYmrHy~V1W#aLNq^h\q2VK~As}:=n;+=zH"CL/F*0.aƳX9!3穼?qEl lRqMgc$w/V1Egk#^PFږuLj>)6`g\7HܞF/)%הw*:[{^ a$+2D &/'g8ї2%U#>y vmΒhد {%J6ɂr{:ܗ춽OcXc:[֐-á|ƞȈBT< lz@s ^\ؾ|q{>`b#?*k_a셵x1|wyQbi(3|˚xAֽu.ۯq7O.辸=뢸=_Yn,gRϗm Fx^ nhB0|ݎ|elS'"\j%;ڵ7=v|;';Akt C˜3m=}?m|ybo|dO7'5s_ߵ veG+&.cOp0>cz*ox>sFmKFB|4[Ov/G%{o91 KTqmZ= Uⲍw|Mqwnu<4oSPI5+n%WmRT}׺%x}Wkme8I"a>z jP c*J ZO(r=6꾑:7mW(TN2|($m>'m]ͮOfe=|~>c8iye (y(k}j8c}~0$GCqжE/(3j(prm*S}qwzbZOi"f -8(!,lXر 6Ul fw<{]/ΎbKu/Ck|͋`b+C:/B("jOm~u!bԿG> t,g =2m &͇y-9s>]Pclwϰ^Xv' i~LcT=+mn, 8N(nC3i!r399yCm}[g[y όŽ%ːǰчEgz,̬IH=2sN@(Y+SAKeMe'shqG}Bٸ:$I$Lx22"Z$ƭܗy= jw(3wqI/ ]@^C?I$I4)nmN,7In[y& #$e=Oq3w;Z|BI$N/I\${;ĤMwKeomG-Rm?+}D~Av$Irv$I$I$I$I$I\7)n'I$I$I$I$I$$I$I$I$I$I[GI$I$I$I$I\:m,IN!$I$I$I$I$ͶWۿ /_ziw("vsl~h6w\1.Ѻ<RN$I$I$I$IK6 {_CM.ۖSDsd ݟ~ [ba}~@ 65%qX}Uxh{P'Oo_lo~~~%I?o?ީ<~v{qX$5gWěAq*/l?8*l/K,5 )͗#}?':֓_Eev{lݏty=m9BqXy9Șqq&.4I4d &VPn"ڙEߑP=pܑۗ_f Q33h^n+'?>o߾y|Cy?ݾy8R* >h~(8sRdr| |}K ئˋ xs0~OKQ=M>|SΗn6U?l}jke{/v;gI䳂Cm羝j;.J;[LKNͳ=#uE퇲_.'H$;0%'%z=f=?}\#?e;͕=hlS̜*vc#lכ3Oom}2͉y{|E3Uc)ݳJ'^9EEN[器^\!^KEE͗&,sMr(o@7Y{AqX~ ~};]:Z hϭrS틟IWA0Qnẻd\?z}~ñE)K(٧87cˎJcpv=X[/|@"W!`$3Z4NJ AB"vMIjW4}MmAxUyֺEEP?}6?"]lp-CY̾m5wUXo_e_ƭ!Fs}ùA+X{ns|l.>+r%[ӕpM7?Ť}yD_KyUfQl'5]M#9KK~A?qn4Ϟl]M1YΙ}t]YqJ~}u=y$qa׃~Km)KW"NXtIC{9`)xm~` ^$l.PnGؙ_g# Lؐeyz~㱑ڇ}^ȏζƠ_^4 1xN j/ae61%m2k߇ΏqkY[? 1p=Ӈ>9_AJۂGgJQr>R6MKb/Me/Lw>+ܗrRԑ EocM>DLDD登֣oطueCG܎o33/^/ l.X π }LcV3j /(v\s&X9_vze}|-˨#X@^$m~ږ#f~4?/~O6_n[Y3=ur 6,k[bvFY\ }X♴~~Vm:`E #dsRd% \`MUy_3N0}QBM_pBrT&_Iy>oUr t{%/Jpӌe׾&qӱO4?z)}5콱_&=6QmNr^щ&MkUUM_<Vmm"ߎ})*M`,_8=aыcc{Cq ȫs۱|qAzq=i>g`_d3#R %e^' a=,^ڽW7LL El~:b]m,|u4J\і:K<_*}c/뽞ݝ2X s= 2X#51H_>}1uwl.:yb6q䌐5{e)ױj۩~a\|(rX뽟leꇹ{:.{<]filzˮ_h?B'VĺxVdήٙ Ѽfqƕv8lcdODQ|M<"l3W5/[^G # {) dQq6qE:R_KXFto^WQ 秈Xm:vmD6VM/J2sR,чACh JKPu HdַL&j_ܾ&2IqҗWIP`^ҡRf3yG\'M~Ugl^K)RoAIllned&lC{5?0;$5$J$J%3 VkLf;t¥mE;>( \(&:"6FB%ŠMBGNCm{:D"N#,gzF@;s+nٶ#cZgfR/ tm_^6цÌYWg>9nDu`O2?9D\k樜8+WpCD.-*'|nmtraMPל >E=:~('ֶkdGwzvKkuϖs/ysȞx/ǀm~ǿq#p6P%lƹO2MuTEf# ^ `D`Y 0 ,D/g5l e'3Oܶwh<8gobCYL(`RN?DȄN(ȮwIV$E%D ܟ,JbDDH*L/8gDdždjXyzk.OK1z^P{rIcdnGΙefo//J]υrF|B۶;\'A}%&Ͳj.Hv C-P}5Q>A?K|23`wrh<_`i.uda;M-gvp.\|q.q֋DK^׊m&4MĎSXzrTܮ<@}yVWO܎ƶE&n샽e 6qڵ/RЦK_RV2} I6?>yԾ"6gn[!Nb!2O QD{!%rC%#*!{8@L'NtqRgr0QR'J%"ßɾGc]>aL-a½N{O5lDI,{; .kPщ1_EPV}%KBS8WKc!vdW!nE`1d߽XY}{Ea=m(珉k=)b=Q㇋W2y(YTW}ovfq88Guv󼒈Ck#]S_ղW|Ձg`x;depP XsyzL1"g-!=YOC2;~i*sa0+WZ-aǟYڧs?p]bZs>F{thӈSvM"*yXT-}Jw _n-'}~ / h,תu^gk"#q}in.p^q*frm (PrQ:EFq쬂6+KJ85ݤ֞Ʋ<6&Q.$6F谍^Wv`>ϗp>yĕm- xA̻g6Vol$;mΒhد {%L+?A_p_ҳ>%ck`l[`K볾ȈBT< lz@s ^\ؾ|q{>`b#?*k_a셵x1|Dߝ%bZ*@(}E^׸U'st_ܞcuQܞů~N73@rnr#b=<rދCDa{VP+6Q_=bh?Tc]B{}}A<OK^Lܶ<6T hg)6m~ Gzsy[| >g;WJ߯Z 7,uK3q/DLCU4tqB-'LrSzm(,}dDԹitRB:mʧv= 7.3zu7gᳲg '9Ls'7WEYSÉf,@&m=q@ڦZ[/ "O'EHQd6Yl, `X\]}m*e;ۊ=ڮig?ۺ!v>YDZYEl y 9@={^ˮnvc:1#F:3|τ_ɼf{U.1vgXG\s};Z A9u!3"(\{Wڶ}qW, q!hڌmb7$r?s>399*KOK& mDžk1|DS&RV'[툑+q}-c8^MJh|i{Ww`$c`_K+%Irm}_ ̍V_o$bQgQq]/ C%,O%+XzD.BN ~f&zqۊ!nH|]j,W6:TNX`n;)nmN,ïAVÿOa j0RN[Ɗ{stIr},""Y qzmK c-<͖&m5u~B]$q{; 7$ʼn)^$7 1)s]f$R3x_~nS<>ꟸ8򧨒QR$fA W/nK~G܎{}푀|?23{8,n'I$I$I$I$Iҧ'[A,:Nk(p>??&@{\^ץo4O\B~"ߎ6w$I$I$I$IDP+N"ֲױ80nlEW1+8P _7#q[ ڿ,x=.|HI$I$I$I$Ir&XlO)5r[ |Z['Wzj LGPXf<xM&;7 FI$I$I$I$Ir6?Q4iN6#rNQv$I$I$I$I$Ir8,n*|=s7`/7?^?ʒ$sTz}8,Kyz{3+ˠn|oz{dpoQ%qsgKΑS>yޟxh}Ic{/Ï?=_E~Guֶ?my ,x^v\4k87$$n侹=tA⸮K < ޓuJ }5W.%` sD./2pS?眀/=|lc^%l>(g?P'||RlsD>+>Qʯv٬_3aZDg]@`Q엃II/, LI/z=f=?}\#?e;͕=hlS̜*vc#lכ3Oom}2͉y{|E3Uc) ⼉љ ;/kQq;|Q乜CL \T|i8zD˜sqWer2mcQqF݌>grC;SNqVq;/b?پXץ}EۏBXV6"b^"]< D6Ta?ڈ۳I\/i{ dLpe9})d&奾& ʶζ"i*%jZHnם_gN;[9ؽd}BiǢDT &w?ZI{jy|6_8D"ĤI S{ĥqRRQD(Hk7npNVkdl ū'̳--?{O삾W`k)Z2gm{hٿ,=NƝ3nڭ!Fs}ùA3F1gOi\Iq3~ƘsNqu9CLq;Iș}t]YqJ~}u=y$qa׃~Km)Kxal%aGĉ\$!) A,_X6h? 1qzPJnw;ʜ>$Jؐ@eڮ%jxla#{7򣳭1(nz)Wo/M2{ KpM`*ϙmSdbDUN `]m>60J*7ۯE^06"a,0Cu1:lYS/1Ρ8Lm;}O}cOOyVc^bas=>vN龬_o_(1ڳטP1ֱHǍQ!^Xhb3Cr|x֋Y^`%l{˿}&^"pvY~6/ih_>t~v^h1>Լl 5սlsrQ^r5LjL=JG])QܹgYƎ~9y)C]]w"Ķ6{Ƙ6l\zD} }|]3~&-*1|s@_٩_os18h^luF禍;3;W$ywqdx1^,YTE>,n}}LE?c?I6Pҋa|,$ʁIWr3]⃚M&?\<(}5콱ƴ˾W&Uψ|>0 ՆKŚHa b{&X_Uc [^Rf6&XtJ;UH_6|἗E/R 엿Cc m' 2h!nǂym7k֧;Șų=wv3X)^?+&"6cr{Tk1V}y/6X>g:^W\%.qhK%/vھgb^sVG9NǞ\k 07|ا/> b1x;W.l?QrF2qyBTK\|(rXS3yal8Lqt!MYO~|۱3l@|1ZhJFnײ97Y}deƲ sLq6qE:R_KXFto^WQ 秈Xm:vmDv= |ˁNn{&QBLԎMKSE"R5ɊME]'H"\BRmD}F"Ndg8U3)RP~X&q\6e8 =وyȽ%5$JDŽM%DeI `GKm+9H0y Зs`DG_HDXXѱ @Wu(m6?YgܝPQiĶeQQ߀XaZ|nZ5vdIQw[,b+272C5''!O70~Vՙ!{'|*}<?Qy{keAYŁ~^/s"ria(V9)ssm.Xe11{U"^ ivG^ ggy2f,>h^x)<_[\l4H\L>${_?ؾb0B2 mY!?CI6 /ZP!I ?dR ^=aUI΂_/4f.@ P~ۄL)bD &?"XFX_펾 -!1͇ Y%1_8bVkEx&x 50g:NJT|>BWE|i1Ս$~1s|i/Q2`;M(#v;rD/5}SG!.w֙ϋ [o嵳CՉU;$7AZ X%c͑yL1"( 9ud0gr>sai$[wm 羒G{:l ޟ}Xγ>Yu >3o8O,>h^\=s3IYN31Im~}&nCevC sKJT9I&1'Wئmx ;IL c1< y/+؄$mHȄ$#P%+6Aq"Ň?Lj0]LΣcNL Ix"(~I%!RE gۥgj;p"0pz{Ǿ=)*b I3DbE?\1(CFZ,:{x70c>ץ/ >sD4ZE/⳨<{$+=LXŢ+I3ň^ն( ?gVs0]|c,< D>B/F*0.aƳX9!3穼?qEl lRqMgu{޶{y^X~o/:_w/ ]X\3.s^q*frm (PrQ:EFq쬂6+۳`',qkʻI=eɆ |k^tk| 3^K\֒*{͌בATʼ|fc`|;Gm[_kggWq%dA~=x Kzvާd,@[{1I}kȖ0 @IcOFUdk!*Ih= D9oc.l_\|`0Ecv}㯰~sbZK{ .1ؿrt龬Do^뢿}#ؘsl,~gwܟK=_vv -wy5̺{-? Xu;Mp=]hh~ XNY7,}Vc87ɗן)n[NMtsR3W]; blW}t;brB{1/ EvlzvtyE3jS,=7b|ϐzTbuEllY]קkxYO8mkss3a}S]BX:Hnm?/Ͻ{ bv?)j_2 @?Ab@2ueߵ.} |Z}Y~q;NeAXŇx#NV'rĘV1Jv\=M%"+&oMۥ.oI&7mS>Q`l|gᳲg '99Ls%7o_eO 'B&ph(nڶ(eFnYVeo3N/QS+Ic:-R,#Ge;M:63;W A_38BǶbpOeٱCO{i={;e`Ϣyp88VΧY<$Bq{'nvc:1#F:3|τ_ɼf{U.1vgXG\s};Z 4d&1,{&ym{B(PnjmrZ9o @rg&'1P|b^0vW_{Ρy1s@6Fo}RD:Ǣ̚ߏ$~#X9>4dr(:rl^;?]T6}|/IrHq;In+mS>Eŝ_sOIq;{)z}:n)n_/'?q}s$I.gI?bR&ɻH2ϷK}6ţEm)6dE Hq;Txݱ}ܻ`ۧg@PNq$I$I$I$I$YEAK@o{{]*\9jVi#ݏBc8/V͓Oqb$I$I$I$I$gF( J$6')ϘHܮ~ߋFP{ρH!m fXvDܮuSܾ4RN$I$I$I$I3bEMF (n=/M蹱ܽO}?e 1_Kdﺬ9mv6|Rܾ)n'I$I$I$I$ə{,"-L-h֑`mV_ϋպ||Fs+pa;ۛHq;I$I$I$I$I.&Eʙ}HA2Z%F7@=b&^Gf:Fa$m|e6}4CI$I$I$I$Ir :ko'Lm΁s Î"r;+?r\mvv(K vm6>-S>/=Wu'QY$=^|sWoeI\#O^ooxEy^o "*$r\p|9rj smc=le{QToGN:!.w댕/xnYqUB,z1"p:YQS n!z,⺉$Sm/WlQZČ;p-qѫ#SY۫$n侹=tA⸮K < ޓuJ }5W.%` sD./2xR?-qd!Gl9' D7vO%;_~'r@W /ۣ!O- <'s% jn5vԿ򫷝v698(Ln}2.95J|~9("”|Lٞ>)ZG_$s}\g66W`M3sع㧏]o4>mZVXɬ6'!yTَyΦsEEN[器^\!^K6sQQed 6 q2ƹ8+w#(:z.sPf0^*_#nF¥s*7Uܾt]0_s^(&SW_F/YؑP< sj_3۾WhzMDb7Sg?{m+L}U܎KDO/@$uio'CQ͠-Elض7Hp6/UoEspm١ ^=Șr+BRY_%be[g[Q5s-EQX$Sί3S-ʜOq^>_TƉ^vT5 sk%ƪs8x~ & |ĥqRRQD(Hk7npNVkdl ū'̳--?{O삾W`k)Z2gm{hٿ$-X_ƭ!Fs}ùA+Ĥ=79m.>+r%Y.78/"nTR>?=ᒿb+^Vܖdk/ 1x9@+c/ j>Xh>O/3۽ *ֳvhf{H®ޗږS_Ia 2$D˵Kb |$%ޏel[h,]ݎ2'3F>0!'0 P4>c#km]AqM&z~ic^ćl{Uyll% &r ~wq`j^vShvnQbe_!`lDJX`bmA1u3ز*B_dclϋCqw8*n7ȭǚ#J򜭮8V.}Y=ǿ3޾zQb@ g 1ybc㑎Cig:'*K5LEk[l^&3_Ѿ}1 ձ3c 0}yA3^k{/KL \|W؞)}G9?K@6>7;~;09q=;倻uhws= #sS?\}yuHsfk3/^/ l.X π }LcV3j /(v\s&X9_vze{' MI<%kqMm"\vP u.`s/>m /dIuuSܾ\.A1b-b:^̮(8k~{__VlNrdAᓋK\$f`R+'B倛8M.F |Fߪ%Q",/mO3]⃚M&?\>~>ONXgD>dfGa8{&:F'f4^ VW7}XVJ[;XtJ|;UH_6|἗E/R 엿Cc m' 2h!nǂym7k֧;Ș3a&Fr0V<,Kㆉ)ᵈ /^?7ZU_K4ϙྎ3F{?RyKcw9 ټxl%r׳UQ:zӱ'~A|$Z=t&&|ا/>xs3w@'\̗5/A2qyBTK\|(rXS3 yal}galjyHY5O;v\z=Ze}3 ~8N\u=1Ȝ]3y0+0qR<9x9N*˛ oOi\ Gm/ִ2ً#{s|.mH$e;%(6"f ]lv0Mu-߼rOtb;m H!^dXڇ70IQMvl2 E[&/n_X$Ը|_~ DLΫ$(0v/ )3 #&Q&=ZhOj}Hfcs{,$sN6dJ3Ǟ ߑ$ѿ&Q>&Qr,џIZ{g6ء.m+AIB10-X61*3VtlB%zG>&v:h۳ ƏE$"w<GqeYT>c7 3zzع[q{xͶ{uEy^\'ElRwr6t!*nQx%}?X4RFvFD5>=d7iƀf<:1vYe7pcw T=ox~Q}!Ʀ\+r8!"h{n>7׶JU:ŌLP׼w9wxb,".1p=I,uHP=[?ѾOats#4{& 8ibi:x?qH!'@o:r~qO>`D`Y 0 ,D/g5l e'3Oܶwh<8gobCY`kGl dRE?DȄN(ȮwIV($E%D ܟ,JbD H*L/8gDdždjXyzk.OK'k[5Q~E?K`i be ##gKwgjKsm 1_;&p2Y7&Jb*)n'gQ}^N2  ڗ )hϥ/)R+Q$q30O˄\q^/?SEEqQރZ!U|>LNc+Vwvyf Z%?(1v^k gxyC|h.yQQl3`F7v/{G0Yݷ٤b$F0:=&"\Wm˧UF'ڈQt]12u8Ym*W !c )&U 4ɟM&5BmRsIxCe%lkIfȠmeu>zc&)$=iv|F~-a_7ybݶi, ^{Lc+|Vs _Zm|\EFaBl읣_6狈 Q4v9oWY{=? {7'f/ċ;%jvx(qʐ0VEQ,ob76辸=뢸=_Yn,gRϗm Fx^ nhB0|ݎ|elS'"\j%;ڵ7=v|;';Akt Ciz_@'_^C)_o95I\1Jvخ 0W\τC;OۀV 3fi^3|b_WGf~^X2p}+c|쉰ѐ^G}/$&۳߅Z)C èjOA\__2 @?Ab@2ueߵ.} |Z}Y~q;NeH??i蠪rFNQ{J\=M%/辑:7mN]H'1q>t5MԮGf'zn&oX$ge߁N4s{炓Nn^֧?&Y0Lhz‴M͵$u m* ^eADZOi"f -8(!,lXر 6Ul fw<{]/ΎbKu/Ck|͋3lz.N)0'bO~߫?yZx̱q[:=Q"Fy@R0zܳ/}k|8;[޼lݲ3W5zv K~oGk!(1.d&\S`~gw3k;~_:n/fhM"J{ 30qy㰱kkM'mw~~Bݵ%noOq;8.pP>A$w L:]rItE$I[/UK ڝD, =*nEAru_I$]##\͉e5Hj)l_|QMFIrX~o{.I$I5N/I\%Ir(27IeF/?76ţ*9 'EKRN$y8,n'I$I$I$I$I$uv$I$I$I$I$IrHq;I$I$I$I$I޽m~-o7$I$I$I$IDlw/C{h}HO!&_#;і?j/BI$I$I$I$IrKHqTᶉۿzVqnnQ";{5s$I$I$I$I$/2s۵/#: ֗կG"{\jǡ?MraqgϷUAߟ<s~IT$I{aY$ȓ۟!^o^uFp_嫼|&./`7AX6`ŷIOo^>߾POo{,Ծ Z=9. \k>پ7_l}R)8G)7%oSO = M<>|SΗn6U?l}jke{/v;gI䳂Cm羝j;.J;[L{Y0J|~9D"”g{Zookӯm,o˿Zۧ9sQ\׃폶6̩b>揟>v9kYa'ۜǟʇ\9Se;92gΌqzGEr1B/'rQzd qm\;|Aq|Tfo/.grC;Sܾ} r 'w3O 6</Hfݦ'_7gR(Q x]7PT~(e%n3(rK(u.eͳ@>*nKm=C=Kߟ@@ƤGC䇒D^mEDӒUJ͵EaܖP;Ov8(s>ű{2ۅ$GEzM~۵Dc9l/ $vOהWEOg[DQ[Kl /n>#}9 R(?e ZWqJUnG)$]|%jlH2EmגH5O@o<6Rϑ_a7&=IA%L|&@`_zߩ_2`"PϪw'W?QGٮe?ōVn6%עEH/0{lc}[PL| bPC>ۧ>rȿҧ֏c֦'n-n`}gmY,"zfQA3i 'BI/!p!0B+"'E^9WIh@I|l_5%%% '>)\0p,\r% 6GURKDXC;m8RgƲk_|P'%f7٘~>&=A0`X)lALx/꛾x,a+Kf^݄B uǾ &0/EJwȱzU9Xp>F{ {G3x6 5%U8K졜psabJx-b8(OMc՗ҿoc)s&u̥Qy^R}m{6/&\C'ގ5\gz8FxڏEvVbx^66uem̹i"#;/3gU׭ ,vE]ƈ,9A`vW`6A<~TjV/=I^6"xUl.4ՑZZ2M8?ErlhC$b7%C\=x7tr;4b2dg}`|MvlSZ. LVmj\/b?A9J`자'vcȫ*N/'OK@9Ychs۔'g#jҳ'Ga2# l_^ (Q69/A &a3-h 1s C_тe~#?caE&T]qdcb=`XdA"rwCq{D^E3F}b=gkGklۑW♈.U:qtFwO9'=xdc-FNlbӌJ١ اvT?ڽ>gy>_ÍSIG2vrqW\Zأ56JUN \۪+/VYf `CG1Ehsy(inGpn5YpD1٤F1ƾi< O ޳s F&M?0ǫb,$QV PTWSܖ"q(xr9ۿc>}c{# $9,n_V1{m[Z_?ؾb0B2 mY!?CI6 /ZP!I7ϗן)F"E{!ls%n~\wCwm 羒G{h-9_~ׁ5< z}S𱝡}ñl}}x5g D j鞛M*bܴmwx.&aWIGVqxNq;3Cۑ0#-md#}2+7s\31Q}^N2  ڗ )hϥ/)R+Q$qsd\3#JD6!ßɾGcZFsށvoO>@"F{ Ž˺MS{ L^mr)xqsbbH£AKZ, JL-gW 8_.=wPۑ=ކ#\=Ɇǐ}ce#?MNS1g1CĞ(V+30O˄\,ʹ☴/?SEYmrHy~VqU#a ¿67F՝n^Yjj㟏pYYG_x!|;^T`\ZC7g;+rB4/gSyVؤ⬋0ǫ}q1{&mV3?Q קR~®'Bx<խ֞iQD~I6uom9]q}c"n8vVAʕh}Yo#*^SMomǎ a$+2D &/'g8ї2%U#>y vmΒhد {%J6ɂr{:ܗ춽OcXc:[֐-á|ƞȈBT< lz@s \ؾ|q{>`b#?*k_a셵x1|`\{ۭ<;/?_?gRGG!> OfY߳p^%.xzfPWK=|uuDh}ߏRPXDv6+ʽG}mG+kq"x߀}޷eAVĶej)*˾k]+r v$FFOhN1 %'c%Q*Is%"+&oMۥ.oI&7mS>Q`l|gᳲg '99Ls%7o_eO 'B&ph(nڶ(eFnYVeo3N/QS+Ic:-R,#Ge;M:63;W A_38BǶbpOeٱCO{i={;e`Ϣyp8Mc|9FgPD mޟۺ>>Q96nX7CĨ;}XF{eb>~M'rg Лכ[#sW}6Xasϱh-NҐp"(T䕶m_ 8N(nC3i!9[DZL~1 wY@sls1/+n/=`3'Ƨu "rw~fcgfMGAs2F` X9f6Ο z^.{n*>a@;6^-V%A ,Fk$F0P2$sp}|0QqW_lKoMGvPvʗT޷}B0|Aq\!7Ғ䖃 LF$]-IkVKx[m;X~o8Ǥbi. n ǂ$ -ՈNGÚDOPZʧXN")>sDn~`OEܦ:Цݓu-Sq|w|M= MG&q#M>|/IrHq;In+mS>Eŝ_sOIq;cI_Av/cum+^JU6mdbʸߣ>ML^9 +! lS?q}s$I.gI?bR&ɻH2ϷK}6ţEm)6dE Hq;cH'XF_-W46=;~`"s߱#-=QV։~6߶M4-ї'm1~'$I$I$I$I$IH!v$j:񲈐X:+ø#Lѓ|U_辚G[k&V3g*<$I$I$I$I$IRvT~h()ї6 =mYqaė,XGvضjRN$I$I$I$I,q~.ˤE/d7cySHq;I$I$I$I$IpNn?)n'I$I$I$I$IrRN$I$I$I$I$wOY\%/=Wu'QY$=^|sWoeI\#O^ooxEy^o "*$r\p|9rj smc=le{QToGN:!.w댕/xnYBIr8I}sw{|̓vq]WxA+'Er'Xk]J6e爸]^4eěZC |sNnTJ6vv{Ol_GCa{ﳟT[[(Ux>O>~)9K"LjWo;lVsqvQ/ߙez]rj9 ,j?rPt9eE"I(<)Ӄ=-|7S׏I_y(q}mlGGejgTsOaޜic}zlǵYmNOSC.1lQ ζiteAq×K9xɾ2qPnm\;k{}=9(n3*_#^'ͅf>*7Uܾt]0_s7Iv|_ *}"K&xC=OmEn)bŶElyGm|z/{k#n59*nDž]/-߿ޓ|7/e_IW8c!%WIHO `)xm*^ ַK4T2Q6v'"o56$DƢA?jxla#{7򣳭1(nz)Wo/M2{ KpM`*ϙmSdbDUN ]m>60Jk"Hi v]<6-(p[VR;0 qT>ozt~v^h1>Լl 5ս% \|WsVI{% #{.eؔ(q\ft糂}{..uSDԹŹ{֣sAso9ws;yYo륟me+0`vJpFE܎kd0N/Hr 6,k[bvFY\ }X♴~~Vm:`E #dsRd% \DIW=~I8 VE 5}I O7;Pq,\r% 6GUeKLDX+I~1^چD+L +f,5M~yKg}d}6 $/}/ψ|>0̎jpbMtN0iXүoⱄvl3/l;0-vHQ@n#I;헿Cc m' 3۱|qAzq=iV֚5ؿ.n1Kr͂b_' a=,^ڽW7LL El~:b]m,|u4J\і:K<_*}c/뽞ݝ2X s= 2X#51H_>}1uwU'\̗5/䌐5{e)ױj۩~侇bpC✪1sf]`CqbҮCavGӎ;Ӟc$7Ul.4ՑZZ2M8?ErlhC$b1B lzQb> j܄Ll&E5ڱI&oLԾ}M`e lS}Q'1/;ؽ#N>#WY|}$_dg^K)RPIllned&lC{5?0;$5$J$J%3 VkLf;t¥mE;>( \(&:"6FB%ŠMBNGm{Md=9EFl{Y vVȱ^9ۺq>;gtm_^6цÌYWg>9nDu`O2?9D\kg%^r%_8DmQڭrRV]^ }J'1uy_~^~siMt23&O~Xʾ> łA,9eܢv \Do2T[ċX a}el;۷\7'@g>}~q[2#*oƼ>_>[k+f15xiB=|*@U_Zlu{  bX?8G1FˀjTN4ɣxL9A Ey\i'|^D'G9pz}30(N z$q b|x5`,ѹz䕘g9h/ԑÜU#`N _υ ~1<{۶:_ r+yp[i׎_wz?/uHnk lDm2,0;9F4G04:2&Ֆ`3;8.>Ǜw/,q﷯jv}_7?;/:__yԾ"6gn[!Nb!2O QD%k@/+؄.Q Od%sd Lj:,Vb{?٬{4vU`x&*Y$n^6A_L2;ױp,Ⴞqe ^0/i$D*1]XƝ4`AmGx0,v,7 wo=V=crK -a[34#{R {Xd Q|s1dzK?0 (Xqu p y%͇F0>:e c,w`}5/|}k+#Zd:2j[wl؀IJ;_wڮ%>_AFKW /ᾤg}K ߷'ȗg}a)bhZ|{@&={}q"}FNU^Ϗ͉ kKgƻ0VEQ,ob >sl,~gwܟK=_vv -wy5̺{-? Xu;Mp=]hh~ XNY7,}Vsx/yO|ybo|dO7'5s8*ic2cF?Ňv ˟oۍK<;/?_"hߧh>0ZP7{=\cʻ{"y4dQ߇ ecwVm{ (>0GKh^۽&T6qh/im"/PymFs?Z|$mEϛD;`nǴv=G5J+n%WmRT}׺%x}Wkme8I"Lt NB>m8&JVD?KTn?J>e_0}#un.yNb|(k>'m]ͮOPfMްI$B1h2'?o_eO '>ϛM,aжikInUr8>2$"E"<ZpDRCXxd,l3cqu '2bʗe;ۊ=ڮig?ۺ!V@_=c!Di{{'/>>Q96nX7CĨ;}XF{eb>~M'rg Лכ[#sW}6Xasϱh-%օτ9̣jWڶ}qW,q!hڌmb7$r?s>3991:yyY$<\ Ktty׭zH?m3k;~_:n/'c=LLfpɼqصᵊ&Ҷ;I?HqZ?kV<2.u#%,䎄k5VX4bOU1WW6?:䎁IK.$ɵq+q*07Z}AVEVGw{{(HKaPb4ab'mPDY/2I;ѥ^Ҽ}Mlfe#yFa\_G"CY]$$I$I$I$I$93=Q"n j"9^+x>w;{w:L7c[=&nͮ7]BU l{~a!igpo,QWׄ׳AI$I$I$I$IrfⶸӾܶjkau"{AWkHkExNܹS,г5rz,n`ney[yWII$I$I$I$IrfVhz, [֊=[ܥZ4n}d)8K,͉ 8A}M9'l~.+ +6Gry|SHq;I$I$I$I$Ie۵@mAۑ`iF-6I}7 N YEFmv5ӿZXO3i~}x|_^>Թk0ۛnDeIxϹw*_޾z%Iro߾y|Cy?ݾy8R* >h~(8sRdr| |}K ئˋ ܔxs0~O9@>z9'`D7vO%;_~'r@W /ۣ!O- <'s% jn5vԿ򫷝v698(Ln}2e<+s8YX~(`y.D#S2~ҋli㿙=eO~O/jm>Eَ{Folcs_?>:(T;3?~Lf>eensb*raw[ b)X\e;*n/ʗ[*n_48Y"%uL!n)F>@yGeQ܊#'¹mɬOhm})R. D(*?~² f Y b!Fܞ%xNL\zOv c+ˡnC@MKTbe[g[ dvs-EQX$ί3S-ʜOq^>v!4cQ"^rcpv=X[/@"WDb$_)j8))P(0k$5K{n' ]5e2b#wyֺE0EP?}6?"]lp-CY̾m?uob={7eVt\ ŌSr=79,^qyWNJw ~2wbeE ( jQY׽v,ЊsQdfoޙ$ spohWř}t]YqJ~}u=y$qa׃~Km)Kxal%aGĉ'0.5rH {/A,%? l!HLFL;ݎ2'3F>?56$PDƢkI7}^ll ۯ^oo0՛K礠&>\f {sf`[/X0cgU+؟lWF+s Jk"Hi v]<6-(p[VR;0 qT>ozR6M/Me/Lw>+ܗg]ꨳ6sy}rͺavu~w;3g3~&-*1|s@_٩_os18h^luF禍;3;Wo, W#nc&n(=Z_s`ϻ/6(AV0 =/% &XEl]Njgq͢2/aq{g.YO酒^4 CCf!a>VDNr:UKU _N0}QBM_p2ᓜr Dz%W`hs4[($(J=|1^چ/%}f,5M~yP(jY{ci}|L8ٟ|`$: ދ56:DbMJb_̎mM8 0-PwHQ@n#I;헿Cc m' 3۱|qAzq=iV֚5ؿ.n1Bi ayXb~ Sk1_F*nu}Ki,3}+f.ϗJs@h;m߳y1Kzgw9 ֣ucO HzzWmhESRǫN^/kJ_`G!kʜSԯc~ SB0P8og̟2ܽ`t?;z>C'ގ5\w>c\lg%VNec#]7[kƜ,>2cXz9^7J=Bmu?^ Y,n6Y*DQ^' #W'U2ɳ _|O3hOj}5r-862ю:ط)1OFԤgOpdF-Aƿ&Q>&lr,_$(Lf;Zrh[ABc(&:"6FB%ŠMBNGm{Md=9EFl{Y vVȱ^;m|QT\]'ElRwݓmCT<21KXsJ~6LioP} 扄k}zn;xyuc ʼohϯAToz$ZC͵vU?\ɗ9Gklvssm.tb\bƨsk5g~cy/ )izEggɳcI bd<3}x\@4/g/M.6~ .&ag$ܞV>E̕(ވke]"~P QdGAKsr~qO>`D`Y 0 ,D/g5l e'3OܶwhAY&3' N#J62AMr$L.8Y ZQ"qZr©MTrX 93M <˼]ֹI~Lcx2e,<{5}֧%2~n P {*6˫"*6ρ?'?_inQ;\b R މ@LP- E,26}A[Bbݳ%mdNW[klmX&e?18Ef {Uė#[螏IryH1Ǒ=^h/vSP9$12i#L2\3߷:_:ur~OlNrf`nQ^;?QPI kbr啘g9EkNݍU#`N d04A-߻߶׉bs_ɣ-)Cu`M:Bd=v>)оX>eƳys5t&g1nڶ;e<0sY9_}D*Ps6qg$.Ю}ݐ6\"koNIɣ9)?s5q+y?4=5q {&Jր^"W I% IF\'Ǔ JVl!()}A4#"Q&lN=2̵׷}|7K( ،^mkfrnKYfS//5ĘC`_Ҫ/tITbj=@};ail+ڎ6abYXo-z{Ǿ b[/8|u"D".^!bg-R]aQűnR@9LJ"\gwa|Nu|U"^YT=ox~Cݞ_&,bQ|ybDN/j[C{s]9d.a1PW0+WZ-!$_L{.wVC{>,< D>B/F*0.aƳX9!3穼?qEl lRqMgc$w/V1Egk#^PFږuLj>)6`g\7HܞFK>85ݤ֞ƲdCɊL>5B/:Q€ɋs xCe%lkIfȠmeu>zc0ǭ@۵38HdhMy%=mX2V5dKcs _'l#EX-^u$gctڞ2/.>_D^؈i}Wc91{a`?h6d%|˚xAֽu.ۯq7O.辸=뢸=_Yn,gRϗm Fx^ nhB0|ݎ|elS'"\j%;ڵ7=v|;';Akt C¸K(NL17vSrjR/ic2_61ێ[dm8tft鼈?gRg鹑?[{i=*{AllY]קkxYO8mkss3a}S]BXJq^q+Qh'HlX&֥/<"_k/ nɿL9ka;oZNP}2FIs%"+&oMۥ.oI&7mS>Q`l|gᳲg '99Ls%7o_eO 'B&ph(nڶ(eFnYVeo3N/QS+Ic:-R,#G)e;G:63;WP"C*|lQ6㱭Xzvv^ZϞ2Zkea,:gʋ0Ľ%Bq{'nvc:1#F:3|τ_ɼf{U.1vgXG\s};Z 4d&\3k?} b`<1xV1ypTvsSٴ ZܑAPw67IwCn(%-LF$ɵq+%`^Ov-ڝD, ]cRwׄcPO$IMG&q#M>|/IrHq;In+mS>Eŝ_sOIq;I$9IOwAo|_2I`$7}oI.3~ R?>M@QEgE/hQ/RN$IpXN$I$I$I$If{ڶ{ ˒䮒v$I$I$I$Ibo:IrIq;I$I$I$I$Inv_U'v3q~}r82%pC0$I$I$I$I$9#$Nx' $>zgMڸ~5=#5RN$I$I$I$I+Uqija6wR}{Rܾ:R^#$I$I$I$I$BzgK":" <9_Hq{ϞoG߃:w?yzb{,I/y9NWò$I'?C"޼ ߿Wyf?QMf{]g9_n8NyhD9s9ױ?_=(* ۣe_~TamQZuAƌK<7ᬸw!@YD??"F~8`y̋ro~&R>?'?>o߾y|Cy?ݾy8R* >h~(8sRdr| |}K ئˋ xs0~OKQ=M>|SΗn6U?l}jke{/v;gI䳂Cm羝j;.J;[LKNͳ=#uE퇲_.'H$;0%'%z=f=?}\#?e;͕=hlS̜*vc#lכ3Oom}2͉y{|ȅ3m<is|CbᐋQq;|Y乜CLʗl梢Kl9&ҍ9\;|Aqx| ~}K1gr~N.7?ў[妊? Kq~& qUkʗ3A}6qVq;/b?پXץ}EۏBXV6"b^"]< D6T=õg~3LzOv c+ˡnCf=KTObe[g[Q䈒vs-EQX$Sί3S-ʜOq^>_TI^vT5 sk%ƪs8x~ & | qRRb Q`(Hk7npNVkdl F\u`:j~鏭m{!D8U;'ؾZ ֳ̙}a.5p8?)nX)-\_ƭ!Fs}ùa+es$˕*zLo*)n_U4i:@B[ۿ Eջ$n[Qǭ Kek3UZsdF~tu}6W7ExIf`sRP{ . =W9uwL,1Գ* bQyOq۹F~-Z) cڮ>ַE`WbPC>ۧM²]5G>]ZqW%3yArEe JxE-6О%ܾ\䅊9G:nbBBsm6qHdz^2/a[3[ޯmOzyI|Ecܷ{ƀZV@C1\egxqΦgØJ8F$>gJQr>R6M/MeXܝ >e0Ǝ~9y)C]]3D6sy(wztνp=/.m+k>v|Yyzg@yޙlǬg_4QL^os˂H\)}KtX UeQ܋wUgO ˈ6%n )ܹ5ZmY,"zfQA3i 'tЋZFc%+8&!.$l_%1$Z%% '>*@űlrD07o3V%-1aq$uxi_01$n:6A/%f77x˾W|r²?#&3; Iދ56:1äbMJR رͼ`_ĢC P۱"EE '!BPܶ_96^1J J nǂym7k֧[YkG`q[q/8;{rCqxqĔZĆqQn㟊|-ƪ/٥]RLp_늙K?mұNl^L<6y*@= p>ka^mhES1'Q]~tbb6(}%g+sNQW/DN X/υχ"9U{?ssf]1aCqbҮCmQc n˾u)tbJgA욝8ka\i6ZMi8/nc_v ^$IqZ3*TOگiͳ#ǰ"e g\i#m񵴬e$@wup~ئ#nцH>(nc`@ x$>'}>D > !Lj"caL(Z2}q$Aئ.3N bJ_v^%A{INH0)vq}4W6Yٮג@{ʼT;[DoER6cY'+}u'PlDM<Նџ!e ?d@`+ Br~VcȶPv?Cmk~˃~?O!F/: vDɦ@&UId€ 'dW;JL$+P(G"0ɢ>fP㜥wd9ƓI6~.cI?߯|>-񰉙NfFB="]KٷڧX0(6ρ?'?3[N$mBwCw"ԟjxK#vG_wh,g/nK&cpX֘gvzZ_gkml{ƽޙ5xy/PLx0VbbdyH1>1z^P{rIcdnGΙefo//J]sd Lj:,V:/!lN=;H*0^ gHXPgn^_`^8oII ΅8W!nZ*am.S//uVƣ:O䀹1iQD~I6uom9]q}c"n8vVAʕh}mHDG85ݤ֞Ʋ<6>$6F谍^Wv`>ϗp>yĕm- xA̻g6Vol$;mΒhد {%L+?A_p_ҳ>%ck`l[`K볾a x-= lz@j{\ؾ|q{>`b#?*k_a셵nA%jtX&~z c ^ž^΢{~k*|ccs̱.۳/rH.|ٙ-жPQnD0V,cW6u"v^]{{3`Ƿsc;Agݰoa/RM;WGf~^X=vOcx{"y4dQ߇ ecwVm{ (>0GO1!Tk{Xq;üK_nr4Ff ս9H(_ $ o,_WKQY]җ M_믵ݗg_&3ә: 78 iђr&JVD?)I0i\E7RI $&gNsҶ)(zUm^ Dr,;ÉF{{/S\p kQp$2 m[OVe˿,1S+Ib:-R,#G)e;G:63;WP"C*2Q6㱭Xzvv^ZϞ2Zkeaa{~Juqi{7Di{{'/>>Q96nX7CĨ;}XF{eb>~M'rg Лכ[#sW}6Xasϱh-%օ8:\{Wgb4uNm697$r?s>399yyY$<۟u^0v_{qxsP>vn}LCt ߭w3k;~_:n/fhM"J{ 30qy㰱kkM'mw~~Bݵ>X<cBZz LZ-nϸ*zQq@]{sQFb^4FON'n kuP%b\]-IkVK|U`n}v'>s: ~Q\,ayFpپB=,sV|D6H1Zϥز>P5e_Hq$nsb~ $-~ W<_T䆑v2V۞ç}K#`ES鉇=R"^ඉnGrr-|]fQkonr·b[+nt9_ $8?KPt!&enˌ_*o:mGGGUr2O8ONr[mIn=NȯG*}lp~ŋFmv$I$I$I$I$1ݯdO^"CIoMK"$I$I$I$I$yGM6?IRN$I$I$I$Iƃ__v$I$I$I$I$IrHq;I$I$I$I$I$u$I$I$I$I$I:_<{ /m5 oO$Iz;ǯno_=˒$F x23"._Ge7ETvI|8sԾA;z_z|,nQ./Oۃ?Gu5'({68~#%&@>}$In>'?>o߾y|Cy?ݾy8R* >h~(8sRdr| |}K ئˋ ܔxs0~O9@>z9'`D7vO%;_~'r@W /ۣ!O- <'s% jn5vԿ򫷝v698(Ln}2e<+s8YX~(`y.D#S2~ҋli㿙=eO~O/jm>Eَ{Folcs_?>:(T;3?~Lf>eensb*ramcZ\Zѹ7/kQq;|Q乜CL \T|i8zD(y;k{}=9(n3Ǐ 5|ݟgrC;SNqVq;/b?پXץ}EۏBXV6"b^"]< D6Ta?ڈ۳I\/i{dLvd9!)ItjrBll+>0Rne( \?z}~ñE)K٧.$;z,JDKpl|ݮ'g# Hd*HL 2%W-@\'%  Fqd&q~iM$a+xLƶXl.0Z^&?}G ^sP~h=˜ZmNlqnڭ!Fs}ùA3JsC㛘csI^9͗+)n`ϮC<,5lt+SN6r&qw|!]"rzV.mcpwO q\`Rr=),w~^dX4[Ieq"BKb ž $Rc۠jTC*(sB;l(ScCOa,jDyz~㱑ڇ}^ȏζƠ_^4 1xN j/ae6,n}}LE?c?I6Pҋa|,$ʁIWKv@%m_5%%% '>)\0p,\r% 6GURK YIv/Kpόe׾&qӱO4?Je_>#{o1-}I'33"LzDGa8{&RF'^ VW7}XV쫗ٱͼ g!})*M`? zwȱzUVrq;#n^#H/g>Z={J{F a=SW7LL El~:b^m,|u4J\і:K<_*}c/뽞ݝ2X s= 2X#an^a*=Oi_ |fsݝ0Gu+K8˗5/32u츼o}he>F>T;/S?ݛ vl1",NÝ`p{;p=V|v8c(sƳX9!tlٯesnڳHˌec-{ڶw/}&nWTGkiYH**7q˱MG. }Pܮߔs9m$PdHd$0\$"/n_XԸ|_~$rJ%.-D^T$&p~U?g|r̠=e^]D;`ߦ >g>Q=9?aZDɱDl0 hmE; &cr,  +:6 #c.v"E7$"w<GqeYT>c7 3zzع[q#{!ǹ KstO{>Qw[,b+272C5''!O70~Vՙ!{'|*}<?Qy{ke6Y|-\Zأ56JU|nmLPל>E, [u)inGp5YpD1Q_xF1ƾi< O ޳s F&M?0ǫ޽mk#޻yCD{-[h~^@zw~}w XbQHf!Z~M?1dg(;IƟ|嶵uvC 219'x8hNtQ) l^e' dW;J$NK.P|0NKnϢLk/ Ï%h{DN|9Ko86'i{m\ƒ~׻_sg}Z"#_<_ 1 u9l*2iYs:(&~)Pڢ ^R+cݾ%$0==KQ2N}_ܖLᰊ[1|x&ڊz{̜=ܵ {Uė#[螏IryH1Ǒ=^h/vSP9$12i#L2\3߷:_:ur~OlNrf`nQ^;?QPI kNǛ{.zy%+3ň^5lcs 7?a;Da!?۶:_ r+y4a߱sg:&g};cv Dz)4'+{n68qӶ)㹘|9^5U5p׿8m.L Huܿ+xj(x8?G咁h~Ul6 >I\?a]2!m%Ej%*ߜ$nG+rlS~6j$V&hzLD`6<#Q@'#P%+6Aپ aLΑsx(^6k'ky>aL%JlF/6{O5l3y9ɥ,{jytb̉! ^0/i$D*1]XƝ4`AmGx0,v,7Nwo=V=crK 3FƮYm+ݓ"fŊ0~x%cP⛋9>XJuunhGaǺ}K=_|T0m7+h>6r=݅9U-_{gQxIVu{~E9WW#rzVۢRރZ!wi*sa0+WZYxpYY2dYP5|;^T`\ZC7g;+rB4/gSyVؤ⬋0E=^"n2zhwsNb{M>˧UF'ڈQt]12u8Ym*W gKJ85$֞ƲdCɊL>5B/:Q€ɋs xCe%lkIfȠmeu>zc0ǭ@۵38HdhMy%=mX2V5dKcs _'l#EX-^u$gctڞ2/.>_D^؈i}Wc91{a`?qabaqt_ طesu~]žol}q{96Eq{;Xɥ/;S;ʍf=і`ئNDԮKvkoo4?zvNv,p'>+87ɗן)n[NMtsR3W]; blW}b|ێ}S-6`3O {Yzn$G!>cllY]קkxYO8mkss3a}S]BX>e~neE=q{ol/R~78(_ $ o,_WKQY]җ M_믵ݗg_&50P|7|pB-'@I(h>dstQ"R.{bFܴ]:v!d P~I|N6S]˗z&>+ |pӞ<:Q}kQp"a7H6m_~Qf\QnU68>24"E"<ZpDRCXxd,l3cqu '2b/e;ۊ=ڮig?ۺ!V@_G!|Ҝ/a 0~߫?u}|?slֱNoQv1,бԟ1n߷}&4N77kglG̕l6vA޷=z=璟cZ(!3r?DT=+}^P,qBq꘱MZg/inqm}.cw%:czR lύ`=0w>3k?} b`<1xV1ypTvsSٴ ZܑAPw6f߅$q";ؗ+n=)H wsPHK[&<+tE$I[/zbn $bQg_& E4)_6]ݷD1G.R>5ۜXo)l_|QMFIrX~o{.g.${J;@/Y:qAW>0,Oqq=w?|L$ɍs!&enˌ_*|+nS<>PԆ?mblYKv'Z wi'B"ѯg_Q"nsG|I$I$I$I$I޾UC?-r/W>/'$I$I$I$I$LX#R>*`B?Kbb־sۖ]=?ImRN$I$I$I$I+@RPElD ^+M+Yo,hG_q^RN$I$I$I$IKZEȑ\"+uJmDDZ'WEI$I$I$I$IrɠhkY@~"2w[ܖcXq*/⥴QZv$I$I$I$I\"+|;{;)1llhA`#7Y+ 6T$v3ġr%+}aH(|<]URU]2՞#Uꚿ$AkY<ޔƜ6> ׵bw=VۯXgt͸['{0rjsŇgv2googf8wȓ3+ҸGnzrf38j%v܁߰lr|>].w/ɯVO~Y|\k\.''g[Ƿ]ہ[[WջPcb\o "no0@HܶSz$TsgI$d-_ǧ8hf#q'G!?Ο/o˹$kJxs&K/d;J20q;hIwJg?pLԛ]T,׏trN{nxQ~YN{I_8.w=g!ƶ싸־'ەb΁/*ņnk]^^̚"]6zM"ܨx9y/v ϙ?7O)燧د5g7hy"1.sJ='ڻlu%kH9ȣKö"g[ߩm\v!KJ[l&]>ByǪ8Ь+|Pf< q}5_mj=Ͳ{W oz.+r+ӜZD趸 "vj|,_ē؝%A{x8Nm<AT~ױ:fEo&, q;"7[/HpB熈VƗ&hdJmj^ΰhhXv!zrB,[EԎ?m`>8]s( b*ucqjñY[SU,~ѩcˎ(c8wVkn܇T|@!_Ctf$U!s6* A,dB! rLr?"p+,7Pomj=xFW\ܾgW0'n"n*n/̋Do]: 'KZ1Gj9M@s3} ϝG$1Ľ q%}yFis ;hjY JTf;/J{m /ٯh{E+S߮fIX1s++{Jokg~ ȧx]T\P׾ {yõ;^# kuX`9 zԾm>6I> qñ)N4zRmt~sx^³!< p]DD*r䳘 # 3,ҳ hW\lTzl5/y.l[m"}\m+n/^+yC@݈ܶەmW ғiO97t6;K̝IA3Wl?'Y7RP9qe:둁kct֣_kq񜘔]+X13]NU[6z9< ?bu_ל<ݞױ`u;]ka$ ?Sl pr,h P 4ފ5co.v"Qmu $"7{Dqy>g`i-\c̵\k#r6}Cd K|3su"򑺾s[rxU1U.y:Qozvy"1]7qKjrEM ԵUgXҨ5j ޠlj?c\]üz~4bgG ZVgoBo_Y1|ǀ-y p Iv9/1s>م-{q{A,NbQHNB4rcohHƿ}utA)?.!Z/5MVXqUl2xQEv^0` ض 5JDqHUQ̬"*H*"lb!9xE>Vqc8>YRxL3B~ċt,,mbA~F9kf$(&~]S}Qb)o m|-݋ۜ?8̢l5N3E;_e-chLzq7_; `*/%Gks]1rX;ys)r7~g'"jAecd/L\v-OWD_3}[vfյúƺuqx1pU`.Yx}g{T,[wૢ+ka"pz|x뾁tLMw ޹oz^zMؙQ߽Ę'XW97\FTaW?10kx$?*D_lc_~#;Q} ?Я|]6mqZ5m~rZ">G7q]Md\0/DmJA`_ XZ\DljQ?ՈUn{&f5ptǙy~}Usu|<O%XMr`ͪ!9ۨϓ_5+0y[5FH5/Ũ]@k5VsVGu1cg\[kmR1qyn˕h~s1QIA28>[q<("?xAjNy%*0&6g]{$nכTVQJjo9T 4>Ncf/l1BRgmEVD39UV }cc}PEikN=8(Vla_*a\5?O~QϼKsޏݲj<k^:7].w/ɯVO~Y|\k\.'' t+3remk7~}qg#q'G!?Ο/o˹$kJxs&K/d;J20q;hMwJg?qa$]7#ǯ隗(-x>k῰= q\z+f17eCmq}O+Ŝ_T &׺zYPubT ":޷- |JJE:vQ`m<>k c*8vXN1}7QDXT#]+a/f=྿%>P("h/T~mب( (0c$1'IjST ֠X&0|mL{}1.%d1ƣ)NBxXT~Y?hp̅UbN5qlx)yUYȕ,M܊}yY;HqrK=.S̈5V _Ղԕz9#u{pm }gmqۭ ,G;_(H qtyH׬?9e88o}Dp˜|%$l` $DVEj%H0<뱔oп AbpcUb*!D]Q?1*JPXwT1c#ن\_+o͘Aqrե*f$"%,|R.ك*g+;LI0c{  cżOy빍zE%}jY JTf;/J{m /ٯh{E+S߮fIX1s+Ky/k"Z9 ,?'^ʵ0*WD?I͗϶W}=< lzʆ69+[sF̎^c墉ZWWZ 1,S|Xs>K@>"V?=5rts>3Եo^klsr^"p-׈2Ă?׬o\bc.bhҎY'5~ ό9}6Wwe[XϚ[5}SGlUqW?5.r9QIENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_log2.png000066400000000000000000003710721471400134000343560ustar00rootroot00000000000000PNG  IHDRER^sRGBgAMA a pHYsodIDATx^=^I?00 V9\U7f 1TBQ 25 Y۴S#DY7j U *-|'"=\/<u//ۥW?|ɨ@_lo޽^?ʒ$Ix,I[wۻʛF;GM'{kzQ~ׁ}no;~(On# _:lk$I$IURI$I$I$I$I$I$IȄT$I$I$I$I$I$IrU2!$I$I$I$I$I$I\LH%I$I$I$I$I$I$W%RI$I$I$I$I$I$UɄT$I$I$I$I$I$IrUW?|ɨ@_lo޽^?ʒ$x]ΓWoeI"O_oo߁*yiԉh.__oO *_nAoNco9U%@16Ƴ_=Uf_Vۣ/Zu~:ړ+{ex~羞c$I=j >l߽jS=}pOF3.&ZZ|63÷\/tehHBD遭C9l]CJ=y{u n^ v}G}s{Iٿl} {WL&{Bƹ]f7!mgvպ7JgW IC,cO1Wy z9`tAՂH)L'yy=g~/=wmxW|+Rd:wptc>udc~c|siL|Y/,d&{䋏Xy9o1Bf{tj2V}]ĥ.Ez姯ܗK$@L̹IH%%.R|un{ܙ3/`B~D ղeL9!uڹԞ|i./^ʄT$Ign찅Pf/Xe{I Dh9s^`ҭ&>}|?#8(L@~~6C;9aۤLfr!wx[傂]Lm~¾˱XmrcF6yK x 7``I/J/ Ş3&璐x}\߇#)Q!nBȚ2i[ $׭]6{CNH;} d9Aʆ4uhp<! 95Yr6{u=缕xtd 0_a)ܣ9>ӡ#G0Lإg}m6C?bo{]&Mݍqv>ofD|@.繰l g}ܷm1ױmdh_c#O$IRB @-ON@Ϙ$"["$.T3YM2*@r|pj dZ #C=EZ9`D RV4ZL`"76!@=5 lKM^ D~V߳f\VJU5۷ChK}Am;~@Igb`@GX/Zq_Li#t~`bUF Zà tZAQ Y-k ]ٶ /HTҎ~( 'ݼL߰ 9ֶN#LH*""@BN]&!d"8g c}NB?/pwHL_rk\_5!| >oϤkn-}f_wD_/|ԥK۝=Wn?vY'}= k6lieF}-luLuw3*%wse=+M5{ď3#I򮿴H``=,23,[nD}>t}ZoWe 9fjk}z۞>>+ΚCWHsO1wі/ckvH_j%>$I$INFR=ATIMHo%XVFA#uJB '!UuGTāCԏo=C% "wP8 `^pf Ӄ!|VjډnE9RK pm/Q#YI=Z}DVCXw>x)>P%>9Ah 6ig6T )' . [k2Ϥ<۳ A׵\\y6a+AXh9<0&M&XFrp3#mCe:&5[I9$"jB 2|F$ ijsrǺO_ki._]]ķKBȽ$b>AC?c𥖯)Em? _tCodEr~"ieJ5@/?Ssu}ն>V|i}QZ郅s]yyǴ*YX"}B{/n:s{K}Rdzқdv m6 h>kl_>gHIJo9bNPjk\I$I$>|O q LTAR3O!%u=wqhjqAκ ДiЭ+@$,4dNkȶG'`,6fО ^s7OzVel>S *l[osmڡ``6A ,e=y .{9бʨYۧutWfǙ_z`]MȘ IRPw1E "6m}{ƕM) ՟4RsɺVB2iL"d1n㳞*kbjDݯoe%wg k#G]ItÇ>Gcx{ Ѓ ;MQ?F&> i3W| '}εAXsƱ#:"xg~\yy}N5_$g~t ϬtQgB#Xٛ]hoNcE@r֔qe/b l6ϩ-f~Cח07khתZh;J'.*h}L$Itn4!ɚ%!ӑ74 U~hb%ʷ;)!UP??}<ԟ;vIM።/1| +"q@ALƁORrL N*F6Ӂجa5fi뫠g1vTv6'[~6i6 El2k5|gr:P&c:E~ LceN׉ &LIHMTI͚g n77E۠Xw lɎm2|+I6{& P9h6|iP̏2U>Lߛ+ VPzW}ʱU/_h{fV3o?ig g1e+sAuΐDy i}c"T7-UrF%;t;z~@Vucg&=i}Q5 +D7Rx{Ƴu׬퉏W:~vu 1I$Iӹg="T$&dy uJB ZgOBrhC U9x{nQ jxvFt)W"se2oA AdUVt܈DZ6~}G_kgfe>Cqq YPߐ .u)>Ȏfhٜc>[KF![F=1I"@2$G$fm7f̓Q'IH-LtDi:NTja ?oc Ӗ.R6c{tZ43։3w|o>G:o_>{z\uٙM٨k ƭ?#ﶹ`56_i[Ա0ǣ7-zZ_eed[2{L6g,.·yy}N5ll.>r>TҶ*C qgo_О!lpUK0ڎ'W|{pm#4y_um ώoz="˒X۞x(3YgyA^$I$WRߒ1A҈y><7VXFDATv[q5hiZ[Q\HHهPz,?7x-Df\iEC%:M=׍˷ouhle^.'R6x6׍TGMxfXO\YجQ:Ͻ zqTHפl(52apݬ[ 4uެxF `[rBB5YPIL,hɉr2m'D(I80d9IZݶLxI#9^=ٶKHqf3ki3}bgk~~ZK;={랭}寈z}E~ɗ{ziv/Lֲc2l8[_oJo^3UjYƺl|gK~{8`m6/wfj_3L?W]/=0R}"-[[xL-۔>L(y>~1}$I$Irqn6!!p0!u)f\%{ G,܋$In{.eMg{xt_N,5z=ЁDj")f3L8Y$I$IVɄԥT ߺL{MLFy$wLH%=c%Ys:ɔSm $I$IR៵{>Iv~& 'gpM(YSj0tInGTLd3TJ$I$\%!$I$I$I$I$I$I$LH%I$I$I$I$I$I$W%RI$I$I$I$I$I$UɄT$I$I$I$I$I$IrU2!$I$I$I$I$I$I\LH%I$I$I$I$I$I$W* o۟:zS,I?l߽jS=}pOF3.&ZZ|63÷\/tehHB^Ѡ.!`FA_'_kIY﷏;` =ϤLF9? :`ۥ.LlVa}2Xۖ|fW{tf}Qz/ߙ=d2>*sŚ\ `롮I]%?IrtonOYߋ?exohG2}G>(3Ν<~-e80{O_\?&S&}*6}j<#Vr<[̵xkϔuf{ _r CS+sn(j qc8+we.>LH5'MH5/d\Gb;ωp7LH%I${͍'4Ϸo CNzD&m?dKH501EO@mz$}o繑x:m?tg6;Talr$zݐc"`|. G}<%&))%YMR@?{ݚEms'_ PlAk<`>G֑^P_(|MhL}ꃙ4k'|қ>J`Eu%wهXc{/m90 m_PaxB̰ك1y+g\jƜ=KЯz1N7rD?sն1?;޵AZ"jݘ˶+:~\NIDž>bק3h]0gW:t_k/{3=o:ўS(yYa#6M$IDsK )HO<::?c|n x^PRoS(g5tRB\8A AX ա>d)jHy)~=ܷ7̉lt~:R-6y5X0:[ lV>\X+yV;fV}l -9=8MGݏXQYC?i#,Y`k}`҂}@w-0kU@T"m~/ RNGMon^r&LHo؄k['K&TgT\ !e'.RQ_DZOk|Ƴ>'!ї68YSL_rk\_5!|˫>oϤ~Uҗ s}TkQ+/GowR_ mŏۙg`<:Pq.H0Ft.óy{mZE`3yݑ>?+M5{ď3#I򮿴9zUϗp|.9bK7C. 6D{mP}1 Wg@c/ntn^~v]u@h6}xW~D7n{NꢜSe4AKlAMSԯF `P/c`[ncyouP(6M"T -Z3R'~{&.10z] d|^Vr?&,#ő!Mht$؜sR5iU># 怇kLHcw'{ǓzI%ɋw/|oO+jY2+eE&+c1L(Sڭz~9z3 A@3VO2_?LJ,Ykg,ʕr&p}Tov&ٜ;-VYᢍG,d@]whڦ)^{'R8mWQszq_7D5up?թ3O{WMOI$Itn'!~HL`%7Ϭ 3Ÿ| )8pуpH.4ۂ4y;VHs ~[4`5d[0t<uvfО ^s gUL3 ¶yehε-j>ރs+g4AL6K03kH:?zVtt4+}KU0.G&d$S(ػ" [i=ʦOKdJHQ&mIUl6m|Ce~x\~`U\Uͷ xG0}@jt<>}b0h?2.,3mlTؙhr12yo`Ks̴l}wk3W| '}εAXsƱ#:aθW6bqc,n=xx<={rڋhS_ Ϥ>.AS˕$I$7dO?HH{P*?WY4A[Wߝ* >ws;𤇦<9üwH>3d% HEC{ d@) KʖANJͬ`I6kmX|*Ā\~c}$X&*ܤYfR۳LHy7GpHHʼnmPxkdǶ[LHMp wMA|+LA9ڟ _Zrß.ԣ2cu8Fʂ՟26t<ޕxrl,^U@L~|eY:s>i>LOΟ]7 I̯a i}[cjcJˆ`V(cdNyGso Cs̹lBDžϋ1:LO]Pl\v ;#B:Q|&j fzP=;*d$I$I"n>!YO@aH8U0II%Y^>vD"em„ٓP%PBJ0| UG9TK[sk>n`Cu`F빃ul g*+`E~n@tj{fmymо#G/뵳doC@dvBbz6(\28,mQ_zcв9E}(B귌zb?DdTIHAo>͘'OZ舉ed+YuyڷbXjR\TEԧ5 Ⅼ[uEL*ZDWOi[ Ƿ#,\1޲ __V;!lpUKpf ^Gdsf;OcLezgb^ۘBwtϗ}`g2or٥]UN$Is oIY i<| +,#b *-߸m-( $vpA? |Ё:!r:M tr) &Pk@e#o\/r8yn̦r<*,0e*}.DΕ2sN(0/Z0ЂWIِi$ f#LH\7+j[ WLJr7+cE;^jQCMx =%#Zrbjm 5J YNcm-8^HxOmR\ Z|_]V "_;B0bB랭IBz}E~ɗ{ziv/Lֲc2l8[_oJo^3UjYƺl|O$$`B٘rgC&J֔2\q?=[q*O LH%I$I\$$I$I$I$I$I$I $I$I$I$I$I$IdB*I$I$I$I$I$I$*J$I$I$I$I$I$IJ&$I$I$I$I$I$I $I$I$I$I$I$I\%!|[yxQ}OO݁>޼{~j%Iwۻ\;'no_=1˒$E޾UxҨ]7ۻޞXewUv%.܂ް2J >rK%cmgÿs{oۿ"?m:./G_$?t?.'!Vd{3}=$I${R??} ?ؾ{çY{lxZny'}矌O?g]L: mgo^,$:ΑT=[9ȵrj>|z4` /3#,,h?>,LDsTe)*oBkےuoά;J; YZbX0 l=r08邪'S.N ~p{?n*^Q/{~C?~>WlGɞ-tk)Á}Ⱦ1Z7^VYLSs5bDlꇁĥ.Ez姯ܗK$@L̹IH%%.RΖֹ->rg-8bl&OM]-{SľfODA$*I4 {HBL ϘKBq}>DII !kʤm &xV^&bw.~ۜ":!U^~J/)>֡}n$9s6~xm 8i]^{y>ڰݝ6hwf}('gC틄gɹrz^dB}ÎOypdB*I$I+s )&ROT'R5R6Z >& -() _@߼ˇvѽrآ#*r8{|F%t] %02qaz BLz>N\E6i4`fmC@Vyאc)lm#]G3_ˎJ7f1zz{S'l_ )Xc0O*l-U+X`]y&먄H @*cELw/o(,e.Jx@'yDbA'̱ɖ`$Hx.YEG[ߏz n$PфiFj?q,|`)Mq _{&b<~QD]#`z| HxBs'k쳐>3lz2y+g\|;RƜ=KЯz1N9_? 4Ͼ VVbF=b`Ի֡1HM̍O[C]U"Cqlr6>υfWX}K/ަ}}q{Ff06m$I$I.-% <<TI"%yMBNHyN*?$I r T.vZ@ʁ2Y=CR:h5j DV olC ԸaȠ @7l/Y= nFeYsk_ mhK}Am`TzTOH>m?zʏ:`N#b4ZPb`b\V_jm]Kd ȶE`]xA"-%vD@H?%g„MȱuaBJ%}VIRv2 )%Ig<[sR?X>}VSz7O1?vU|ЍחLˠ]^~^q8zgjm/~g<с:Ӷ'vAP6cp;ޣިmӲ.ɳMY\x^n:@zfsЗ9I޶] RRϾe>"gX뵝K7C9ާ#Zscb|,Glc/iϊP+>CS]xs;җmg1I$IӹфTOdRCR[I>8DHvr|IHUQ+t0!/qzAPI.u `d3ώ `>d`d{ID7n{N#ReeS^Fl!zV2b>|, -R6} lKY},#5eئI=YDv^H9^pAE/XZ~&Eɠߞŗ]ʳA [ B[I(5A`%o22 Qi,k1O7hJ9'!QJ|Y3"OkxȾV˄;֭~@pewMF.Y !}aHԏ}×Z,| տ1cmzHo)=D{LjLwy֙ˆ T XuG&k73j;gq?W>4ч+tz3)\W^1ms6zuž]hnG:WzYӳuL5:޴~ Kh ] ~_%ګϚCۗY1h)eR{6x'Ե+&c$I$s; O?~H@%pd2?-yD!U~fmw줄 oHI]gqZkp.4%h{V>be<.sp}AE X`pˁUFu:'>7=$hBLL:"I,hɰQo3lJI4KeRIfp!qP_c,>?g=Wc"f֕5x>UYĿ 9ZϴOk8>9ڏ/-3mlTؙhr12yo`Ks̴l8uTOdy gp< uUǚ7Ű7с/U3?u_E}͑s%9 gzrE}LmYyg TaYbJpq[Xvо]Pwaq/b l6ϩLu-f~Cח07khתZh;J'.*h}L$Itn4!ɚ%!ӑ74 U~hb%ʷ;)!UP??}<ԟ;vIM።/1| +"q@ALƁOR|*%(Ao ktc6kmXA|*Lߛ+ VPzW}ʱU/_h{fV3o?ig g1e+sAGvݬ3$}?3c3װc"hCK0+gZ 7tMY: l6Aw^KYsK~/g<[['~ڞxglG^@$I$9OH|Pyy,NLRaRIϽB{[$0z$T)g;Z?Pw;XB%Xֶhi}FպAf A:[P6~Y3EɀO~n@tj{fmymо#G/뵳doCƸ8@ oHGẔAWi{dGm3lN1A-up%-$p U#GwPwO3ɨF$&:bG4JV{޿mn*0փچw+ʟ Ľp~[z:|_ds~9Y~{;/{ToץȶF_1nyJۢ9彡oc|Z+s@/$*1cgܴܰY<;gqY?F꿜j]L]qϤgҶ*Cg Ն _W[9~گܣ3鯹yyڜϵ"k1g.C[<7VXFDATv[q5hiZ[Q\HHهPz,?7x-DfyhEC%:*V۞[J:4BA q e3ٍ>qݘMxTx耊m6^n@DΕ%ܻGuT޾zMʆLN nAP# ~Z`}mASʘli%'$4P$}iɈ,'AvBdCXem4%ޓm({i<m&063.+&,K;={랭}寈z}E~ɗ{ziv/Lֲc2l8[_oJo^3UjYƺl|,RI$I$w%?kg9/}$9Ln'ROܗ <$2Q?~a>*g?H"?(ߦʄT$I$* $I$I$I$I$I$I$idB*I$I$I$I$I$I$*J$I$I$I$I_?lۿmb7V%~~ۯ~eֻ-~|tϴ?:>q#i%RI$I$I$I$IrpqݒAr'%RgO6eB*PȄT$I$I$I$I#hb]v?4? Sٶ1KHe-Q طou2!|\%!|[yxQ}OO݁>޼{~j%Iwۻ\;'no_=1˒$E޾UxҨ]7ۻޞXewUv%.܂ް2J >rK%cmgÿs{oۿ"?m:./G_$/YC}gmo@=8s {rOHgG(e:NKeBceqamYԵ Cj >l߽jS=}pOF3.&ZZ|63÷\/tehHB^Ѡ.!`FA_'_kIY﷏;` =ϤLF9? :`ۥ.LlVa}2Xۖ|fW{tf}Qz/ߙ=d2>*sŚ\ `롮I]%?IrtonOYߋ?exohG2}G>(3Ν<~-e80{O_\?&S&}*6}j<#Vr<[̵6ݻU_84!^+WܤQR q)>:gq=W@9{]|<9j>j]-{SľfM߾M8G yzLA,!M<U}.UdFhB Cob{EBjvÃP=:>cPJA{V=N?tC򄾫tJ%>S!UmvröI)ͦB.}cEڰݝ6hwf>i?38 ʳx/ϕr'Rv|2lu|8oCaO-U9r^0b4m @BIL`z>N\E6i4`mC`Vyאc)lm#]G3_ˎJ7f1zz{S'l_ ̓ [KP2XڃuU癬u-СmJA#u_r1emхbL)W:# 8)'`}L AtY6*:~U3Lw$2&N6Rfs˾Ԧӱ>#!5g {Wu>xE75[P"/ϑu?~#9E Cu!>?q,|`)Mq _{&b<~QD]#`z| HxBs'k쳐>3lz2y+g\|ά1sz+^LSF|}oe. ?+o}ն13PZ7%1>o]vcW̵Ti6pGlry&א?mKLJg_|mCewƾmY'ړw%3?+l6}Ħf$F"/6ے:4.]^:5".KHP#uZJxP]ĀhMH4Db]ly$!V{ӲcDƔ3deRd4r(m\-}Rj xut$A&'vY\X+yzofwkAmɱ6Ѵm<~IgbFe_+?$s &h _=w:m-eѵMl[ օ}Qht:uh"|g$TƟt3aBJ~&X:Yr0!> );It$xZ3- M.IVSz7O1?vy/|^_Җ>3s;/wyM>y΃Sʞaџyz;G>N۞ Cو5sx64{2{>M˺l&Ϻ;_gK8ry@z3#I򮿴9g<_B]#r>u{k;헨o݇rOGS7n@ m60Q!['ۦoj+ip|^'_3|PJ_9eL9x MWd$څ%$|j(QLEc'?f ]`F\&-a˼o"AYXa+K}JYcaϰ/#YuߗLH'ܧG]vKYi7lcg$7 L*uhB|+Dz4rɮSRX8 ;`%& 6?)8`CO=a i|,C/?:c 4d>?k='uQMT@0Xp%YI &gx)W[xn0O{d1r- fm:([a&Mfk*pa@|GTl mȓAQ=AuE=@ .VrO2D> +x \Hې`YIMxAVl9 PB*xZsC\&ܱn;?]ﺟe~uwMF.YL^XDh;|x{_Q—1?P[)+z0Y9fڦFn C{ǜglx@U/zZ߸aV`!}Yϕ+e!,M w9וwL[✅% 7PW O~+͓6WzYӳqM~B;θpU\m| uM:|gʣ?oT9Ϫ-gjm Иk[}>{V>h \m`f` .tV6>i}iVfїp`]MȘ IRPw1E "6m}{ƕM) ՟4Rs rL0l'c<%wg k5DqiD׾q1|sK6@v*L49GYC\09fZ[лKWD֙Kq ͓݀YY Xu9i}Q {H0 Lg\W?\yy}N5_$g~t Ϭ♴hy}.]xٛ]hoNcE@rvA]uw ·gf>?se:.\7 =ƨiO5^?c64 |'Ǩ1QLɎnb;\^k~-붋Y^<㒽ԫAvn%/gIY ybQ7KC ̓ -;q4O=ԞqY/ٮ'hBy}{Qu6Z(oyk!~v|$]FRy^^B> )xo|j@P*&X"|껓R%ZN[x`4?ԙ4qwP D0A @_=I 1.pm!`^{l; 52{_֭_ +_0ǜp ~oo”KD!`;߬Yj{ )Fhp 8qSt8 *qy6v%%LwAGwI3|?>hr>GKK]߅zTf~̕/بa\YSƆһSzB+6i}QVOu=LX}gDW84־^ML'~xbG~sЄƥ*_WKtyq?*d0VRb%A"B;f8YtH6@g=!5jx69ovid}{XY۠Æ yWn>!YO@aH8U0II%Y^>vD"em„ٓP%PBJ0| UP9TK[sk>n`Cu`F빃ul g*܈DZ6~}G_kgfe>A/ ,Cym;Q.epXڢu/hes. Pl1/o~&ɨ9<|1OF0:'!0=>Vo;vS 65XQli"e=F^n%Lu"o6pYsw\_UKm qb3n F[q;jE s<{CߢWl_IU(cθaiyv^ύʿ69(=Իq.gRr'z H۪ e3Ybe1^o=9OU1~&5cP6m^Gdsfzzc_tjevu TeD~JȄ^tB][ L+H@{Vf)I֞e]؟ɚlfաKYL4qzL{ln͉SɌ4Sp oIY i<| +,#b *-߸m-( $vpA? |Ё:!r:*U+7]\ʷz 0PH׋Z/ f7uc6Qa-Slt9'p, l(1tuByт4޾zMʆLN 0| a0aB YAPjbRsY-T?2䄄&jX!-ђdT#hNQp\7`r+0lmFr{mSe/f0f겢1gs?_-.bB랭IBz}E~ɗ{ziv/Lֲc2l8[_oJo^3UjYƺl|t 3@&O$K)GI `cCljՅ':M;}|:uDܗhs5$MH}LH] <<$yK?4I[^K߭. UĢ\RN5z]cRI.zAxB.qYX/mɕv_Bw';Mo&#7{줄i'" )̓G-!.k!Rt,̓*.Ű58$}&RR-\ $OhdQlKIrȄT3V5 #gڒ"ROl0F"c?.G'xBD%V%KUe"vV%2xRB# n ewsJB2ҮH4I-ϓϢSI}{ZkhQ*u$R៵{>Iv~& gcpM(YSp=oĵ_OZjMSy LfB*h~q= 2JHYVB~ŒNIHՃ(=\x(vkW'~nAS 0X|rrςn3)/ardOhAt8K]b٬&d-̮Z&[*_3i{Ȑe|U)5#@0C]/.ZIx8 7'Eg{7w>#Ev^B_n2n=ѧ{̯xy}.u?>eed>5|+9gY-Zğ>3e&&.uuO/r+?}\"b}eME:_-!>.1wws[|s y1|6R ɧ&J)rr% ΥKs}V&$I>s )9M@"HЄS}$g$R LLPu^%IFyn$&0v{/7Y$f- g*<4{/ۣ4H`BA{V@:[M|@FpPmʕRmvröI)zͦBx .}cE7[wgc ؟M}`V8i]^{y>ڰݝ6hwf}('gC}Ɵ'y 7El;oΚCwLH%I$Iren0!=D dB&XBFs4!$Ez?%9%!@Up.W[~Xc,a_WbVѹן9ş{3痠_b:o?9~63|նQx.uh RsS@8f>6оqlr6>υfWX }K/ަ}}q{Ff06m$I$I.-% <<TI"%yMBNHyN*?$I r T.vZ@ʁ2Y=SA:h5j DV olC Ըa @7l/Y= nFeYU5ۯAE%ǾڠtӶJ_t?JGe f1n y#} Z:?0P1*NE|- :m-eѵMl[ օ$*RiGMon^r&LHo؄k['K&TgT\ !e'.RQ_DZOk|Ƴ>7!')~X}O9Sn毚>L|CsR{?7gV5C7^_Җ>3s;/wyM>y΃Sʞaџyz;G>N۞ Cو5sx64{2{>M˺l&Ϻ;gusyf=}C_z${vy_ڂKJ=K8}aOx~v%gI{'Evb2YLiT>/-g͡+W|9§h15i;w/5]W__f?oۯ+|, -R6} l rncyouP(6M"z~B *z֚ |3/,&OE,=xu-W eJrO2D> +x \Hې`YIMxAVl9 PB*xZsC\&ܱn;+^:I%u!^1!~wR6/c~RV`"9F?s̴M2hY9:sݿj]_>cq(dB8+WfCY:Rٙgs+/Xeч6x d@]whڦ9UdzқI>6.&w}YpYs`9>F/LU:x{t&0tV{\}kTBW{$vVۋ([>a"mi6>GHԧW P\&Iar; O?~H@%pd2?-yD!U~fmw줄 oHI]gqZkp.4%piA^ʐ>wWd2|k=>}b0h?߷| oԷzSag9/1ӲaоS?u*8ߒoAqd7oֹvֶ>VkzZ߸8vDҾT]Ht_E}͑s%9 gzrE}LmYyg)TaYb iC]hoNcE@rvA] l /b mD6T&:m\ΡK5FkU`ă~-xr{Rn}>5n"!52m([Rڙ-m%'Y^=gIOmzb 6!~Ŗ%!gg7IMHa'{yIt$ MB,XbNJHOok;koṃxC}xK q0 4"q@ALƁORrL FL>V_`ֆԘßQQ8Vwleؤf>G6X]`O=Ʌ@l=/t53}i9_'.+0%y 5Q&u7kڞ%dBʻ>Z#DB*NNoce-@^';݁oI ?S<Շ[k {,_a寨:z#}g×g +#>]u_Q` w%ާ[2KWlfՁ9>&zvx\OZ?/ϩ9ӓg:Cg3vN|lMKc ]*DDK)V _'>^ۑE.ǻYL@ַ7e'dr$ %h.MфT]CW?$Cg6m #ɥYBjڧTލ4!՞$ɇ'^> (L<~ &0$ގH콭SR={DJH;ȇ,w,K[sk؈n4>#j]|3@zl\-(@- TYmFs#ڠV3kke>j| \%u 62dm@~C*8 ץ J[ ;jes. Plc/o~&ɨ9<|1OF0:'!0=>Vo;vS 65XQŽp~[z:|_ds~9Y~{;/{ToץȶF_1nyJۢ9彡oc|Z+s@/$*1cgܴܰY7Y\빑7缼>E6z8S}3k9]=mU2.-e콃9~گO9>kؖ7 f12NUx:c-zK%D%ױ=B'oQ3g2H{FK<\R4%2B<]VkO M&% R1Tt>mM,nؖKڇO?GGRI$GԷL4bOύo1Do\ Z{ZvVR!$k yx сY{`A@ F!qVR-qxЪBH `vAۼO\7fS9f79:}c=seAn`F b<.@Q#^!8k@ul5X_[z2&[¶e*m MɹR'yfkERΣ+Z-δ!mcf_3~E rJs9&dE;|'0e|f>3?W]/=0R}"-[[xL-۔>L(y>~1}VZNZ0$ %t ѱH NVb.ICNJI y2/3 -(߳\_;v %eSR([Mh&IxdBRBo]&ɽF[&nb\J;F&䞱gLH!Sɩ6˄T!rGoMbIڷ>ymɶ1d?$>\"!fgB*IӹZB*v!V~'IvA0!}γI!%kJQ?=0{$s|mLH%JHDL ZmsIpgk.r|}3Rf,+qtnBj*u>ҺI$WIH%I$I$I$I$Ir].Za%!5#Hɍ*F7J>%(I}vlrTg@ovNSI|xdB*I$I$I$I$I! &?G^A=M^圤ڥhXb>$sDڳ%OC$&J$I$I$I$I{ M=QђV`d^Fl x TF҄'$+IiPuX`59sRW+'+I$"RI$I$I$I$Ir ^BJRPKԉeגSV©$UJ"d|nc94Q?ic85!%NMH1jvRlI$LH%I$I$I$I$=Z 4I F4$gkOr,H/Lhg$ABTa'i<,[7)X9O"KHUNh9I$+dB*I$I$I$I$I KI#EK8Qv!*my֦./H5=My'}V1DvV9 )븝0I$WIH}V^>dygw/7nZeIx.NɫWO̲$In_޼4D|aﯷ'V]f{a]/7̱߂*~I }Y*3OQo:%_w~iPyYs)39ž|x(JH-1OPd+ ^BNx]'":OHqZBN|BʲA$Ɍ%~~@<}O%x~O?~h$juj9d'rXIu# z9Drl'׃A} |݂'}g_>r 4h?2`Hqn3ŲYM`m[]MҙyGUv|g!X_kF6s`^'tb$q%IKy=g~/=wmxW|+Rd:wptc>udc~c|siL|Y/,d&{䋏XyYL̵`oa`a84!^+WܤQR q)gK\Gs M&`W9?Ϯ#yK|{D&FRh5%(3KJ5^J"#TL,49:i'%jNIH͓LHU{5@.$IOHirAo@b&#I?*I2s#t4!~z"!5;mv` V}1 ϽP+@Ҏt0K->SxH!s 伇m2eSl M˅8m }v?aXQ}z1#E ]zAo${:)؂$ǤI$$mHJdLږ`b5Iuk"&z)|RNa٫zN3Mnf teEu%\˄hvw/wH݇KSXPgy fPs⼽dB6idhI!;_0%J&[ D귁@V㯪aB;M dxܲ/tY M%Ľ53޾zQ} eԺƳ sd鏸߈|NPݤ?k~Dܟy> ߧ>J}~^k!(_T'_~}he96F8e,d =sY<:Wrά1sz+^LQǾ2a۟ٷjPZ=%1>o,a25Gg|C+m)4I%IarK )HO<::?c|n x^PRoS(g5tRB\8A AX ա>d)jvІ  RVZzoo<Y%.2PbJS ,F6+g.|K|>׮+ٰ|~3ߙ`*3^o S7>]^c~}&m *חLˠ]^~^q8zgjm/~g<с:Ӷ'vAP6cp;ޣިmӲ.ɳκrΕ=* gF8 }陓m]i rn/+쫞/]#r^nD}>t}:ži af2{mP}켊=9=Wlhl-.gsLoJ?-qrQx SڔrMZodM+ j'Dl$%nҷvO ].!5d!MB]*Y$ύ&z&J"J$é'F딄;NB:X ~͏oA XSz7x@:a0A !bQmI]s,*TY* hA ,8v۬$fxE3<Qtԧ@9ؖmXFk[-ʰMm&{f~~0Hk`*Eɠߞɠz ຢAAy+m9'F\@OH.I"k,k1O p1$"jB 2|F$ ijsǺIm\Y\?L5I|d3y1cC?c𥖯)Em? _tCodEr~"ieJ5@/?Ssu}ն>V|i}QZ郅g!}G6uݎI+VBЎ7"θpU\mA}q]ԡÙGpSNW84־>e] g~87?O'e|+'Z(^2RCE' ٤{l$%I$>|O q LTAR3O!%u=w.zPIe|[&!{`MgРv/z C;&;dPhv}g6dΟ*kUP<2f`cwh,+@cvmQ;[f>#@A*p}`"a\^{,@wl?XeTгۧuYG_­ 횑| 3A3I<.&JHz[qeSJF%\k%(6$n4B6c#I4̯1gc]yg6ݗQi ϧ k5DqiD׾q1|sK6@v*L49GYC\09fZ[лKWD֙Kq ͓݀YY Xu9i}Q {H0 DrΕxTY3cxf1IAWy}.i܅:]+hŰ1Vzf>?se:.\`U=rs׏ǃۓ|'Ǩ1QLɎnb;\I$Iq )LWC+&9 s?? ]Ge\ pzϕ?elx(+>تY/b3́7ee4P׳˄ųsyy}N5șȮug?yzvLM1tgaDK0+gZ?sus̹lB8Tج8W}BΈЯN}%ph}>ßoOĎ/=I$IOH|Pyy,NLRaRIϽB{[$0z$T)g;ҁG9 {_;Hn TҖڶ/?#m]|#zl@-(@- TYm+s#ڠV3kke>j| \%u 6F"23kGẔai{m)>&CƼdRe/3V́dTI,&Y;̓Q'IH-LtDi:%Q4VJ@zmxo(WY_LKrB֭DqNq3's~9Y~{;/{ToץȶF_1nyJۢ9彡oc|Z+s@/$*1cgܴܰY<;+y=7R圗TPr^gꢏ{&}-g~P n<{6@_1^oo8i eγJ(a#uZَlmo1ƾy&mT7˘L"Y܋x3_Y@_As@җ|lxLS?M.c*#I$Itn>!-)k$؟s#ae[L4DeW֞]ſun8<A;t:PX4D`R\P%jfKV0QX*}zA8yn̦r<*,0e*}.DΕ2sN(0/Z0ЂWIِi$ f#LH\7+j[ WLJr7+cE;^jQCMx =%#Zrbjm 5؟# YUYƂem4%ޓm({i<m&0v}ݻb?|l1aC̄^lOR}У?s澎/CH /{L{e-aIxzVz lUڮ6e+a6m TK_ sz\)/gkER|!CBL-gZo8W~ܷqm& v1ȣ/G7-mNm8`g_V;E1B9&i$ O6{n0F]x(^ʼn9OlZ_;"_i׉7te\lɼ|;e dQ:O'I$Iلԇԥãyq$ xɔCI:r]nuvOz'r{bN$w$I$Ik KsK  R'oe2(6˥$cdB*I+ɚ~τ3m $I$IkqTgC>O$$`B٘rgC&J֔2\q?=;gZ+TJ$I$WIH%I$I$I$I$I$I$I#RI$I$I$I$I$I$UɄT$I$I$I$I$I$IrU2!$I$I$I$I$I$I\LH%I$I$I$I$I$I$W%RI$I$I$I$I$I$UɄT5޾{]7ۻޞXA>v{T'ɝﶷeIrsk2߼怞ow<ؾ{矈_m??}`i>~v y}Ъmw>SõEO5e7UfsN@ sxo{ן>^_۟w\?w1gw˧w]}|B7'egepWeO#wiem z/ivc{&R7+DLй!BL; mv/ ]GV}5/ޮ)Qo9!˽ ˙~nrL/m]x]d~ϗzuɽK..yN]Rܻg )HB&h"$)zr&Yz@Mhx|@RG/ɐGDz^u)GA]p$PfӇ9( g irxyzҾOXy}%qQ HHyn%$ ̚RJx%s#Rw\'RYLHK3RBS[1-\.ޮssMٍ2~:%rCgǝdR:ݻ޷Kv1nLH {72^6;weB\=7{וdHh9 {=!ۑI ԷUBAِLz[Ji7R׬6R8oggok\e遑{)~~L^M#%/_.<B/~2mQyDQ g womSg3s0Cm欑+]q9ylG+b5ǣ%Zhd;Ç}>֎Wog-@ϣ2&|e<.sγ|&y8Եsl{kG=ox6;msyjow]?\z_qJ"{=~sL>MYP5w1&{wrnz=<+y?31Lcp;A[{/䂞r~{QqlF׍&d?CI`ؑ߰ )߈ɚ m313>CSޛ뺒R(cu<'d[3h}>n^4esTlrмl?KTs<q=.>CkKzZ3ԁei$m V*?~Y|9&R6 _6/uq~@Ƕ՗'s?ar:#1dgٖ3<s&WojΎ :]?߬EW[uhkŖ%މk&| gleT w!egV{f#Xl|6/m3(xI6d8 4АrH^9^X]]f0ϤqG+(>g-.c?Vǥ$&b., }|6<}ǫs|Dj]V!:@rڡ+gz9jkA%O3]3aALi>ǂjDL ُx-V[ź,E?$[/ՖwDx_ܳȻjBJ&zR˼MMXDH$K"TIDNJoGzΑTDMV!J*x}Ȅ9Zda};4bϺ44/e '/.q+/Q{ ߣ=RoMt P!c\wfRxo;Y )D5 %TBZ1,cD~soqnU ElyV~.O˙t6E`s֥ 6gŚq u v)d38by {y([CDd371G~gra$ٻVF;5Wl\06m*J&M|n}e<$MI,:-rG7C054[^: )DY!%$ 6{&g-GY\ʽ{7X]LE ƥuy; )zq -Y R6}j:v_NRw}roH~qK`,>?k_̳EEXB.z8yǚuh5'jkڄ2keD}TayB}_anW!kkhj>͜9 mfߙd3r%hXN+g]8]Ņ#~y۴ zK+{HI1ksa<-A֑1c؂w<;y|{~Ը9i٫;+1+K Fg]ӳyv3t94onh{:|l1O3hEl:2clqH `'q!go.. _EmWׅ}<2&Mgb36d-,]Z?#xv&}mT+NdZPgٶ7mɸgۻ,( drcl@ص}֔v崏a:z*ektΨHg1,7c3&zk˘+GE9'h˘Ktlտ5Cd21|3kͅ9W0,ak1yu,PlAzw>XFL$pFGb\sucLtkmYʦю]ќgyBlZ~Nu,y?Ol&tBY~h3. 59̒C3 4N~%Xlr]LQ=PGuz@CQY.*j, lm(ϋrƣm"یc(X63LgΙ6Pc e`N4y6v@]A/){2"} qGPsnCVm[h租u{RwkS뿲6=5֍xrܐyVڽe΅_ORG뀶hvY EnI= 6.,DIfXPRaXotٮM,֧LЈDx62񮩗?>f7%*B7%B8TR|3\^Q.]D}t+KYB]$EH?÷%r6 $9V^/|'.yVeY/&}&W.UӋs@*?!oGץ_Wsv晍/<3i3m.< d1F6eLXːkì ȡ ^>9E6?(TݍulHMg`r{a3a^m[mEϦƕ+`^Udyg\/Ǔmo^Ze&O^޽{Wy`^ x޾z"o怜7I^H_oO:m8_? {/ |xv{Tk&٫عK=:f9yFZ!z>닀bo_Ε3*lϝ{$xny\km{~}k$WKHx`WgYKh2?v1Ir7XnOz(@;R=+y~xe}v/ CpC158|9p0jS^V6/܃/f$=[/__x1-I/2KvGB34ShHseW]oU` dMǪ1u0!G;y8c{?ϝs|y^zs/b={3͏Mιkή%%&w  $vh©?TPDгg b &h sC$<7O'&fVJbƛP;[O8EՋ]>h@9qK1@/l>Oq/IpA} hѾi3MpiI(Įqp3NB5z?6fGG(+ (c㜼;W'AcfACq^B%-a;w_ ˻'2  )ro1"ً1̂YEd ʻMm΋߼ XRw[ond }uiy %,*RxLmΉVsF"%oj"F^M&Ȥ 쏿z۪}ea )k`Fx,У SϵCI0Ã=k.`=}}Y}/IyI{g[*oUe;m~}[iߖ?/xc{]žJǂ:g:& ހº삘 ;|< 8f^3aojy"ǒzQ䲍1muͺӲCY䄹@m>0cMpd;˶QcLƔI w扴/ra9bikSB)D~6EM%X׹ 2MՆZ#E5&9|sq@~w(o+Ѷn#2@6)}ﺂf+^ҞlS'`5W+h΁c i.:~l_ q1b]m9 _ym:O :PXgi/9\f6ۑ}6;\ӈxDl2G s~#6IRj xut$A&'vYr c쀟Wl@sff $I8?f'oBG&QC5 hN 1"ϐm&'\2!ec2&qt[K$2~<?b'T'O.lِϱe_ʷ.OfaDvfOEU# mm\0ڗma=7ސwD[AbiC<Ss9o&6A弛)t;m7ñ&6k8[XcÚܰ!ͯx{wL1^n{lK7PMH529ꏣ?YbΚvD6i/rmmW]2lA e~^t}|%mO-1Ǡ XʹP?q$LK0 9OlCH[P7Um9',4CF<[xck6 ׃'zp #gr.X!eU$FR=ATIMHo%XVFA#uJB '!UuG )dm ^/G;6oS)g&7,FޙlxW Hh &ׁ2,QobPDV9G6}kI|F`ތ yW5b^92p끎y1tQ6rUZ6}ߜTu9J2@IDdLDQRUEo0ILw }+QB*$MEr3"HO_6d ?_0!/=k ^wϐ>RWhFtRGï>`g׵W,/<{&mGdxƼLmuYyp@6z\Ƴ,8̵9<'QN<b: "5!VikNBŹjԹʞ8^tROʶGk9n tsYyuz[e"_/Ɲglxh;0ގx\'^yN)SSz ccj>b-IR,?7Vj/ r &^\3=|CAlPlqIp; O?~H@%pd2?-yD!U~fmw줄 oHI]g]]xN;vKtc9t#[I7UBo%Щ dvZ0Ԃ&Mֳ.eL )InR|!,ƘC.r>y%X0jXYwzS{C 9 YP]#Z6]/ڡ\'uϪAb&o:$q Zg$]('9%)JRc`%fz{ ^׉s+%(j8hLcgj1׀~j q9`|{>hUXѰ׻ZW-sk,?62v / ]>.k#ߌ@ްdR]zI_X2ςỳ.{רukbGo}:UZ;xG9lOF4ϟ"zb6jy"7uxlTw9Re?ϣd$,_ynywkegPe'uۺ~`7ٶ(zrXTz2] _aj6kdZ}\ӓy;ߨ0s%׭] m\콂5/%36g|93]a9uRr$ٞ#?o@HN}Bߍp6@Y]%YVGc` Eۙ"{VDz`{ϴz{DNзpCͩuhL }Pۛg+o6^.F]*˙B&cM+?>Ʋm<[֙aHʼnu"IPOÄk~mǃV_fk[0WX|6kzPem]߷ÊZxm{Wn/WDˮc:_ϼ/~G#ǠɛK˙mvD>ڀz(yT%|}Ld<:GVB`ifzvCR(xLucԫɃ6lIr׸g="T$&dy uJB ZgOBrhÑA[p~ӝ83Wb;6LBDOgrYh^0voڑ]0!Eee[fP,16Sm3u$Nrx:y!yc~#Z!CK#>dY5Þ ׮^2 622M%`7jz}+9vzbEﳓg$Voo71ن,,ov+sN75LA79xd:>^̇]ዦR~Wg-L =˞cI;6ujn A^g̳<벰-҇u٥-[Q,+[֜vLhw9D[R'ԛ}5!OEj?c6([Yc=2>JFmrIu5A}ZdX6Բ_{s*o6>)2kN0NV/d:ɲgOZ~^ԭо2>3O5u|yS _lDZ˵aCr|m6qRPwIQD=ǹ.+6z6|9 $qLK(֡l3lѺ-yf\'-I.'%ec|xn$|&^|jӲ˷7xGo AN;,|#]z`bCM,{:䟃O%F; !)[NB)IV.Qfkܱ=T6+)mL *gtU[. D:2SAoജfT{<ʻQ.zt}Cն1qBnUSI|QAK@dvO<_F7@Ak0DodٞM,ާRYrFvwR읕RY7m4 }|o>K@װuo/R~X{J$_ nމonYx]_cHّg^U.mlCW1/9j3_l' ِvкpy/6Gw{c=/`է:eo5/tF9LX*S!HBf1ߏֽtBtZFp1 od9LzkFҽiW7l8'㱣=sσ"n׸__ KŞg`r's!7'6݌96Eٮ枞K0Tf"uی΅7/xTy}kٞSyhfbO iK~Xhe~bs͙ڦm%5لԇ:ō}Y&W\(I>sv0~Hͣ^ &qI)I 9;k'_}w˝>yϴI$ R&R f Hc/$Jg7L&/kU;Yq !dsKdB8J$\-!Y?>>_ jH0/W漅o`EryM RO)|q~r)w]&n+̄T$ * $I$I$I$I$I$I$idB*I$I$I$I$I$I$*J$I$I$I$I$I$IJ&$I$I$I$I$I$I $I$I$I$I$I$IdB*I$I$I$I$I$I$*WIH}V^>dyg\/Ǔmo^Ze&O^޽{Wy`^ x޾z"o怜7I^H_oO:m8_? {/ |xv{Tk&٫عK=:f9yFZ!z>닀bo_Ε3*lϝ{$xny\km{~}k$WKHx`WgYKh2?v1Ir7XnOz(@;R=+y~xe}v/ CpC158|9p0jS^V6/܃/f$=[/__x1-I/2KvGB34ShHseW]oU` dMǪ1u0!G;y8?͏we~Kxٻo~njv] `vv=Gl SyOKkϩv=-6z-+X2|KQ׈! G8loDd]24Hp-p5!#{ѵڗk8Mç՜;=>%%&w  $vh©?TPDгg b &h sC$<7O'&fVJbƛP;[O8EՋ]>h@9qK1@J6?xS Dǀ_.h> ?iL7 $8Ylcvt~ވ2[zY2=ɛcfACq^B%-a;w_ ˻\hq{YCIr_\|'tjbד Db w{фOF86T! p3sPeoؓ}4e01 !A#d1!u7z;mZA!e\Dn-"clVm]os^l^|Nz[u#H@#սKk3g/IfYV2h?gosND*ߟK01-/yS17rPj2hG&-X7vGeU۴VC, dHNH^ 62ci<g؟~J9X@fx߾z]}¾n,@>⤼ZL*Tòsx?6tԭo˟Ez1JĮb_c3ocsa]vA̅>3c/癰7ɂ_({9ȅ1hv+6 93O$DIT7Fbaߨ m'dbg6RRr|.Ҳ1DD8g-%Rqi,#ֺ+ |B;}RBy>Jl )[ȍ |+}N|?{n?@d&zOdw1lT4Y?= FMooc}gs yG5!66dσ15saxm"оўa[λI["{[Lvh;kmceۨ1~Q; 2߉xIH ܃KZYBj&@Q֟M1gj;f"XC6ꆲ{DP/g'~dcPd,\8y&%̅xߜL'R6AϡU (LۛY*жEFL!`o98cfzpV.xLr~5jC]\$;hB'j2ԡ )$ZJ?z"h$NIHa9$x:;!,s Ahm*&h!b; /kP"$v h&F@3?o6Y ˡA h*Ȇvs"7 OبZ̛A=:/Y+6WÑCn{=q:/;.fCS{C˦sRr.66GQI(Ƀ4, H2"ZBT!v &wo%JHEdUHΰ2|F$BL4 &e}? 59|fewgHT|+#:lDf Z+n[=ж#x2~_cX}Ѷ:x,u<8 GKP ZOk. XfY\[KZ(4^&F]O elce(=y"7d 'o?+GBSb3;!Ubyl{6ۇbq'u;&kS퇲-;mZ{1\ǟǻEm|^]7|l^uˀqYۯs87 c:3ɟrSETXfKqՏz=+5Ǘm?Kt.x[Y Gf6(6e8g$ԧo?|$n 82ɟ<ߐ*?6PvRB 7..wϐCUc! 9fټs,s,Իf ,k-sPYJ:dg 1@T7Jad3].F%1h3=xDιΕRXF @F[u|~Ń3͘kgc},=8j/!U.H(D0Y: )W1p~Gcm ڤsY{j=99mىaݶM-r6m(A+LWڲǾ4مVdN7j;̆\hcuk}6CWy7{`͋kA@ɣ#;cLGlxXuN$%\1׿Il3=iGaxL[ ?Mf'FRy^^B> )xo|j@P*&X"|껓R%ZN[xbop8L'-dMag6c;$)Td!aR&g4el4ßIHEqXmށ@=8yQS~Gz[#QW}1s縖M׋53%׷r¨u_:Iœ,%iߚ[dHH6qjʶGNhϰ%R 8kX[H)}ީA軑(dS@ԑ=ڊh~,|lh;VdXqo=^[߷xϘV5 35>~hz9m6PN$獧ba;79һ"Q>[?O^je۬PS0p% 2i#(ӤX~׷c'(V>;qFBjծؾ6[{>vmrf[h2^nQc_3k}; Ⱥu*}ޙMm#|/^[(;: 6{==={ܓvm?vA:Y 00gyea[+6K?,[X;A W9#>Qj? r.N7)<kB4m~ƲmP-^&l{2 e|:ږ%Yߦ "_(G ;9Vb2/q?fcc X[r.Ps -mB3,[xuuL@:|,/=]q_3_Yןk>P/ |\J==$f'%ukz~Esb3lCΗ9kcefj[Gն wnIr9n>!-)k$؟s#ae[L4DeW֞]ſuL;zSvja{}FW(jbCֱ'D*!U>OI:pI&|&ZLFmq2P۬0Ot<156ѡWn&HˤNr&SdꋔV?~lGe^/ǁW2zw"@u}uRvdc晷WUyli3aoKs}& dm6m} '٬ȥp]6sK1oH?|(e_lGCS6{}ZH[xDL-S#lt6hsmV/I&>'>p.n22xmt@I|9ln{*0yݻʛ+/? eMQ~x_0I.Gz{bԉhyaxoxû맢^3gO_oo^]\73 xs5uX_\{rHVa{tS%Y|.pZk+˜uoM_$ɅZBw/~>kϟm=ZB-QIVs>߾Mv |-׋%DY9wCn< {K ic0Y?mo|7~Q_ɻ;&_/ۖﳶ }KXѮAs 6U;R}oy4'*yZǣoionOz~Yc>~WdBDDz+Rxog MnB'>VַsG̅ <ڙ>fa|#f{_Yǀ;璳==>^z.gs5]y&9w8Zs*rO`/+=p _ec5byc>[/` 6/\ !u~8g͟tH?qtmsZ?Nl+}hB6$Rs܁D7ء HRA="A϶I2%'궿JH<Y=(Mo^B]l冋<W/voؓ/@t)ү<Oq[$RP7~Y.G^^jI(Įqp3NB5z?6fGG(+ (c㜼;W'ARۺƁD0IV5Oq`#c0cMpd;˶QcLƔI w扴/8D{]FJikտ'yB-n*!UƊظe>wm6)y4 3&(CQ}C]|5Guc7}I{0{U_ȗd:Ci"_iWݨ7ZG ٖWCɜg[mZw6KImОkD.^/Ͻ4;ꜙق'Id$O#߰ od ~ڶC21sH3d )9>LHi٘ Ifk3ݖԄd ~`]ea'C6QH`lHز~Cg~\/[ssR'0{"caɪy|O66hz{x d.c˶?v氞o;͠ !{~Ϝ 7kxrMr}g:DXl5-W,F1aMnXǐ WLo_񎉓X ):zMH529ꏣ?YbΚvD6i/rmmW]2lA e~^t}|%mO-1Ǡ XʹP?q$LK0 9OlCH[P7Um9',4CF<+#5WX=k39~Ր E[t p]p ɤR&ȷh},+I#pꉠ:%!V섔v 6}w3|mȊeL6~}+I^ "#R7^dDu==[ .rD{ԣQ9jżbsze9=d&cl6t1媾7l9,%BacsdhI~,d_[;mD~TB¿O˹{A& |}W 郊/|oD-u4:{`~{]yҍrcgvOڏk>VZϞ.dSw ^Aiͅlb-IX9Ll3Yʕ6#xsqc6u κ9R6A)ǩ&I$>|O q LTAR3O!%u=w)ΆvΊ;ak/,3&mе> Dvh|VXР:un4(36 f,"KssB|RcgHk塪1K憜l^9̹V]d^Cɢ//\<.-9Mnq1+FvTd[Ѧ:M;ͱmQ1q@Pm%ڼ|#";s̯5W0~h͑cDn,k-sPYJ>bϪAa_:äp g](8)JQc`fz{Fs (j߽hp_fo}DJ~خ3報WI~` {NdžyRsla>#䬱ۘS)y4<8ݻ/k܌7L7,TW6KW,;3b6ʼl_ג ,6nc:UyD!eosS‚ًPT2"I8%h+Abr/Nd%J,{cE(_a`!~ăx99w)71CEyT n: OG"p7捘#^!<H]MKkuR,f`k75Þ ׮^1 62-`/jz8v/f +z+sBAjծ85[>umve[hsJF`hqi_3} coqʜewC:V䢩mDyޕEٛ eG `m}zeog5 g= !!u qĨwu b.M5O|MvY]g,es>6ܓX/(ږ,@E e E} >/qg﹮acw|g7S}ooXfs b -sBO-<{ҾM&nmXEg+nk+8\). \J=q=$f5:"֏纬L)rqXT[G\b,D,lL{oK~QvKq_H[1Aш|r \7 VF~D ATv[~q5hiZWQ\/HydчHPvR o]^MSo&=k?^1 P° BR~p/$nXjUhmMoЁ^n.@֙KCHa%6PΞn"j] uy۫/{#ʿ ga,_3p \wuƐ=g3שׁDqo3a}-o%\6us2ٛ ia{XnxC s<2}=Z澎v|mť⺔H$:6Vz~2~ LsрL6{\E lQOm1r 50ٛl@_t0wo=.KW]pn/GcLa. =Zs׊_lXkIbǙ3I,f{Qt*y*t,Ea5fE.Շ2/[^e{Nm!ДbO iK~(δoYasEY$$-H=v9l9H820?sutC$>q__58bx/i/iXGJ[Q#Y;Y걾G wxýM$Y:7Y VB6Cތ?^%)IK$82YJ_ָU;Y5!u,H' RI$Um#j[$Kی[o &-/Pjx(/RE^r 2ήLPbyJӴ/o˙_n.9,F#ŴT$ "$I$I$I$I$I$I$idA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$()H}ŗFw߿ѭ|8χVŷo?s{^6ᮏI_2n$^@^}"pZxyw{l?=y)48{F*vߥ3ٜGZ!޿NY}/c]Gg;yg1fy]xy}x<鯭>Gw?VexG}f;+%&g=ҽk\wי==N{wpXA꧗O'o^y#z]qx2}/>}X$j}j;طnŊN*HфPnZm&1FK*FxfqX9b|Wy{C;;`!8Ƌ0b[7v(z!/@A NA j/x[޻|^?pgqgIDAT>kOOhWO/z'Yc;|'2KrٸN}""O])v}~WV sMnNUvmË#큯)m{|19|܇vNyé/Z.>-\l_z.õs^{StFy=f;B_]*>r۽+6uAęGlSPc|j XL6;;M%iYqZ|T~?(dA`xxxb-Q^gI~uMwϞw{(Hiq A-8H<,}~XA)Zxnx.Udt6 tdAjGو5IA>MW\px$)L6f,7RDWyW_?>,*cnypip)Įq¹P"~,a;:wo[o/ˇ#1&y^(@_c`XыП.WȨcwxRE Jo{lgP2V~# ;CZh[*\PoKǠc(VuxͿ AzX)g+urй",^Zr_ wer.b'˽q"BóًJY 6`eO w̟Ix.G63|~r/bͱ9:e.C5 R{I~uMwϾ|słw]tjaZDbo9Z_>'-( a75Y@)~#fd7ⓤ0٘EYoB:;`A4^>yz;?nE) gÈNۿ=en]1'PzqQҲyyLGm_Z~ >C_̲"eutfLD7UK +8YxS N1eyn/~L^Y6mU\;hG(K|Ρ}).y볊횖5aWV|8SA oԜ[{ Qƿ(ylkSm ql>d7Ώ>Y8^7Ky[w<~\Uz&v* t'c>nc. ba>38΄MlDzQ<_rنO[@{f{{㴭@ep(?K t07&5[v˶Q>&>sH^H9)/ǯK8,vEA|F2#p Jom-ٞ 1?}hfCȅw7|Tyכquc6zҵ.<ˌLm~ ر/Hk.PtJ;5>lE7㭯k1=>zdo> e _NbAMomLIK-asS_ qH]ֳ\_|X˞Vg*ˆA=G<Ư|/ .m-𙍌2[.l),{q%uߓrIi]{pۖ]ͨ]  WD>& gkk:}^KH*/1nk1yO)(/<>?c|n ^P RzQj`cP`&B c`m( #Â&Z8 eڡ\ۇne؁?Tqhdgmxnlxþ(_ M%Xikr.WCImuDzKY3Mӹq^Njv#Rv51ܰsC7ߘOCq7Sd{i;s7C_l5-+m|vX1d7y&wLT~(@E($VAaCW0!ËU>X.{ eW`_uZCB(궍7uۼp̍fm~nN"}㹝Gِ85mF|?(:^fl]E֘ڴg*~ Ơ`őSd3v6JڔfȵYBz[1C̣ R=<Xj;!)$?3`wX3d*:@l<>#zd7 _׈k>VZϞ/}XWzZ0`CksoY:p,k1<](/Gs{ }t_lg+r>Q.$r!_gdr/LϢج<#44#q4;Zo'zuɎcja{"6>]lk95.4Ͱm9ђ'c ]]le?ɐMs )|Y|:Vq\#vXs-GɎ<V9c-dn8?Dηl} }ql({8}qcEpsq1@8]bkߪ]d\ω]\Nm#u,5>Kh_#or9ܿ0y?>=|pV,TAnCQ)'ή@ w HQ視P3 \;<䡧A 6A JAlemX"bIkPI&gH۽9bt%!`ŕ ,Ի ak!ِ} ==eb*Kɕyb0ۇ>WBTAa_:"ɤpC~RH8~MCVm'պOdu.V0}\Bonp>y/ PqW>ب/J8yow0Wj-1y5VczsuJ;>/g@G{%{MvX%&sYcIJsqYVZf@5k8zZr=m_TJy~) ?x6= ^>-{Eb6ʏF\ ښg,?GJ|ޚ^ b`?|$&~y]]1Ɠyݺd1<ڨ{qL|d}Qڤ4?v)f3s)7~5EwE KT~uQRk-Snc| ]t@1MԀ!d0ƒ4nަ߈eoL3ay0mx_xXӃ=dP}^NNA*yTÚs z(?bcj?HIL3r-3g̍q-+5%׷u̩'t+@0/,HԬ"SOLUؑcؗ/ruaψ-Wƛ.Zacu Aޞ9@HNsBFy_,2MuRGv-+֣ sȢLt[=c0(zr5̘V5 ƙڬPݜZ$N2^_.F_*ۙB&&J㧱XӈaI C Eʎ<_3sĊw%cTx75a{vkdCo]cf6>v{jfsLcٵp_X1 ̔ځKG@ˣ6=G˦v?D0/m&ڻu?q\~MϪ)5>KVf=ө_ ^ϢT"d{ uLA Zg/BvVhRAt>I>?bmJ܌#ಒ# 7iAe:c]?wys?Px99w |ŋ:gH ޼OG"p7捘HWĨwu b.M5O|MvY]g,es>6ܓX/(ږ#&/grL ՖQ@S6Px1\bʨ9HEdCѺFIJ x6e\lƙǽ9 vtXY_WkOr%gq?&e^6W Fܷ-o{lqe D٘u=fq\#Sx,Ts NuiTKlU枯7=FShl_AUZ~`^vLv!t];ro)۲=l`St!lf ̥bnG YuK:.-6 R!mE#1p(Xa-Qٽoՠͧe_k_E_p] 1:8J8 &yR6 Ww}0D; Xulb9腟y⠑W_P:SݰhXhmnBmVSnTgDœ0O#m:xrqr܂2֖;z؏zjέlL~m#ivnK'Z+Db잸 u*Vbg7plϦMa"CϯdŒon9 RxcX"m玂T|D!(Z)%!c\ c (gOo7zJ$_ <n"q{eoDv7,k}Ҷ좾TqU(t {f3lﾥX f3~N@&{!m3up^\o:anGgPVs1.ۜ[\ bXjSOT=wͳTT8) i `!|ПmǑodC#5/۰ ɭv*(ye8a.Dg}M&ޛE -Sryz\&'FY:7Y y6E?SfCB~k%Ina㡼\Av;Oe1XJ.<~|,H%I$I$I$I$I$I$ dA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$()H}ŗFw߿ѭy/}{Wf?W~ziyqx1ƥc?v'Mﷇ7/>/ޞ#{sxa2Uo8/ykHK7?bk-xq9?<+2vo.ǿǿOׇgll`s+m~ܬzQܧw_8Fo4mE |/>ͳwOmKm|:v/?{N?;~d|Ӎ;lo;B&|Dyϝ[va6xiHrEO|nw^>?N^^8ܻp 27.jkηO[9*VN{燓D\Qטe{:|i]{*}uD<$ۼ9;q.軞fʙ/raYuia\Y閹8z\ e_(}z ;TǏ"#'R bh mW)qۘ/ӑ!kMN^8Ar{ ) 擉A^X8ó7 /E(2fcn8~j0I^桼8u,|H|Y5/}'`~'Idgg(Y dMȘ !AW/ E`.UދϞ=t}= ^>r5'gf #fP/~vgBb}@>p.[cΈw9wveݫaryv7e}1aGݻ $Ld_GwkGM)|i]b|.P>sA/Tf:)rjS}1wgj Ϫ>#=+%W,Hy/zAz?YE$Vz~f[sR"z=ݮ{\`X&WNC>$yfyy1(RQHt_7{yeqʣ0|N^wMvH<[1 ;L"+2&)b\)z\ TloIkŇT°K[)2ؚmmn@UP бVHhER Z{dkdO6*mUt=O)А>n>䲍x`Yb԰>/8_3aWR' xSs+^M|59CWT7+ٖ(9+g{rl`\lGvڍ禊yeq3䋱LELȑf~[Mm//&Oc4?y׾?ֺ6s/ X(">ҝV~6|ES)6X.ֆ'W5}?p=%bƁ.T7#l=ϕ"LOF.Boffb\g|)t߾29{?*_K +Njv&}d,7Ջ΍שH-+?ƚaͶ: 4~;DXnvhg1  ڠ?ՏZ jT7L|es{wDD毤X}]>F¾>b8Q,[5A?Z<CЛ7s#:ƚxul5]FmfV†3gUP `tZ?["_۷1g\1X} RPj^x}t" Apv6Y_A*"ӥ R&>x2di+A'hP~p4yaT1~Jm@J/3ۍޔY&)~޻qQ|m@<1"c}_?>ƃ ԝ,vnr.yĭzouF7 R6__VV$)EQ+ ŒYѡ#""UىU0amG68 R?S~;= *M5ˆZi/c@>>Oxq3>O=a|  }O||&z2R6涡_-z89˧}RmFLvÞ%y3;8TO\k}m/}Ȭ .9ljC_%m&lW˶eY+mǕ{ ٺbx K1穱˘m&}-dKY&84='1H]B R_ ]ʞrA;ؒǂԻ1m"ec6o oliZY㤞eYՅgy¹ٺ"[5li:.FscoUf^Ndľř퇎1{ՂT/bRC RWI?pꅠQ: 8)HUQtrA|.ˁ `$`7ds_^%# _`ܐ4YcI ˤsBܴxit lbDclل0VL1 ? :O>{FR ]x|c?D |ϣ8!Fp/us{{{ \?="z\^Ps.#ƱUl/=X6k2|z3&"g6Mn2З}rQqdOAO{(@pd1?xD!UfsбvTA /.%iHe#(42F-{bo n, a{ Ʒw+D>^dVk! =/:B;y{l8$GLt*u:՛к@X{CɖL;*X0 d¢NȓzqS9pmN_7 `̣\W6 ba؅ R\c+@Ymf! P8|R>M>7[15tu][ϰ?frm+n3>W0%n_l׸\Da"²'an}Į&5ٖ/oA0&scI}Lm}oAރ܆sB7"exuұINmf{Y 0ٸtщ_mOS ?q _St"0AcAɋ9?;W\LښǂXKe3/35y`b9[O'Kq<܏@WIi11qՂk`k/!>)7~5EwE KT~uQRk-Snc| ]&jxcc 5(XmozJh`-87xxV"c,+-8FOP^n$$ (6a 9AorC1;5)=M;a:D..uV0 u(ᡋWv!IV .2d9 R&Z*zE5@צ}-HY% JtxPP5sJ.kJORy%?ǯ h}&gc?$_zs0y~tnb3 ҏLvP61vlV|kd,Y˺dOʞuQy0+pnG~*zѶ uE0LGNtn#M@?5ǫym8oMAp ˖ȶ6CSc5h]f=eڄ0۲Eg⏩|}S5O?lʚ7]dxuNm6ӈɻ{ާ `k/l1Xllw?J{#)w"ڣvck"&d3&?+P?=rrS!BQm2`m;Sǘ~qbe7=[ymhyr݌_3]7WT d `B FcQ6+&Z{AO.⿊Rທʖ(Fbx\EHlma'~aXo x~Qo:EdyXSݸ|f(L4#{ͫZl 9i'&Yn1:8R:W'Cmڽఋ\>ȸ|oi{7a._V@b /nb nVQ ,Qu+`r ] l{nY`qFRQBq܍OJ:g,`y(`gA `y+F|G%<Ko^9+9cUQw< =%>nDZZ^93ClyMXK} G֛ہ~_m9o3nW'm-.^wR/Z;_ #ozϺ|}|mtnra.C.u_g.P.I',}̍n"gfrb|uNp^ hs}a\/ 7b#Y}O-JuQXbDV/z,ca߫X#v*G8`br=H\Zd]`2Dߐ=6_f.Lɾ7e:_6I|=3e3n_ ڛK\$gб0'Ȯ1|݂c`wA]$]@`nݷ fI\{xܣAv~kyxuߺgob8޽$iE{eώW` X>(s"c! R& RGu3*@o+/z 73Βx/EEN/~]W|fŕׅM6tc+/?r<.V YwV֚Kn?S\&qrDqZ{7{~'InF;oˬ|$iO\q}|^b..gW\9i3΍sspߏ/fv^p {=On?$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$I\ _}5~Go?^[m߾=ϕ+^xEڀW>o_&}y@ ȸIx{sxahiak [x:g6v{Uؙ;|I񮝿 s\jo{~{F=[3$Ir&.Vq_^k??|t\k-޷EFG5:/Z5mgzh< Rx텅hc`#>'%J̱z37#o<95ooshw?Cax9.ǿ!/<<֜ ~\SV>_74_+zzhgG_ާzzx>9|͒?_s7u }JÇڟ>{m"u:I{79R0*,:5zrFXesl@_z_z.õs^{StFyAx̘wdT|N{i/VhlF~~<XQײp>gQs<W봸O_CaQ~D>?DA -<UmX"714l;𙾀ԺE(Į!(8s0;qw;Vⷭ*Aq1yח (p A X, b' pYK!g>N٧>@Ϡ@dȭ6 4Gvs FCaSq)&Ges y,pԾ<;A熟8h}O{ ]Oٿ+ _&2? Ǝ|\}|vY;|R/B9*π<3Nɓ!W1[.zR gōf^Gl2y&J RuK^ЩOjh}^O/$'6 :n.~fIqBY $ooD>Dx`x_,Ha_rGۤص!epx[Dopsoܻf\l~zV>Huo Rdm)l}6c"AX*BQ_xśZq ({ ẹofN^Y6mU"Y 5~A e@6U\* ݠ` $vL875'lާB8)_WYy6_[s2G,Cne~[-;?|*=ws:g17`XX]Q@gyg& 6q"}I(/lçO =YqV2?W%LvCl-d;e(9 ?}wn/N͜uHlhWE"<|ɏ_suo9)8ȆⰁ ʩc(jm+hsT7f3 ɵw]AW|iOO0+ʕV{Սcs/@ƚ6i>mfbh)v{ɳcrrM >U9,"sXkސ2ن\{_cM#baU%Dq69}hc˼cI\{*HAxxDs+Z]d}իrVL(HIsMz4a!7l7QbT }XDcRVjeavctC  "Bra榇Х_sMbn#[*Aoymӹq^Njv#RvL`_1&5V"NAj`gpOѳu R}L;F}?)bb7hc&ڮeD6AJ:[/0}Pė2gGgr]I,f< :V*j,A+h=XƵ,8̵Zpk׏ӣl}z=ۅ>vlXӈa(vr /:KV* R Vs{3!>g/ ŞsU֎}׏&սWCٖuh/sn41Ȣϫn]l  n\7*88nLof՟7ɏ HJ[>em䏣tXyxW*ƗmKo}.?fq}1;HŦl'M\)H}>8|(,`/l:vڎ*H>uvݥ$zLV1'ēR<ΐ6˛ƈ!ĆgW2Ƃk`mEM5iZ } ==eb*Kɕyb0ۇ>WBE6 KR`n@BW,)a(̬1fz{Fs (dǍds_Ɗ); Vz_;w\?y9hUFjӱa[bϩ9k<66w b}^. π5@.K(7#. K&UMƒ8  >̀2/k:p {tX?0N6/R~le{:7"_|Z hŲm\ bƵ5=>N{9kY*.,02)+W10<7i,;{O'g?-;<ɹE,{цbRt1r-Oy\K]hqOOv*Qa6 b֮m6Po6`٠9Ph1?13|NJMbI< +9xƙLY5'AƸjA 5_|x"Tʢ%V*樂T)@1l.AځLB4F 0If11 E+$p Be|>ѯ@v)>o:xbqӱNP~&q(@~zݑM+>~ײ~b]Sr};,?0jI^JQ@xu@u9WS~L>A6rTCԊ?Rʋ=guzCK,w?ޞ9@HNsBFy_,2MuRGv-+֣ sȢLt[=c0(zr5̘V5 ƙڬPݜZ$N2^_.F_*ۙB&&J㧱XӈaI C/Ņusg(}ܟ8c-Y g'wM˓)lNܺo7{ȵY^'zl/(mc RX={Bӕ RfR$ Gm%HL#Iw̼\eo8<% =!j*2u7.O<(L*KssbyS`>T Ouꦓ:ɸtq!bqcވ99һ"Eߴt^'/urnf}SS0p h#Ӣa_!c/f} UZ:NQ@(-Q0S#`Mk_Gm狍1rٷOeβ;!+r6"WJg73icцW:O8kV¹VleuXlE1 W:vLbT̻1 &'&DTh3mEʹxpImr yZ a-9uejy&^E;٘瘂}{S25Oli'2xlٓm2ukt-=]qd]\YZN)u1XT!ٿ7<ԭ]q|~D8efBO9\ fb3=Qĸ:׮p13 [撟g}ݒ|\ LP4b<׍_1B_\ |ZvU R^2c0x[FWh&eZpρWTAc<Y(eZdrڧo/*m,c{S۬0')L *3U7|, D:6SAഝfq**?X[hub??M51ݸBnU\S)||և"j1KℼL-JP&Lٴkz~%[~jQ)uSxӻ\b+_-FIم\0 R |Yrs \2?E*+<<΀rv㬏W2t^}Qmݍ> ci߅ẫ39/UygU%~wxC ۇoi#.LlH ;hݻ?\?%btN[A?Ժ՜6u 6k+.ץtF&1if]=?y)\4g,͞3Y|^ lQOm1r 50ٛl@_t0wo=.KW]pn/GcLa. =Zs׊_lXkIbǙ3I,f{Qt*y*t,Ea5fE.Շ2/B򰏽u-sj l{ZH[{5|Dq}43M-Z%%nA1 dxK˜X;M 1{J<}yy~dӋὤpc]c)InG_fd&8ٛ6IdAdAj7X4 y3rDXxق/nӗxs$I.E&dA*yDd~YVd=;|ֹ' ,H%IP.V YwV=5|Mn .Oo3n71$@㡼Hy582AA)_<{wO@dUX,g~ryu^\y|{3 RI$$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$I\ _}5~Go?^[m߾=ϕ+^xEڀW>o_&}y@ ȸIx{sxahiak [x:g6v{Uؙ;|I񮝿 s\jo{~{F=[3$Ir&.Vq_^k??|t\k-޷EFG5:/Z5mgzh< Rx텅hc`#>'%J̱z37#o<95ooshw?Cax9.ǿ!/<<֜ ~\SV>_74_+zzhgG_ާzzx>9|͒?_s7u }JÇڟ>{m"u:I{79R0*,:`z~2~K'7t32H)=Ͼ1>TEn@M9l;n8C2kp38 '+?fGG\Nm-e oPFcL^$c -HB@F Iak9lxRy{ϠSO6Pt3(rM?\=6ѐeEaq(ktu&g@;_ʙ_xÄٹN:7CIPbAʻ.~ :0ɂT-"7ނ/vE$6TMpо,)Bh7+=$>a1,\)bo1"ًhRfэgۿEdn]1'ƽmEKw1{@nm#]`T/-?khπ/EfV2:gn3&*ߏ/WpW Rlf=i[P~^09"an]Mk?s+m|L|$4ݹ86sS GyGg$y:>,3ɭ_$RAظ2mקM5#EGp>X66A9u]t E um{o<"d6+a5/6u^uEjQy6u@Xf]6ͧt\ tm1.r6yv "WB 5ڇb0BgvXkސ2ن\{_cM#baU%Dq69}hc˼cI\{*HAxxDs+Z]d}իrVL(HIsMz4a!7l7QbT 2)H7z)ePVZlM;@2 "Bra榇Х_sMbn#[*Aoymӹq^Njv#RvL`_1; sUxkh>aڨn(K{+|o\'O${A!^_X ~u &D&ZEڪʹɚWeLY=f*98['Εfp?8g .?8`SWCʪ6s=uI^ Ť҇ȯhl+E# AuLA qSV䂔N,{щ;T %C7cnȊeoLT|WJF,"!f"lzhs\H"^57A܌ Ѹd1*,JӸ( ]Lj -7cRr.曽Bx xЊ^ P IF[D+HUJ!)`c8E6= QUզb=嘶k-裶] s RH~0rrA%zB7Ȗ:y'=X5g׺^t#X솶qcGJٳqC6KP ZO+hqc-K%s-g{eb0A_xnv+m4lEn'݅d'~/DZfσ3}{UqRc[;]?ޛ4FW^e[vbXǿ"S>u]& 2q=l85v`0U\'?.dS"e*yJoaw>f#ْ?ҵyyT/ r &i\\Lchw5ȑ M:N$ S}p 7PiYO+_Ht7U¿} )KI6f'xҎN%NzCvyS1:ؐq0JXsu -Իf7CX Gl_̽Ce)2O fЧJ/FT hQjtE;!,V"LoO6ވ}"csœql[ql=>7X1eYu09_˱yOrث$?^c<\90SrXym)mt\Uok]H5QnF`]ʿL[_e%+q|f1ZiCe^6ukb_Fo`|Q*m_0(٘tnD&x&e(?q"7:kkڣ;{RQxrQa>~)HYqny׏@ܽIc{j?99mىa=M-b9Eގ6 ȕJO+L]mftZB{zS:a ͗\vm3qzq+.́GLjm6Hyߵ++7mf'm .x0cgڷj3d}֜AN),|^ B}4 R0nj@P*X!ꛣ RZ8ưiNOD3I YD*$MXd<ʯl(D NE}!i!~͵͵K;,5p3+/F:nF_|ƸM욒a!Q ORHŋ,^zUi[d +VBP^\9ި |Ψ eۓ#)siA(kEU [ȮEsz4<~Ytn+g},EOj<8SC_ SۋK}y;]ׄ8^i4˶Qzq"73aA\Y*H9 g1B³>f!V/ ~ܽ7i,O^S-;s3_"gzuz\ ] |:|-G#}Mbv{bie6%꽔?WIMa98z,ʈifXvOK8]ZT'hϠ^M)ek~N[ן~~ $>(mc RX={Bӕ RfR$ Gm%HL#Iw̼\eo8<%(t‡ ԡߴ#<)*KssbyS`>T Ouꦓ:ɸtq!bqcވ99һ"Eߴt^'/urnf}SS0p h#Ӣa_!c/f} UZ:NQ@(-Q0S#`Mk LΗlr8F Y~g9cE.FJ)]k^Pvt/``fا\ly&m\3ڰ@:Y 0gpͳJuY8YÊͺ(8&Vgz3IyWQ :ޤ=\ׄ(muƲmPY9c=BQmYE/'ߦ"KxVAJ~R簋c< gsjo69zޔjC@ [ L',[x}uL@>r]l/q}dW< V] 6߱ͽmV[>y"iԩpd8s{e-w:Ț[n G*_>>Qb %i}ZqB^_xU(q s&zGld5um =- ?:)d}KuI?|gLfCfAy.=u* B_洹c]9_[q)l.30Nզ+H-1ds{F?p1ldo}i ӱ[ ܽloTh.^vkù1Y0dž^j\+^|D vc&^gV`r'n7FAΫӱ4Tf4^ T8 yxW1{Ѻ9نCS6?--?8Ӿeybs͙ڦ-f\ ]z`]YO$I(+H;?ۚG>&HpÛL[_P^䋼d] /߽;Ƨi _2*,3L<\rwYGN/<$IE RI$I$I$I$I$I$IȂT$I$I$I$I$I$IrQ $I$I$I$I$I$I\,H%I$I$I$I$I$I$% RI$I$I$I$I$I$EɂT$I$I$I$I$I$IrQ.R/K>[xqxwɋo~k/?"m]/D} d$Ir �90Dp0ε^-~~{xR?h&q퍜UKW}7f9yBzO변b_J=*̝w$@x9n.7̽^?ϽCk$9+H/Dy:[ODOm|z63pXIu`])^fɎy/9g:>Uv_YO̽6QsFGd~vjV3'-( Hl,$}偋YR7&oV{rC9I}x8#9x`5x/  Rؗ7nE)Ƴ"2|s.ǘ~~6e= uF.F{ۗ5gǗ"lHt`3] ZR  +BS8@٫)c] `W7sb~˪imh= P R.C5/6VL 6W_B_p9ދo}Ss¶o^}/2E{g[joUe;!cq~N9>^V"~s78W/^|>squ qq&Mm`3'җЋ6|3ۛm*Cy^Xk?t 6 B`MX1)@wDYO)_"񳮜]~^+/>˗KP>,6/H5Eֵ RW5o+>uo9)8ȆⰁ ʩc(jm+hsT7f3 ɵw]AW|iOO0+ʕV{Սcs/@ƚ6i>mfbh)v{ɳcrrM >U9lZ`q`ڋyCƂfr}~1gǏ5U$/񣍏-B̏=&Ir= V'Gg,ϭkjwAjT7Y-2] E'y`5dIR`$3ݐGeoPB䯒6ʤ %0z)ePVZlM;@?D Ɂp(|+`+7?`nz ]zl 9$6:"76X;6h="eC/ry33[b,hOOŁnrQ 聿mV`>AY^ sTAjplAۗ_+6~Kص R0?C_@]ej~a2'!(cdne6lslYKVz~vԅ,L>lT4Y5?3 FMo|Ăk_Yϳ:61f~Gl mȮ>51ܰsC7ߘOCq7Sd{i;s7C_l5-+m|vX1d7ޗ+SX)W9=K/\ ʴc?ٛ"f*}3v8֮mPMt?h.ԋYɾr? x]N|)c~V~$q&%B|nNl&)chi#6&kV^1QdH2lj|59WXp Ǚk2RVKW-HQ-&> E~Dc[)^Ůc R؎cTu O'te ^$v6ޓNlޡR(Ls#DV,{cr}x@bJ2 Rf"lzhMyڷ .xAkm7p~4.YJ*?b1s4#,<~4.;.fCSZ{C˦\ f/P1^G)"W,TB RURr 0_G9z7Nc+HEbUX5f?Fo9Z/bg1?ycW0:N}Y 'G |ȲAt2\ ߈ [hu6`y\{_zҍrcgvOڏkčfjm+gҗ>j,A+h=XƵ,8̵Zpk׏ӣl}z=ۅ>vlXӈa(vr i9V^^~Χ<Z9=)H\'5vcIctu~jPexn?\`/u{-8|(,`/l:vڎ*H>uvݥ$zLVAø=[ ~Z+HBӸ,a_u5F[`y7woXvO~e[vyXϭvsXFѣ >rғb SW[ٟ0&㞞T~lB%׭] m\l\{Acs8c~$81bf?baYa_}c& Dم<LLVc&Ϛ Irc\ Ś/>KAyFA ƍ_@ h]e+_]}sTA~z|Z Կ6u @鉈`&i!kC$ɁޘgPwE!Zp¿/F,X!~͵͵K;,5p3+/F:nF_|ƸM욒a!Q ORHŋ,^zUi[d +VBP^\9ި |Ψ eAy"Gx{r#e;!~mȼJ6dKٵhX3 3mEe;{״3c"[ט'X[g?hz?@usk8>8!wʐz{q3}o^uhq>珩NڷўAnIr>_ ik?&(OF ȯh!}/m>-Z*J)/1P Nj V] 6߱ͽmV[>y"iԩpd8s{e-w:Ț[n G*_>>Qb %i}ZqB^_xU(q s&zGld5um =- ?:)q__58bx/i/iXGJ[Q#Y;Y걾G wxýM$Y:7Y VB6Cތ?^%)IK$82YJ_ָU;Y5!u,H' RI$Um#j[$Kی[o &-/Pjx(/RE^r 2ήLPbyJӴ/o˙_n.9,F#ŴT$ "$I$I$I$I$I$I$idA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$()H}ŗFw߿ѭ|8χVŷo?s{^6ᮏI_2n$^@^}"pZxyw{l?=y)48{F*vߥ3ٜGZ!޿NY}/c]Gg;yg1fy]xy}x<鯭>Gw?VexG}f;+%&g=ҽk\wי==N{wpXA꧗O'o^y#z]qx2}/>}X$j}j;طnŊN*HфPnZm&1FK*FxfqX9b|Wy{C;;`!8Ƌ0b[7v(z!/@A NA j/x[޻|^?pgq>kOOhWO/z'Yc;|'2KrٸN}""O])v}~WV sMnT'ixsGB~P/>{^_W$o΁}3jqݛgC|ٻ^<ϩ{]Oy&91?`wm^<<8qg9Ws}m 73,b|ZmARfgU>9=R?Nˁ2,H,ϟO%ދܣ 1W׸pٳ1|):-@!Pء:~?b+H50E O@mJ|a,H(QAݽ<)ȇ[)}~jϙ$ƌeWFjʡ0?jrbq$pa7UҺS]sD.dYYvtuD[߶޲_e.ZOɎ;Z8cE|h_4gD ?A}yv ?pYg4fߕb/wcǿT>Y?^T3yG샕=y.3&J x;!#.^ȽƊ5rrZ|T~?Y(H]&}6ݕn>Y RuK^ЩOjh}^O/$'6Iܠdo4~fellh߈OdcoדydA _*:;`A}%[${Cw0SoO[{c̉?f\l~z'tF.F{ۗ5gǗ"lHt`3]эDR %~ X 7 P∘&{~C+/ަ_yv@Q/H 1)-k?qDQέ#/!4~~훚}t| %~VmmW:܇;9 x[ߖ?oB/K.ivloos { #b50&{ ?7َb6ħ|NOcߝۋi3g]>%84sY9KH߳_TZ|&hk1%d C;62^F.;KЧCތk#e6Гw5Yfej`߷=Gcg'Ǝ}AN_sҧSa?f.d,o}U^@Ik {[I([r j0_nz~[eocϔOXboJ5Ex %pK|]3I}_r8~s89eͶrο-gu֟Yj8G\R=-1X+ֵǾmyEO،uoN~Etc-?mR`.zv9M%} mbXsTP+ P3V W.Q69 n2,49xIK Vۆ!0M=,h򧅣IAJ}>*IY0n}}VC&JIvֆƦ1E$\&K Qq!q5t_GU>t>X;6h="eC/ry33[",BOOśn_؄@G(sE03#fAjpzAJ*H"POu G_bM@WFAtOҾLū@kߕ\99)sry9F[fC9ǖ-:qTnIgH]8DfOEU8# mmΗq?H,e۟X.{ eW`_uZCB(궍7uۼp̍fm~nN"}㹝Gِ85mF|9?\+>/d٧SyzORwX6`*|զݐ==Pc0a s%Sver_).7ml7h^ Ť҇ȯhl+E# AuLA qSVTLɂBlbсWI[AdM7[b/4`;`q8.[ 6} UAV9+ree<.0ɑH~g~*ѱ*gTr+t~#:lx؃Ys}F>}K7mnh<h?7֚}=K_;dSկba6:޲tX2bx&[k'Q\?_&F_O elر骿b6JO#V}]I&e$G {WdyE} dfaŠ5q^Wײ|c<ۭk}Lvý pW ;df9:`[Y-uw٧mTom9<7ce+;0DI,HlOGv`L(5kWAN>ya IJv0k!s!r]IJ3ű]ίč=ϙj~ L?"tr}rvs='vep9 |ֱt",Y3?}p<0>G xZGsӊGRo6 mGohBJ:n q2x#ipFZX\z𐇞:%pb).ՃdKHrR`+clNZOg58C<%C. ,d1g`EM5O CX 9̆,k-sPYJ>)^hEQT$hQjtA;E,(Nd[Eޞl1׹PA8lbVl7lcجA`0Ϸr>gxg73報WI~'o:Jͱ%:sLocNi{ z[trDzɎr3R0ްdR]d.k,^`03J (FG_K.G5zTi82/G]s#5˧0W,Fшa a\[K:I#_L/*yk4z18)~oPxuu5O!:vZ3hr39b Ei:`ۥp񠷩V(shiyTom-Z''Pl( bC|;07(ΆU\!s{&BY N>_2A7nΙcLs͂յ}GؑGx/ޅ(1tT3g79S^3@e1X&,Y~ G G hzyZbl ~cm oẋ(<8,X5x9 0 kz0,Pq8 o 8!wʐz{q3}|S/ 奢~ȘŻl7LHcݳ(^#|Z30cSk6c*ˮĎa^8flT=X:Zqd9Z6!}nC6MiܵOb4saW53ODaFJ3doU o87lX6ۀ({Z};Qj U3}c^ /z.a#hkٕl rXq=5?X&,YQ C5L~{ ?_|Z?STXTsGP61),hUYJ)$v6)q3`J hoIJ7$RUdsȮLxQYb˜{a>TXIS3$o^Iƍ s8F[sw+D;K-ףc;:~ksu5_ {6\zŨ,"ʴCW +z+sBAjծS;^_7U7u -r_L5x49ֵQf7T^P%yNG Y~g9cE.ȕR6ϻ27ּ(>{s^ <̼O LعfaU25u`"{!g8p.ue-[Q,qL0wÕ+f󮮣 cA%uBI{扯 Q<9.e۠lr.&{%ARp>rfR1 cݓj>oy9kGNm e[>M\l.DlTha\#aCGE~A/hy>9+l#b *-k-( 1FGiZ$oXBWh'P,B_Ji/7[< 6߱M( xraXtmROn5?BA$?=ld9dTVy>ylCY0ugMa:n޾(6F|ʹpC]ԗ*μŁn;slCݷ @_s}& do6mWk`1 ]'̭ j]jN:evR:#A 4Xml/'OGk>I*'1e65m,8qs8>qMCtyĺ&te6!Ue%'̅¹|5D6ؾ{_eZll2aGۏ\?Xg[>~C%@8QXCjgȅ mP@>MAK>?\9ρ.#ץm`ړ}Gz]'d^y`Mw1O(gZhT.۴?ٛ>v{+M@͕, f|ۍX[`6u RGHep<$}(ZȃPI;q__#7fW3[>O/9^n y5CjIr)ʇtpϒǂ"^s7erlsKU6m|0O# RIbqve KdA*I $Ce+H_`_o%{(9-tǹ;,HIqUm#j[$3em-<$V֚?˳,$ /YRw4% Rͱ+s%Y?iLj:?ٗŨc) Tޫw? $I$I$I$I$I$I$'$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$I\ _}5~G=^mW>\y ֎wW_[ym$7;޼4D|sx{zvkmgזN|ݞou_&[a#ϝ?. ɽ{:3z.M//5j ;+H/Dy:[ODOm|z63pXIu`])%b@S{d? +*q||y]ޣ?؍gctN~<?N^^8ܻp 27.jkηO[9*VN{燓D\Qטe{:|i]{*}N8"{mދ8W`]]O9_ư0tW K=^>=| ZpG yX)T STݶ\ЫmFȂleɵ& / ߽ydXnĀ gR G_xx}Ad/#G 7F*p"``'>Cyѩ˕qV)IkɁoXb`kz_N>O( ?x +}Q'@Ț1Z*@,cwɮ^| BNoa BP= >RȒ=3! ۉwřD  Z=#¸(H[cΈw9wveݫaryv7e}1aGݻ $Ld_GwkGM)|i]B}#_6vB<#+f3? c.yϖvU?W}FzW K=X_"N-L~ UH 6G볷 礠Ez]~%9 6L/|HT[ Rxy(o $/H/K-#]V0|N^wMvH<[1 ;L"+2&)b\)z\ TloIkŇT°K[)2ؚmmn@Uh 1-bBZuH!,h푭> Ƕ_VY̋`76EXv-Qs큭qqʯ/)@q&l}k s9 Rp75Wnuߤ/j_C> Qyi[Mumy򝝳|/Ȧ}DmרXqn8P7Ct[8_Yy/Plb90F3|akn3>I!_-݉썾ngWt>0鯋ڙbo)[bm+{r%]k#c^"FaBu3f\) $jD1"f6l&u̗Bgh{*cپdrngGBX{ONn >oSxڈRbcovl+#@C8ތ1fvr]͠ SX ϨŠKٿxk YEneLc?{<-{s9,}U[hţ/=Y|1G)?ky']VeAVlFM/oU,l81 8sk\+ A׮'5 !۽}[zyq!sv u ݇ V'Gg,ϭkjwAjT7Y-2] m⳹![@$~؛ƯK YH GHĀo %e_f)!M ~s[~aI7:,yBqcE~}<;Y\w] ޔnl.!ץ:2[#F`T/_?{' bxYdFT:^v+ޝ^_k_JF R)\YAg޵zAJ'~ϗ>.Χʡ@4Wg)!ecnzۘgϼ|g umzd79Xnޑ13C5{~A BY1zmZCl6{]f¦}l[栯hv\8PzX+/ǰ}+}i+fB%llAs}]%D˰!Х칡+S끾-y,H&RV)9uA:NYc q,L殟9چ٬fklS~P.rM;YjNs b.S&ƺ|/G͚5?ۛLT*Oekf|q Ѿ`.%攲g1< hlC]!cUy`zvQ3`sIU]x?׊-+Uö=Rl$?8>k\E_hnLOY~ X OyɺÿW-HQ-&> E~Dc[)^Ůc R؎cTu O'؀& PھA Y 8SL6E:Yza8^2TJw5X͗^}g7D8~_r?qNO0?vXI#᰹Hʢ8Q Z6ns0-)1Yz.nSŽjdcRr X̐* PB+HM] k}`q٥9,HԢbjic/x(*H~XEƙ.gvfyzs/R~6 k'ו\:Y?4l* Oɦy ZumvغQJ7>{*>_|y? :|cc/*3ƈ7$Z1ыۭk}L6"dc3[hl6[W<_Kl`N NԿtn`:Sc1\Qwg dNցaʄ1⚹&-,?0lx]`k '8c&]e⏩T\lбF ٳ4JJ8S7_f3|S'ܿs?2Fe8~>_75=RY,pY~,8ԋsy9hCrTO۫w:c` X1_zsP>Z7!?3_k^lC_Ab8F=q?>=|pV,TAnCQ)'ήɆ# җ䎠`m 707 ( E)/ oo7`oз|g+~$D<4 ApNO%D|6M@2ѩ):?VoB"kRoc %[ʒ:2YFR辫`щ8F1bDXiyRV/yFQe,fth7^qŕ >Xk1vi|n2 ukdJ~3`'gc8/3A}=k)yNuidӽPcp?Jޜ?*Ŋl}LJcoCotu][ϰ?frm+n3>W0%n_l׸\Da"²'an}Į&5ٖ/oA0&scI}Lm}oAރ܆sB7"exuұINmf{9ea q+L}$?~.v E;a.ǂs~,vDu 5׋v a 1?ѱJf5:_;f kv>h %sb3N䏩7⬵y@b8Fc),|^ B}4 R0nj@P*X!ꛣ RZ8ưLs7@jP &Qb6l^puLřvE?/(7H՗+?~O@G39y`^&2{E3+sa_h~g^!Ϸ[dK>^#c\ֵ>f&{r&Wʃq_@7Ns8Sыl`+/f?utҾs m1<^ku13xn ڀu`X\Dv-îx4eE6cY,&ܖ-:Lm{};ޟy 0g{VDP&SǛSwjYFL޽ >뀹ߕ\>WE=s-̍N> )sB|hB0f|沣 6oc1Y&CŞV/:,=Yl|A"u1Nm"KجeWB9?pns}쁞w {]\U.ŷTviwmHb>P'Dl6^X>ĺD uX[Ym0,5&M2cQ*mZǛEƱ\g0^՛cj}?ٻn˜_2W_s5|$gp}ox_fݐf+կqSz $_LhN@i_=P1ϘO8vG D(8@<1yPO9X !G۷ъn?Xv]L}scʼnalqd׷Iu3)v\R1f%d _ ik?&(OF ȯh!}/m>-Z*J^ R"([xmrOK # y-,H2p.7Ca}ٓn^bKȑ\NK=1p'ԑđҹB?YojnE]:A{Kػ wr϶D5Z/(",nVQ ,Qu+`r ] l{Ţ ?u\*[2 ^ 53.sgd,t$9?TS\B}K \'-@l7Xh*硆g( u#պ*ŕϙ/FfF8kZC?ڴt,bnk|u2?lCow/ĽW־L}Z& lt=x{k {w vrB>㱲r%gz =@ǬhڻMG|PX[=Ge7ͽthsqwriN:f gn,v9Ǵ6k{s*߇K^@{;?z7e0y_͠mZ&{jQʿ KsjSv=fc^Ua#XI]f$.vj0"oȞ_moޯcf fdߛxraG/eky|ݤtDę/%.3X|OdnA1 Sή]nG[ui{3L$IQoQg X;?5n<<ٺo׋wȷ1_wu^ك"Ʋ \g+zrn,kq籐s)ͣL@쏕=|gIrs<ė""'Ix?ܫg>3~f &\vB1ޕ +H;?nk%)q~8u?ћv=ݓ$7~ eVIw4'D>> E{g_3+ ֹ93;ywa_=Þ'OE RI$I$I$I$I$I$IȂT$I$I$I$I$I$IrQ $I$I$I$I$I$I\,H%I$I$I$I$I$I$% RI$I$I$I$I$I$EɂT$I$I$I$I$I$IrQ $7?|Wv*?u#oo^$Ivpw=۷(~1VEYf᧗O^x|ѧ7OS__~kq$3> /`]W8.G[\2<|V{r=3*Ŗ 8{9q"(׷?y4xϓ.PË~rod'Ic9>/ -{owO\}lK gQ>HݴNMn}3hۣ4\_oE -HW O_O_|o/xo=?׿'8^\RŁmN\׽B_o_m@2fB//)=E`vтV)rEey^FfA"9||v25~W×}˄9 >sqe{{qȾy RP /(@HыP0Ћ пjA+XȡK%(=z{dozz0JnS,.Q^sӃ 񶆣uWEO7"yY")Vj@A?7UoJ Za pԽ1;X3RKx[FpQ,].1jAIy{\s>\y/2-N}ೋoxπ_ E ]^ 椓%e13=?Y R'ʯXI%.N&.G&[ 7dCm *kS"QLGYum:c؊=khʏצA 13k3"ȹݽ5fuňX.gc:ٕ݊\<͕`Mϯ~f"ɚgy6Տ1|!w"6ܱ~7w n@L"dx`^Г+;żKBJ~@ZJYYAJi8zFdO![(ə+<ogn,Nx|^RvAc׾h} P Nb5j=W0@X'}_6軜Sͦ{p l,F`Ca]׍ܓOGBj,PC"+śY-*ȂG6Y2'z *H YA*م=0sbAj 7U/\6_v$_E*T5 x]>wM>/k(@Z^b¾Oyw*QA?_Eѿ/2b,+kOuF G˞1 Jm-lnv$ZOsZi#{iXb=L<o` tEܴM )71]! xSnM7\vdvoX6pMB2M)'"Do%'쏓';1ibB6(aʎtvMT_<(I6]c庴-mՙE<-{ _AcOcFo|>]{(de{ʘN3][-CC2vAbb!?Wױc.^|Ue]=`N}ti㭹clډOٶAk9; #?N{K|VtFO3ݦSr.%סbw-fXi S_#|_0`>Ⴢ{2per>.j_`k[|9tcg lXvöi͢\Yj" claE RXp /{Q)Uۼ ^ "^R<)(H@ce=8E(XQA*+#YXϻe1Xݪ鯻yqK^Z/͗0}<~׌g2ۮ8ҙG nNf/םh.qA Ok' EQ@-BD*h]%_v4&qx E_*YnhŲ R3Oy!?y7m&og7fJ|\q3Eop.d ল3nYD$_S׵.?G=|__?b~ˮփc%ڊ5:1,{!` /61T0H}z9gZΕfزu߀ƻP؈b_ѫeZ=yY;εgBLsd`x{|casYn,rffoCL7Ke\ [muҹeO`iL9rΊ%aq#lm],ôG!9aO¸`]sΗ-=`m?ڞ1 Rժ^WjN(ߝ|96Nn:9e%΂\NJl'F:vC`3Ix_GJo k-nJiF;YbϿ-1&z{LA"eϋϞl՟.r1m%ӜG <09FCcȞ MD9VT>?o/ҲۚAsڊ; Q~jOpe/l"zR x@m=sD97T0T8 g .xqFBuAޜڌk}YL` vI"(\`(R"K)\룋9v$.D>^v2[P95Q{IÊ׊ZP&/ynb>CAJeܗҼmmUvâA$ R\~_ק8ײJ;Ӷ6T,se/2[WZ2~Xڏ?1kr ? dg1*.){c2Y 5nveOr ڙ+ߝÅxFmkaNQXqHg];!ϙʹx/\}q[BdFQELToǎӂ': 8 hPG_R'B\㩡u[.F9bT ezm잼S/A/A66Bu+zVA,7X-H_+(E5n 8"B'ɞܫrFfLӸF `e G)7:k7~F{-ח2A7gƮ܌7*[o|CSzP6׵f8C[vqy~iϽxbAƨ= AǐM ^/RBë5?$q;C=k\־_+ ~ރ+""u>vx=c~iׯZu\P)sO.we]C7xC|j]Cfq"eֵnq4UIb|}lajC9oQ7rɺ:0n&ǔߚmvlcGM?.:^mDB?k}ui80?p.){PYԩ8/R`>)G͕=)r+i1 ~¹g>:mB6ƛgތl:Ɛ=⬶+~L5h}iL,ƙ R!az}h8֧'O+/Lz=6Y)pTj~)[~l(d&>?my$ۑ~/a熼u8^E0wx/dV5c^ z)ӭ z{(E""E$ok_+H(%^Xܬmw/ /'48?6YC6bݒ i]k+Hmԛ)7m봯PɎ(7~CUnh{Yo쀦`7>v7z3|k#|P߉8a]qGoe/CzW d5PvJ.0d}znؤ= Vۛ.,l,] km_k&ba[7f+ ^On;f3&}zF>Ɛ=!Լ\"lnB"??kH{#>d6U}8OzԿ Ȯ4sM`l^hq[C|8ubWˎeMLt y_l߇!&mҸl#-pINix6+7[vS&Knfm{7o㲥 9ތ3goq6-ó>eaͤe\OбK+>RIr(6T$BQ(yX\ ukvm u8 {9}_̇$gb0azMc8l.IH<<{?K_&; (VYJC9w 3H= u?oY|>}߿%Ũ>9 qh9XQ\Rv$|/T,ReA*I$9Yz`dKO@/2nQnt|#&}Fl[~:9ŻX:[Aj F$cSfIN!$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ[۟>+}wº򑷇7/E{$;yuz[n`޻;ڒ_^?ww^$I$I$I$_=='+Fyr=?==kв'Z{e>1ާF]|b(wMnZ&>in4} 7D"]ʋwYO_O_|o/xo=?׿'8^\RŁN\WvrOóZmrI7o[Av RSEgO`%I$I$I$Ię RP /(@HыP0Ћ пjA+XȡK%(=z{dozz0JnS,.Q^sӃ 񶆣uWEO7<pp Q (H\ R_ɘYS㿐>7L.WzɂT% Rf% RI$I$I$ɻŅɾk(VpE S {  *.Pq7aaYZT+mn0dO >AUp[[mU$ G+){`Ă,n syqZ_ĶE 2\_.BUC/CǗom G_طV%*H7^1PA__{EvMi1Ach3FAMc>c܎=DiQk=pd?uT6w m86\d=;W"o+'u (D1kA-&ycJq +M|E:osтn6_*:g 6$pMjHHvo.g}Ѕ,'3;2n `y-I$;~5_l?N/O_nJƭ4I$Iu.Z‚yߋKh]iߟ(xI)DYE([)Gj RX_~5/)HBGV}Nu6 erz!l$_旗셬\o:їd>oMtB6{q/lFu RXx_;Y)j%TAB,8>I${F-HѢ{vqZOsZi#XKy>F{&^O˹/E= ##*İb+ AqD/XD7޳ut|X1, `ؠ&늡zȃyQFPG{,H%I._;?_qm/Mau,sV R8V"oQ47_/YJ$IqT)^~b2M)4>rG7CX0"DoO")HQ ^yٓSExݍXT|qX ?_b / erzmV/ I!^ RE>@[m-J`ɋc*t*No9ϱk48l=I cnW}=#5s=՘nk=G[ dҽ+ PT'ʹ R{_ ,(~0B4zE(^(afY VXdp]3b_5Xjw}[ ݞ; RI_߱B (&e ' ZlyQeuVZ*y ]_(` sI$IPHA7 X02TR ~eb]>s]|ApMeaRQE: ?lRO r1ͱsB^(֋me>g.H‡ER"XVwI?Y80 RpldmSelQa]wg7rb93a~6?y|UvnbAܺ*)[?nkh@@FNѶPKoPۭ/ dv!$~JHw)'RދhYXn_{%ޓZ$竻H9Om~8u^L}QR]]%:#܍Ǖ|ߟy?l2_R>n$Ƅ&[Af8Ө)R7qZT9?~p$_Ȁ$/|bShW̗d>>r㸬{9T:uoHgC)@(Vx Z<01 f $IJ|l ЁN \־U({@j@ @ TaE|<vdԲjK*~-\HYe)$!Y)q,a}+*Ӗ@Jϧk@L#_&-_f|aTMuu>Ykp`h 'Cepq\ e(52@GrGR6Hű>c0HO5n{I-)co s@JC*lwͽ zKdkq#:)`%zDHwvH`S(-ch_ Y!&p@ʸ7 .vRI;{g)4[)D;t۬,8汾CJTy= ;U=`?)`xZ.)sQ ] 4+Oqd$=K~>R:@JPwHXA\ǐ":>uK Xl>T,{Hi$LiRRcL^ZLY-N>F<Ղ7)"n>L]G>{3]Y]'uwzu~ߣ:p#aN Ze c@>WNn Ė*{eWW?N.~^y4 _=<;oܯ/O_ʺ;"nx雗yezOۋ#OnݷO8֯;sgzʼމC}B?//ϯWy?/MoյlVvAҩ_˹)QjGB P 'E/qR# |I{hK~ M.=iݣZù.wF-]R#HB,P@!XC%dYW~ @k!GAAK#`} `R9}샾!>~ȐpXm'vx\=p>"X&G1\ H>"ItK8Ϣ .GWͿ3'Hq\2#w$ex$Fz@Į{|rO([+ĩ;&c4Is9HU ka$:KAVhS0 I~J%h @lGMqQfS.lYm͕k+5p`'z~Ya>vR~t`\-N7o\EoyNbQ-ܾ2[qgV$シ,~ݤXdm0~!(u^@ճ5tH= p,̲w%T|zG]sֻ3C% [_Y}÷ۋ5'1կ9sqŰBYwNBl!eI~(%#LV w;؁4JYGZ% ;!X(c_k_D`R>A@عc5i;~gnx,;d` O5}BV sm0B";8 wxN zς \qGYʾmo.nvY:X 2 eX d>}mg r.-<2xT[z,o[`?F+z%5n??ާ?Cqm˜Ql8.T1Έdc\ϥq{ޙI .$.-^d ^Rwag'C)8` %N\-@d{ IXe;9nwA+jY*/e5yS`nO 5,,y׉g?Vg#Bl- Ţs z3nub]'ʥ~ »_8Y@O~H߈rjVWZ H4B!ܻ9_8n@ʿ{;fSh;+=g1)_ xdaT/)nswK2M21zSx!˛Mu;:=@JuGuzu;n< ?ge}@7*z xUW$ΙoSWR Z?*\dżvu)srcu {^[Y o}/sRXo ߙ}}wob9ck?s"x0 Bd>02B,d E=xIk0!ڡOKB@ͷaqcֺpO(1ֳo9SQ{o= 82Zlבܸ:E|Z7sRGBgAMA a pHYsodIDATx^흱N?!?Xȟ>\ F7СDzFN]D֡ϡpk88Y_Ƨ[V%ҡD ȤL9^˸ɞH.8-igTƶb C3_*0m'c& O1XWB_K:X뚛kja޳o=$H{VN.|Klqr{vC^_`!؛,c'y9%9/_GDPc/ ߵU)",,tHwf5-Z|7oIȺ0CC*l!$/ '7uyb=5ӧ/Q-z_ī)ֽʤ LU#[ԏ?юe*̨6O{yJ4X5=yNR{{(yόV&tcX'sZd=IȺu]ՓLiflUaKj[M_/k]3s1 Vu/j_ ꫯ>b=%]2NV2v1WbNŁBݏ{:{3+_3g}+!yn XI.ֵII}z2تj WIZ3ݗ뚙/=v-{xy QtZo>})Z/x\Ŝ]gxR9Hy/_sQ`tȇ?зZO#IA&t71ú?Xo9/FWȉy܀y%P/E}T_+%bLs}$yy/s+:kr%F߸' Y1P_T*`!Sin"}`lץuAه]2 T %:"UmO\mȜ_/I=Ycí:u;oˢP{+w:DP*3syP6w3trMn;oxOy" s[tϽn7Xm&N$C~ΙM6̶mpf2>AX+l,ڒψTsV 0Wu[ˢP-HG:7qQ~Ń r9dŕHX&u@gd N%afmUP<}.s$X~)N't\)r3^~M6mEv/vG66[zFrs6]_sma{_Y>zW9[)(ȇu/p&g.e8=*$t(Eywfam2a0n]~Tv-o]OsUcO='d|,6~R!'ZU> X$am?.D:?З3,'[ײ ;Rt1)'b/0IbTC`:Iw빏a-񰥪g7n]~T=\k/=2_/eK_+}ҳg-gSMlav·n71)IM&L{S"Ib*mUK@_J8!+OH:R:Zc`'yP&8ui7Tw͜_#?0@puSwZIYdJsf>5`IW@8-/@N6u_R;7d./x/*K%}+qxѳ")Iǵtx|Hx]1(ynw}YأsbsUҦĒX+lW>YK+KA^OO/x?. Cgx[:7.B#p8u[ &H8tꀝ -m  WtLml$]:`cn c F ;u[I7A&HruN~6AЀ&I/E`R&I׷[7/%M;ʧ?4_$]߽ix itY}g_읇$]T(2읇$]v/8ᢟKo~$] xXҕE^T,Y󰤋J^,֗3X6C.*dѯSvet`4I>'DP_$W_?ُ*`4I$] ht'?z[c4˿߽Y~ !T#1\҅ Gd3IJR!Y3 qӹ_'sp4cSI,wIkiSw// 2@Kmi+* Rݓ ka`תCqjz>}Sp'5A>D,2?kiSҽ)s -4KmڟO){R {{{͑k&1x+9ARk/XFSGghΒ]w\-DAil B9QU߮y!OEx9iLHT" L܀zrC9Nb]wBme/g|O{v%Xxvګ̿n^xϤj|َ1`bkS:Zvi2p~,!߈]%]d@рlX:C]Kc.!egќf1]mG%-jrNyDy'U4|2yLdOF{^%֍[٧۩.f~{%cb5g=ٌG q\]eQ7Hץvh/Jh,N2nYM3 b5N%KVp y(b!tߛ/˪M=Gfu~bm^3"msOۧ)yq\ޫc[_N̯x{վ7毙%OQMYn1jq__cD߂,s؏u?zt ”L(*M"qHuKVKl[ #6^h/c&vkܵ~vBEX?^Sl.7EUr5 [-皘;8n{zG09Ưe0ڈr=d-L70gpCӰ950*h[~''k ӇEG3vt/ ~{pNB _91'mPļ5ApQBn(<ɞ\Ϥ߶ H jd =Aew%Xxvګ̿n^xϤj|َ1`bkS:Zvi2p~,!߈]%]d@рX^r+p ֤/8L79 Oj>r,ymQv/KH}' J5_ogqz16uB1`dɮuV)rbovg_rXcŠX,,hO6c&CW-dtY c?u)f ;Į.6Zfe;oaLkuŒ2\n yXq='˲jKX&{<~bm^3"m1>Nn'?m§qyme93oUޘf>QG6Yg\ݓw!{P~=n~ zc?( .S621F 6ǡz#-Y/m%7$[ذz|[zYrg zsRGBgAMA a pHYsod$IDATx^ϫ6U?ąnnW .fY$daAf!ĉF&.\E]pՍʀCdȌmS?:~~|?PU]u9]Ns_̎z>I^ @Hpd׿u/U2p ]dw !5Hp &Ca>ն \Ӻ㖺mE//B8v)kT-:Ő]J:_'Պ<=cϑ9^}5OO!)op%Cqjк֖SdbSA!QkC?%N\D(<_[\u M>|VO6!3cs 7PM:yc-?K|l_&!XÝiButQ1laһ5֑Pi#^:ƼK碬ɝwRWq:8%[C}&Ul眝tZxwњҹdZP"Z_:q" fUw.Twl2bn<$ֹө|Vτ5W&t2JΉk'l.rL=Iz>qTl5at`=P[MIe#9̾C%u#Yg=vg[tڅ!>_?,P>dH.>kұXkEj.6.tиْ ?"_g};BŲy 8Q'u': IFتk=+'CodE輶u}ǩ}I2TؐCR}O Yp"+",PZ+(y\ $_Kb.rq3@ }_:i3)lcLL' 8cg,t(O:'_ce$סcUłvʚClIdֻ1_ZH_ܾ:p5O{pݦれKОD&DDNb"F5 t.O{1Q?\nv>uHr _ icÒ?[~>.7sIF@NЧ,~UMΣt?\/W\x 1FduO6uNe+Gq46 >]+n}l#L9fH>=dA2L&}:0ђC/ILjѝ *pd":? %AD@2O <Ľ nlcdw !5Hp ]dwM3BAAAAAAA"ɒ!@> m^A2@>}N zY ;&e$Cttqz Om^A2@>}N zY ;&e$Ct_y/1;9N_g[f? |/ \d2΃7TuR >t͒ ߻'g>=ks[ܰfgq ϮsT^nHkT܃Y'8= =L jo(|fɸs-ӱAguVdtpP2ly [ 7ltb>'cpq`<)tt?IV= T*y+7J|t ?ZcqtRO eg{1'{Gf'qN@-7du{ٜ#_+kw%ژ7t._S(r6L m#wf՘bz (.h>#b?ux򛡷'YSqtNKtN$~vhM8 jbX0O98P,>KpZ%V*9:7PF'#HGk/-IN!/TqF-"OΗ>ƑZkDSV״.}^^kGWΓ!o)S! ۜS//c܎woY%q C,gf8Hk[pW6"KzIv]/ӧ9*~<=e{Nj&q}47)}oSMKU |,+Y S" OK%'eO+b&s>('J˜=&P.|~6MҽC%lT&Kz{~XvRkd|bcҲeseRӓs6x9MA pcT7yyc{N܇YtޏGy_)D)kw1,ڀdHtKerǹĐWMͯyzDCL?\"4MOؖu;ef+6NuV\5$Ca܂Z9vƥ׆@#W#XB*e&l8,YPo6 ,07UG%xz[IŖxSē}N zY ;&e$Ck6K>GwΪf //7f=%CW+O*{{A i,o. 8%C$wƟ̿pvKm%BPbpvKmo}9!s[2D?& ]k2o~LFDZ]!*d)2u0?c Q~TKPNfwU[h,9,6~Bcf%dw !5%C|4￞Nv^7^Zw_ gdrys gbdS@_pC8{*yzb~r ӋWκ2Y+,𰌍ŔA89"nǚs՘swr&v2UGH*sqPma\gFq>P`k~Ϋeic&'7ے$+zпKjNC~ ,?[#uGbd[ X!nF$x9Af%Џ#l)j|S{_|yX7Х3dIs^ jNU;|3M. hO52x߳drN>>ʦ\K[>^\LǓ_DM1(x۰Uu zz1.rvZnO'3\w2F"ԣJF G z)}@< >[kA -~\aUjǕHſK>u^c{aĒ+z\2$dGu(;FuTHld2kx?&ѦS.ebQ{Ǘ2>,npfsuތڷBδ4:;RK5S6vz# ;rPpɟ{"vPOe@z-QϯE߶lfzBm,uDBy/g)6ƶoӎ,6l_w8Xڿ~vd\FK1tv{Bc/ir1l }ì9Vh =:#2^o.{`n%C'C!#=]2Gyؑ풡6O &풡R4RN0ZGHjP 5I~ח_V[v99"nǚss}Fpz?@2T8 /b뉾-f+B U)WA|T5~=)}%3t l%Cd *R #4.Qz^ *@kN^{CC:9Z㳱ZWQg9V|1ޣ,+-ڃ*:5s\cɉd4ɠc()P_| dž*TI$vYl,7#ꗝ׿B2Ζo&Z%ڣDʿY߆.M!{ORP;uAgv~\囡lr1LF?/B7`T`[Չ[ɧTٔ`iNjxQ)zs-a}9Ǻ˥ϛ2<7y=aᏬຓ!6X%#>0 DuDXd[r7+s*U1/뭐K|916ˆ%W'c3dHɎQD{w kb@'z RxD[=a(%XHj_bKzE6onk[!gZ߈);f=d Wc2 xU4QvoZB 5*6zkѷ-ٟPE*(&r<u_+tSlm~ݷb1gg+áxM̈́P5e2Zz# K{ ~|ܘ?K>a+4ƞ "^"]2p ]dwv'~`GK~<Һj=%Cl~||zu|[}u8'Cy3?lƿ_1!8 "spɐJE%CT(!Ҙ]!Rw_V5̹Z%Kd)&C{莿g6/ rO>N{e~]]1VrK\8'G/c\5u߁V[] ~_j1 ma =oj|y3ϯ[{8~F=-P-VNJ>*5~=)}ZsWtߍsǏsqz?͐qXXo('T. ħ؟ /BOm!Id6>cJ}d\?/Ж,iW+ϓY>mFDb*f,p8MF~慻$fĿDOt4^Pik|6V*,Z{tLnќ':n-yc(`dtXb"c`s´I/>qL.cy XXvzI&׃Sŝ tȮHuäk26k#4u=~6yUh|~Ϥ[?|ߕbl'J~iƒ>9@׿BY\-PM۫K ]z?C4祠vThp%~TV*1NdAbf*w*֮]SfHAKvI&mK =פWٽWuAAĒm~ϡ[? ōǿf9'N\||cyLX#\Y{%Oг4Xv)llk} {OXg#7drNJ!6|_f$Ly4ex%$mEJG>-[K eJJudIJoWmԾo߬>}[b z^N#qB?~l#fF,پү=?Az3u2oGT'=[ +;8:(1YIf;ƈ@”rjˢXgP;Wv=A]nviUyQ=ߺ09lw@ݿI) ~񥱌_suI7hƺn\}_X:nSBl3eaެǘ=?`dJJc29~@-'eFMCKSL"eƈmr.hKɍޠP)50!g~HHl<|SHD,wWn/~ Cӱxg*UPLX~QHVXxƶU[ؘX|v(_x%*l6T}Mٹcz]!ݠjmv K2$m%C:Ev88'Ev-cdK_^*opN6K.$Ck A2歷ޚ`Ցe|QfIENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/llama7b_autotrans_8to4_train_log.png000066400000000000000000006151621471400134000337620ustar00rootroot00000000000000PNG  IHDRe@sRGBgAMA a pHYsodIDATx^ϋeז /G%.51 E4ta-"G1iOII4ȁXC5# m^A@ x 4QvtCN$T75;}s^{{^3 >{Zk{W1?￙?4=Oj\x=z4/o76}*xz3TWL_\m%.˫sG?+?^6ϗ}~&{co]w;[gN-Y>3[`owgp9e[^w~w>1wlo;7ȷ!\P~6?D|Ĺ/fϙ8`]%*?~8xNo N]'<؟-c ׿D/?[sp~1gZT!+\s̃\c;C&EƝ3_齘9p뼽B{ཱྀp ?eA3 ߋw pR \IAp '͛ͫ+}7ӛK뾳y=]`%WMkO=dۭGtg|pL_~N7K=ޭ(mw1S;G>}?.0~hK>SM1q򨜴7ݩ#O-fٚSsT:X7O=pzyC2x}xz N?p1#׵?R9SXp6Dµi8B)Vw8 ,"F&*+_? >*u=bjWP_e;ݣ#'"b+f~kqu5VuEl9W3f=s IN8O}KvJY6&/^+r~V=WLAx+8 W!r+b[tDžrR씸b3iLwKtj ;?pµ۱QGV1+YM8QfٛΫWmO~wԫ(}dGYҊ1V1'{<}Y`}z]xC~x~WO-߰n p\B5몵?f/VɆINa7<3)Q<~ψص9msb8Y יZµ/W}$_޶u>u<W/\7'x9eNBu/-:)ZQb"܅ه4wŤ(ŶǍOBQݖfa*ڵt=bpƔoW1};ҌJ-}3E_h=nπFXDzv,.~(瘤[Ϝ+9NgGҦ,ގ~|]^y`+g!\{Ppߘm5!߷C"gEk-'\ ۶EX-?M"6uouB{:uc!j wLHēL){c1(qXoNQ3M C;K[{wX;X= nʾHtnjܓs,׻\kC -w{02,qDMRWSRl9pE-9޲mj2^LҸOr"]:µ!޸vv:353rOXDCbK>µ3[ }˜Btnޚ3!V?Kʩňw8hx=izm⏧s oMaܖOo.n⏦S`N*\!hٙߞx [0pEN[f{txCcm{-q`Ysµݳl%̵k-:Z╁l[N_Ya>3x\Wno8xW ޮw;+ƭd*UƳR1bųx7_^noێ??vC كb<0q%=7ϴ]p8'GZ- ݹ!o}#f#"lmz\Ex3yȆw.J[1="\9 [.{N^l4&/DbN!9%"k[iW`xLkrTg/8*D(xg |ՊhW1mﻮp_ΏWqdYq6ݱ^~9+f8h=szcmXԞ9fJsijsL[uˋ?2_/~3AgQ[ 5ra [%;qYC}ķ{=kDuH5beŸOD:h3v-V6 D?}-\kUKMVPՎYQgu! QfE,B7_q7ӕE`c(TR-0 #~޺PD&9/* &gQ-f=wcµ#kaXG6+!Ӛ2)@>Gj/z}*ZW-\]K5;:c=[ u$&7Mj)ۋ'x/~ sLO8!]p}ynX` ;a[~ƿ.|_w Kywӳ=׽]\abyqMyc3ZCP9g akpzh1"{h:<~wnyFDžܖusl}\hwُB~CeYzp~ \dEQGW>j>w^(lv¶{;SoŤG8}AZgb-3B" 9l|z~#H|}OB߶P0f ?N[R.vGSOZ!vx$kO2"iZ9ZRn] y@sy&p} s Gkmݵq-Bl?y+)wY 5 'go/ {G[W|pu.w9&9r`mw̹NP찼' ŮE.kb7%t^ɚ9,!V3r-&1sЪB⇿V1D?}k;zZ-m;TnG} ~{3AR %(49\0D\JJ*D^qw>$C,j?ھbK[KڔHV{btZs{jOBlb[[7y_ӊZt{uwu꾣vb<3yeB+ϭil7NE!ĂgC)p7A8amp8_zcKCqT}GH@يw[M:h^#H{$A`eAUPej-2i꣱9",$_-z'K1v^^"Qo_Z=wn6qtXszX5YcwψTO{/z;}Ӣ3Q?fdehG֊dzTTR8O9^v;a^a?߂.>m9MQp0Cp@Ƈ{Җb`g@!#K|v[6M-Z;[4e[0?5.B[tߥ}| ReϋK5u~L?TG۵h1O{}/2:\= k'1RɾUw˺Cj>ϱcp?9<yAԒf&zkg2~!>O]|hn[(ˁ5_2>)[c_"ǰB|ÿbKb4\]Fc!/|,G~}Z{vϔg+5]Կ_kK䫾aرn~.,\/z59]V>Gj=Y#@͛ͫ+}7ӛK뾳y=]U3o/.UtÞ:C.럗>S/_ni^l'd;Q꿖P0Faj:\ op];̶V?8֬(O=b>q&&dcYw)hC|<:b j-žbbGZG1m %5%m8VRPO1/{7>/o~?}⾴ZiZ{p4~j&ɿsM3?' sy;ZsR8N'\h|g>Dd}}ݣZێvO??>z?3}]n/?rc{74o=ޤG+I /mߝ:j+[un{`?L?هe'R5Y?8Ѳ.Vuu3]Ym}x~|9/P}?w'^ȶ,<0BBTp5q[ y J*!n+>+Wо٣qr{OMܱsY{$^5#-6fd'/Y ~q:o⣎>$mA#KlKҾ[W 7ZoZl[|k$]#F:?깷s~֬i1۷H- x&z1}DµiC) w; ,Fj=~ QiC,i]Wَ~h :V;&.;Wj,6(\#kU 5,zv$X'p%\;}%,hQ n'T~q5]\[׍p o7^v߅/>[9l=4`x9LWr`n&yP;&PjDX5\_V\Q!%ZKr_ hҮ;]-j~Z-织߱r9qkQRmr_+#WLvx.}F4gQJP4[kEnXΣ׿x3iLw-:( /ob FZOofu3]Ǐ ErBH#h*[Gyp?&"MRI"ڽ_Z>򣔕߳('(np,N|[avl[

zޣYs4fs퐤[-凳}[5f@!D-2=Tz9idi%_cBL//ݖfi|z"A3ԂnӜBlɺ S.V`>FPU:e k"S֮e3>ybY;鳎AϚ<ǎ>Mj=ˬ6z Xct~C~:7wzko6׎}k~,1 r}m5],[CڡzwwM0F:`և$P|z1:ҊCp%bO2oѾ|JZuEaXl1l.6*ы1YjͮVx4R5#iծ~?jmR~FbvOȩVc=jf\o9$C7Rx8 BLkll -n.9=(D Xu}߈ ێ+`<6~g։) m~8G?MZW$uO0fTiž%~㽚@cM8)DoLU)7opj%\vDz+7*(V7gZHqjoT}M7N 911>ġ\G۵|}%e\;]Dlحq[`0XBdp5l[ԉz+ؐ|m[-BPyChLkG}U/5jDV~{>H#kkSbK|V$V:Ooey{B=|=<ΎWg`a\/B|{ugOT}M\5`闳y-t  cx~On؞P`M|܍<`Epw<"\zBrǮ%PHҡ*VBDnl;c}BFzyLyj jc)EiĖ2^%5vHwB']uҞyoӏMV_K3)vd7jL^,hn<ǎEyx\mz[ON*Vs:Yj@rR:Ԇ FC̶ސUc딢oNlãmG;,Q;4WaW vw]q=7gJkZu/vp5q}fL3so=/Άukkocy9"ح-ZjWnvޭyW7߳-_CH`t\p-ÿkkN}kFK=T^|IY>]bg{M$|<߮0o.lK] aֺS}[y[-s=MObcQH;ſa~tm+9fZ9}^}֟7)N-UGy6vOeNZY6r2*Z{(^BН"v!DҾp]q:5^ݮVv;n_C>+: ;IdIBDe`"([`ElID9Kd?.µ|z9*\Ǘs_ fQxC6Q1m.<KKpUQڛiE.~׫WSK=v}Ms [5*gf~cN޼.ز7?fNސk"kxqt_j:to[oݟeMlG:ᚠ_kZd>#u0>CM m)>m;5ߤ/@QM/kDB[wWͶlBD45hCӪ:m)[QYƴF1u${ߌr:Ys؟ ,ZRj)^WԱ8v8Z>HN$\GTy)V[-_AZԌDsEfwaoZLl0\4V`HVC~/5ZuuvشAD`F J؛Ӹʛ|^It:յ| ⅟.\{';.ro#wT YO]p⵶{kZ Xk[W {h_+|+̨92i+/#^lOuP6c֩) #쏅{x88WۑC~_㠶v=MAuMȬ 5k%>R28WyC5ߺ73S=9/&3,{j}QĖ}{_*^{dڠoV.}po?|^`LLS{u+ne#Ҟ5y5Am IHEp/ZeR맢u`hf']ߖгU(4CH4"LIlZ皟ZYbRƎž8j?Rik`Ƃv;ƱĖvjjmKmY] ;i?}v?SsLr: {?4KDOkO՞uѼZYh A;Nzc ogu|VoV؂?}p#]a#[~v`t;Zzc؄۬C!e 5QI?1p 'N 5k8)pR \U>^y3yuefztiw=U;juvium?n+tVr5[?cjK q:GO_G>Ty8}&/|'kMv;|re>~us\yXyL&՘8\yTNZ|iѱs}T*\_M7;ۣ͇ӓ0Kg~pϧ'yK5ǟOgmwu:`{8a:G8W6x+`p}~wWONa?j{u38kힲIV߇p8 W!r+t[tDžr-v[Nfbm/G5ha%U#Dn\_~};~W0Jm}?rXMqډ&ͭ+p-? 07! T6>nCy|;>uixkmD^ / Y^!\?|ú)t'ޒ`먬jLD$oO-sukڛm3bsA= Yuo'LViGCxka$fm~umwr~>~-~sźWcC #vG|k1XUj̾5-~~q>7no\k`NY Z\ o*N7k'^|CwQõ,йzM]{)\`U-zC>ݡ?B;ׇֽ4f>4ׇu<$|mC;Q؈mi-~k>OSȷ=bpƔoW1}3/m]GQj8ۊ}T9 c\b[Pn[[{sw}Fc[@=SL7V̚Qv E<6r:٭\lמrr_Pel0E~5uй wse"R֧Cb/S3 i9 BLkll -n%9=(D Xu}{/\o;~c$bl3Q|#:b淎su-lztKzgZؖ}w>5eܿe{#cRcc YRcՊ`7οr^-T u˶aVԡ`M]N#msURcS wfI'xVyB('k\|9p\Nd&0o!-(A2lweSžnGm¯w_uN7z w]#G:N vql QRCm{y 9޲mj2^4߈ m1o.o\;;^JajV fS+Q* |lٶ"cZzM$ q؞c>{Ɣ5Z{ 7bz|jg\ljl\ϓDbveʦM;=fNo=_>9C osLj _NI P 1z˷򆬤[}pEf[-\o=:`uݡ1uZж{ښvqG{!<7 P3K͵k-:zb‹-{/\p}㺗-lN{Qb;237 M-|9[y<+~NÿNg_B^nnkKzָ<{.UTéƖ1Z.fV;c{eLx9|?בk 8p]=ZBnovo }M6Zk:.ZǾC6|uQnלq.-Cލɣ Sjr([G0X)^C/Ga@}O/G}rkAԌSV_d?ۺGA uzFϫb1.Gvkoq)/*ٳC` j}ay^B,Hd^-j1[f̷庌d~}Nv^|Tî=;+80w>W9Բۜ/Ͻ!cbbEo3DuH5beŸOD:Eh7p3 lblesb~/ަӶEbj{<ת֮kS> šr|vjk1@k us kȅ %wT YO]p⵶{kZvŘNR-Ujf۪iMmJF@ʹ͜l#?~ y\床'Jy1PwabS3㟡9K3:]ϾԂ:쏭k,ؘ*vulBݜy}d6a"%B}`ļP KTr!б9w\/pP/1m )<ӭX"DaPpqZX1«aO:b>]8mDk86oo{yP %,&MbyϿy#@53@܇Gx{wNܿxH-Q|ty݆Кog^pE{5k8)pR \IAp pWzzՕ>ӛӥu?Wk!8Sr_׵ܻ.oV[m۷I/.6M~?}Sho{4[kQb'G臨wm}'~\alqaі |0ڛTcsQ9ioYG:v+pUnvG'?a񗜗>_டOO.?חj?=> u&qð=uLq^mV!vyV9p}!k'`(@/Ja0Yuga6PcWMb}TLMzԮvGGNhPR1qىjlW޾' WcoUGQkתkXG+A`Nu /z.[Wb̂E}%1>:g!f?}N^?µvO(\Z5G+Al[q-f:c9O[t5NG@w-:( /ob FZϯk7q \7\ˏvL@R >ͣ[^c!}: }RzY';lg~Ϣ]!\?{<}?+'ߖoX7d[ly5bCI-}v b-c[{Ӳ-b|Yl.H򽧟$Xݛ+ VEZLK?Rs] #1kSkܖ!k㭟+ֽj8n9V+?R=ٌ~Uj̾5-~~Wno\k`NY Z\NPMډ+]Dp]~tD 5v]pF=TEo?LB;ׇֽ4f>OSȷ=bpƔoW1}3/m]GQj8ۊ}T9 c\b[Pn[[{sw}Fc[@=SL7V̚Qv E<6r:٭\lמrr_Pel0E~5`zf/e}::~)&ib?>Y/g!\{ASߘm5!߷C"gE뚠 }Wjq_M"6u"ouB{:LW>pŁ%XCja$Cm^kAjq\'-~ ܁7QѢǼ~vi ^oO[µkG^!|9}nLVڐU.(/-j1[c[[{SQP֘ߗqpI5dq^KU+ݜ㟋{P3H7->y[Q5v}s;y_7bbUKM5ٛ'F/>*r8RsEu"sA:h)DǶ֬OEM&fF!}oUF9" }9w].ܵGJ^͇?0|p,Ł%+heWc{|$R4wS/#/U6pgo1Y9=&0f<1 8pjCb!f[oVEȅrlR ©3mqxpx%v8 a#[jn27o\XBOvu@n^k1c^ h;8}g;׽nfuXjۑᘹVn-m}iڛ^Lٓl9 :m~~mW >{ae-6Yb)vVfS [dk~YJẌ$鼎\3nɄhѢzu{ =C|Cwoۅ̆OIumz\Ex+{Ȇw.J[k-^zdC_sN8䟯[Zo`x5NFq}Q:~u>Z1H4D"u?= b;U3z^=ێu9[{SKyQ9ϞSPK ۰__O "b1FD/'{} j!P:\7l7c-e$svY]uŘűu,{2*&^k98/N$\GTy)V[-_AZhDsEfwaoZLl0\4V`pVC~/5ZBᗞ;om]J/Kp_7ӕ;b<$A@(TR- 51f }Kbj߻1km𵰋^bL'*5mUGضc\I ]h[e{Gs]fNjܑ]~0c5Ʀ}` YjaV.:685'k|+Z96A42puKD\[6$tm&VL(Edؓp@;Nzk bj;۟%n;BIx8I|Xop k!+!}DrQ!ޝ22c!~K]h^!e9ٕOnp 'N 5k8)pR \U>^y3yuefztiw=U;juvium?n+tVr5[?cjK q:GO_G>Ty8}&/|'kMv;|re>~us\yXyL&՘8\yTNZ|in֑ZWFLOQfg{TypzyC2x}xzN0\loG; c]9olXϜGoµ? $\{Sr" B$ 8Q^XX-6PhKbjZ#/vU=>rBNUc38M~:"\9=:>v_ sN~sqc(bspt,YY1> 1As{&:K_7ۇvq&EC@oV鶘l :Vc~+p-rLz8vԯGyvߋcb|;>uiN3gQNQ~uݮ=o7By-߈\lq3ܿe?OαDel{vkoZE̜/_/YXݛ UQҏ\iH>+:|HmeZ6xuƆ5GʏT9}]Gٷŏ/ ,ך1opµT+tSv7t5\M#tkpg;Ow+zhkăiw ӃP}nKc}u o^Cry1E׳id_ǶǍO<|[rFCkn'],k/h 1q&xH,hY`]4] ێ+W&cُ:7:!׽og&\KK|}ht6֐y-e^W%/56pgoZtҊ{S6:F9~9rr: =R`4ߔ c[ Bk' "x\u, ZPUlb Z̿zvn;.*@E. ܽ>8L18>"re+!Zei/Q\ļE\tkӓCq{uk +UTl[5N9GZ쟃Nc&Yeۊďk%l'!gXO1e|n!ZYA3-۽;$jxy;{NYOG?:gax.11bދqR:ԆFC̶x [}pL[-n=:`uݡ16Z]{vqG{!<7 P3K͵k-:zbba}g;׽nfuXjۑᘹVn-m}=$f/DA'Ysub=?F*}uv3^[rmԳ SwlͦN5v1b=FuZ_k 8p]=ZBnovo }M6k:.ZǾƛC6|uQn\Nq=]8[)"?\OQ)C|qQ8nku,:ma@}O/G}rkAԌ"5g["46AT]y\[l;b6hnM=.E<{vLA-1o~}=6/ϋB߫@-frݰ݌\q ֯.kϛjص~gucv!ycmΗې1P11_yq": LTH2b\'h xBk]X`d+- {A7}-\kUKMVPuVYPz9d\g^ܡ!& z^G1o\ I$\h#wT YO]p⵶{kZų"ŘNR-Ujf۪iMmJF@ʹ͜l#?~ y\床'Jy1PwabS3㟡9K3:]ϾԂ:쏭k,ؘ*vulBݜy}d6a]c56S6uZXbj M6Nɿ x5}piMLn]GQ=%2WV/ḵp7bWG)"Þt|5q[ӈpl+Wd,qܡJ›YLx?Fkg \ X#ˏ*- [%F 5(k)?2up k8)pR \IAp '͛ͫ+}7ӛK뾳y=]PCpLۧ侮k w3]Ym].L7Oۂyo]P_\m ?.~:٧3~;Y{,m_hzֶOOmL}.yOܯ؞â-`zO7đʣrBKwtkNVUnvG'?a񗜗>_டOO.?חj?=> u&qð=uLq^mV!vyV$\xM5w N"E(Ydb,0EEC+G? >*u=bjWP_e;ݣ#'x([_56+o_䷪(bΕs~n ׇ:>v_ sN~sqc(bspt,YY1> 1As{&ZqH[q&EC@oV鶘l ͣ[^c\Iޯe-Oclw=r[vp,N|[aϓMovmuTXFbCI-}v b-c[{Ӳ-b|Yl.H򽧟7K_iGCxka$fm~umwr~>~-~sźWcC #vG| Bٷŏ/w~b|kXqµT+tSv7t5\1!m.\Y_&\`Q mM8A(qb>D1:7!<ΘŁ"ه4wΣZB7> <|[tsݲcu(Xk7uH[))&\TÝk!|I+5Ba}Śv5~q9rr: L`4ߔ c[o{IOv[7Ecy_6Dւf{Kbw<(||_tYwp!d^P {8`Ζ`!<<-[ Ѷ*H ~"-ڥ#\?\kgGޫ}][(ŢQm`V;bUkb;юg˖m+.?6|Dp?Ik>VgLY0[~#&=gȧ*}Vuvu,': 7b1 /p^NP:u,Gzgp\+l7햶3k^[ ٓl9 :m~~mW >{ae-6Yb)vVfS [dk~Y^O7N2/gyf΋ UڣEi-zl(X }M6k:.ZǾ[C6|uQn\mf28`Ay8^Rn|HP{>[G0X)^͢vkrTg/8*D(|=e5hkZEh~d{ ]wgz-1r4lr=;ܷa^E~!bT_N @BuXnn|[8HdM5s1cǗ116sm{9r": LTH2b\'h mm-ſ k1Mp1Ӷkj{<ת -a=:лpIv{9frZ x^(TR- 51f }Kbj߻1km𵰋gE|k1ZUiӚbR>s&t9nus9vk{sGwuO̕()bk3"Ŧg?C?sgu}O=fus][sX1U(.`S 9jmÌ&UMOӳ013+U\tljpjN ܷ%,@`?Ǵ&FtN#b*b܆CĊ^ {pIoM#ZA\~uGxS޳sZ( og1o{}rp 0b>H.?8߻sBPCr7o⣫65|;[ up k8)pR \IAp '͛ͫ+}7ӛK뾳y=]`%WMkO=dۭGtg|pL_~N7K=ޭ(mw1S;G>}?.0~hK>SM1q򨜴7ݬ#s.U#'un{`?L?هe'R5YWcGk2sbݾ{]z3~qi:N$\;P^ag1 bkMb}TLMzԮvGGNPR1qىjlW޾'\WcoUGQ bY5A`Nu /z.[Wb̂E}6|Z՗B ק?"\G+Al[q-f:u9O[t5NG@Ew-:( /ob FZϯk'jL^ sȕ^XĒg!8qU>ͣ,FiE8>x|; cOj~]/kyds7>#߳'(np,N|[aḄkUk+\n[''ݾ#;LvޓϤA߄7 ׵}oZLYrV^3 (Z𰮣XcopgɾZB7>AAt[֣Ej??>~<µS]/b<ztGopͼ~u&*^7wg@a#cYZg?4#߫,9 {3-TQ}[ϔ06o>'Im7f7g u ҋm[!ʱuJ7Z̴Q6ۣ+Dg{^c~K\-mwP㜹pf,[ s-ZZh;8}gk̈)}FcwpJqvx߅Zӣ5M [F,^1[vsj>q2*Z{(^BН"7>m="\u\}lx;ݎ9jn+(U A'_Dg[\s-ҷ`=Na d@}O/G}rkAԌSVjE_䘶]W/pǫ8Go{꛷vX/\/k>f?zZ`{3+nfqL+1Gg9p R';Ns": LTH2b\'h xB3E4n ^Mcmq_a o>RyU-~sZoj!y"/BQD`HĘ|LWND*NbnAu-50=,ALξZ{DŽkG[.'N'Nm5o2a~=ϑK96z=U{ؾ2y\޼ex ,s,8kOS͏ fsߑ?k싵foKss/ZWh?F)Γкe/bE\=1g jG[s-{ܽ A*tyfC3 ;-!IfqAiB8fVIkC)],xjxg7vL71Rz5kwUvDc*bM)̓֡.㑭{jo=Q~NX?g2%5R|A3>c_l]/π%Ccp MůC??حS c}z꨸8Dl8g2VbL/5Frţ5H!-s$'[bk}͟$z^j!8=yO,uY^׮q"kk }fq7p߶?&&snkhO/~7/ۋ_Oo`ׄY`:ZW ˾% qG&5b NmOvM3V۹kT֘q-L}lm6׊R nQu"z5^?/=/ ׭ugNw.\'{\Ҧd6ĚS@O/ܿ]o]KoHZ8!&षֈ1P1p_cz!yKB3)VnӺ1ۨkڴƮECşL/"FA@io(ZSc0G]0IqbD_6&ε}[D'}$f>Kߝ#|.xE=wM^UVV|k%i_M_wwkm;J>@dwOwď:9<,ҼT{jLy<*'-tM~7H>*G8vT'?a񗜗>_டOO.?חj?=>V{+7ӕwTT!q7RJvݝAނ8q#I\Wz?aбp|tbk/uO4u䭌"LoCDϦ>6s%9k=N$\;QP^-EYdb,0FC+F? >*u=bjWP_e;ݣ#'x([_56+o_䷪(bεb NE{$X'p%\;}%,hQ n W^O?n5{`wF'37Kϝ|G}c0*(m7d^qr""ElKs$f~$5iű^ZJt4QVZ֧cROݸ:iL+g[>3OE,[8,ⲴM$(u:}ځ`<|*ϜckOGwe|= \mq&EC oV鶘l :Vcʛk1ݯ_\M? ȷ?=N@zu5]VтQқ wd^4oF2gJQ~u Y^!\?|ú)tQk8SWհFVo}߲'_~,7GMe{^?uAsϢ|=>$@;ⷈcQRip_EzVPezEOJb?CM0.)kdX:@7 r~Xz,Ibޫ Ɉ`7o[ٷ>{BYe![ּKQu~y~HFr8J^,m4rhZcgW},W?$9KZ q)\;rܦ`:5v]pF=lKkxĨݿȓ׺ds(BiwF IgɾZB7>PZ4RsTou:-gϣX=.\1_L"j8á\ оۮa>k$ξa/yE_jm7ž#{^{VᾛY m$@w th#ZaHB{(mebÿqYR 9QZHG%\@_ǩo}ȑ8Ys/r.kh=xq|k+ݚިy?[q߬ݓ>hfjK nX5[ĭGmL8p\Nd&0oѱ-(5lweS,:m-]-*jk-_~pw~oȃw_uN7z wmB}E7 "SLwkk X[e+!Zei6j~Tp>yvϧ'xّl_$05a+_}1Du˜\,Q̿`>$Fܲո>-u[~ȣhY sU}| C_-k= 5'8,[DV놠7*lHgVb=5@89kKc4(XYkyY#n˜_ۗ웷Cm)r?֞ HId.vּhI[>8p>Z1j׋ߥpUxY,v&oI P 1z˷*B.c딢oNiãmG;,Q;4WaW u[3Ue+a^k11"\ M{/\pw\֋&N[ףxDT°Uׅ庋o7ǖ@)@)@C6FfVq>nZ'$܄`1:VG,bt ?:mLqk[Ķ7k7ә7[罝0>pMV){`- _GYKi=/FM955vuW_'jQnlm3"\6k%iǣF/duUhQZ !;7DoD7}l{Dh_hoenEwz+7_sŭ 9p PC"!"-╽[ƅkKd?.µ|z9*\Ǘs_ f$PZEi{uuM5a Pfbe[1.%oŴEڏSgǴ?m'şC ?R ?Zz(^nfwaoZLl0\4V`pVC~/5ZBᗞ[ߒ-I/D/*fkӸ ZEE JQ-[`]u:Xrla brBlS?{>&\;xvG(>ZV ~,'Z"_={k*N;n )֞OVj~ l->?4W?]z/~SO2a\ܖor&y@>[ qˋ?7n[M_ bAMnF?">kůbE+o/Bk=&Ku}sl {ZD!1_޶R{XAK\ڊ#q'ƞnO&x|Ko{>c^~ZJ["h,ub:^%-ɿ xߌ8Z_D1iqf ^hvo[}?U?8x_C_i 7R-Faζ2,JbȨcYx̬xY,XVrZ[j/շFZm,uxNH}DGnPv{W[1g'#vkgͷg|>lyVlgm;o4o-&F94)?>9Igcp w45e[͙ߊ5ߘ]1m/^Vµ Yq 3mk;{-x^EՃ߆QzIt@Flj$'$ֽ^Z1 - Ns+0qoE{mѨņ"4fF-%[o_du X8譳qm:/Q%} 飲ߊ8l}9<1S YKǗBc 83&ҽĕg=feֽ0e`Rݸx8>KN_ K2'2FwNZ.%}u"ŸceoχJ }>B!>S|J4jZ@\[Sg:RFOOukgN~?s1O"ˣky8pվ5^l 40*Jzɱ>Y' b\Cc,8ĺܵQ3-XV_Spk!k 5|<6Ը>tޗQ`[KZǚpȣ]ke p \g6(\S(Áp 'N 5k8)wϯoLo^]<|=]Z}s'jq{ڳrtjSs:GO_G>Ty8}&/|'kMv;|re>~us\yXyL&՘,dUNڴs=2wmG-fټF9p1>_r^>з~>=$^_>^?xf?k[%/.{WvLWQAksDµiB)Hw8 ,"F&*+N? >*u=bjWP_e;ݣ#'ܛ"b+* \GElf2{%\{6Ÿ@`p˟pmF/D5^y=Xjq;59s&EChEXlp]STt~;m5.`V\_mwcw= Z{uy ۾VfJ\7c_bg@qګ26DŽWĸwBDp'{<}ۤYL}z]+P5ggqzp Ge7v7O]=ʏVrokY}OZ}v|]o?mllXR~@8ORtՊ8x9 ӣoJ׽m["^W~uף9VxyZBrv6|VCcR}f 'E -NtBxJ"c]GJG3(|[p=NؖI oC.Qsbxok6i N?{qڍ)߮b1nWQu|f ‘B(v7kϚHދ~%⮛늶fQ]#\,k/ 1q3&(DԦ@Ad W~om"_$bl+vD9 +'М@aN"*:{2BR. Ex?N Tb>o[*e;Ģ'}o.m iKab}?nN$!?: ]}ZbW}rR:xFC΄[:px%v1V|]!qr-v>Fq-b6 n3Ue+a^k11"\ ղwpµ |yj{ŞZ~ bܹp=j{9{bSa>f: \3'8Y}XZ{"qA|Cwo[߈n[ךh>W`s*F1ݎ#})\GA7{elJ u1\-تT=Oae@}O/G}rkAԌSV V߸Vcg]]zS C Zhػ>VBv̊:+ !8 1ʉ ,5tLWNDZb1[P]Kͷ,yWz;ج}LvZ۽5|-p-Vբ5<$OU$VvrMyo9S1;m[K9չ=+] nZmpnN~tב>}N/fiGӻ~ 2}~j{gv٧ϻ_~~omi{I5& }6\]cшY fG58NO~/9/}ۿ]?\/O/z<}\1f?Dp]88'(D@/LJQ0Yug15я]!8AmQ8S*M9^CJ}e'/D'klG[ƽ3vcW]f(~>b IN8O}KvJY6>/.s4 \%•h^m UyJSo8?^~kÊKz@~iѹr-c5cjX8fvoNq"UMfc*yY#uo?:Z,ẝXԃ-\\-\ P),IX-ur;li!(^>}/D,>n|дVV)EpQݘ/EXNmCŴX-8_3a/'YiӖ;Ѿ:[q}=bRdOX{?#cyvHϫSG8 ڋBLLll -&)g,BiYhՄ3$m"_$bl+vD9 u&r߶p!3E؋"Q!ha :ůoIG'-~ ߜhl2Q)=[%L*m iKabjd]w3-T!\{׽i?eKߓRXs񖭄h3k?W"ZiG/܉yKGG~>=<ΎWg$2R|oE7k=6y_>˞}GVBN̊"\1Ƌ YtRbL.4fr/hQ̒us k?oŧAtc}Kbj߻1km𵰋^6aR b^^߯_\g z5iM|,{2 ޓ˘4ֵCĜV-LD>gSi`ON۷%,ApOhjŚN5q[ӈw 6ۼ!(#\p" -1@p 'N 5k8)pR*_O߼޼Gyz3z;D\xm, ^^W͹<_3f .L7O{L1X;Yc!Nݽoκtgw{~P o[}]͹{ϺCKقFp8p璘#}<>_wwkm;J>@dwOwď:9<,ҼT{jL2*'mڹNǶ@ڹL:;p:Qf/>_r^>з~>=$^_>^?xf?k;2ف>n?9twP|u]1l2!io )YWQ-ۊ`ϓ cưuU~w=c:~<R6{q qY6[|Yus(w#HvB0)E "cgE(*DG?v1!zGEമG^L lG_{4}{e+* \GElfdo>؍]QGw >;W_\ܷoĘm3?|!=xD>skyo߭k'o!\?{<}?+'ߖoX7Ҫ@>t>e+|ч3i?|qnȹrH~umau=7-c7+ֽcq~{ynlY|JD0c>I|Ҿ[w/2'JUj-Gw^gw7וi7i7]kS3qI%սPhρ::Ѷ/Kqcyڊ5l+g*)mT}mu,yaW&5/u%uQ;39oe9[W[U>1/vSn =/]zrYR@@;ձ_YlZ l{] EV_sl;{׍K7VNkQϞVwV k3:' j~m:pdr/:s9?d|MuǸTЬWTܼr$? t2_Y*u[Ҏں-;5smVpO}V:_[%Mz2粆`kdX{4ZwF^Gu]3.ef:q-;2Y c.]|vZ?~ul1΂>v[O{aOeV2c ul~epsOXGv-] 0S̬ˇWaݠ/.1ԝi9 ڋBLLll -&)g,BiYhՄ3$m"_$bl+vD9 u&r߶p8_0/ֽma8qE'-~ /ʶzo.m iKab3E-.\ޥtS3RG^Ǵ-X6fv뒶88[5E|tێʺ| mǬ1ۯ_\ve|w]5Cq1s ;K{_>+V͔$:a̟39.0ZkyИ[vFX{V}4h!:32٬?雟GצWe^v5g+3s\/luev|s~7o/u~mOc0ȵ7~}z|se]_}UץQkoroķu]{m>+ {|&u):qrg~k>fvZ8:~~6,,3;~QԊѽ>I_[kՙ&Us.kXs5}_‡uUsi:o;|^9^rcgZ:-?O#雋`||Dʸ1W>I Zb 1hSS [~kh lw>Xzoc;.bcg'ns8#l˃/ـXYZckcvÂwVxX>w\d0|tja9>dC΄C|6lwbVԬ3.@SwX{ge>WV{ڔ1(cPsc-6s#k+ U[zgvc]V1kz̶{s][j>jFwþS^Zܸ5c{rLfH:}*CO[cẼ]s ۋ194lɘЊGQ+޺J?H7^3/:Q_[KqNYy?0*ss{[e8FjoO|Tϓf8˵g9n֪I;_^>7bRe9 pLd V-Jkw\Q7>m="\ǵ&Z{ϕ56ʷ|ofkxk?0myGb*~~A7i/~_/pg:(~vu1\-تp:Qa€p<^ O/(ZrkUV&5l|\a8H}]~XDbS=ssw/Eh=+S]>KL˃QN|s;*bj#iIz~}"nsbb̦,&YHsx齡mE'w=N>dqik=淴w}\)lkixϳO?o=>浚H6*|yŘ%~nv8w\/nߖ KmĊIWw8VugCWxes{JlC[=O"wgۯ7~75q[ӈp$A跜vbĝ3 ~kr#\kj9?>nwkGp8c>QbkwQp9A}N/fiGӻ~ 2}~j{gv٧ϻ_~~omi{I5& }6\]cшY fG->_r^>з~>=$^_>^?xf?k;2fC.\A[c\Ik.kѳcвΎ?ngږɇZ{O= |h^gmAϵ߽gy a3a&vsl~v<󟏭DµI) p8 ,2FQ&:+G? >*u=bjWP_e;ݣ#'x([_5VX`Xzm>2(b˸7c&|n:KXZdA`p˟pmF/Dcq BϞډ r*?N-[|vu?,\絨5;Juv(^SgG<j>opݨ~-?RφoκL>*\ `іXMq;%3iLw-:jq Dk;.2;a;^3fyV;L 8 Xݡez8prTglQXyd^4B=\>o)8B~muS> p CloEyϳCȷ{+#D9>^W'?>gXE)zXb)t\{O·G?Stl]癴]Kx%s?ST23̇&nͿKNm̗ט/Ѳ]]W-Vks1LyIqIal[YHmvMԚژۖ\ 54Zy Xծ뷳D:muu>>?/[+ձ*f3~jJ9˟ӍuGl,c}Hvo#u庬c=M9Ԝp q)\;r@T wEu#G {ۮ |ֽ됯GsjvWCcRp sݶp&`Ń)weT0K6:h]/ ɾ?b-%fpYk>KPh N?{qڍ)߮b1hZ^>uQCrA@64/Of+뒰kgouש^'nl/] *;7b:t>>WowFhưw#'M'+[XO3 ?num k*DLUn۹NZ1\蹳55>~WQ wP+XCX[` :]1P;뚩ݏޱY}\:?Os]vnuc: ~xj ΐ#kbByXo+b%Y;9/VN}|b{l֙[K]8 ڋBLLll -&)g,BiYhՄ3$m"_$bl+vD9 u&r߶p-a<`LX*a/;XZ+'|:SC5Ph]µӖp-yWG&va(K$ηb7i|{ٞ2q6ud{Ok#((ks9?5J_.g*[wl~>8Y7Ob[ا)'9퀽Dsrݱƿ\-YŶo:Q̗לU+h#}+4W[`v{] c0s<1+|m]J.5q9Jq=>Zu/k[Qg<>?}7yT̝ՍڛEܜW]1p&\7ex[^IEMYuͷ;u +pw~o[3vWF].ܵ!z1Bµ?ZāШu<qZCb9޲mA\S+FzOr"]:µ!޸vv:[Jg'=7O1?:ω|9Nr⃻~sxl [EvZvּzLs#ξQoEeg> f;#sj=\碶*&!O]Vgvm]aLu5Hc{Q-bVւj}(2qr}~nۺnL}蘹>sNv{] k}9xb]me@ϐHm]~>wGi!ƶbWIM\O糁YsnSk7c;xW$ۖ1o'IϘۚ5?9pDKC b!f¢mʱuJsyo mzKTcCr'Z|l[~dۭ 0F{a<[qe*ZtٽA{XʃNײwpµ 'ktGI>Cvܛz?@c̙3 +ZVe<ọ۰;l}E uuGswjƩ _cA9[Tٱ]]W'_ceuy#v6G*ji&mo{ٕs9Ng]c+ϧu>||9ˇq`VBN̊"\vw 3~'ߨ7"PrbnAu-5ag-ʱu(;ج}LvZ=6eUTkD2>(Nа=}p!zwo/ߑme|3_e<s4-bl a(g_IQs;~KE [6Z>ܮh>˱eM=; hY>:k=χ\k_wLTTm5r侷c l:3%M}ǬQÝӎkõ[gԖ=D.Xqjw7=sl_b%|4Gq֖6jw?,jM#_ 65(ۖuH߲N2sp;oKX%6bm%NKpdX@nQp w o \IAp 'N 5k8)wϯoLo^]<|=]Z}sp"._k?w3]Ym[9Sr_׵7Sի7fYSy?7p:GO_G>Ty8}&/|'kMv;|re>~us\yXyL&՘,dUNڴs=2wmG#fs WoNO~/9/}ۿ]?\/O/z<}\=B`u=R$l?&Z9;,cϣ֯--Ϧg%\M6`8p@!zaRA(E(,B?QT~ QblCi]Wَ~h :V;&.;WU~?X%V':^ e;2͘>|A6{$X'p%\;}%,hQ 8^X)R$ \-p?9sϡYaþsϟ;58 W!za%,VESg{\p,)Nok v6kl{7]A +.ZgEN˵c"t׌Yd^ckw^Xq4[WWᐲGyaf@:靐o|Cq'{<}Ûv|h*(uhm=o7B{Nx52;XG!n+rHE_]۫kxgʏYݺ:c&*l11-1iW f[b-aǼuܯݴu6Cr&e ae{Gwz\*_c?~URU{.֙qs>Z(k p q)\;r@T wEu#G {ۮ |ֽ됯GsjvWCcRp sݶp&iw7bT}i]lmTC;p3>n|: z\jou:-gϣX=.\1_L" }jG殂-J<|Ā雯ltpdBsW 4lV={]g:ʏevD@m[_ʘQ`K?;c{}k؊]ȥ;}Z?[o=x"Me:Dv W59Q y6e;G5luxٚڷlݘh(YoƶۢlQq"5Z\M>CjA_߶-5M"VRo`nz]>Jt0?*\g"m @U2[iq0-YAvv`f8q.oPTwd7opj%\vDz+0s\$VmPD8|wԸg?waWÚ,ށZW~?Íίπ}kiv<2e}eszJ̼Զe^.[WxIdЖmy0F6baNkUuyQi.f9W#uPph)ȶ+ߚH.om5k1EXsx*V|̘s"5\r ѻ!⛅⠰8@AyxZhq-[ Ѷ*HJ5?*e1oJG~>=<ΎWguuAm#7zh`-Tj{K]mylΥPs8[i"Z0A_a5$9D Y> {ݸ A4@/6bv!,jѦ[0~|D:ޮ1Ɗ;$.wr](ζW1O&x+݊suPj1<&NZPZ=V\bc塩?o;8}gsVLáO/5:f|Mhqb*ؘN 5{q "c &U!y2Ͼ.3^ eszmpަĬCͿ%7¾Xк,rV۶\9& |>k{[>$>FX|}bәsu PLd V-Jkw\Q7>m="\ǵ&Z{ϕ56ʷ|ofkxk?0myG H<<-/I(#fqW3X׭Ss-ҷ`< v<4i0 \>ϧu>||91C"sS] o~^9,l[>؛0װN mj]q4DߣG[ı=z{sGߢ%n:_|5U݉i>ne 7vSc-?7Xq]3`#'XDżY-e}^ {۶"۵&[56{!:~w=מf>&vbV=:K|X)4frjVvᷞQ-[`>$6Clla brBlS?{>&\;xn<}vDžΚZXj j* Y>Ҷ-TlwjXﭛV僜ȉ[(jXimk7RL*6"QڭaELJ6Rv{]oo /?wo-|qma]~So]}#@,ί\z96~^c"rZ )OĉjXmd1԰'ns/~R < %2~;oKXEGfۈp/Gxq^bDK01m:k'V69ᾰ]49\r{ 5q[ӈ@0)qI|PoV;<1yKkXռ"mskkG Qb4ʏAރB"jnW6asճak8 \IAp 'N 5k7o7QLo~.ξtj8/^5{ߨCpLۧέjo/.>rtj[!mi/mx ?.~:٧3~;Y{,m_hzֶOOmL}.yOܯ؞â-`zO7d!rҦ둹tl;1 ܬUNO~/9/}ۿ]?\/O/z<}\/G1-$"n)|}L?n+-ZyG;~Mp=\o7'(D@/LJQ0Yug15я]!8AmQ8S*M9^CJ}e'/D'klG[ƽ3vcW]8d߃ 0':?=-+1fAی"_>ǡ]#s \:\߁:<\o}~oG 0ę|T -a*:c9O)vJ\-^gӶXc r ZuXq ~<]/-:wr]e,vvf"Zx Vtx}ˏoŷ`z' ޘ}iE!?cۂI}RzY']A!\?{<}?+'ߖoX7a=_'K|Y#!H!ۍs{E2VGNbov}bu;Mڧ;/z]+~y{r(n?Zcp5&ߢ_/=Se0~]ޱYZ=-U̲g%_wu͌WͶwZp q)\;r@T wEu#G {ۮ |ֽ됯GsjvWCcRp sݶp&C$PHQaݛ`,:>ه4w}lKk }x)<, VX׭?>~<µS]/b𵠄vd*2۫Da7_KI[|}2[I@K%a/tw,Y˦mt\Ϟj{=jF[L~{BKXGbW׶\sK%q}q|{.}+9!xI3ﯨymVpc=!㸠m,f~mr-`qµ 8Vr}[M"jS X R gH@- ۶EXUI !V A\GsGLm: 1ALa=K({!{.Dx?NDo;W3 0lF@ߞk֎(VcB>f3%3EIR۵ #5n9#lq0xESu/3_l1kLd~vΖDCo|1~1onNe[c#~~1scuT2wjAp+?şe0uPph)ȶ+ߚH.om5k1EXsx*V|̘s"5\r ѻŠT5\!)ƒgC)m+O~+Ϟj{5rݵƦZ뼴[>o~7561fߓ뉤[sM!&1ϪD?b;j9ceềOMӃ]:__-Dp757T!z aQ6:px%v1V|]!qr-v>Fq-y2[эV!Ct^)Y9_rq&êƘ*Z|98fAw-Ou~nofJag#1Ң]hOcClTLcӏY5םgJ>.՚P[Gk͢Mo-c*e1+bv;#.¶NԂ.)6q*vjܓ_eo[[.k+Z1`;7rby!wDXYV)V>ʹ6ܷvNd}{ *nKbxtl/HkLRC)_xy\ϊU5 3$; por?n-DDVہ*BS>t|w~S #5Am p]1m9n>N.[2]`=NvcK`.hڭ#/(7GK!'[>}3ƒ~W`svzjGNG5q[ӈ฻|9p6Oq_^,:A h=a߰L׺0E.89։}#p@{ByA> Gik]F:c`dNnzwpk@Dߗgn|wv~0 ib3G$-"o`&_[iX"ojQnh@VR|[vkɽJZ5nlKMlR{= 5@wY'}b|${^q=}$!޾Bbx:֣,l|f1-ƩJ"^hƾ!nir}l7׉rRw)F[5cqDLw13uyf8 5F?^ʿ%o+u%7VGś1JDXi eCh ~max.By¶%ZZӭ5ZQ "}Z_#8?8oe]ܛV|enS.ż5c(F(_0W6&Mk[ ¥_߾UkMKkr564#p@83n0v^1,N)nr\<"ʭcoj-ʍ毵VZ>Uݗ}t?fkfߝKFbj-Oo[f8=k;;Dz[Y" r[IN$[/ɷdǒg`K$Ihafz[r nWZ+͢b"[C.'9Z9('Ti7^|R@m -3a,=2}#p?A|~=~fznҺ{^Oןv8C.Ӿ=ge.{zgqE۩-+\@08-:"e) |ޟw isxQ-1hמ 2Z ӺH"\X %R[j,kqt1bjF--k9FՎ=}ڃ; ?.~:٧3~;Y{,m_hzֶOOmL}.yOܯ؞â-`zO7Ƥwb8 C Эdcen WSyl>_r^>з~>=$^_>^?xf?k[%cte?/^A9v[喯#d!$$\vcjMĞ--DAr5\#i1Ӟ-ŷ$-medFrA2Ʋղk15V^Tei_e?}OC~ kmҳiO6zi;D9pDZ!zaR Aĝ(.΢? Q5ZBtTy"pZ#/wU=>rBJ#5תwlG¯sj|b1^?1n$X'p%\;}%,hQ n+^! kUůw7G1őEزſEKEk>#nZ;(lrRx<2Wz9m;>ɯ&q !>.v|:>{531yM|w֊fY{>?G+l[q-:c9Mk8?^t~kP@#L|6_=Y^61طvݛo r?wn'N= ĉ+›>ͣfiENoz'Iؑ}RzY';lg~b78B~muSۡ&V=>f-F]:r5/mҮ+ r_qLkn?۾wύg#ˇՎR\qvY+?\wװ(EE ꋃY(SGZ-QNpj%/-i!D-ޏRc|Gm/_-4|HGd#p8KZ q g>fqxQ4ͽg^4KD<̎;vu}|םmµϻvZÒarؽk_$Q(*Ewmb`,sc*TɷUIJ>Y ee,/bL_^{VкMZӷ?yqoW`KH ^Rƹ.׏{Z,믴B(ϲBZ iIV!ֵ%Nkr&ۗS]G7oB > Fѷ7"w m{R !BС97!g!5'KOJubQrQNƒ*}D9Ǒ't"^[=5>Oy(qQoܖb?'m eKzꔱcx[UBfA\ |emUbJȚRW̜on/\n; 6ƛbSȱuAD6>\tΥI,D9 Z-Ha( 9Jd+d-׸DD$c[ ^Y;K[fߞk7,V/mJ' Y צ#q8 -8nFkI)^mo-~j]{QmG%{ _JDLJ ձ+AA݃a!۫+I,E"Z E沾E-c6"F#=Q:-QEi1#Ǒ`Ǜۘ붼4;>uXe`5^rp8pT-HQU)9KU*rE\0{ pm܋Οy( 1D Ks߯͟ ./VB"W]zjqE Cb ^ERY=[Qm*U(eFG$ڥ#\N/5ّbI먎 v>µm__a&ehA&b5XlIΞ"}8sod55MٗE4ŢZܗE~|c]樔"EGrͽBK+O,X#{p9pDCGdaDґ7o1N+(mw?ۣɇvn 9D1J/6޸^D$u-vw,X.BY 7?`X߸>W"Y_^+*h Ĺj5w/ɬ~m\+oBI:m[c+ [o?},$xEXۂAZ\,8 #\9ۮi%Rh-j!T_QkXG> K~Z9 I׀ۖ1Fc`غsɄPf(mqѺ'W䷱"i_G##󱼙݋[0nJNm'G%(bŽ;[ I2% 5y:}:*\g}3'$鯏D9CUG؜L}o="r_oϗh~m\+Gp_g[Vُy-@?3^g)'c uxz_ӌko 1"V|M6(7*u*qq_< ׄ盇'F )uhV3 ʣ`܇5b Y uu)6&[Su֯7D eweoQL-/ dX-m]zv{>%'\rA~Eg!`I<*,eİ3Meo\]O$`XE@*헨c|K, eEZETۅ$&Q-fw1H⵵{kp.<ώitoCyv-\ *#eL >5 ~e/v_0.9o)웑e:SSSEg%6'c (3ɷ.܈x,r}(gw;0"\r춮s<|kbgi]oFm x"W )S FB-F5㔷!ZpWp @pR \8)N k'5rͳ >p9]x1jp\<^/+%.ҩs o>ȉ$BK.@x_Ng~3=CV޽n!\gK sϥؕij܊ԗ[jp=?=K)Ro^4rK(iE#8HPFN?kE-׭xi>sk qjc+^9V59Y++J4mپR:o>Jȗ}磵.m^毷\'k|o\E7 P1)!-spo^g.\ /WQsNc $8w۬}eO(۩>&~ sA)|jXX1p?o:N,\v/fpk[H5q]2#BۢN%I1-5$b7&@!oPP  u;GFU;bpE(f}ߌ^bò}g3l'/PpH8V p+HL%V˜Ӓm˂AƵݡ1\?}epwc r\鈚 &.}H13aGẞC f j[l-~kʶ"َOoe5yB,, qmQ>(\SY%_׋P͟9=4n(Eϛ0%>8 GXU-}oxz\-;&l#:U|0Marm=Q\gJi NQ@fIp;t>נF2 i58<WSC>!@pR \8)N k'5rͳ tyr=x"s3];.5^2N$\'oHg_:vr>:u69s>"t=S@ *%0bzKկ7w顭*V{5@D=8,&o|f)_Z[Θ8?}Oĵ'??.۬Jt3<%&Tw)q6e[U>+q%JkŜ-}'\롈DX)jv@ /uMŖ뷎%꘭t6po.{RVrmFmGe[N@D5p @pR \8)N k'5Ip ]ݫŋK|'Ӌ/ :gnwM9 pfKݯzDGmψ'~qNjمS8`Tziq:[O3ym[(xUUڇk=Զ;hJ٫ӛiu>|-n:%>JקLy0&u qo*oͅV~g It!oL?W_j> .gBq|i?WeBP{±c v37&/pMsuX^N$\H+H DE"*RED̂o$*r bކ˾e\X lg_?z4IbUJ?~m)[EWe~OxG+I`.uO?*{Ϧ6+f?^ \C>rqz \sL*`ot[%ۣ"׏@GL3i,l͢k'zqI~P< ':[磍7o*ͳKulɟ_.o`yudc[MvῬSٯu,OrweQN1~qݮp?`x7׏Va?M섮ׇ.+╬S֏o$[. =h-뺻5I"~roG*'4?r^~܌lgsm0sly"ZyFD/KQǼ\iWggrf{fm36nɟIP iO\ |$h~{,Ydug۱pBdpq>z\vZ"tx:h.wmy94w?A$Ňw/Ŷ\VƒPr؞u(6_ki˫ {oWŗ7׵mlǖplBfA\ |emUbJȚH ܾp'oM!w[is.I%Nڊ8l!neYI䁶#HT5;V<<z7=mgQD.mj~{߰&X*T!J ]^xf1ui-]r,ޯxUxq_ٸ3qLƪPۮe߹Y3lj{@|񓱣>\wn󭎿msJ;1\N P)9KU*rlEOu]v0בA< =?omWP -jc:0=9uñ;sJT0֍zY@{Pg޳Ue!U(i##|n?CqMv䵘^vDe.5%}L)36,nm~%bk!s:r#1Kxbfm\S~8>E|9ir;o3uEy:溓Ùp TN#JM"[uۘZMbc}qxCk lf+lcc^.P_(?w[Bۃ|nk;U[=>N_Y55hQփ%}LO)5N_~ elƵ=^̖6j{5뷏,5^B{>\kS\uXm޵ʶ3p|N&\5cEi+="wD}߈n:?kOsSlfv/nm{qPC{:?S ;&T"K:_>W} ܱHߒ+h|! \pn_qT87 '$<鯏D9uL۽q-zR'*Wvjnubq? i9mϋ} |Q|lۘ,~vbBmY7ב?n3r縔b,t=ǮeҚi3]o8mXcLc溗âN;g}lyq^:S 0HEH1ޢ~,&tS]YvbK= 3]zv{>%'\rA~Exg;.A6. lѦ|.K:$|xʡ헨c|K,oL=d۸aIbBl{.$^[^:h\Ŧij f^?i]^DAn|vmW=#Ϛ=gI_fcRP܊eJv1reY-s'\R[/{yq;3/n,oW6S]}|)ce5TNO 8 kp)oMCpp2g @pR \8)N k'5ʻW͋Ӌ~(O_\MuLWrr)p})wOc6>f1^byv9 /'^=s=]zun)in;ufݎpeƌ s0~pi[mgSN|~-/wԱI ôv8-#zw^>o늹1iKx{ec_\ȳymz4]|;o:wD{>\ͨlE7U^Hۮ{oq;al/qaUVM>fʛ1]{Pyn.P;yи!ޘӯ|\SW0};ʎEzخ矮9C =y`G_@r̵۸F=[9 ʥSt9Lw[9܋oYv}2GwSހ,ۜsY۾ϧ1'JSi 痘٩]^·ۼʱsz^?s^sX}kcl%GYrO$\H+H DE"*RED̂o$*r bކ˾e\X lg_?z4IbUJ?~m)[EWe~Oxё7I`] ϟpG͊E~ t3Ń9=,ڗc<;r{8t%9;Xn0XCrIDATQ%?9XOc͗[#%jn8yh9ùp~ϋin)ßJ~zWu3> oDp|n~A8zIGflHyV~ɦʷ {pS5FʷMvgXs1a{ZaDG7gq-’Qfb&mf5]]$_(tiF˛V7^u=]&v2͒fʍ hsXZ:وԃŌllN)gֱhlk߼5F֦[5٥qȸ\ך^܏Y~VuܪuIcc m,~g\CjK}ϹS~$}oG̚K,_˳3'v>C`V1iNZ_5ϩz=)gST-_v5?3=ګ?ݿעqvf~ַ* r uor̛WNXULۙ󮵈صce]cVbآ%cwy5n=mxuLeAt[s?y"Iƃ1u|핡o9g ֋[4'c)'s\1P9 A}qֆdt^y_k6 3ٮr[g;fqe˳My<0&kQ[~rnµ>3 8 $YUos}%; l;u޵CZl7G ]K:oxyeEu&6u_l ^5sú&m᭏ꜣ>\m晵mr!V|`KvBW^b`k"y7o3RNJO[&llMVi ,PӞp-5ꥭW Gb}:_9/yFE;`D67jUYLc)*ۮj8ǬG?ͫk?gmyΞ!z :{PnF9| r:ۑ>_Vm6eh>: M4חɞkҎaڪyVNc}hkkѺh%)~u右.ָzv+4*89X+{"m~'ZK.: Z^Bto.U")"=ua}ø#\Gk+^p]C/ȷmo!\Pk6 @ sH ^"X*ހ@y LXe N!mD~O*]:5o\y-fq-s=ǔIsugDnn LlSWh9U:~DycY9VYw ٗ,7ȊkNM6\prʃzCyzK|Lc掋juYEu='eNoXwn3fL3qGbθ-ȱ)jҗ޸~'mM^Wb}Itz\.Tv|7>3S.x0θ|}D6=͘,C;sKIvj Ʊ ilׁxlŬ>Lgb&+ٚj<ol/t&`|q>޽˻&zMGj#V{ۇ|lp=n;9bÜL}6N<^5li&qpc~)x뭶yVv,,Gwpх$:) ˎ$[E$vZ1Gi mO>tvm6 v=>Fm1 a@nZ{r6ok.[Z%pSkVjL33͘{(N+V._NDuuZwJ^R;guxl{d}*&flŬAߛ7f;ar@1xf| <,%#4zftݯxrKRqbuڟ^y~'?d'§ԇb3֎YڇymU1{^H.ν3.m+yM޸16=_Fu+g\uVz5q{cl!i,Ʌqq(vl晱퍳g\l ޢoODuH-m]]GIT d4[_T_Xm (9纗 :f+«MZ{J/RysimUqkMޘl}`Seև *ӾRY֘YlW1ѾعH;~xD9܏Y{wWd[IvK79Ŗ1rjl`m~!V I{cH+WZJD+Ï Zoj%96|r.ƌ(qqW9_MWe_z_ZۚFzkRə*v119gČ|tAʘ:fDgr0{~f6lkDٮE,vxrrtq1m{hqsLWc3|Xs:ǚe~rݙ|3K> kFet^Rp :1lv^<#jIDygr)ٯn{78(,n:N?G Paas9|7]v 6hsҘqkL>/ Ι;ltyv?o}ߘgkW+_p i$(obp:Cq{2 ~3Vo 0}y ٙ6@1i_P8L~ஃK`M"wwimSm~YsDX8KA|ݓk5Ip @pR \8)U޽n^^*q=b!+n}& 6V~Mۖ_bޱ3C휼].G\ ן~T\M?mV,bQ8 -"_fWp` cW7b0mQ~7b&~oo ?,f1A  y>엙 e{n΋!L&Qzo%zs>ȹrX[]۲9;3W?.]6oYo%"|l|)<,Cq5 B \k~:EB._wEUM"k2^a_+M)v1jۇک+Vɣ~0"DeO8r$&]LrX&35n' џ-|#VHw5~vi ,PӞp-5ꥭW GkW Ŀp͵g r|W[v\?3fE D"3> }̇7O ֞c !qr: Z,7xA]oYYt,;;ud{OcK{8 ·}{µ3MXp7?3âr\w]4) @l*o3V;Bœ7B:5o\y-fq A-<.t_bTtu99V~DuhlmyXoI.~2)Lϰ]ߙޞB:cO ItDHo\s]%,&t޺mL-`&Ԋ#BgL[h{alѦ'^7m#MXR S5&]kSW]lܱc8ZN_Yk?Zuk^/B)1c5+kO)2¶ ;:j,}ooO1F̼p8N&\5cEi+m6R}#޶G8}a6룼ǭ%\E>rn_㊽k]Z-haŗ"dcF(I*PPElU)l< \pn_q lgVBW7ε[ W:gӸlޙ^o.~̬0c|*k7ml;(>#lT"_:m-Ѿ1Δy&N$\gԢE]GIT >hbK= K[omkW{ =u.0O۵y-Y ^48S򹴻.IxKK)B헨c|Kh#*EHbBl{.$^[ǦOYNs96Y.u/Tlk76g*R>cPGN2=>/"lZc,u9}:$溟+se{[(cq+eQb;#Bm)FGњ@ p( \׷}%wu-kBgW^oID<:!k@E\$>|-5*zBtNq9*?^coeZb G"p>gzp >ʻW͋Ӌ~(O_\MuLWrΒͳ y?)篟xe}~uco{ų̻9Vh7D3W~[l,&måxNµ]"m|Q [P*PoZ"h:o{o{sbQ׊ˇ$ʁ45s$2F9pG'ߙϼpzG_*õMߌfj~]kz[͏No~VekI:ޅq?_?Ο/>>5mӷ#_\ O[- ׼'\O^؍y ]?ni_~_fzيͮXtb?o-\/«[Eߌ G'VC&":n\]Bes筱Mа=_es"C!(*E$(.bu/!/lᶵ b'!"p#R&׏MojalcB, m˯1tBr4ޛ#hswODI`.uO?*{Ϧ6+f?s$8/n1?{{|IԵ}K](olmގ-\KQ_EGX rFEe-}cp]!yՏdz [ε}\^ϝſ-ڒKݯWy pmILj <6EI{BxW2= ; (jC$kݟw5F0<ƈpdAD,>QNJ R"jg!x`E#DP.YɢI.+cIu}U豎IW]T:-۟Ho\{__&}Io29^zXAu/\{vqG"Zq [T[D[\ T A\8 M!*."y"!lB*&5 ֑Ju߈VmgߍVϛbSȱ (Vm`8pb4v+q6)'nK΢8Z- {F(Xe[q;)Do-6,S~kiOÚbVثca#9{Ug;u.$;6\,Ngg\0rm=5im-Avdߩe*߃-1ʳ=Fbgky֜1*Æ<|m1srDwG?VEݗ$Vک q.qX"$.cU@.'"'|K4 <|ee-QUap]rsY?^ 8N.\'U ETUBtoZU)~V[wE,7;u>pmΟC/&z,\Kls;o }|>g{Z׻VRqOD`Z Ll*o3VB!-H췊kp:=8dG^y\f^k-撯:L@_*#LjѢdoPy]}ᮎ[~ kFdL$k|T|]8)#fCy":N*\mWO+fA͘99"(.\o圅nzξ"|p3rZqr'~sE`=d;Ţ-A⻰klὠƐlY_>ֺ\2o%'\r{Bsɛ(TN#:$UblԢv~Goagn͘8܇.~D]Z- H<Eµ w,XD(xӲwp5>o\<忿,*,&{s9ٸ|>ն͵fu977 Sٖk־_p֗GM-,qN/bh> π@M~U&V{kk\C։Y3G;1srDևpan|miMbO!\{.z}Ld%V˸r> u 8 XIb A7V.{SycěwG{]_uB[5ɄPf(m^[¶^{oFtӧmQq\{H6f:6vVq X+PTo"dȆ'U`Z;[-(BQ·ȗ0 \sӧu{ֺ}Q0sIˆgF9\/٦7m;96[yfG-y{{Y?"IMa]+ڗOP԰k~jPkr<#Nj-Ͻͳs'f'%7cmIẼ-Ej]J$t+u^uJ$޶KZN"}V5E2V{ָcJI ؤҊ)ͻ[i^x~2j3&'/^9Z(؉p/;D͉,ZHA4ޢ{UVhDs(Ŗ;`EV (}m|1 (9mݷγ~P XIx(BaF*Q7xӵxN(_%Hc|Kh&ފ=,ILVZTc5kkعYӉREb?ۖgj+ yΘx[}Nw;/ܵBuX sFD u?z]uk@g3:[.mVdZ%t9$[*ɲ&*~A> VDŢWwE=p ZײH<<)l˼]ϑ$"ˉV;9߸ }[oD-KGp6~PZ?{{nCwH<!IQ<u3žhd Vl",ߥ/GB)QUKbn!QmJ42~6X-9>׉ֆ+Sޚh 8K ۇx^!9pmi h1++ԫ>elpm9pcA~ٗNxy>zb*tǾjڏI$Pu9[r <hkQ B_C.m?c7ݲۦ-1$M V zX˱`a01"nik<g_Dh ŖA)2ח:\22i^b?p@{ .I]\=p]D ]&l#Xoіx7"&sp="1}]22"$jbۂ 4>;ףb$Gk".xݢV4W>!K$.Z+2.ba~խcE"RZSK#o w< IbԷ@ .s&'o=Z%lOaT@.8y[Ko2hmtp@bBXT&Z^X(x#R۰H0֩}9v /69_l+ p@B`[UU nRh>p҂Gzݕ]`p+̩+vXEk@c +(`#߄>Pk'5ww//Q\O/.d5xv)+k.eͳ P_e?l}1]}@p, ti;q:[O3ym[(xUUڇk=Զ;hJ٫ӛiu>|-n:%>JקLy0&u cq65ybzKe|*n7''u5%6/ޘӯ|\SW0};>/ !\kw}~aZ z/ I91kq"A!Ph($.bu/!/"b#QnM$6D^}T.zBjWd;ѣMmW} ܱHߒP iHJPkttTZ8* rBSG4cn2^ < ^N˼wps}v@w͸J"\q짍iԧZܶW;x / vܻ`mm\v6i{lO>'@jQ¢#$zvN&mKLuQ_%nlH47(+{؎bZlGa \ecQ# µfcQrBuVuˆm/!:χx> ,‚hS[>v%ۃx>rxo%X"F"*uKߨb;spM$ڽ58x";Y{}\7e u"m_ӿl{o]i\bEZױ1_RowΞSW(Fs϶Y=kTkX=mo\pX Qph \;Nyc5wN k'5Ip @pR \8){5ݼx1x~闏zzt]'t)ų^WOLYYt).n]e*Ag gc_r^w{1?.yLGEYZ=`tM|gz?۶Q~ޫײ?z4}3*mwxuoW7?Ҷ:}[NuK\?|X~O&aLxnmk9ZmdԘM9pT:8`=Kͧ?>?_?Ο/>>5mӷ# .ZdvX]1fw_f~}v ꗆ$Hµ%-ayox/yAωk+@,Y\PYDEnA5yQ ]awG7IH0J1_Ӷ֘K:cf|/:m O΀$ E.pOuO#f"g?uaHq : ųߤ M{_08g|_@P}{ `3Fx,"l%GEԺ㛁ꏊgӶYbw=Ot añ~omu">xr^rīg{|aU~;%|5zov~/^):';Q~/c]!\"No~ú)tR Zȥre]^L=)ہcp.¯#f*?V.{:Gv0KZ˸WɅdmK_UiLs|%fa&JHBTYu_l~}Sz6Wd;߇uWF6ȟIP ǖ("x(j[ moW숬1ζcZ];pM6-{>%vO>&A!E޵Ev{p7-&Jž rYKCAiL_R{嵘p{7o45߮=EBQ"e^7Bhm{Xkב`;ْ]=ۋ)]#Ȱ7⫍{cǶ~잟U'*׻7ۙ{Ƣrwb1~u}UY,h QqYTe}QmqsFȚH ܾp ySl 9#`8pb4v -{81ǵ+p(cC{&]P\U|ٞɃyџ-|#VJjNY淧=Z k"K[a-3^0Qkmዚ>==_ Nݲ']mily bb\5I6xo?ǣ9aFd|,1Ǻ-cͽw-N2ص y[};_N.\'aP ҋX(F$x$N:RU[YH_{ow}lnfa7.׾i!_؏?8}%r`#>xgkՖ"ik՟}0,\o\S Su)\w=ҺD sDOwvnU}a;#u*9'/;Vuf"m[ op+-oKİo<7њ]ۋݒƟ% 7'Cњzmy ۶C~{M="\Gq#qۘNj4֏?wEtMS= 9A9;C%m[ElU)tK_D845y:}:*\g}3'$Qcm\&+K'inYWuL̸E4SsnۈQ;]6ȶR.qzӸt[XiJ9-|/7[fyx{qc;'eMGvأѱ]Nc IN$\gԢE]G ITMږꢾJ .(Ŗ{2,&QEݎ0 o㛍YG =[-bLU4/;t+YE򹴻.`*C{C9U߂p,o |Iƒl=,ILVZTc5kk\y a#jnr֞m_5MÄcc/꾯+LiӊĚ^c_c}9.߶>Ɇƾʅ4n(Fs϶Y=kT=[:/^ڈc~}j\\{ڹs{e昶{wk p{Ԣx,@w6Dk%7[[o \p h ^V \8)N k'5Ip ]ݫŋK|'Ӌ/ :gnwM9`|byv=.kZ~p4;D{ͺ{8$w='}x?}پ'N֣Lgyy8#/{UZGoFe3o.ڿ5RGv]7{ic{o5S$IXxMع#Rmۖ `|ޘӯ|\SW0};ʎ<_ٔg+D]LW_&!\}g)6gXp}8p}0M{l=g~ 'I%RLb"VgqB" f3OboCeGE2G,vM=$!*uD`~Mۖ_[c.錙E-qfs4w>H4;'t)I`.uO?*{Ϧ6+f?sl%נ ]8DAkkp&_ވU´/[e[M䨈Zc|3P}6xmf5]"%;4χcj ێD}hHWed5^OW(o]!Ғc[vա,?T_gi2Ɗd#[eyp[Y_WX:$ELuv\b'L/7B1Hb3# 2QYg-D5Rf c۶l֦ۙ5i5n5skӮ.ɮ<ۦElyUx̪<H:6m1|Ov.Em}=v?!_f}yͮ{m[e5F=YnxҜ_Ic9^NnAY0y)/>}>|ù%ՑIXcZK}?`Zӷ?yqoW`K9ϔ\Ws`+32oJ]iggZUc"-;.wCd;}_ [q󠄓ߣDb̶qؗhi\vgWٳo:ffy|LY1>_hǬ\g~"FlK//7ARbeqci\y}%և,}EHH_.K?L_l_{Jw\.x1tlY,h pYTe}Q]qsFȚHL:oD+_+M)r6Y0gѻlǟBx{έ`;0PwB$kpppe{n'Emu0jNY淧=Z k"K[a U8"@U@uM~Uq^9ȳRGP q}ה9n #?=kAZ:V ?=iTׯ1K)%&9 9"GlcEPSǘe3_oc[_7ڏh|.ٖݳڔgh}d\g\?`?uOlcbU6ql/0Ʌ$jAJhtߜNg[M⤪#P*xu<$vv p-ǖp}AOdžxC-k+TWµ&tH+hqH*zXPyH>xׇ̾Al*o3VB/8VH7>CqMv䵘Esdgv|Ȣ)uyW[<w݂g_m,xk.ך(l>'tߢ/b covQ:olL뱭\{yƳ[MgZiG/Z1+2uXrv0"I_'ӊ.~l}'أػұ~.yњ[Kkcmox 8LmeY-Qq?V6u7l1[-݆X Zog=ڞ|&Fs(jCĿ@~okߴ /lͷ#S!w<8L9*cZ%ik՟}0,\߸VuDhN{sR40uui D/a٪oj+I+=T>[ؐcD4]gQ~8~J#Wo{y)0'Cњzmy ۶C~{M="\Gq#qۘNjq} ܾTqCr>-:tи:l&sշ- ;ٖC1 \sӧu{ֺ}q1s\sf"ѡ[liK 3msr) "-s,}Թ0<]k:odJg2jǥfdu;7ߤ}ծM_}ʘ`ͳÜ}Slm9UntqDL٭cF pоןx{_ֲ8,ul[xdNu`{|TL6J$N78,p:s3]kgų^O^:^yB.n]e55ntM|gz?۶Q~ޫײ?z4}3*mwxuoW7?Ҷ:}[NuK\?|X~O&aLxnmk9Zm9'u>uޘӯ|\SW0};#E^'[;=:Cp?)/_`p]p ;H&R,J2XEHP_S? }˸v7ξ~hzVcI5m[~m3fޛޚplwr ,\O"XH'\lhbmf:Hc[@ڽdr ;垲+98$g{޺jvl _3'mw?x :],k+ gT%Jږ$r~v;"k̀pXy!\ lDca_}U,HxE.b#za?YE\Va(}^|(}Ree,V۬E+r1[%\1}YMv-RĖzzY)ʺk3X?rM]g!2,!,گ֬ku} y5 ?أikj>^ p =_wg!\)DE@dQ1EB._Eb*&5 ֑7}zwwnrlq9U,3]%B C"؏M!N /)^8ZU?_Nd{n'џ-|#Qbbh ,PӞp-5ꥭW I"cc-FXl] _Tk"jj#0C>kŵ).F~ <>n_O7oB̵3??fxM?^uAW^+h^]ǯo_y-/${_ܩ6+S܇-Ke,_j '"*}s: m7 BU?E]9z8 } p|'\˱=?omq_ӱsŴ'\k޵,B%q:^~UpmVvCd~B9Pӿ\\|cΗ>⫧+aWTяµGO-rK&J;uaнp]oΉ8'N&\5cEi+m6R{w7>m{DGܷ1/zk; בcsb(8"+5}r&Nc\`\ un:?ʉ,ZHA4ޢSIS]xBe?$ݕտ=lG1-Tߣ0<#(}m|1 (9΅ 3vk!ZBl '"|.XDS>v%C~>" ~oA8ȷ" pVĝ1FߩDŽk"0Fy{Jk,qgJovO-㕨=_WTs+Yq l+~ݝ"Qھٯdh:ؖ:1c>ٺE gg3ߙ:!aEfèC?;z?e<&^ B bsiuICkW sOΝk /%F^K'>k[}΁pmݟ?6z=&,u"{ׁsk,6xz*7 _ E뤱[!_.q`'-T-\o00bKEzY mS )!S ~_ $Jjp4ʓWӅw37ջKͺ&]tM~;{ϼn]B\}% nz??Ho WكsE`W[uKyµQp'}쑷Ss:[O3ym[(xUUڇk=Զ;hJ٫ӛiu>|-n:%>JקLy0&u cq65ybzKe|NqT:LwI$:oL?W_j> .gBq|i?WeVkNI;,gnWL:L0'q^6%\$ֱV~ oc_^!B=4oS{ ײ^}Zǫ;*\_^m7ŬSq"J!Ph($f.bu/!/b>#AnM$6D^}T.zBjWd;ѣMZm1/Ō HFV77 M؞'*u7%kdp-YbM}qZpwY" "!vqFȚHLMъp ySl 9ŪMG.y@ Ǝ?p{!sG0;w>`{`ϸBlJ_O[`ls尟ES~kiOÚbVثopk,H: xkG^puʵd6.|[-/wmʘeN6amϞ޸F`I\P]ݟ,pɏo9~vCqMv䵘 kuDO9[zU9lʹJk p[;ng r[l&??tam9WaU k:Km{DGܷ1/zk; בcsĽ~W@Q O0I->ݱHߒj< "KQQ:=kݾp.1p>Ј4(2Ǵ˸("dg}Lerբ׋D̈<^=wBڟkñs{rWT6>ڏ=5O,뜯>]So?7θfZqeӕgm~hcQ9̇(Oؾͼkaw랰"V]D}bf{h[sUƮ쬜^pL۽qul~o n(Ua8඀p 8mp Ufpop.7!Zp 97WRW89N k'5Ip जT8}W'oMʩsWc_LWze/X; ngn{5ݼ*qĩӢ~闏zztᕝ3<+;\Cɿrj<#^gR^WG>I?'?#s 1o7?f3(_?=}1mIwk O>G=|z< kΐ|&|W;v`ss?'q8tOo;p]?Et]?D[aP^8PUA\/n]>Rt>6ͤ>[ 6󗿘^~(W7.LSIQ铪5.we?yߟ(jxufLͱ`3ݶmj=q#IXX~XY?bozpG'ߙϼ)Oޚ,uzkvoGoZ [Neƴ`ƷPqsk>l cq617Ql<1tְҹ-{N? ʾ ״Wr`1}h=+;k>D);Sͮ?n!+3v}@~MZ{i']}г<ᘸ3fm>x \;e3;;g*\sg)~!\v9PW(;&\{w\i;Vˇz0`Cy_-!@MdWzؑ6?v>qL7˦6/XlVc0C?Ct:[ҁ?z}zg[շՅνH_}bQy/m̝<Ġ=ډُ%`?3k󹲷63]3t_LyOP9}jމIsهwec(Nȉ$חcGq=qp}cs~3ߙ;3o?a^s:򝟛n޵stOo;C:X=GUw·|}7>Q<';oN'ac$FNti4׭q:}Bs AކtY.X_Vd$Nѡ9.'HUjį5>MH ͘gWux/]OWO͞IF%~0(egNC媴;<؞|W?k[ouGe]KhX3mIDMv)/ ohvlT+vi+µo;lq$ZňѾ*:6bo^X=G y{ͳ˼{CZj]ȥsl}$.^h>$Dz?,n/`KkuT~u73Dysη{M2S=Kh=5WFx\QFD)=7X3~5S]ٯe;P,&׈l}?a|P71 榝od^<Q*lϧ-.mcnk~odw> 9G>Ovtؖ!]k6m}棼PpvR.5 KnϢ~:j۝ⳬ\l2* j|q̨=}qEGLE}Us{qf{|5棴LO;}@ڷuF_Y!=1~X彳\c}KsQ.3[[|{S9>+D|.#U3yY|Ҙikz|\;SOJ\usqTl6*Uw%}1 Q&]=Ko~fL߷[՞V5sxҘZk\혵H9i9,~Smcr:Wոl͇x~ӿֿ3C]:nb>燐s_RR{Q}hIү3UBu'^|o-m*_Ab)_TGLo@\lLO8]!0[a۸5X|; vs P^sz+;˶uCsZ/Zy>!9"7QF)^mrYˆVKnی6^coI7)Z1:G"nSiG^z[ߛDT#$[:ngY96*.Gµg\nrd^N8`/_>wަz}lqop*Bޓ)_})g7p[Okyu}g&3/}e/ 97i}-K:B>]ڙt|K}۷^Hόz]ٵgϊe3}ː}3]lgGc/$v׳j^4ǚi]2ΎV~Q :1c#Vk9.72v48Z۹OĀ3qcHF7 J )m,%$ֶ{~җglh3yNnWX6FM qU{e[k{cc)b3k|LבyKО|u~zcp-^6W?D{j~#_܋sM/=xoɝg$\þ-_Ԗ?tclL959K{>vٯ;?u|8dPߗ߽-֟1ҙƏ]+\?D~^iį7=_m]n#3Os3vZy|4~Ʒ᫛͘Y[sܗ|{NZxO~e'j~p(u'L%P>/-,q'QA\ mw]ދYe7[^? $H6v6k&Qo/aO2ʇtro43G~17:!Ќeqy'I4Mji+&qSߑ@;Bq[~%tmko#ƐƷ9\&gQq|9;~FrK_ps-so[Z. lGkRm-p 5&]C~dz]RoUej.T)K~([-;8w꘎ݙkf>T6GRp.yC?CyM~Nrdp O\<2Y6؎-On_ٯ*wM ｟upcڠg&c-o+xe\W66ctMg~3:[~7 A 'F|TIydp-CjbZ3.-s)BgW-򙚞Mk'fu,6<#g&u 4ƞGrGZ˵%|{zOmG V{rcly}E#6[$CKXnOt\7hG+Uɾ۹ЍK?Cmu9_F\lv͖P]4_(_1>)۝\bjFws9^Z16kvo3*ent![_—.2È}XR.dT>x36`a}]XpހZxdS,Pa%C@ OתBum9>`<;.>,澜{µ=:}^ǖGr *(˼UOԧ8lLɏ?^jt>c7fFc{Frʝ~mᗰ=_=Ljvq;ٯӭs!V.̟M ݘī_\X1;΍;^m8cK{帽9bͳ*cƤݹrvcNZ1fdNVN?ʿh?B:tͶתj_F/#t&Tw}{qS|tC #'NGS]+P q=벬~FZ8/͵i}FחG{Q%zmKxq>:9c0r+uzN}+k!"z~Tr,4F3.udtd\\, ng%\ Ќy؈Y 52\>cb=ʼn9_UzQkbxzVﭵ?3slgᗰVۊ=-1%m/!K}ZףӡP/=U=xPCD~P=D 4&('`'>]KGtN{Hh>63!-ysn6qeY | ioJ *ѵ3.EUkǝ(c{ǽqM#"3v!#7fiط&hƺLk|6Bs]]~KgGoEm[|qR,P$Z3E oni_$2_mm=x>R[O4~cۊ'A5ޕp]v\H[{Bc9=xd,3<޵z {g^jkO՞meϱ{o~&{\ۘ0M~ _/_9#۹bwH}NڬgBr|_%ҞVfy]ONʙ^̪ ^Xx=qqKS)Iۇ5򺍙%peɷOe:k'̑6ŝē=haZ53PP'PL>ėe\D gL΁xޅ}棎77+0Vs\m _ygxN;wGt-Zؖg6ƇX?|Ot< i*_/;{ͳ<%|yop>]n<+9T|{]}78ݽ1ls-=+c3s¦s9_k#ؾMFz;4o1:7Ҷ3. &@;U9=ʁQ\v<ֶ@Ϗ͐s\/~d.[ϟƬg&+6>[T~|ybBAo1j/4cdklr|kQ}(me!>DJKWNA_BMP.:1]k?_QyN3 b#ryLu[wӸ=T3>ڃƸthٷߢ̝RGk/7jCmwUG\8ܵGYvmesI󹲳63Ќ'ZZqK&/v菉i/o2|H_>cׇ짵DRVZQ4k'S͏溋t7a?s'N&/[)}1m?9~1u*s30\r߭|syXOFe :1#OLL#t 6O'Xws'ϘѵIRϚ}Ȋ\T[1k+l5wc=El_W{ '\'? /ɣ{oM-r'ץFa.pWR0c  >@>?Cqo\ 쇦tc}h}{A35ˌ|@~ \pXxO( ey^GzT -o TwgS /q5s؁[g=?Cqrz*:54'3CN3$z0gi,h,b 83VA`Ϸ'J9Ku/ u>h\wXrZq>)pH+ v;_~k ޱYNk 9p hkW+_p p @pR \8) /*_9u`̗] qkgt-w_%8uZ/Q\O/.s=]zeG }1]}1Kt(w#T\ڸm^v>~^_yD{!#%l⼌/G ̣_P׏tfc|YH?1)'#s}٧gÞyϽ3ȣevHj\s@QogN@s=#qnhpqN.\VuCL  '0)urRlbh"_J&xyʇoYiO^|}~ӿT_LДϼm3?V~/Wrz(ʺtՍ˃'Tf|R>~*kK]fòG)**^A{)Ss,X|3vL@j=j'a<tȸM[O3ymSK5-YGk?>e3Q煏Mߴ6|󽷸ˌio;|xnmkco0yBw~;t3Й;=;Q=˭]~;|;߻O#vҳ~)5q;\qcͱxGkhzHµ]0P@~`&PLؚi}2/bu>$]e!۫НF ߺ\c|)bk`=m~W(ohEoaz(Hzߝ8H%V ^?'1ܞ>_j;$> VNm$^} oM6C7sIg^N˲)z8يG =yG;;~~;!<6+S[٦~fkwȧa_bR 7[5cб^55gG=n_(> Ǻ۝cŬZ_l[/}lKW&osSg$t?RfP/s >X3hnM%I."Ưq-~yZ1sDvmٮ~HkOK3μ;{²^{s.3O{WD8mz>[Ɣ|~x"r[ϕ}џ`Gc>ڧ\~l.{Gy0Uͳ1x]DZp{5{P2o0'Zc>gUQx+W1jKe}HcѶ53Uvr8m+W^;q̌ _9}9ڤqg"`/`Llg[cW?^-yC y٨b3$dݜ $>Bu'^|o-m*_Ab)_bhl  V:DP:\>K_sm $%\۹1sz+;˶Yhk>ՙsDGQm r6w[/kz r|%lf\ܶ.ޒn8 RXMo"u͑ ᯗ&Uɖ٦kVh Mڳ[mcjgS {W~LQuwfOq?J{}|}-پoWD~`Ʋ^Ͼ,\[[W|YRg\K;ӗooW Q?صg~f*D(mgŲ^*Xspߌpdƀ?Qkb[/3M+$/9 Z}v\ٙ?_ws8k+G;'9"f.Yhgb̿*ɗ1K}E9>qq;ޙk?Ǭ_9V<yͳMׂ^s |˼˧U6}Z gi䳿.Rǽ8ۨ+1qU"_~Zk͇@l8mw=7U7םcrecs@cs^-qV@𐩲_sWےBQK e8-_Ҹ|8ȹl8/j{<K+ڥuCZu;{w~nK܇q2=>Qϑl_B5&{CGߖO|P;~,R75_TW=j~.pl:0Ϩ mhɖ3)u=Ysm}׸隷^̚s{v#ͺԟmgUf]s\A^s>l\>,8j}Wy-كXk~ a<=Y>OgEz6TepW=}1=s!\ٿ ͟M?ىQ@!27T99*_(`;/ڻ.䍴^lZc/yB-_!FsWRYO\ յUXԂ(r Vs:]De^ew \@)ŋ?^jt>cǼAf׹ i[SAY&}3e #Щz]Eg)ܺB.=V Gf=5-U.y^m-Ҿ6}F7 x 8=g=~fi=9ݫV[?H=I}9=G=F[ ?ɨgs3v3r gt笶] ߙko<҇S~AJ_h^^1Sj~fZf[|sUV#zkʣQ5pOKpr7^wHDݵ|T"_~9GA؟f\z*m1x01s\b"<2#8)\t *۞>q^նPuF9hm 0IIثx0dm hRz׈5,u{ x?.mu}߹/uɇzc*`νs 'fvx\Q}MF4?NL5xչ~t$Z-PA׳$Uh֛Ǡ|<H{c&W6 7jf׹q>2۟ o.dQ1C@$ZBa-2oB/ײX[{'ж!Q6~5DP+oϣ1uloȳ=qm{{?JAlV9~~#B篣_]皮vZ5^wNbnx">C1Oe_V_]>ה{2y&!J}]|xDc|tHƈjP⨯ױMU7ץr'NqTIQBsmK UgDg&\xډW>|^qkƈ}LmɅxmuӞ\Û؛Ҏ6:r\¥s!nw+6G]b/i͵xcﴭ9y;'fԱהh11k{Sť7Oqϳ*;BV.Pu cD?ۍ84cFm;y껝w\Uvp^ӟ_Z5Au9 j^?zؙ&\ۭZ7/Q~"T@4JAҏT=p G~.?xr4Q5˃NkoLFԇyt 銝|mg>”+R(IPRPq)^;\w^׻Z{Uڗ=u޾ۿ^p/_fb{smcbÌ7-|XlC䌼n纊MMQ_wϕz>.s,YRVV1ؚ8熝e9Q]vFrB尌A\c]sr9qwzʣ>vhRx1mQ^2s~[rrBk.caTk >ꬂ|g>> j[\Hb1߂9gbh;RL7lCu>8o1u5"@Vz/{{{S}˻/x}hc!ιl3X~.]o;vgQ7}ܰ1K|9ʟ\ѹ'}YtT^<{r7>?65ammy/ wrccٞh޽ຍ<׷k1xЎme3pwYĸsŶ7 c|P׋/v}>ד>-z[? $-WgN"+l;@.\$o!\ˀq?nJ^0 ?IKRm/^З~t\`yſePWA^>L9ヺH\e!6S ~}컂h=T3 >ڃƸtd"E, '`DTK-ׯ. օFElPvWV[]޶N(PGt .1KWpmsv\npY6,p{?jܓG^@گ+ao\K{ݷ^ڇ-v>^S[s]~EFm޿JC ylֶ}FrXҶ7-R^\2yC+|\Xy.ǡ>.a YX&u~k^|pMg30f\mwyҩy񶇗='kг:`m_u6|{ZkۛSg.my{lX*fk.BU{ǽq9Yuܗc!czDF;{?\眜cF0ъlGRcF,c1kSnjQksn6*y֚TV?m! Գ\Mgng>:^\VcKhlsQǭ&.3+l}e[?fƶ*}GgL{l9roO@;]ep[LsEկ~uW \A!巬p=Rgkpu9ky{5څE(;%ۃ|( \{!c-3$'W_'36By%ΐIua M@u]^h, #^*;z"ZTlJohOE q G5|O}N*\N k'5Ip ]ݫgHڿ#mI0 [nzgAhA:r$r%FC#Kk`duꩈ̬|:u̩|:שQnNWuy1>5/8n\[Xc@p|zvWb|NO狑3啇M^YI'1ﻓj5N~bO#g<>W?Ro~>bzV6S'[׺N7>vڮ}^,x2iʛT}R;}_UQ9pHO<-]_FOzT~q9B/M\N˟4[ 1˜N.b\}vm˺1MMn`y6]9">~_hu\:q\űph58pMY,EEb"RE#2vh!"p]awU֯/wIHvW+:#ӶeZ34sa`Q}/b`- ,g揦/It6uY8׸$Lpˍpũ_n lW(z仟s>$ m=:]|t _ \W{.p 3SAPT´iAQqm_B dggbM,F90_]5?/ti^"&mr.cc;8 7>_<ħIfƨWG1V?^Ir2WKeTv:jSDٽ$Ec}$0o6/bsE]۞?+|m O/Ř®( kJS<&b3ņzXj/M26.l?Qf_+Lϫ83fB =o?]|:$}kD7<nCkEgtٺoMesg};Ds^ ̓}M9edK35_φ;z0Z3n,k+^gT+cS$| fQ4= ΀p!\S~YU?Tkr=,vy 36zbғ16q&SB6%T݄c ةNDw,IpCS"<$|X{eH9eMqLem>/p% I^-\?1]$dcd|dRkqqς7݉1\yK8Q糶ǡ{pp͈-op~l1̟h[1Gl>v6~ɶ?3u;}{}gyK볍6ǙYqgHtL>v.p͂1 TV/#+PD o}GۍVϫ|MEӺ(Hu X[a=L-Cc="aK (0x]7+ڄk2$Q񚓰R7]v2,6JUbv5^PoS'Y t=Z/{(A~MEq "fơL)zon5$]]h+{ pfI;^;>ܗ?) o? 2.1Ƌafmv:_W1omx}zvk1fe}" Z8)a 3b>G3'$oLۂb?UoeMhY{xI[ p-x~ XD_X{Ú} Qzpef8y,$qZ8IY'Zԏלm7Qen3WDo$).Ql'̓|q9韦gQ<M]-ҿMyk:}Vb}}AC"q? إʖ^ Љv=HſaK?C> mR1_1}#les̄1Ƌᑸo){^wwk6~ٿY} uxԯ/Qؤrܶ!mv׃̧}iT#Joy[Sv5mO6\ц\= q1."CYQXH$uET&$'&pLrU׼\mIqNL o./~oU dcFu"R.h%rǷq,Jr?W CGl"J@['Y.Vbx%^ lf;Լdjd+Vâw?M!t0]6y".hI~M1{q.ۢkaƸ۪8cԶzt}JN&\WEkƊV˶[K}GD?M="Ư'Z߇#XpxݼQ_}[DIP/&|ĩNR%E $w1B{\-p"m }U+Tٲ',.:oZϟY ( gنlI 5} FOiS~pm("D&86'5 %aܺLp@p@w? 9Ip @pR \8)N k*Oo/_N/G^z;]yשtԔp,j1Sv__Nw7^\=qs>Yt핝nj3o_{z[X<{^/u;~Wd>g.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'nmoݥg@Zc Kw~鿿k~.+]__WM?ϵU1 A\dl7ӓ>[[$zƍkM/w>}{fx9y-3Yޓb=;`zB)#ӭ>99+wB~~W}WN$\(D@&(E":REdbMt+GS? mӼ.X ~}1KB+R])ס-1.>]E/:>u>KѯA`Nu~~&{?6g7QX09Pq$'e.(I "qvT Gz:@6PR) u!n%;NPQHx=R6bYJbq,$"dgJ ^/q c5`bV}8;W)&|D/܋Ƈo}qՇce̊Yr u&}s3]W[81ڜ{_gؼ^Az !\ \'YYImG~9m [^ `8?ޮV´'ޖXa :*8 T߾ ^ Lhw9CK煎0޲":'mr.c1m>uxE1n㟧z$aÕV,cl=h+RNIlmMO}+ĂcRC_8BM{O:ɹ8q꾹q,c]7&B~'6?D vNd =zo!2 ݵ<3CUeGxMd_8VhcהrNG{mgpfSЉ#ץF+.|:5qԙg:2owCd!c"8#W޹_8L5ӳ~'dZݤwr?O{J~ߙݻYoo>{N͋l$3_`<C]ؐNs >Ų4PG><3֒Hah LYDžkjS]_G`JkbCcI$TBl8<۾s9T% ~m"ZAɬΧx3ymy9|wIx I-d(sD{;Sg%#{rKYgUu6a55#EKlg丹=U>;Juo3%~sW|,|/J G^ޙBxukuZI^gozʽP73Eg*\ҧ췤c~k|쮟98;Vb&L7=?^f,0NaޓE?k|~eo.Ws=<_-c݉2;u}pz wzwif[}xv~l윕kk]W1[Z,k ,hƲ}QVJ,5VHOP}֊yoѷ?dg~z>*?b|pu|I-a}m5@$ Zͫq;p&6z6}E~ ׶;K[fߞkXQ^ڊ hĤ'bm="vE2h&wv0]oZM"_ʾW~/Q8OwI55|϶ϔw`-99=*]swLuI NAkNu>4::}YhnO.tgG3;.ޙ&l1*ΰ&WeuI^|I\ 6ցΉ,M]ZNb=O{~*L+ zL\V# J܈C4'32ʙ32ٱMw*}4QX\gܣYдri1_}vTƮ7ʙэ]ְs:׶3UփnC^flK}!7#9_OrqiɅ jA:Jdtߜp[ ⧪oeM\{8 p|+Š=?%+B{ztLR k~TF<4Q]]&ۄk!Ulen3WDo$ Q&?@wk髫=޸~:IxkjCTo6?-qQ!$E=X:žrmk?|C'$=rKe2Iݏ9%8;S+'{TKqJމz>9YxZD~nؕӷ[26(VY33MؒcTc|)bk1y|aͩ>G+u {W^AEwХskx#|Pɺ9N{2{s]8ش{dlR_̥Z*S==f?ݏҷ_l3p B/vYx7>m2zz7?;U=xxeV6v}l_b _-^KN cokּMp޺8س}ܞﻂ~oW,c-mF9pRz[Wבh#E;U|%m+"hnPf}]iK]`cvϟGD=~;o[ozRLYg {.uz,NE>d!=Ccz؂\jJ0 eۭk b?ljspM{|RBE$wz5@_cev9%?%"Ypqtm؇nҵJ/9URnf1ߓŀbđӻE#!/ʙqL6C2gs^$X o37O8L{>QkRi}ks=DWV*Sy1Ϙ,c"CS}(lgeYZqiwM"nP5֪߬5O2㱷qMק 3#Zc{gۈ{׌[ohg^-ECq9yeDLe⌨"qok8?3s҆=9ݲ_X=<7/FeHgly>Xw8X`+UK>eg^*.#6O_ce{b9/g)nW/}Vz O˳P|p]`qeP24GB$6 ڒ@9eq5gYx*vIy (@ۙb4s< \Nzk5ʙ'\_ f+ڤMy=Wg;8; []Yޚ>G1g: 輡řONI%q漓p !<8ٽx}H? o\D=_G5Ip @pR \8)N~bO#g<>W?Ro~>bzV6S'[׺N7>vڮ}^,x2iʛT}R;}Zqd}nV8>=)GA{exgo>?U|iߟ?\__sK(C^IG*U3~g.c;osOE7O?e7SQO/<7&l[n==tEO++Q'o򓿪rftLODmۣ7]2u8ӱX=Rac̉k H Q]PQ|vi!"p]awU֯/wIHvW+:$-1.>](bW$X'Hg'\sh}F?uǀ:߆zqCE)+.dX TAH BkbXĖ_{]Erc}q?r6~ۛT>_fm㏉Fۤrow}.c%$$(ʷFXXH"#6VX#mTlYaPƋájbq&푨xkyˌuVMܓ#a!LUĬlFoW-1~w׾/^ZP^ݷ })8bs  +1c:8R"&A"NsyM&r fQ4|]#t_7]x޵=k/ B`kr=,7)j7?1FINvs|)AY$1gEzؐNs<&cYOB5c- T+ %\3gQMvߦ. vGJr7*Tq]Nqez{S:?v)G>$*;5+g4X>[$*S| 9$Tq>ts͡cɅ jAz )DǶOU' ≥(ʚ>]qs[FsOs {<^?]C3/+B+֣Zy-sUn϶U@/3É3 twM dI5'&e{(vJ%x#yG͎ ~ϦxlWd[7ߢN+FPWݍ2|r5/OX ŬȢ[-ijyRXcxµgprGƀ<e;IۜrX ]19{b`dK27Tc;\ M zwH<x{HZƞ+Iqekab.#/u}_yjk[ hLYGDؘg5,{ : Bqza28".1gx/t>93ۦzQ/vWcdў^[[x1]bubin1nʽs~g,.sӍZ ǘG  ?'5cEi+= wDC߈ntp#um_{}8?2YEo+w|^PL{o7!^$Egj$#cH2rII?@,s$;y']k<~gۗcw%aNpt3R@ x{1u\MVϖł5emhwo1Ƅ!zXuŎ^ 5l<==}CrMyﭸ'Gl F0{cR+.OBOk{[gk2{J]C*0}=14۩cb筯) >|쟷4Ƹʉ(ZHA4ޢSS]X[Lo&w@5أ? D=u;o[ozRL6PLj"3$$KN)[>vw5%(mA)i%չl DjŦxmu%bc5kaw'bGѯ\k"]&ѹ1S̍K[[$MڛEygn>,s.vc_c7!&b-9xfjG TDFSy9E D"RIDS=3,ףX w_Xs:7cXخڌP}BDzYٟ & cM>{O޴w[PL޴v9tף\ L5бr7gյ.{f8yVyn϶0лPVBZcۇC'9+Z\a m?؂pmx@@4Dk8N737U VsJ@fŸ Yp NkOCV \8)N k'5Ip C~(7w_o+:ϼnrw7~VLqrzʭs 8=\Ow1qv''șIk^c$Iq5N'\u1]}?ӧ޳uO7?e~}1Q+)rzThzkwY_r;m>}/~qQ <4M>)fXRvyQ됔Eϗ|itџO?뙏/S ]ǟr|4~s9/o6c~:=mF۠ǗuOϱcpݲOӇmx۩sD}}|;uub9Rc_:ќkcq"DA!xXXEpFGd iCeGE4 R__LlaJu`~Mۖ]k}}Y_-VGl(Vkz<kf6]y#Lz8}]7\3o yZBm|kN|?JIg,d6 ?^/ozudck3ᕄ-'sE[)$\IlmMƎo1EKY1vyG f"6G[Ե?ypR)슢iްd>ų\ooz)6:Sl}:&bPk݄,oSb&e6پ3c&ʷo8A"yIfEH2|l߲&8A@6iʗQ$VٮPO NK1Q2>2)Nu}ٸʸgmp<%sYW=Af7_?z6B4# Ss ;ad[V흾彊>3?&}-^gN.\U ITUtۂb$U$'~xitEKEsXཥk} rna~Ǩ D_XEdwm9IHĕkeT&e$jQ?^sDWBe\ifGI;0OBGMBrVhb4}BvKz[k77ݾlY-2^/@.Ɖw,xcv+[z1ߪ&A5Dp]JoqlY"\g\3D"N]ڭlJvy6)~WP_1E~;eCmQŋafmE:k]]^#SW]+mђzvkǩݛct(L]E¦q;2Uqd۩m1ǑmqhL֌k[ضK}GD?M="Ư'߇#T[Ix7 s˦c="*^rMS6K56I D!b[E&W&9\eOX\tߴ?@Qβ -kÓx뙹_n7GkLm ׵ױ֡Xqς7v)Bh[:1/kqmO{ɸa0q)h]8-:fcǯ6v=jGzL^+\̟K{{>NP7K9רAqϖusܵ mSkZUq ^< G;c(};S^uN$\GԢu DoN)m.O[Q3+?Qwͧ/5vM,ަ+w޶$\(ă;CD*dD%U&e6!J(TR-$j[+,.g!:,[9o;VNZ}φ$`k:%Sk^E@bk?4v)E#;Ű2[޼LW$4+Oߑpmɶ{qget5v]ߊ)ηi?DrZ}~S9?U|iߟ?\_ .{J/v3=o命E.iȓ:p)滆 kō_>k8/mSvqv|N:>miw?MU_orewLKy }8$µݻ]U'<mvi^1z֍wo&Ř,:bB ;N?C,{ÞX0=Kb)֣wV2MWN$\HkB)wT[ iCeGE4-`j IDAT R__LkaJu`~Mۖ]k}}Y\^-VGl1䫄8;_? ?L\Nm2(boα8arnhI6j~.(y'%Dҁ("(BO(19^JwRcdՓO(1''?2I^oH-"HL LI8[ɂˊi}w0!b5ٙO}u[Y6stbԞ1Sq}FkG;d3qq'W{l=3GQ5^Az !\ \'_s?sx9<+92{@~2r&*<i#xY"lDGErcꏊdggbM,F9x`W a-U~[~KAĸkN 6JW[:3|KI8Y']y.7;?^ ~K Ϧ- oX7=$LƩYqtqz$dΉ ?Gܿ ;R?Ἳgc{J(݇)D#Md_؏ )KuM\ܤ^ۿ3Yj1g8.8 A.5Z1vQ#Ϥμl81Y|"[^CA1ʜֶ:g?=Nɴ~Io@~d3mwB{qpm+'.rxiO,{)S˽8k9L/{OnڽHqiQ]yܾo_ޮ=R櫄t-e=@X|m= pµ/3 k&gxvKpoW7E|Ϋvml7'Cw?X֡.і^?.\?!=DyqzKXY봜!>h "Q% <Ahp㤖`#݋ޙŸ8+ٓ[?2Ȱ=Őc* 6c{RULCr=U>;Juo3%~sW|T%d2}Y1azyaz(AYK'?G} [|c[Ln9S$!_IhGqKl19nR,y,M9mZ;1ʿsjΎظ 1cOǏr>D3/cX짵dwŽ_ۭ>kzW XwC]\^,>Gi]ڢǖm+>8~=yBt ?3O̪yxI-a}m5@$ Zͫq;p&6z6}E~ ׶;K[fߞkXQ^ڊ hĤ'bm="vE2h&wv0]oZX_pȟ>yy!򅹮+}'a6ip,ۗdQ#NẸƟ>rΔ%'bGe&Da$Dܞbfy%%:pY SU݂|yQLǎw [r3=Dz ,E:{z𺄾}QgzqCa,@FEX޾߅8S ;$s>-x\nړ-._-K1Fyhp8D_q/񈁳c{T^iS߱N;nWߓϸGݳriSiSg8b.kmd]opߕ3 ao=zthmgqx!.:c-ճ*|L ?sM4u Mi  :IO,f[wE[I,]pp]{Os {<^?~5}AbjOwÃh][hV?tNr*#خxЮ\m5VIXX* LJlW"7X}` ;5o\S?Z$<u5?*_ӷ_woMJM\dB#Po8*֮?Af\[2Xρ N;‰2U!x\G{{gJ@wp|@7-(|Y"X?^yl[- U9Aڞ*OKY_4}k]}߯{!C@>R!W|-<$s!Ν-U{]X̚-Ga@:ϦFYal9ly ҇7vcg aMp׈8f5gֶ8c!^*v{%ŏjz &1<M_?^řML819)[x{QZ2|)x=Z€CؒcTa!zg q}֫G Z)Z瑋_?w*ãTz{W_sQ 'vv)/11`g#3qn{չ/d!=Ccz؂\jJ0 eۭk b?ljspM{|RBE$wz5@_cev9% $"zqh؇nҵJ/9URnf1s*.DA#!/ʙqL6C2gs^$X o37OI{>QkRi}ks=DWV Jy}g )RCaK>^.cW37{km=255㊲@7["}kRݿ>X֚vҞlsCڞ+E23Nkν(lƻGɢl[{[$M7Rxwt2Bн'[WkcS;^գC_ 錍={.ף Lc弊~ǹLKYEߕҥwD&y=r֣lO۔y^`Q'o= |Zs@BmiΌ$NH3qt,Db-/Zk cR&Ws葅Al`ܛ-ĹL!|L  8w5xष!ZpylMz+DS~%xPqؕsq<@| Q9z<딤Yj; iÌ@+m)c5`OԳ\+pT \8)N k'5Ip C~(7w_o+:ϼnrN[bzc+ۉ|vVp? x~.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'޳ݬwKҮv175ǮrTĽWw~鿿k~.+]__WM?ϵUI%82~rZ5}wLJ|?=HuDߦE4v9Xtwk,ru+g}(c䯹|A7MOfz"ʕݠI>qv:N ˉki@( .buc/"b|k"]!A0mluBjW]w]c?L_L¯i۲kroc;˫ꈍ=&|ĬD89  .dlQD#1 >^ep}@BbK ,J/ufe,+zdjvm9y G'IԖb!n7DЭOeߥ."bKqx<ākͱccz.:ӈ36gJ6(q-AtTD-1Iv~}&m'z~ vy=>خl.X[E^ޏpMtE|mTH8t{udc,꣖OoF&IuR9ur먱Vv/yQmu"BC_8BM{G :W$ۼ׊T^,2$ -2d#\L, UMݛKH3+{Fg_ٿZp&]~U5 gq-ymR]>vc8F‚֢fEjEA#(ׅ̖~i e[1ʨ~noά&ogYٟW }̈[g%t=9{/L %uo A*f?17}kK^xiA}xmt6{3٦>dL=G ?g)\[?Z0&n>,|7 mylWdqzsYj=A' tR\DZ HC?dWgoښt6Y @XQK2"\}~7r{H譇˔㈸zoKԫҝW{b3AS'd8pCgO?u{譇kz.r:4_7^ v9ztYuCڍgqNss=ߤ]lY!w{u^ML_ix:s vooykr?gʂ73϶ZƑs3p": # zvNoKLuQJ -7;OS_j8[Y,Nm7 Ā)&ʵ6>Ӷqf:"QYMdDdIԉ6EnMR51(%4D1:mH4m1YBlO5|L&xm=:N"V,(vkM$:_!ryx }qľ)Q{(mԇ%trnaLf=dD[z=|,uV ?=bZ,QhM'+3I`M\7{LףX w_Xs:7cXخڌP}BDzYٟ & cM^ 'N}v EG*Z1]gXLGc-kcn&Ϋk]p"i1maqVB1kv4Ha1g@+?l^yG[~ޚh 5؉f憷!\"}|5jVc޻xW Ls!_: p p k'5Ip @pR \8)x<^|9/nzt]SSz/)_N/_u56סgw.7x9S^y;t{̞$|;)_qX.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'~/77kݣ jNwfM !)dk/26?5~3M_?6hr_ߪm tzvm_==Ilzhu?M~Gmyoʼnk8+`Q]DGoŚmW~{"/>*y](vWmbzZW;&ӏLۖ]+}NrkֺG.w4C WOޒ~MaqX zI`hDgScqK4 ןN1[ ׉ `[QfnZ¶a~uy"'\LOiz?9Cֳ7L|?U yվ72k9?DA%Laߜm^'qm?*fv5}(.`~x~ vuv97]"^Xyү"1ϯ9)+%ßt$Dx1ՑeτWmDpY']y.7;Qv/IdQmub+<̛͋lQ׶Jhz6_[K1+yÚreDL!^ @A{vM-=OJdz>*ΌBO+ߺ)A&|_56>>4К~:6q.[}S?ęGD_ќx"ȯ?:{}ߥsgNْk188v{LEVk⮃?p9K Պ`3I:E<+ w:µϻp]O.\wbhP~Ք8[7I)&l瘬K$lؐNs7&cYNSj!IcZČT+/F -k(d#h|/INbj1 @~$$  #T]^{ݗݶN X>_>y=}>݃kF/ny@ga,Dz?b`?>ǰqKe[ޫ?cAGg;U\ZmW}9p8kG3gs ,ku, |eM<*J17AȴBz't V+UIDVܚZ$_'v׺`,~(x[{WD–Q`nW dI5'akndY'em$6}0'jNަNzP?/P_"\ Q|pppDECR0&vݾkIV m?vw|/OwōS6@ߘ$'e\i ctڤluh34;bxkb^cˮ^y*NE6ǟq6SILg>&}-^gN.\U ITUoǺo-RIxb7+":jC;uA< W??.!td{8}$pef8y,$qZ8IY'Zԏלm7Qen3WDo$).Ql'̓|q9韦gQ<M]-ҿMyk:}Vb}}AC"q? إʖ^ Љv=HſaK?C> mR1_1}#les̄1Ƌᑸo){^wwk6~ٿY} uxԯ/Qؤrܶ!mv׃̧}iT#JX B۽k. vZѲ)tvi =ڞlha6ڰ84׺  D0|-".^rMSwI2aˉZSW]JsZ%V&91o/HP|UMkՉ$2K"\(E^)\og0(QoEg[ҋx1\P6mR[b 4=vʆf9 +֣t]tֺFVޣ%׎S맻7 Qę"WoMvdnɶSb#љ!(9p]+TZwivoc/76p]מh=~3o9M6yoUD&DU$A䮛-:)Kmdk%lBs1-UM`LsRf˞ i=f(\7e[O'X 3s߿8oJl ٙT_'kkc7aeCu&oRжtbx%^ ^g,ēqM}G0c/\[uy{z@,By&k9Gc{U PJMw5%ZmB$Nm;.Q-Υf[@';Idն^wWX\BtYsv׭p/7j/ IuK:= r8e'Z1`iR)ŋF wa3el=gy)ZHiV.#0\m?X;:jqSo=Ң$丝.1{sxLkt.6Wψܷ.{}?g}ǾnU\1q/m 5FOiS~pm("D&86' %aܺLp@p@w? 9Ip @pR \8)N k*Oo/_N/G^z;]yשtԔp,j1Sv__Nw7^\=qs>Yt핝nj3o_{YjXW5C'ϖ.g!`st[շ3}=[(*\=/wJew}u%veiMSޤ߉Qь5'AQUvUrg3}M]÷pCM_]*~lϟMʎIH$F^bbq0KLOcGnydoK7}q q);Zߦ:3ƫZ>bqFgB:u<䷥4=V.}ō(#F;SnR{^v%BIkw&Ny$o5G.Cw8kzmϘcZ=z7PO#9oos,x8l3p)V9gE!\+,{yո3ܫSSs~?'I$ R ""VGq" F&(r⯩ކ˶i^,wU}G[%ꎉ#ӶeJ\ŚѵˉBKМ$X'Hg'\sh}F?u P!ix ?4]PB#\ "9vQuIk0;8q$|BI1k_,K/Ne: ,LL6 MLIIC^/+2-'=+Ʀ=joFD9v%NxkA~W2Ƈ6:}\glJ C}Wڞ5.\Z  K2B ":l̚:y6sNyz]|8p+9+\&]}vv0cvLTxxZ F7g+e[׉zc۷kv5}.`YK+ w`^zu uo"q/7ӓNjRIF7C{ rC,-e*λkyf88$2C1I<7} bRNH=",d\SBK%G{mgpfSOd\(ǑgRg^k|s,QqL -|T|4CZƽaY׾й0iO~z>9iw޾߁<)=~gXS5qd׃}I?.R8q=ǖ21/_q;~O>ƽMƈp^踱ߡ14l=DbʞX뻺_ey>Qݛ Z0cWUxC)+DZ/?q'ʵuXhD#EqOU9}b{pѾAy_XyXY V$VDĦI",7|] כ z<u]>p݉CIp{t.[z payP y!燹 \B|XK$=KK~gϢX=.\SM]?a=ZZؐk=Xɫ[6$A*}rms$B //9Dkkw>봜!>h!~|wEǠ|b7ɫjB~w%JF8bԞ׌E sq>Sxmkώgl;\c"8[!8֓ܵ{1]6vk+q$ן#V޾߅-@-&V)%5 k't7`:X'48c{g"~s iě.7gL?q,e2}dϯ9-= ߅Qt|f?gao2?9ڙ{r(wzIzT⹹qWѻ'e tɶ|q'Y,h pXVr/ʦxTbnYiD!tm7Z?M""n|":Q=c8^m #a,>{ՇK4c{Q?^=P{u5n'x^[F~o jNY淧=ZDB>atKm72Ā,>} N ג] '+:ClYM,/ue_ #`b<'rbrNJDz}I7Ċ޵*ŋ#ܿ3%M#ٍ.^9;$}%%s(ĘgNRNߪnAN^h7yQLǎw [r3=DzJ |Ő$tZ:Gx->+Ƒ>,o_=.i]L}jccpyh]yW׌ͻ ;W(Һ]Z:w>YUB>cymsr:ZLz;}[6m7NKY]EM!#\{1b}AOmMGֺ g/VÍxO{r*0<{al{> ^_ Q6sJ* G͎ dwCp#\M_]5#>NXk<?*_ӷ_wo?WH$IB&Zۃ~)Ϙp1,k[cQ{ Jȩ]챽/VDl"G{{gJ@wp|ɵ%@VTKbQ[<6vĨP՞*sam!3@X{m1۵hQ @}=qsY%;kX'~l0N*\!OYXvl%,!s޲mR bcG:mGۓ ~mUQ{GkÍ pu>ۆI}6|s>@릮pϵhK %g2ow5lX>ZW ھe %s;t(1X۟}[m7)h9| 6ILNEH JHX~/oLp|9ך'h$[#G#cݡdb35>;J l['悡8-Y΅ }Nb=ξJ>{F'^[Bm==_چ]-3o/λ]qa'%)ؕ[3{eSιF~zȮ|.e@:ϦFYǡ 0kqfN>ܽ#dAsH/P!Oc}Ǭ%:I&"K#|;V+ch]#>1<M_?^Y p&Wv_ qc?8:{}Gh޳2I|53HqQ=f– q8[c__lr8{жҷ*ٗ+փ]ebxOlwR=ۛg}}=g !v>YP[a 1 ={=uM}G0a1=z,v޶kk.ouYO_B(ر3kOdxp $-SAe4>؂\jC~xpmu%bc5kʧb=8FdY<5@_g/r811:?+q%B5Bc&Khҝ ^e,=/v14{ZӻJ 0QΤXf=!c~8J8~v'jM=ϝrMcwgҾ*e{^}1YBDԧ&PؒzXb=ߕD߯{s軼oV׺ keco'TO3fFf`\bcmqFgq\]o3Gǒ=9BS:"*c~Iń|k-L㌞= jNsΟI;4emsڧݷ_GZ Uȹ5^"uvpfߜ@8q .4h&JTM"Ph '"'%cуl+wgɃxqvspo^8G v>3嵠.,a9 <, \Nzc5ʙ'\_ f+ڤ)My=Wg;8; []Y; L9OukGX/p䷃!ZWs W^DSyf; iÌ@+m)g-Ip @pR \8)N k*Oo/_N/G^z;]yשtԔpbw7^N7_'3ƶ!g+^N'\u1]}?ӧ޳uO7?e~}1Q+)rzThzkwY_r;m>}/~qQ <4M>){پѿYXPsߩ> 23}M]÷pCM_]*~lϟMʎH/%!\#{֪?賷.cPMQjOxth]C'jDL;wӵW9Kẹ?'?O >u'c4=Ne#>eotgG-q 3'I$ R ""VG1v" F&(r⯩ކ˶i^,wU}G[%ꎉ#ӶeJ0\Śѵݡ=<1k ,A`Nu~~&{?6g7QHp H#mG,\#$RXD[ 9 AX#vcp*B!i|AQ"96"\oYaPƋájbq&푨xkyˌuVMܓ#avW/sݸo<ƅk#ͽ/y4ذL/}zk>}X@gb9މ3'T,46ppmOLjEl$> A"j|v9̀pp!\ םX ;:G+\,fՉkI6ǵr#>mk#x0(zu]tK:Z({`==<\G~5Gп{Ob1.#>8fsP`adz؋O%{o~!*ᝑqN^{^cuٸě;[ekg[bخ-XNQ ~]ucY,h AqXVr/ʮD dZ!=aq^+UIDV횯Z$_'v׺`,ډk7wC$)v^Mȉ6ZV~f3erFv2'6JUBv5~i ,PӞp-k"K[_!| XK_Z{&ZAqO6lvx roXzG>{UAm͜T>=eb؇Cς`Ee#(j_d#\3Y(Qo5鯇='9ֽPc7}|ոy {bljcSںSt]bAopǦrO}#^Ckmblqvvyk,-kDe3uY{uP DUvlo U$'~)"r('\{1b}AOmMGֺ gw%^ 'DMd5&\ G'^ԏלD,m*H$e5;*prS[KG?kG^y^Qme~:XA_Yw7&/~;ȹc,dw7#Ă=Ykf+ϐZaHYGX뼞;l᭶RcmQE&JayRYyF|Qn؟Gw9ܢ1`/υ}N6\k -':<{o=y'Esbڽߛ%q?=9"(V>7g!\!|˱9 6{#P{F=;,t, ~#Ο AvS1[ Ayw[cfM-h{p6窨=B+͘qukM&Nr9pNNW?ע-1p"k?ٰp}nǵč=Io=W7&\(?Ӎ1ϺoŠZ %p E 1sceqD\vcxX}^7%U}ƫs=yjTcx/Ặ?ޞ mݽWw譇kz.r:4_7^ v9ztYu3uPg|AcɄhX kL۶C|{臾ݴGuD?amAg}kQ[8nBH6fIF$^d$/~&X-p"mI8vNl9 5y60*\Z/8 Jœf<륀2fC]-!bݛ۹"ƛx>i )ρ>#XKr=|-CDQ-a`:[)ỸEam`=,Î^ 5l<==}CrMyﭸ'Gl *}VsʻN3/ۧ#sݍ_S\^ o]LWilGs٧__Lofʾ;D}+=Z]ǗN۵Oߋ_?\qO7MyOJ'F^kXhcjbKUIY$[|ϧ_/O_#|=e*~kSn󗏦o.aOg|>6eslĦ&\l~wD~.vGc?y/N]:p8X{XAWEN4XH&1Q,J2@EIGe iCeGE4 __LjaJuDŽXi۲kIvn;"" wGWOޒ~MaqX zI`hDgScqK4 ןN1[ ׉ `[QfnZ¶a~uy"'\LOiz?9Cֳ7L|?U yվ72k9?E%La%EX+N)1dggbM,F90_]]q#z8}]Mk(Z*V'g!1ϯ9)+%ßt$Dx1ՑeτWmDpY']y.7;Qv/IdQmub+<̛͋lQ׶Jhz6_[K1+yÚreDL!^ @A{vM-=OJdz>*ΌBO+ߺ)A&|_56>>4К~:6q.[}S?ęGD_ќx"ȯ?:{}ߥsgNْk188v{LEVk⮃?p9KڊՊ8H}cIxXvB#ZIkµϻp]act=" q}S.O{m_bғ16q&SB6%T݄c ةNDw,IpCS"<$|X{eH9eMqLڦ}*_>GKXZ~cdB=%;I,"D!Ȥ8q^e*olk=bn<ϗq~gm_C[d:dc? Ѱc4+آO},1uXY_U)[>x>Mpef8y,$qZ8IY'Zԏלm7Qen3WDo$).Ql'̓|q9韦gQ<M]-ҿMyk:}Vb}}AC"q? إʖ^ Љv=HſaK?C> mR1_1}#les̄1Ƌᑸo){^wwk6~ٿY} uxԯ/Qؤrܶ!mv׃̧}iTB#B(*[SAܴb/x{Jvma?WEqh$\s QD-jsv_`IP/&|ĩL;ITOL$zDy}%ܹs-+✘n$(\N_>ߪ&A5Dp]JoqlY"\g\3D"N]ڭlJvy6)~WP_1E~;eCmQŋafmE:k]]^#SW]+mђzvkǩݛct(L]E¦q;2Uqd۩m1ǑmqhL֌;.Zm;ķtyM{D_kOsw?b1ǎĶ~חF6c-=.$Hz]7[$NuR,(J$+0bl;i[諚^r͖=aq~zGQo86,f˷Ogqޔ3 N&\^n ZbM? إmJŵ>Y'> y˜kĥ>u'㴄蛍O_1zvp#3.mђ\:}C,^=[b6q>7M-hWY{zn7܏ݣ&>#rߺ-nUqƬǽCg(|v=4r_2b ٞCq}2'WB4| "m t_SXD5x7!Zpp}7 [[ !Z8' \8)N k'5Ip C~(7w_o+:ϼnrE-&Wpcݙt+۟/`'U8<3q&Wn*g}q,"֤g|~@\OwZ{g}Oދ3{>:x=8ptLF>xy<} ʯ>~<}Fl{ODݷR٣ݯue>o||]EY7dzӔ7wbe{ݿ6Ư/8=5G+9 L_;\gWW>~0mgӟkcD=r/1Bɶ%n'1}#m]ٙͳµ{{dBȽVCk 'nϡ/6Ɗ{9y4&?s>@|Cg_A>/N= ~pۜg<:OliV0Iv~}&͢k!%UBGj/uR9ur먱ïS% 7l"NguSޓNrn doq:KG׍пI1i;'JB27s\KKJZ!*ɲ@8)7<7}/bBC}LoL1qVZ g6ř#] '!(ץF+.|:5qԙg:2owCd!c"8# ?3\5s/R}oО|;s>'&}yS{δϽ|jf}񺘽+5m_k=֎I?[`,~ޙx#{Ǐőv)\7}{qۯw]{{-IW y^~/|i.~ :YJsfl/>Gó;}Z?}EuUg<:Rb%&A"N{}cIxvLFoW7]x޵=뺰EcesiמmqFx+\@!{vXz!|lubC;I\B<|XK"ˇ%\3gQMvߦ.N)u g%P e o8S@؋ރIi?^ y^G[h&~0Éo@᤹tg1vqRK0»^YȞg?DDMF}My(nd?GY۟z.6bY*%:7w+Dq6,Dro=kq}CXk^]k» /|+zoBb [ΔW;~3,ئ};3+cs97GEēcP^YfX318t{-3Jj<81%~Z{O]^ia.|0u'vs+T$c~*g vX_- \IρTk$b~΄5_֞9*fskܗ,.+O9=eodп+/qs|Uzn3<\ǻV+ih,sﭴυ`^rsn/ i_^T~-}s6M鷲olwvp{gKg<@ A4tS1[Ayl[}XjEK_-h{aPp6窨=r.jx?pԺDzh"={$|,$ݤF%%5;E[bebD[N_? g\+oc{cG $$LKLjݷv=-pF'jYym[Nx -J޻ #ܿ3ʪg=j󭭵+XdX/o24v^XRs s ҙ%io [6}Ym1v Qqv$֣~Tꃿg3ů}q$ &No՘.9Mk3bNHQwd7q7i{Keo&TU{{0~hoa e5[yٛ)ރB>X;+Gݫx6'5cEi+= wDtW]EækGcmĶ~חmѻIDkB<4"}H-C?4[sF}E'u*W, 1ƮK,b<딾%aӉxSmmX;Úף n[.0pWϰ> "W,\w8pRzVבBeE;U' xBkH47~3پ4@$ _S|]gL" ^a?*C<= Y$}ȔCztk%չl ,Da"<ʶ[zdT ?1F'HdYP[+k6Áп9ʲrJHDb81I{ݤkr_sT1!׫ܾOc$$Zӻ#]LJR8k&!k39Tq,Ǚ[!0$=5)>w5iݹoIKX+>cvOM%v/fřݛzxbsQ>v}N??޾O5_f,̌kkTc?Y=Bk+]6F>Ab%kJ8JtΜDLe⌨"qok8?3s҆=9ݲ_X=<7/FeHgly>Xw8X`+UK>eg^*.#6O_ce{b~ߦ}K{ U婭W|Z3`%'?6Ȍ$NH3qt,Db-/Zk cR&Ws葅Al`ܛ-ĹL!|L  8w5xष!ZpylMz+DS~%xPqؕsq<@| Q9z<딤Yj; iÌ@+m)c5`OԳ\+pT \8)N k'5Ip C~(7w_o+:ϼnrN[bzc+ۉ|vVp? x~.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'~/e,1~~ia9p}+;W?5?|?ՕϦLZ숤[BJj?Rhs>{?%_ˈoy}v_:.$mx[~M̚qA3yM^Y,N ?7Qt_syko27Ux)#Ү@Dg}qbt6F:]|.c鰱sD5 BdQTA0\(.."_D(.DEnB5hyQ8Ehӻ$[R])ס-V/ni熾/!>/c;ko5 0':A??=E3s$8߇6ksj” ]B\`Qbna(I""A KqB9DC9oaX"rso~~ff >Č`"okC{߃>acwMbc6d>8sOliV0Iv~}&͢k!%UBGvcpM‚֢fJ\>-jof!6oL޲/w+C͙-L#Q8+$'Gz񥃉aY?UiSw/b[w_}q}-6}kK^xiA}xmt6{3٦631d9,k+VgT+'@.7v7mylWdaqzsuZ]C [0V?1vgw-qːtl =7ɗ)iϟE"yVpXX`[Nϋ i?'c20eP,P3֒OrOZ;ӟ}qڔoWmkq|-w N 5OIԾW^s|j eh|eg?ޓ˼|ely?:q-VT`grm8i c-^7- lW`[(E:}q:op]ۨ9ݳ}"yuW 6/\#և;gO9 {d w&O^k:h_mᝑqN^{^cuٸě;[ekg[b؞{=w鳸0-a:acYV\$4RYMȍ¦'#+D LS{;M_޷/Zq?M""඄׉rb=#AºS=o ŚX7"!HonBN DJ<5)3=?1Qbx׶;K[fߞkXQ^ڊ X?Z3ՊM ~9dG'CwϷKHp{K m=rܫ jkvh(눵k->g,uo)`T,L ȒWM vA kĞ1'׺vվJx澯oԶױ>s{OMmJ[i8e#{s쫙S׭GcPZ$[E]3`gwF}Fgxbcv{Fq:)fI^Ο A@ՂtUeg&m7md7 +"[S]}{b{O='\+wcp:$|џ4ϧ:_1E/3É3 twM dI5'&e{(vJ%x#yG͎ ~ϦxlWd[7ߢN+FPWݍ2|r5/[A{Ka` -&{pձ[|?޳?[XsEy {9#c_ kiz$mk@[Nxuyފ{N1WT}Fqo Pg!\!|˱9 6{#P{FY9Ip||M -c^Ӈ ?'h舧,8:bc9nٶN)Ԋ8 XtM ik*EadD8~S{,S#{Ɗb9/QJw3^C;PrτQe {3 #1.F/\죺N,6MW]z5]Vo{̉)59Q]o ?'5cEi+= wDtW]EækGcmĶ~חmѻIDkoeK~ ",?S%xEN"Hb1m$I 8Z( \èp?kݾ(+ scShN˘ uhދYwo6lo=?3<7b-ZYz 4@O"\ h=3.{c^-8|/[Ű-_C9+܇=z-ފ{r$ƽB+L+j?0{ө[7~n܅ʺwbJ{T3ՙchS~{+[_=S}-9?"t<Ř6=66P9pRzVבBeE;U' xBkH47~3پ4@$ _S|]gL" ^a?*C$*̐,:ѦoMהI@&(TR-fBQݗ &Z?= DmGIrT|ފEڎr4vDK>X.O17/o5n73jo坹㳄_ι[ۍ}9߬c_♥ΪBgSL‚EYGKgOWKľ݉dgQ>,ףX w_Xs:7cXخڌP}BDzYٟ & cMN /(7 #0EіIciUQ{XpXn鳉Z=3iZ<7z{sgb jqFX+bTZcyS^Ha1g@+?l^yG[~ޚh 5؉f憷!\"}|5jVc޻xW Ls!_: p p k'5Ip @pR \8)x<^|9/nzt]SSz/)_N/_u56סgw.7x9S^y;t{̞$|;)_qX.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'w^mkmĮuw9c_OQ됔Eϗ|itџO?뙏/S ]ǟr|4~s9/o6c~:=mxvm˺1MMn`y6]9">~_hu\:q\űph58pM"?Y-ED"RE@oMP+_S? mӼ.X, ~}1K½+I5m]Էk;km% ǡoٗn> p%,%Φ. i?~bA8~ֵpm7EWO|8~"r܇g_o.?~}oe>r&*kf5]`WW64Ys [ֺgBijdubn/_sSWJb?c$Izycԫ#+\ $l9+J%N*gn\nb5v|)^Ȣ>Vx79 آmϟplk맗bLaWO5%)zxK֙bCA,He5Zc&dZ {6a(/|U3.Wu|SIɃXMµk"m}|h 5u:kml]9~É3H9D_]tAK&Μ2%Njך/cpgqZg=p]rµ'3 k}}C7|.t\#벹pm#'\[k6jhgjΞĤ'%Dcr5%mMlJ [9&ǒ2 >/6SXTڧDxH1#$s~˚, bڦ}*_>GKXZ~cdB=%;I,"D!Ȥ8q^e*olk=bn<ϗq~gm_C[d:dc? Ѱc4+آO},1^DmsW5:r~}1; "aK (0x]7+ڄk2$Q񚓰R7]v2,6JUbv5^PoS'Y t=Z/{(A~MEq "fơL)zon5$]]h+{ pfI;^;>ܗ?) o? 2.1Ƌafmv:_W1omx}zvk1fe}" Z8)a 3b>G3'$*B p[ #P5N_e}WDfQscbgG='\+aycpv|8c"Z7{Z˵c{ڄ/3cDm%+ĩL:IԢ~$lZ(vJ%x#IqG͎ dw8aO4=繭,h,-hYMo;o\}Z eR_]Y.VbxN l!Dv. XYhʎ!CLۇu'cLٶN)!֊̾8d vu|AAdh%d@vlo϶!" %w݄o8 {' D/5u5;E[beRӭ˧[$/Y&XQH" Z-y-5\klvR0VĉK-^55oc?&ʽ+ưhOcl8m6pb=x1L׬M޵Hg˸kkdk=ZR{85~{@}E)º~uZش1nG*Θl;-8-m Uњzmy ۶#nFtӦkZ2z+c׋x#:mءm]3vc(E$QEK "qfIFQ&](D^>c ܹHB_ʤ8'k ˟Ng?u~Ya1[emtxp=3ˍpͶ^Mu2Ặ:vV0^:]k2Y.EmK'Wp-ubO<Y \#.8%\Gl}ư>֮G R {asio{ xf75(nm }mCk*dߋt|~6зag Ή(Z H11ޢDL{}QXS"V|Rc` `կ7]_g?m?gCoHWސ Jpۄl7]S&[Iܶ؂\jtRDVmuw,D'`+<'mp שSoِl-\DZDyߣp-׫#QqRusOQ.Rhp6s[x~;{˛:f{; 5ٶqS9ӺƮK[7#-:HZ@YoowJ;ʹGbnqU}r7{zVe&s'gD Ҽ}Ɉ1d{řac˜^}N\ ,P: "m t_SXD5x7!Zpp}7 [[ !Z8' \8)N k'5Ip C~(7w_o+:ϼnrE-&Wpcݙt+۟/`'U8<3q&Wn*g}q,"֤g|>+^Pk|rśL9u4 pnN~bO#g<>W?Ro~>bzV6S'[׺N7>vڮ}^,x2iʛT}}զ&NZ}^g3cUrg3}M]÷pCM_]*~lϟMʎIH$F^bbq0KLOcGnydoK7}q q)ЊX䯦$xkaX}Oιд;}[Iلv; ^YE(8}xq^>@<\)Agp}aAڳ:c}6WޓXw??z´[e[ } 86Ev~}fMkh k]]:/X~f]Z} }gjiw!\GEo聛* FSNЄ>jzfm{N:zQ4c7s!QvymR!\"N?WuSޓNg xoQ:KG7;NI<-=ɇ'P_? 7KwL$yw gNR%d"K Ě}N`%mgjy1\g+|]3/=ZA{335ks*,k# =&YXYְX^C{.4vq6?ža=s2|7pyOA;)e~ zk(DQUuL caeVYɸmKpLjx^ȸߡ1$d=DllgO];]eaZC}ݛ ܵZPcjUXS+$Z藟vNƦcP+{/ȩdTr8G=B')\kq>AM'B "'\kۄo9]WsZk6bdgn\?Bomv!=` ЬI2?+)ɠ h[y`عzK,%G$Or}|݃𾴄뷧?~,uhS]_YÈ(Vd_ྞ?S,U-V sGz.;5B&Dk(sҵ;#?-gϋuy$O#^vXI U37+DJ^Y׈݊ޙB8ٓ[l:uhO>ӂ"Dqxlkv^踊Cxmk՚g[Z`KAp)KdZoʭ}mU]MWtp[[$?b ٢bu˙BYU轨}Ӹ/=e_+-Ɩ}h⹰sL1;+9gvOL\wO.gr>])XQTl QPu*)+o$p; Cl ^߃Xµg?|4^?DZ]3kV6w@؛}5JgExa:= YZWPƌw!lw2ooaen5W^R&t^Prʈ-F%ۓRLH)~=ڣEab͒Xd#cc;goQ"'FZĶVفY33Rbo(O%x?Tf{PFυn[$Sym%O;NŶ \E'z,~- ñwL(6?B/S\Yo^]넟yĦ$Q{ s}^?n%C=Av%fm|*m\q֤7<3%Zl>Fbxgw^#}c{:]˻<]`G*\GO q 16 om}j7 ZdQi}6 hY Cwxͱ9* A˺kaxk5=WƜ=תm`(9SVVɰp}Z/o%};%s;v|o/JJ9zokvJDb3m;xך ā^ޙe`po#vQe-#c{v@OC<ҙך5;$ik2[6}Y ĵzn󇳷pυۋ3=/'[q꓿g8B$f{{V̱s3-km3xN o?8kuqR_[b:ۉ^9ѺWF=%kov {bͪf|*sdk9yNvCmX sBӲoGa㐏v8RgX}'ݓ{-&\5Ei-+F3Y}CȮ|.2 \Sgwuֲ}=1t״?gQ_Љ aq­=p"ƂkC'Q޽HczI>[#}`BJrьVJQ'ƙ63Em׎mqu Wz_Kń\[xDlwHyqP?sil)1ϰW}݌=}'T4=3ҷ:*X2sã8?sJ˽+Z{Js@~|g1äsϳg1)={3וu:ubS}?+{*ͷ6/n\yޚUu޽71no\;>/|Gε?4T'9=3Z[ 1'ͳh>ޓZUX{=INFډ:YWZ AsowBޚأ0HW_}]sLTv7Ʈ.D{x  Hx9aq1A{KhҜ :JƞgD =~,hӻJM9ʉ6IYǚH8k:mD2j47}}m%oEՉkFg9r̖r\ǚϋ37+z7M6n cvre5eZ?޾$צ 3#{DLa֬7_]k{}&L`۩ce _rTKv<{w;TIcx<3!!=_׷'3֚3 ]S8pjlG/y2~M_݇k'`imv>ˤ=nged'7/qq .@}W2O]!Q6p2C ;JDOGHhυǢ%V$.{g;p,<8ćK;s hD0 \2IHQkpoECNk!lM~D)\ܫ8yxhqveyQgeb%\ q @ O? Oݛ9$\yC:g;8SlΔgwS p @pV \8+ kg5^L/_^pcvzں'^N/>Pt;Sl'ڛ nyڬ_M_y8ޟtG{7D~}wcs;_Ⱦ:}~j'ڧe]?}Tq#7UywM"_mVq}G]{Eͬ#9Tqoϯ~}lZ;??}2_H/ƥ%!\%; ?fo};=}gONoX}vGNomi捏~)8=R`-ӋtϤs;Xe/[\oӓZ[{x:=b?d-ebO܀M>qq81:@;:]}Y \2gHOEkxx"&j jBU yQ8ҮdW;AluD ܮ7k՚w| !uE%۷ǦyV`Yps3Ag%\k;kS}&:A1(/e\p}q}=Hbs+u=D%ڮ(WX\p}yѭ`pb$ Ln͸*1ytGyP+m}""w⢢g2fH]. SW!L+%XEp]lS-]`ggfM$&9(%%WkV9٥-ٷP!wƮ֟}un_\o(!ߖoOE$_~S{/# C2 cN.'^׷u.i oOYDžЦ~oWG}Q|䤯.7̉\DžE97xCk 1/_v95e`BN'ykVl}aI 6'_rბ5cmk#41IKpG[tOnbEHJZԋo?sO|ߎd;k/s4g ۛV6edͭ:4Nւ9אh顅:\=s;{ga/5z#x?3r?2JߓŹ7"f^[k~zo%sc5̧!8 4k \>!\kspHeH8BFX-pfBSlWZ'5_!:r~u3Β?a L ;+f&XxNDsU?]I:y9S;cznf[HEBv1ni $PӖpu K[_%|'Z{*ZqO6Gtvx zohzzbi߻=֌UAh$v5[@]}ښuHW hP",#acFlCJB???9'uq_b8.ڻn{| mݸN0AUͿ=6vh֧uR,Vq\b*ٛX:#=}^W?ј=kVdLm7Y杯q2O#ZE1|M,uY+L4#Qad.aVj.a YBxb|+Ӻ֧ۀčlw?P댎Gw9ܢĽ\1ǯυ?ouZ5`0Vv➼Z%`c2{(3{crgX`S0 Փg)\a!k~>K|E=҇=^ ǬY9Ypfl:qiW,N|*\GO q 16 om}j7 ZdQi}6 hY Cwxͱ9*q\%2K]oğ\kF"f1bϵjI3(ow5?/\sg1QǸBc$e>>k]yy>-~DL] u1(@5)f(\8CcqX\vcxX/b\szO_Cڊ[ߛg-z+M=bžP׈ƸJKѺ&1Er<t̋a[. ׮hMhQZ JLV7O}#i}zRDo`zUz̶(n[j ;S|-(N!V'䷙ͬsh$#SJ2RHWI?@fsᶍ$;y%]=[a@:ϦFezCvN^ߌG}dl+E5V̚{q=@v&֚>vMHP$`rBSfslDcLL<~}bLl\>&\x=ZN"YV,vԾ$:_1Jyx}qľ*{*mt ;k9s+{/Q+qlK?,Y -ToM6Ŵ$,h%7|V12v] v1QnG s_Xs1lmF>#cY[k?i&鞜cK;"NH+7'(Ct7maޤ晖b,k cvz>8w}]p p"ͱ5Äu/+ckv!NĜ-K`|y]DmB⵵9@5䷢!Z1@;V1kZNķŘw.u50\hLPOY稱P|zvL)qvgš#gk}'n2IIp=5OtIy4ʯ?|Ci)w^Lw?=fL>1컮w>vҮ}^GUY7xzS7q$"զSvĺwwuL"t~ ?O^ _'õL>}2/}o6Ljc~<=Jm쩴A랟 67e-tQ@vv|X:sX{ny/"\gp pgH˄@0EELb"80_C% QՏgC,%G<+j_\MU+傿OU[nWj|X;J{V}$($sa]/q23Ϊ. i?BD[⡯>/º}GNqz>G|g_/|lguް| \BTx2@o:t[ }86E?vq33o|8LXr c՚kNv)}=jO[VyiyQy|O$v3F2;?,oZuxk3N–-_rETGbJ$2d#ElN6g!\ez6_[Oؘ̮$ _Ŧ|H_G y-eVmlF?V㟯W[39̘(d[qm8D(V|_Z@^oy^ScM.>]kgbw(?8hk}<ºٛ'.5Wqf[JuQZy pQ;e'zϘJT m&Q(-r6O4;}Zg]&\eh|c-Q}B[HIϐM՜8։[3I9j&l甬K$lؐS7%SYI:}N$yk3r}N2 [A .pC+A[$\3Y^-\?1],bx|rʸ+}*g[ۭpmgs[ZES`Yd:ѲI0}fac4+"?z:~)?3흾*|g{yQ(k!,\h~ [!9'cͻ}쎤 qڜWf?(I0s!z ?z[k97;}6mAP/D naăk--?/y, Ylˎb}NoeC,&\6V u¾kb[ρ۰GkYu'4nbmoIrwn[,m ?5#ЙO^CPsV: H²!f qZm}j7ZdQi}6hC欷ck_$XK$5U'$%&PNRV׬2\Vm+┘nAP~d )uՙ,2sȝ"\('pM^)\og0 )P ފ81ς5vm+bسV۸P]b}qzd ͦKWŊpMd]Kt|]]+h_q<~{S=8D~eZٴ1nG*Έb{h[qk[kj&\5Ei-m6RVO26 ko|][.ت^Շ/,Y7îhxfI&QB'M!P1|fsžʤ8%=[叧қg3oCb6Z, zfO7Gk%ؙED_g=hbzuk YƮEiK'WbŰ:bO<_H/F\#Zq2NKNkZc]u m?<#S ޽D8򹶷Gkt>9"Yc͡=|ꖸkҦז?Uq |~3AG;()}žvINbFfYQfD$:EMMw}'iK= $~E5EbcvjTμ8 %]7&C9"/:aN7!Ѫ$qNnq w.mT$E[\," k:_7hYu: _UHU̵?UyrjbXm[3[ּĺ[3DiV|kmX;Q?J_UqSo=ҢYPq;QW{sxDkt6WJ߲\vz!2=)'s5;9ij#UKۇ8k2'ןkv|+ Ҧ@kuD4Lp4=' %a(;f&Dk \c \9Ɵh p @pV \8+ kg5^L/_^pcvzں'^N/>P+!̉}mzzU?_^pVy~;XeƑqf:g]q,"W>N']+ Q槥/ye3u2\N}I{YOUeyMU]WO[ޙW$f/P:>vCx{仯ߦ>?><}dNOLeGE=q'1m%ٞNmzo7"[NO¸u{;9B#q2]ƫ/N}uqvHs)O>Q獏~Yھ͏#Q=eeѱ_ݡy }H,µ޻FyUGwo{dou8uk)k\{ڛb)>*y]PWljz'z=T_.}rB߼Uk>vޙ8OX C` w?=c-_Y393&v*K0t`~,@vq! X "d2!xQrK$bG!1e'%>.2I7&$%,A$Q '";Qe*6ok݇مOaϕI{l,KLnk&b# k9нxꫢgFؓ36k(\7qX+חp.gv<]a+{??ʅƭ'V-jGnqG٦G.v~}fMo k]]/d,qZz.- 7}=jOm꾗k=(bHJ 鶐uxg!Y[ pN.'^Qe(Qq_ ~H|)3q?1ٞת5;$:f5*_=&ٰx=גcTnkȦ.X|}]_ JzW$?b ٢bu˙ؾf?$}N-ugw7yΠ8ηKYTp9gݱΎ踉1bOƏr:X3/bc짵dgڥJu5kzknY>臅^,>Q}׶Hy[g1N߯A_S4Nw/Ort{ƹZ Io2OM%n[<.rrQ4Qj&E}z{ve/AɤփV=_jwfK] %U4kރU7_vግm[-\.o-mjz{$ziHI#Jdoe=soc4;;Nw-8Ưtz6g%Z¾zQ滆v,"Q/ɢFP2K%$jܩ3)99%'b3(xЈk&c [Z܇wK{Fz9<;tatGM_u_>N£u_UBژ{cG^=gM\dB#PD8|O.cX}KC;N+VB(c{Az#^(T'J*hl\{{gJwpgzqgSĒtdb".1籱ӷ[Q<6J=Ukvy֌LceS| 6:3E=̟y' q;<|(jO;N1k֌ qOJPp?3$cxMki ] \J Μ{[:;v=C~Mo6q25˹\#d{Xk{p!p~mz?h3NEx9l}Y74yMl^ PsV:H²!f 1O-FS̶81q9Cm5zc|לvsy_n1 K8Xֵ:I(I3$1Df1bϵjXXVVɰp}Z/kk1<7vyt3 N2N%%ݷoB3m;!eČ2s0V{{g&g=RNbbHd_`82gwbq.1V:Zf9mOf˦1kξdeG+VBF'[q꓿g5+Pok2{ 45N6|^C?1nyC>=v-Wr8{+-21/gԶCk׶M,lo۲Pe7ރp}nqsx&xn_FMvEkBZ-tI\oD7m:_WO^ lUCWaWk^0i~=4"|HC=4燯[=x>Fϭ8s.ܶؗ 6@-0 \Sgwuֲ}=1?ř.otb8Cc8}{cG܇8,*\Opk4SΘH|/(aьVI3kmzg$௯L'mA0Wz_KEF.bJo9;=]aJnT=5#3GKs0[J3,Uo_7l!G?wO6aEj{ pb%;Wi/;g{1<:4ܻ5kSF_S="YF8h["FN}gOm9s:cg8:OE 8KFbp'XoٶP޼_Y[ ;޺XXk{X=]hXg(8tf }꾵mmgy=?P۞_\gsXs.Jn{% gXTs)bcY;QGir}m[-jno5}Ga _ѯ7Hzr\mjߝ'x)u~Ȗ,NUB?daxtЫ Tc3ܹxE"'qև޵D1Yj&6SM{.QONHdjrm?˶CnjHI }LҜ :Jƞg4?G'(ad/;w! X *Fb9I%㬙l8 XIƱgr*!g&nIe>iݹoJXkFg9r̖r\ǚϋ37+XbsUk{OSiW +⾦cLGw4caf~pc{p{%Fkr&MAE¶z˛Ia1YmoKi[}\ƘY6ty.z|i?vh'ܫBzk]ȱM>/y] Dc弪~2޿1/~fU};I_/gn߷i9XiìPt;Sl'ڛ nyڬ_M_y8ޟtG{7D~}wcs;_Ⱦ:}~j'ڧe]?}Tq#7UywM"_mھ>nmW@{k^*F c]ơ{ oO~|TgײWןLߩ^@r|1.- z%/W5{7Ms>{vz#9N u7>)tq33iN7VY"w,1/[i6s6tz@2~9}%s&:L$$񐋆Q]$ .cIEjBUKQ15늄ڮdW;AueՖ}Z_o;Eμ-+ƺXb@"hLn tIoAP p# mK,\_"A|Z+v1qDN[E,#cQxѷ^"UDDSlqW(1.|?qo6m~!Ab}")(F_"28)1OqƝ>Op.O7n=2@oQ;t[ } 86E?vq33oY8LXr c՚kNv)}=jO[Vhlun_\o(!ߖoOE$_~S{/# C2 cN.'^ ZOK72Tlbai⢟l68Ǒm6w{ ߫VIԭkڧ%55ij(Ѡk]Wۀ"_-E1)\ .C{Ru/jhϷ֍TkTXucc6j}j[ }ZonN{BWXD - EfG>"}b2b;˼0WiQkYT1Yx8CQdzg. QX4 B!,7x )d4[W^I4dk!,\h~ [!9'3&}ݑԗ!^RX$$^* 'XLd5&^ G#^O׌D2ح+I><;(ƭֵKGM_u_<$@wo=Ke XУfO _K]~D Hu/?wXDzM7SVx}{Qc??ov$.p?q/{x sa-}qr漖k /Ueﭸ'Dlfq[M7y=~7|VtyETOys~cS /mH>FX{-6f};kcD ӧ \>gh$,bL֧}ةEf[-\h` v5xk69X}{ ~q,ukMFrpJNWƜ=תm`$9ϠDf ?dXq1c dRaDR 1Oؗu5 kkڧ/3u&2k|#όv1j%a&={B=n +־~s^Kw3VS;@{ÈzDQ'!B ADleǮ5|z\LWq4_7^v9Zd\3E,49kzHG' \>g]њТzuy ݶCz{oݴGko|][.ت^Շ/,Y7îaѺ[8fBJ6ϡ͒LI*HIH#^%Ta΅6p$tl9<1I(F:{3Z@ x[1kٹ"ƛXk ρ>#^Kkϡ5KgQ/\}g{wD@q^ыafYؗ'sV5[0-]ފ{r"ƽ2R"I[7K@ʹ]ZJ?4߇U{Ts3lVy,{`ٶk;AbKki}Lu+5O5:\l7k'dS__A#ZvMHP$`rBSfslDcLL<~}bLl\>&\x=ZN"YV,vԾ$:_1Jyx}qľ*{*mt ;k9s+{/Q+qlK?,Y -TSSLK‚& ݾb@]Jy&CMC[򅹯r,Bf6#},qV54vhtON1%*!. wf-(澵MgztQ>ahc1?]~zT uqpݲez˨s o| ]|;hup9VDq7kqu3@83 Ad hI|\DP8h]!6QmduEblWP_w P}cpU[nWr͛vы3EoW` [̮8DgUcv4 O}l!\gEa] Qf}__ O8=J\> ֳ՗MO>v3}oe>r!* /ӳ">?xdv%Q1{\pPUGn m&Q-fz{,;BOG@ lxֵhZ:~.kDkn ׌s ׵8IIϐM՜8։[3I9j&l甬K$lؐS7%SYI:}N$yk3r}N2 [A .pCl)A[$\3Y^-\?1],bx|rʸ+}*g[ۭpmgs[ZES`Yd:ѲI0}fac4+"?z:~)?3흾*|g{yԱAtS@OCuP EU!2oNGѱ-(FAUɂxf鷈)Z귎uַnG<;M|yL3AHN[ L]놟C-5f>̋ʿPaK"0? ص–^ ЉHe}<~,eG1>mzo7!ytΉ Wp+Gkaߊmޣ5~vމso} uX7P$|9;q඄՚v̧}h!9pDC$qޔxnSQbO{}aLm5[Dd7 1%YJ *qF(^'l)QkV_s.\IqJLۋ (?}2[A:Lp] BNoqlIǓE\3VoEg[ҋX1Lvy+m\[b 8=2ʆHf% +bpm%:Hj{{Ư{8V?ݽ}U y~[tl#s[gD=8|̵55gbЫXHVIDAT[K}CD?M'="zJ}:B7ت^C,wnkDkswzX%A䮙2),$J$)0*#& s.ܶؗ^g˞_zzL_p]pmHoY?c!\}pM;shlµ{-PL/}bM? ص!mJZ_5Y'k1׈K}D:Ni im6Okbay~c*bzWY>h?N'PIJ7k9}ԠgsoSwsCڲ|*"o9v|~6ط^ם3 I AR"h#Y;QGIr}Ok5?$+Db }{kc{B#\'!Z+ꍭ-\]W䘐Lu9 noh V8'jl;϶LwȢ_wWH\.BtIsv׵p/Zg}ϊ,`K:sk*XYI*Z쟪<] 9xшc1涌-}-k^b֭W"4+[NO µb\6wǝv˨%/qݪηi?,E_W}~S9 v>ӛ&w.2&8KOZ9Tr=}MUoS߿M__}>_2}O?z#IL qo~ X8 ;$ KUHv/$dYr츠Cb!$~6ױDHv2V? I1C,'(qvqvSwQ,YlcR!œ_OVwQbV}H"CR}:3>.s׭=#ǑHT]'{EsϘt3gNq8 µ@SgvVήꌏt{=]`O.OǷj0J q-FQrTDQ)Bѷv5}X~%%WkV9٥E݁VZc{WXv"F7 oy@#\ש~[\'hB5=l='{x\Nv(syꈱ޹oa(6)djWן~ú)tI'3g@}|a/bN4vt}joO2t6̱LɌ{< wLjx^ȸߡ1$d=DllgO];]eaZC}ݛ ܵZPcjUXS+$Z藟vNƦcP+{/ȩ-Tg~8GSs?pKz5"k- :4{µ~I+?M-- 6 r$~0\XԢm1~^lca.=/~?xykpcs~RGPY%o3>/qV3'ę^Dq_3-*-W3q?1ڞת5;$:f5*_=&YؓCq=גcTnqh$?b ٢bu˙쟏 %C <9q}t|Z;={BJɝg,~s iFś,WgL?v,e:}ϯ9=Ɗ{Yl[A=}}so6תp}ΔvcS1+u'[&+[Cp⹹2iUWNu26ޣ_oKq6z}kkEZ\DS'&e%^#417QBz&1.޷-kq>ZLZj=ؚk5e7#x.{[.q/㰇0ĭIR$ Ø^O׼jnF+v3ƒ[<CnQ~kiKf:-#N&oe%xy}ȯ -̽#|w]KTw:a=}꺰^ ;drk}I7$кSg;SR4wm^I$ .9X!fa01gIRmNߢnE8ok՚g2vL3Rba1=_d֪\/+g42O|M{~[uA/s05&e`KV^H}KW>ԡ 6eXߡLer͡wߘۮg}WT܄.;ߡ<뜢kzüعli_jسVߵ?f"]_'Bw4uhΫGo۶?}\\S6z6@M몮f1bϵjJk2,\_[~mm<0Ǝ N$&0:K9bڷݤvc$vjVi -#f:"S&$LuoL2 b7_Jɤ"ń' bȄ#.%> 4Jg^k88l=f-\x} q?1ξߊsV=Í̶b*Mgg/ |߇1eX3kn}cXI}mk|l'zD^x3_>2?q;'쉱*;7UuΑvZ ;X9oK۽/c1 }vN˾;C>㌢w~gX}Yyeݽ` ׮hMhQZ n!7DS߈ntr#7hت^C,,B֎3ݖ>Wc{xbփU=ʇ1z遮z }Xk`|\>͹pb_H+r,5y6}7*\g-cC'|MCqN/Y'K`,6Dn~k4Ӿ5L/n,0厎ьkR *ƁqfMLDu[E-Sk鯘ԟKB7v0%:X*ؚٞ=f̖ q8[HQݓM}QwD<4jiS.;|/ qB"g^bry1XT<~oNNlJgsOv-~?=[^sû1͘kQc=ZC~ś(ֹ9}WqFAkc24dҚ&qWAuO}swƙ$j((p4 SIk -ğBkH4W~ ž4%\GNBW[^c71!kvCCz8+&kS|.[7PpY;Kfb?1:kv#^<ʵc8l%^N'%Fv{?f@zc/4a^u z_s1URihwDcя%`.gSLr"'dIYǚH8f%֚T1$cIc}LVVZlϫfq1!)Ikl)gui8{w#􇾇wz7n)3-I״~,i({>|_f,̌=_3Ec_pf}r>+ޖ+}2 to_rTKv<{w;TIcx<3!!=_׷'3֚3 ]S8pjlG/y2~M_݇k'`imv>ˤ=n8:SSlf9Y2#Cq\ $p:`nb2IJ -P"b}8BD{.<-(ɶ"qwܫ8ہ{gg98$>\ڙ{ϔA/$gIBj / \{N~5wʅ'\_ f+o&rMK韺}ejyo15=%gG|<=lg^^"(?,~=랹>b_"=A9\«WNU-JɾPh}Qz:RIu?Qlx)>b/a_n2t)[~HкWͶYQH}-Ӻg{"Y/%=(R_YW5CxzZeW|3(6R<b]ݝ,m՚k8|޷G* Ic` CE"EPt>/3'r.,sVY^Ӷ- {< Le_?U"tVm"lij9Z|7oIȶ>0CC-_9k(6 ʾX{cņ+ƘjGN>ξ ?OgskE8YFµi⛎NYP}- ׊WCZtǃmn ڿX+1W,xk}F:51~_Eٓl[CP;-霛o#cuu5b8_ZWG?g?K=GHF/d"H#:W ~1WeKIW ]OI==Bl\4S!8_n׫ X\KׄͻAҿ'Cr`نqe]9c5u\d'LԢdV KdgY]$ڰ}#ȋJ [f6< U쏵@_s]r1tݐC؃Ux<*.˛xDV)I&<4k<{0W.)dח;X iŕ:i&9,*?A}tpW9{dRWA@S\ӟ5D=AE,{vڋ=9;9Yeotl0ZZ7X}Ñ}sƇyƩxX'}X|)F<]L>kJ X]cїcF6;gU?[r!X! O}Q27Vҡ}IAfk-܏U2)) *U"'\/Nrΰ|6Xܨ~9yїgfy\&_/!?ƑzQfk/|/ k"F'?wGQ /BJҷ[>'x`}ZII!-/2gE5O2<oq&V_isU}b/趈YI?(cfZNMn XE'|<h~@ANc>`bR\LFWG̊4bSPaV&^EKݺS3%=M[zGuet'wBZqm9OCXr;,Kޖ<=O?¢xx>(v%oG> |ˍ؇U 򧁀TTqT@UHm"HٚK֚-/#絿xI:dOKny2TX˿(}6]r_P g_MnYp[i\y{tv8?#%(r/3ACS  hAV4Ny)|D%|bӊ+Jzω}M>%z>/xb*|Xy_Z8T%(aWj{ z,vǓ(hC w$ī9A\ǃ"x?='@<  ?='@<  ?='@<  ?='@<  ?='@<  ?='@< $A󛿹_ؓՋ߹ֿ?$=A~{:~?MZE\_$AVqǷOO?+<`"ݡ(TiyE_sD X9 / $AsA0:χ% AttvMY`AtPCˇnPEEO~Ya3zX5X]XՋ #"/&EЗ/_no܃׿N#e"?^y> !laC!8TB[q"*A4 }Y#FʟYgI)J;Lpӧ'>|sq[:nY&C᧷$g ɇ:[il>\Deu익jH^mpV{ Cޱgڄ m^>rCa"2nC&Tg\/eDbC|$Ȩ7~>lluzH)IjwcAPHQT(Y\/$p犘r`j󾇤_UcʻƜܙpyh6N׮J.(?h~yU(FPO͂Aߊъ_o|5V*ڬǻזj[lg%(엩?e{ Ð4E9ZXZvP!+ yH]k(xfcYd3_ 1R<KQ?͗NFRɚ?݈7ެH;;Q|2f>IoĘֻ@F㫱d߼k Cܒ^.E%۱ktIw] 'D4E/63iU2NI4l%X9SlI%eh}U{ |;D!tݛȪ8k'6 Ҹas4o2|ZWoUdW//&ƛ<&x{vו֜zɻzhwZoH,s=aEPRl",r P;ϑۖU@meɉʟ.3c[۞Wؾ"Loi9 zp74X꫽Pjr]:91yf<ݱ1nb{x0NKRҋk2X˴3œiȱzWqTk`T³qg''G|!֬67aͣmZvYvu*P\EF%'+~㭱L'תۉv\sn~gO㝜g'tnc=A<<&Ly''A|ԓK26}0Zj׬՜j AO}X #]ކ?J_ ?u{c^0olګ}AG\yu>u&KcȎxs{hjO&{MU \DfN>~d*YWwg2x$ty.fٝg_Sg`tNdx"HWxedU_ѵzmiFsZ9l3c"~ ${W0&E8;Axl!>DP`3S} !lǡ B؊CAtP!E[cʟYgI)J;Ln[˚q8(OZgb(4}c!p_f뵾ouױw֪Y#q{Yx "h{Ǟiaky(CL.<+WkX,/e#_?'UTdԌ?nA߸]#q|'9=!NTWB!!ߠBJ}!S8W$GFWcלּ_UcʻƜܙpyh6N׮J.Iۣcz[뎯ƒ bD j/drKjz'X~Zwlǖн'u)~AtΤW0;%57D:&{`XKO%#̣U)psuo"Ú7c]-6gNcVHM{;ۄtM} "zyN}x1?5D1qxtƳK|d~OCVzuWGuf;t یxT.BߕXfdcTpXyܶdZ\,~Om.KNWAX׬yfrmOīilSFp_qCZu74X꫽Pjr]:91yf<ݱ1nb{x0NKRҋk2X˴3œiȱzWqTk`T³qg''YmnšGXvV7SUaȇzsj_/O7W/d?Yoe=Q/VN4״os[ws3N^uul2{x$ty>M oUM- NN 'rel`ԐY׫9ʃV?#FM S'v&͛moN^;*q 6aEKyM1w}4H;~cͱj<^=I6U9$ ڋr9Q@}d]ݵt )FXCBMfwj}M=!nu9˓E ]}cU~Gͅ" hD=͊ =?3t]E AGSwUF /`K6+Z _()PE !l! :"lO $cAXc{Mx?Fo q<IYq!'{dH>zocn[#޲Xw:W(Fw6R[w8A~q[Z@7ϊE\`翪9W<獟?Qp.Ց8S'*(!WB!!ߠBJ}!S8W$GFWcלּ_UcϘ;#Q1MUŃ/ӛko}^|d(W^ε)_ն6R_m7M,"2qlOwSa2x"GU K O ?dŵ$+p l,4Ls՜^"~@v#.xfvhwCvb濜4=:.Pj, 7A$`B+fx}?u|Qvl{]I_8MċMLZx SR3x#8M$cbg V[aj>`Z2坜%QO/h!_Ws5==~Fc-vyOlL7ۛ(k~$NvB-E!)殼Ofɰ` !56+l量ړɞdSC(ѹهJ]ٽL  p"h?$dvXqYYWyto[ھ\(VNج7CU Ip4N~9[r/B،A_}Bq"ETcVX'sr{Ys1a yR}ı=&\l/2nG)},{0=2$wzocn۠GeZ5K}u$}68=!Pp3m¥^fQC?Alա@|C2| ap! 7\ R47~>6F1#ջWGLNrV{ C_  %+$N\PQ_MR~EV);wrUx|?{sm WczW"Mgӽ=-E`Lhg-]oؽL )QӼYq-C \C32 'Ӝb5Ȩ_9Ũ]Zi`Z2坜%QO/h!_Ws5==~Fc-vyOlMl.(m!JvB-7poyM1w=74HmCkl9V^=4K}'=ɦ*D{Q.s3'J?ѳ{}A<+lU.]wjo^}@q+v4$ lַ1Xww큣ޟjwcAK'8K|d;vMW7+Q.|=7!\_46DXT+P1l'KNի 9Ɵ(f>6~ss4E[H!Pr\cRIG83C* Gv5~/]+`,NdA\4֫h`jl=jLY1"_/m|9䗼^xXk׋WH~TS^ \Xmho_Lk?ޑ#ߧ~9'E'Ȓ~sޝ=-]oؽL )2ė69C[]y*-N⓯s5<, 4gXy/.y^oKMF}}t=6)E~]!.eJ6W39zsTM(j=s%e<ߥf,~TXzS:ِ.rMF822Z~]pqCNuxg,Ӻt|Fcwwrr'50鎍qăJA%ę3TrWҧQ$ %yj&kU32iadPj-՜c0qWEFe|8Szxu-kk{'ɓ~}j? \2㭱L}+WG;os[ =+W7v,ӽ'GB;ظҧQz !SIPD=y+c [5z5/~Fc#ֶoo`aoAe|#:bl?j͒acul9V9i/:ޑ9Lk9AӐs؇J]ٽL  p"h(~&]A:$w~Ě7DZW6O+d\J\^1 mAyLwt! ^i(9A{`_f M ˲Ip4P%AE^K9IENDB`pengcheng.mind-test/docs/feature_cards/assets/Transform_Ckpt/original_process.png000066400000000000000000000205031471400134000307560ustar00rootroot00000000000000PNG  IHDRB9SsRGBgAMA a pHYs&? IDATx^ ]Uy  ixF H+mblBeU̴2U@qap5vZVN ZB-FLWB@ 9]wϹ7s{ZSϽy,1 _knCLdL '& c2ZYD0 0 0)BaaaL92 0 0 caaaaS3 0 0 Ør!daaƔ !0 0 0faa10C0 0 0)Ge ɭW%-y}~kn>%-ULϛR-;*#'kk9Χpܗ*սaT-m׃:R2H 0m ]}*Ȗ_T79w|s&߾bQr߼NwA~M\w7z/>bJD5=9Йi ,92"𠟳'I g7C΁1 h9v/HlAg[PhAk>gFdv>䰺s (h(+e<dW5ćϜ"E9~#º fuAV&DzxA٭YN}t!#k; 2v71o@q#=<}"eU5/m.)8\zJ*St?~S~lP2:a!8S~塆k>qR1jTYs'Y/L)^=I:6Z:κm9~!E<,,y g+J&*;ݞk4AÍtBY)"StfDbeMYC=nҦRuۨw3ͽ =|-MjEZvj0Q2:vР]=q|ַ0|((($u_N3q<ŅNWt4۷B=+Iq^vaa 38D@!:!DvD}m۟U8^]Ȅ! sp xU>F|DVa4`,NDHS[C#/%zP:?wP>ՇTy^1"$qxepSʠSFbG-F KH":ك&)ddpRyB}O K=k9_CWM~ DF!e %yM;L_I*bt|3>מ~B#AȐ!*2n5 /€B'tއ܃]Sc8dhQ"FwW?vn.& Q~dF!}(Z!X]bP oѵ4IblNqO=p"Y2)U;:e *Fǀuj_R |C1hYv}kdE=QTAi Ͽjm\@&Sm_t[<ʰt!}esDdAVD(KkDROY!fA ,]d^uywUYY1޻| oܕKU#.!=%m4=ڥ7`dYk-ZE'GgSXo? m%=Osh KɝLGk-*c b!cz7^:ò7}A:7Zfh+kFPL+.4/X>`2/0B\ܴ;3~!T%KW۷G_<#*w 2ϧx]ZlkzȐ/hB;0%O~K}rCHo0W]{MKF4E].jd*Bqۘ)wLYRSug}ɪ wvϐg,yв,X*ii P|72ɿto|E]Ҟ^ u亵- =yǒi_gBy^hMc^bJAy]1Ԯk["[:2/=0H !Fy ?O);va弈My+!+dD{1Rd^I ZKƶD!%PNHO)k1>~]KwrT"E,&1f|zW!-P}ƒw ?ʌvƀrN%)Y` n+ѱ68W"e4ITQ>'Ǧ;gQ#0|{@u6:}K7XGzm_ ?ɋ(:??"ktc(u+(rd6+6CkN[7ٷ.6Ԓ刎c$5?V g.n锳!K  E ⍷wOM@BIF;dG§QDշ_vH7|n!$)"%OS x]E,EHMDoii,_R*t5E#7NB-zOȕ{\snHq (S;k** (w[P*d= WhD|Aw 1b޺pN˱YyN\w"0DE %٧+z. ,Af`g,ԅD((tZtRh `̸ITOCc'o0@ Y砑Ϻ+7t|)1uw87@(uZ~Lkfun5ԒdNO+$7o|h%WE}EiLyv֊ڈ(XȢ HF.1Y#>B'E>hᓇQ?ۓͭx"F"FP;1PH* W m按]dpozQ3M piFEmC{7AAUay2t? t`tD+1 u#E2xkG0PXx0+˜$/EByX]K5д}z=y^ @J|,z%y.v=|"hB0-1sBg726r(DGoz/^dnKBnEeFޗ1v͸!ge,DBgWȋTd]Hj3|$Fʵ{EYϯJe^]q67jӖ݊uV۷}rDHZF@N14mvຘu/ޭK1d?r]7m;"z*[Ɛi绝2!pZkڒ`io+,1z98d5e;0:dՇ[<4PI"䩩2鱧O' .M q=Uy:hyt1F/ 債p@?Շn.nt4ZȪ Et}A#zjdӧ~V[7eS8BH^3bsT2:|'VoʋH]uYYF+ӟJFP>Tj55^E;Y#Tuu0"k.桏ECn!shyp<;R_qhg쪒#'}FJLwDHx-kR{pEv= +T43Z0 Qo޻x0;1(0FP; _AЊ'\N}ie%Ķ_ K)آӑmgp~RU!ixHC|EVR۝}%&:Fq5E\dQ'R qnxTQdu7`24"BV$UלvUi|%>IوP 2{e_[:3ׯk.W]$R ɍc 6=3w~ BޚYdD(nzV}V'JP,Rrɥ}RTNa"hd;xͭɠg<畐R#Ȩ>=Av m:^w/RJxfK#6Ev"FJㄎSvepe8R(9RP`/ ՅNb$F2%t)"93./jU'bEVN7`h8x10zl+l|AVtkOǏ!onZC!p0>aaa ,x/>o~z04iQ_Zp0:NM=_j@./TOèt>t(ߒc84 Ҏ?QvE\N-+ 1#]Lԛ˒E  {;V wVC=\/ S nnU;0ځD-0VN;uƒ@=856MPmhE\~Dk\ נeF,cqTU 0@U!&7Ńg5Qc VheCXP2 O'$0A Ti56tV#~fʋE"*T+ (|2K_dmE-NܢP4#T7RN(XK?opA  - 7g6ک =Vy$vxc1zpjH0"mÌ0$ GHá$0A o#hY% ǫo.O< uQΝlRH6y5pc:NrR͌$nGˢyM]Ι眵2 MK+M䵳|7@zD.?V/ho )(1lew^bˠBYXxn{ 3 I%@"豘teI5"S$ڑkϹc υh{1BOҿJ r-EtyI{&O=G^ۖÈN=Zfxwe_G cS1  !Lߐ1BBgٯ_ܝ}Y@CPGP)oL{'n !Cc}  YtSԝp < W4<*9YvA0vR0 !D t:e=i8sFdW?8A&+CHgVn\3dpREyAy`(ӆ+mƗc ےA!)9׹͘$BdyڡC%.:"$!miCg֭8V#yy=_dufO 6'4<w|F uRȓc(y”eP !䁎4zRJ3/t8Ir$"$7YClhG !Qꐕ8;1+yInKY~xP;҆tNg\mM?gVщٛ0ɃzUf" fiAO0@#[t'\ A#hX1`x$CPܑyi!t'_%@-mnC9*[(;NrZyIU`tĠ4uwk. YSi0 R03ɘNLd8X0 0 0 c `aaaS3 0 0 Ør!daaƔ !0 0 0faa10C0 0 0)BaaaL92 0 0 cQU Ø>|3lL & #E 0 0 0 0 0 0$k\(zIENDB`pengcheng.mind-test/docs/feature_cards/assets/inference/000077500000000000000000000000001471400134000237105ustar00rootroot00000000000000pengcheng.mind-test/docs/feature_cards/assets/inference/inference_framework.JPG000066400000000000000000001367751471400134000303100ustar00rootroot00000000000000JFIF``C    $.' ",#(7),01444'9=82<.342C  2!!22222222222222222222222222222222222222222222222222" }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ?( ( (gTFw`$-/<7"oiQpC1_ ͣ|_U>W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`Ea,|K' ǫBVu,P&W `\)QEQEQEQEQEWNol4EPIUu)!,lbY WpV8H!3<Mj I/oRjmh,>zjϹeHAL&iPE$}@IE((((((((((((((((((((((((((((((((((((((+:Au}N\7V hEe-Ρ/I {uV(_бR<),R,GC=>=-#7`  {ER(((Oxe鵧[&72Z-=&}MnWMT-$ zz Gb* ( ( ( ( ( ( ( ( ( ( ( ( (ke{{c]hka.|X˰i O#*9`yS7EƭszEH(%*T*cle `a;^'Ԯ, 3L7*֚Q!.q,/]s yBa$S5+p ?$c̓n2Nyqk$vzea w0w*H GZ6ne[%%Iz|C|ǞO]}7l bXn *X.W9;x=|est4˛%m,51b[YM!@i,UnxLo=saՔMy=1ø(CqÞIz3}յı-{a(4aGq UKsOfudtyn.[wޡVy0\-|- >u -;i C mR3`p7 $^M6Px [vڲc%r:3du=/&d?MBCgvZKuu!u 690ak_[DuK{AK q`Iݽqֵ,<m`!{;mFQ lk Trz+B,jZ&bt9c9gdxn(/<Y6\AmJP J|%c_A=7"yewF_ެpW N?At0~&Y 8V-Yv7m[BI#K,Le96ns Gz?z[1u-noXè-m(3#2,nʠ2GB>nZڵ/._ i(lߐ5"n5=CPݴe|*qEz99'Ok_GsiivƎ؁OM?Z^=և.ɨ\]I*Y;$_kr]-xz<m6 O1]5zse)e&qVM;خEfl1sӯ?§[[ح7~0^^0ܩ%u2ǃXQ ]/D%+#B]}F9 :w}De-I|KW8[e`{u9uN9wsahcTMN\nE&0.cذל شXW6T|lg,KkKR)#0t < {Y~U9*[#t0AA(]68OsM{I*Ki,drJIy\pmV^Od#y{pUx *'=+RD>EPEPEPEPEPEPEPEPEPEPEPEPEPX 3DGr Xr5-6ݥK/ʃd$-ER((((]#gn:[HNZLwF9-T,Wf9#%'WU >e)+ϥr,ԯ10)܀ OGTV'L6qie_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h_kFLG|WA2M]ݢ h)h/5_I˸V|WA2Me_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h_kFLG|WA2M]ݢ h)h/5_I˸V|WA2Me_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&KSF}սWlߜw8Ep:+ +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h_kFLG|WA2M]ݢ h)h/5_I˸V|WA2Me_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h_kFLG|WA2M]ݢ h)h/5_I˸V|WA2Me_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h)7k:1\(ۊ?526m]}KT3BQf(`(Pw=O@ tP=;j7 %?|WA EUފ?|WA Eھ+"moEqھ+"_CFdGjs_CFdG!_HC\G!_HWѿY/$QZ~'x0X_;Osmt+ r:;1r*!_H躷l[/ˆa2<:t3<C&jA<F%.+= sЊ\2j^(5;UUw!_HC\G!_HWѿY/$QZ#WѿY/$Qh,(@vWh,(_4oK}V{hw;z+_4oKj7 %>P=;j7 %?|WA EUފ?|WA Eھ+"moEqھ+"_CFdGjs_CFdG!_HC\G!_HWѿY/$QZ#WѿY/$Qh,(@vWh,(_4oK}V{hw;z+_4oKj7 %>P=;j7 %?|WA EUފ?|WA Eھ+"moEqھ+"_CFdGjs_CFdG!_HC\G!_HWѿY/$QZ#WѿY/$Qh,(@vW5ob3lkn-4 g>ď'szWBX.fKk"{ ?RGjV-4SER(((U/}?*B)݁||Cx:U1E8.ު!GBF~\>hWlPƋ`T <݅QV@QEQEQEQEQEQEQEQEQEQEQEQEQT&4ymV97@=ak8JrJIl`Mn-TV.cwzsӷ>)5uaϣA}SO,]L3n"2c>/ϩ^ 81G5|dBzWmwSTE|Q=Δd։l TɒH};uz"ƊP;\MGURzkc5}֕th[G ye`bNHBy|+t.4it"8.eW+>e/"jKd0 Tʷy6e_ZKHQ4eYT0 3޹_%E-{ϏF`z׷%XPN7!U'p?OTjK?vtvWM.ϿTA'թH;#h`ñr? ){ Q*#Ưm`? <;}%I{+w'Мql’p2 ^CYu2K [CFS؊=.֖8oSӮ+Hԍo^mxUVōoR}cb8WB }{U<ik.mo57$]偟8O=[j(Eeͅm2O1޽_U_r7_K-,%X~Ҧ8-UK2[3g9dԭ!we8y}qZVSjjrC1!vT?fVM3Qleմ9bjiXU\9YK{H/L`@uWd:6kehdd ]19,ǹ4r{0屡EUQEQEQEQEQEQEQEQEQEQEQEQE 2CpLo5q&9WI $I=,{rPB΂0G:{5o֌o#+;(((( cEW2C2zeO@#:u-:cuf4l=HQ=0?kZT[ΜgĺjQ`gT#$S(t k}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3o'd4}s>%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y>u0p֤[;l z--/T`r@F>W]F×ZY5V @r,2{JGE$䌊먢RSwb((((((((((((((((((((((((((+ ſax@ݢ((((((((((((((((((((((((((gTFw`$ck%b}FeܑzossX8 WM]B_;Ws'$oRkzT%S]\-AFA{QkOoH%je#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>OiZ ߌ˘b 5ڜn?V9-g8lV&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>=.rSrkt(g5n+I%dy݅QLAEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPP-h0jBW=y%BeJpB5r5?7_Us]YHVbBp~#Dկ58n-uo-UsL^jj?S5+,Ui2r:U֝W)S[Q\\gsH=Ln(쮒 *Op}<*zZd-gӧc% [tccyVtetarZE8|:( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( œRje oJj1[TTn4O4i:us m빘~olU3_?+jqwHF͠cǽlG$o{[ޏM)-RXas󞋎δ4-֓[ZI34e ͌qjQB[{6>CWIoh06g&.~h_kV\b4rrYWnsv4RqMI76Yy]\W}]m㍸q4|ͨ_]ȲC1c t4Q9HM LOܲ"tbh(((((((((((((((((Gx71i鬟*C?_^k4:R,T}zNu6c+HI+,[;W.&Q[|ƅQ^qQEQEQEQE^x>i{ 4chGOr'ֺ*'t&GYx2R/*+N{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REPvqliﺷ"͙"~qw[OOhN_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8n0 ۊ?qL, ıs޺(u>إ6:DeXvgrIQX}((((((((((((((((((((((((([ k Y nמN񵇆av%5(Vu`wazc0PQT4MZ-wE`x!K\(Wz#~ ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( >shOwf!AP̥y$y N7+bȱo?UTL_4oKj7 %u }/m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$V][Z|6Z0:E`FAGWCEŸ`F"cDD@Ut@t}h,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$TZ_Ϧo{; 9"u2H&VzmV'TTMAY$QE杁EPEPEP\X*:akkc g?WÚakkc g?WÚakkc g?WÚakkc g?WÚakkc g?A't9giѴػH֨YN:ET,t+p#\~5^[˝6W{bHE`$?/cpMiEQEQE.WOHأ#.x<Xl4q#lb֑=xۤ2{_gf\Uy`Lat䈱د`v'ZtP*YGG}|glcsljkg0:rDXWv`;|:( ,#>uk>qȶh9˶{z~5^X53u9",v+;0>pVʖQo8[4X=?,:;8d}+NK(ϷZϜl-,urޟA$ᝌz%Q\ N;}+JK%a&O\>5խ.KY' 0$wzW9xHaA%R R1@^F&A*s00CƠt)=QLAEcxG:e+eqo9W!Q Tuy+I E*tOֳMl_'tzE_"K{F/7 W$Eɏ>֝'DG?MxmZ}LñUL9$wWi~![}(^[d|1;U }A&Oxĺ9 9IbpzyY5u]lWiu=J$񥆉//u-帔A8-#ɍ94+mfKp C+!~Q\jV:.vod/=* OWmbҭr=1߿*m+4/kx~ey[VF@U,OL?j'ZJ|?&qs kJ$ZXW:+μ)]OZȶV;M#\ MT}G_ƭ]v y6%H c qJZO9&Q\/#4&*B QvdsY$ҿ3ic';Gv֊*mtWe㛫Z@[.c9o*RӺvc  Rujh+>!Ǩ\Omqa l0ocYJOz:>5a[ɣ%.[sp?Hһ:Z(fvم;y>8ϱGpM?O]COdn pr? YZ(`ީ!\CjEQEQCCc #4P=)z쭦M[S0%h&;P~_3=hR +esc]kޯo2j #$)G9j]_\-b7A8 @Vj[/EngEq%Ԛ &7A3G|+ܬ$w{T~9Zǡi?n-nDb7nrGJ=?\3Cg=GUyHMܳ#0vuPϧa4o Sej'rEm!l8۵ͲPݝ[Zv<e=9Hךɦ]YڛfA Qi7wi]h^9XJH4hg;q@1T_JLl'5:EIQG~ UF$g:kvWח öV}n i8_L|U|dgZIYgCy$j0J95.=z.aGxnL 8ȦYQ::V4^K_JTd |zHHFuWw\ֵ9|Qc3ڃ0ѰyJLJ֬h)<Aa^KK68sm}?L֋)kCШJƮj6Z%en"٦,<8@eiG@Sʎ+a#8ҫҿ$wTWxJᦽbH“;Vn5kK kS}lggS搻8#$<(֋6vTWx{V[ҥ/m;a8QLqu:G8 ,"- w PIR[؏_FZӨ:ROC 2@ {7_^m2R88jkTCVv*Z&UsqyV7PHm'dY6)q+J (k rl|A EAJl.H_^yz ݍ cw+kx@D(((((((L$$r@%88<{]&.%PReE%\~ GNͯC)5MS0ϾԆ^3o[Ug~U?Q^AQ@Q@Q@s9c*cQUש2YEWyycqj 4m>c}kYde ʿݑ~W? Y3gH)67 hϒHw鸄3ZEeu Xdr"9o$k>wP"@K:WYECmX&8?xGVg[/ xJؒ2xP~R k&n;Z1叔½ ^i>vrkZ7:ʹ-c}Gu۷PفuB9r7u;šבEuw,E4 Mc'՗xUPǍ,cx[(LP|sqkRi_p9Uu(u-~YFGr7 јm'ߠ'%-;}ذ8Q$E$~QۡOC9?\ZYKt@x#F1=k"2*Yw ,a@l/ҋV~N9I|3z߆oD^G8,rĠ_FG|U@.4_iW0oM[g9q]N26o[S+md;; nv*1zqxfMxh4D9bxlN0QwBro@* l{p ChI[}ɔ0GUTPUF C( (^I- 4,-Qԭ$bf/`>8888[̑#me+q!^3|EG&уLH^UZMBXRH˺7 ` }+Ӌo#-tmmJDQIc D;t0 ;~E2w I]`L»:){8smX4j{Qqhdo,b[4ka->]Fmu}Zko-,vU{fѼ!i֪WDe##瞕IQu$?|iְZw+k(Ga@ ?-|AW#J[Y H40Do(ϡQKi#> ԏ +ϴŞVYC1#z3j&n%X @1O]mN^ַ֖8-hhZrYlAqO 玦h^twF.K)&F(T31;cZ:(ksX,|1#P ݼJt2yҪ';}Ak[X!}$28_﨤E]xOV5mfɵ+6;`(ny>޿Ax3Q>ޟZ]3om{⻚)q_g\2*Yw ,a@l/隿/o_[hX 4psoN;}(uuǖF@T?8"[/M>أb֊^?AI.4/ Of,O*vQ*M! oAII K/_ܙ0-&߼YvOVꭄ|c$bUIʹ&PESVeR gJHZHd*3YsQӢŠa{ t#;'929r=0 (Q@zł&wЕK<ɸ:zv -Bu}BHEG6r3[tT8߯k^e)XEA[ǫEq~ĸV98; EdQ@Q@g3^Ks#I' F;eEgD8,= +KX@GeR-V'U3?O%?Bh諸@(((+cQWG\X*ƽI( (2Hwg ?dƓ iqL֭s'!^?h+ZO1V̟xzg d}? ?c'j( ?h(GO1ZP2~1Q/Lb.d? ?c'!^?kQ@\GO1G1V̯;c?LCۤo %1z -|j( VЬ0Fƃ 0IEQEQEM< LGTPN~_O8xRdL2բW{DŽ ],T\^zƨ? ?c'j(<$A2`ʤz׌~5G!^?kQ@&!w U#%c?h+Zf 7A X&2(QGO1ZP{7xIe1HEj'C"~>í]l+3 ē׽LUB @[G4Vi6A yIn*V'Vu~]?Q^AQ@Q@Q@s9c*cQUש2YEWyEPECswme("^W 5C-9c7s3l~j4fjY?X Ow?nH熧7E֢" X Ow?nEdEc<5??"  3Zxj+tEc<5?.f Wsxj+t\,j+'+ WsYVO$V?S]+p5bT[MJXӯmi z( (((((^_/.XLq}SQo1EdEc<5??"  3Zxj+tEc<5?.f Wsxj+t\,j+'+ WsYVO$V?S]+p5H熧7G$V?S]fkQY?X Ow?nH熧7E֢"PE}:GjT -R}VG@Y袊b ( ( ( ( (wL{iE2:UA"4VQ ?u#ߍ2:OH熧7JfkQY?X Ow?nH熧7E֢" X Ow?nEdEc<5??"  3Zxj+tEc<5?.f Wsxj+t\,j+'+ WsYVO$V?S]>#t[dʹP 3VdSEyn~`TLfV=8O`&\^H ,GVQVgN_2\?n[ڐJH̗-%;omǜ{ U(EP (+[in'pąݏEP2MCj6W3 Fppx<a۩jl$Q:h 31u$n+ 3iwb``'=;Qu{jQEQEQERMgNRQgXrj ̆{ 6N{'סzpqN[Ywlvz{yFLlρ#b`H"袊b ( ( kƌQFYiՓiBY Cc[dX4%b0ʩ^F+ ԍKEQE1Q@Tmq\GnFH HRS((((( tx㕮twMCSdɹsOP{ u]kYq٣r?*sn=j3?O%5[f KqQW~SQEQEQEQEW9?Jȱo?U]?z/QE{gkÚL]ۄ[h|@)e&w#VW:bٟMF.RCN( /j7ZW )|J>AvOyJ h$gE HY沬ڧ+ob۝_ͮxH{S-u7),Sf0n={bi5/kqk,:nr [+gA5q:ܺ[BŢjj93p+{M-4CRy5,g9IoQFqZw|W]Ak=kH#&qa}X5/mlӧ>Q{Tv݄8䜞wl = G@={W9=ox>5KwhzvjYG4憯a񎽨>+k붶A@׌V6 -umΤֈYNf"EKۤl0[9 9?J~[ vI/mY+9?zRzo <.DHD{]o_~յ-ZNV8~̱R19\=kjz5w$ 0#>((( ZE싍d)x56tZ}ͽI# 6ʠc2 *21^+iAB$i %,Ib$ud#hڲ6-b*ġQ'Q\4/ uIF $"&@s ;֦2 ˅JqZeޓXm8mu &Og짽_>׬$8QZ7ZKq;$.zIZæ.˵ĨzrHSѢ)((+*<]dvzKZ67\Iam) QE1q45m7S-kۓH+}gz+>n4>{;y,wYU̸ZQ|Kxs]4Z=Y`)+  =NmwŚwmm=άoUT#>l?q]7L漝.._50q09%kJk(e2ŧrZ$.3Oy䩦 o߆Sx"{IɹKW$eQ{Tv݄8䜞kڎ"ki =x?M{ A}WOk$ ;9:]c?K0&-*_6#a͐ITӌUnzyxCcK]b4[sss5q#t=io <.DHD{]zs¶ͽܓ]Z^ۂ#˔)W8ŽIM}C6|+b NyuXdn, @N:`V<]GSwiprp;>տe;9}KU;@98<ԺGlH/{<忸Lk/=M\RZv: (2 6?ԎH"~ZֿҚLNZP!c֪3o[Ug~U?Q^AQ@Q@Q@s9c*cQUש2YEWyYr/T4mp'OLa\ (VPV$ IpF^ժ{Yj qeA= ?ULN0FO&hjY?1}C~cZȾSvf?mh VOE z4d_ǩ߻o3@X֢/cݷ"=OƵ}C~15?mhȾSvfEdd_ǩ߻o3GE z4jkƌQFYk0i!C#EJdiH*n.'4b+rF<doNs*PKEQE1Q@Q@Q@f\C=FeNnmש?O^a2 [{|iT*z؏CS wE}S`O`_D*H4;(g[K9Ynd2> ЍVMbdDh2#5:yc|ڴQ@QLAEPEPQW0< x]( _K"SY7p;+~F 5Uԫ(ea"Ś*O?V'+jQab['+jQ?iǨϱf?iǨ ڟ{XwAسEV ڟmO=;sYmO?yB?{9,UyB?<[Szk=h<[SzЭO=Gt}4UoЭO=GO?Vúg>Ś*O?V'+jQab['+jQ?iǨϱf?iǪR%:h$@ygp?wfúg>*O?V'+jQab['+jQ?iǨϱf?iǨ ڟ{XwAسEV ڟmO=;sYmO?yB?{9,UyB?<[Szk=h<[SzЭO=Gt}4UoЭO=GO?Vúg>Ś*O?V'+jQab['+jQ?iǨϱf?iǨ ڟ{XwAسEV ڟmO=;sYmO?yB?{9,UyB?<[Szk=h<[SzЭO=Gt}4UoЭO=PިYu1q,o*'kʡPwbab['+jQ?iǨϱf?iǨ ڟ{XwAسEV ڟmO=;sYmO?yB?{9,o7_-<[Szee]xFm KYg0`$cQRLBJIwtQEyghQEQEQEQEQEQEQEQEQEV,mn,mnEQEQEQEQEQEQEQEQEQEQEQEQEQEV oa=?K!o'd4EPEPEPEPEPEPEPEPEPEPEPEPEPEPEPXW_>i ѶXW_>i ѶEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPXV>j ѷU^]|LZ|ZnWQA`#xi aPt[9( ( ( ( ( ( ( ( ( ( ( ( ( ( o'd5^}7>{{RbrXW1yAhZiCu4P$G#B*QEQEQEQEQEQEQEQEQEQEQEQEQEQEu#`k[3Okɧjqm-D̒WTo9P(((((((((Og䋉.mѦBl⠂q8ֵjȗ"x]w,c#0;skpcc%,[|qFx!PikgLni˓Fpy|WA2MiC4V-b'3ź) G4G ղ_Pf{vhqas_kFL[Qp*Mt7VA٢ drq4zDp[%gfwp76a}ѿS/$֔MulTݚ-n>XppyG'ۢSPPmյ۲sn)\6qN6:osԂī]lufrNM4xvbdyO;T8E* v}[ :Hs?֯l~:Xycn8۷9=z(W {kOLdsJru>n9Zq*[X,#+cXppyG'ۢ/5_I)#/3=4[|󁿌N9E;RhẶK n7,88gm}1&;#9E+VxDKhm191H-Jgմ02$z\̹7?C[4Qp .ݻC4)l|[p *U5u'S};psb1lw,1qJ.US&t91Z=@QIղĐYI>C<[C:7PۆKdeAc~>%׎TxX6G;Wx h)kv9=+9aʄfcq$u>jГm:,aAh;|X'MbM'z7(h(EjY ʏ}tU>vls@$:@ k{\7r kna(iOE̐zz3=!v}?Zi\Jws'/3]tk_;u.xI'|WA2MnE_e_o4}ѿS/$s00 h)h/5_I(`a}ѿS/$_kFL[Q/5_I+֍&h-R'Wј*h4}_L"AlG'zQEcmL9Re8*}Na,sji IpH?A, 5/-3t$`#h QE (wXѝ*(3=h;,Tgs+#SW9spX.~SNRìݩ0p0#SVчL,N=?YAѣl?eʓ+֍&h /Z7 e> h)kv9_e_o4}ѿS/$s00 h)h/5_I(`a}ѿS/$_kFL[Q/5_I[_yInQE~g-~gKuXY H?X涻fK["^JH ijP.Dws AAL VvIy\%Wۜs h(øο=r4V wI|p c;zn5ԍ`NmA-V(@I8_6LU(TQUz Z/ /Z7 e> h)kv9_e_o4}ѿS/$s00 h)h/5_I(`a}ѿS/$_kFL[Q/5_I+֍&h_kFLN)Nq-Okn.E}̇l{x> ?>^K{Xd]d0>it@hsV ſp>QE ( ( e5%A]y"xLMFk(mLMiG;`8g,s -nc: >Tӭ$-#FAVkt}2Z.OǧVot U)8x4גjkGV[$b 'F>n:m_OE3N|1w wmJIl䔎O8| TChC=AZx~kuޯ,d||qz֒ˠ鳾KSp-y[5.뚟VƸyq) X`$~#_ן_בTW)F6K0n$|Wh铐HJ (~"k^8́%O6ToO]CMA%Q?֓RK .R8 yX@I9֕k($p`>iQE+3:rGi?$?ܬM$~$ ,7*1օc阛4P6袊@ [ڼZt8齕2*j(C .P'6۽k.F9<uW=J&'Y5tB#H¹ Dx3Q;[ί4ƥeܖp ܙ>8k>Kymvޙ5*Z,vVYi+`&v:yzѿ뭿Eymޛ} |-D>xV.*fvӱNSNϟ>y{&ъlSTsp1;/]i/g{+2iEpGn BOA7S]Gg>ֳ2B#d(%Gj#_ߗWh~)6姷}6h[U20{WVt'{,[[6ֈ%Мp1vSШ:<:o.tX?K"2R(YĒ)qC`⮙jgWwJ#Gt|y[GFːM =VNVV- 9>zgTnɸM&{0}42?x]/aՠ_ku!@Y w98`Xsnɤ 'n,H+x >fIW#fޭ}~' NͮDδdJ[89'Tҵ5i|vn#͉ 3\vB߉;Gy}-l>C(+9lsk'WtffSǗ#G{[ן~`Κ(toQOFUXfX wmZOH_-e C(g4+R,Ι4Zv`^e~snFN&Dž& "HLe< C`sO?; EyEunM3˲r smͧ %gs/^En[[#Kğg &nXz mR;+͗MyK*H.x ֏jd2~o{+;ܑi,67 =_o|Eif iP{7i ~+ooYco塜qnNTq0{]I(?GMIxN;?색:5X0?HPoZR#Q%8 ۢ)SQ"-y^x`'t,jy=2>VꦣE[$+NdXze}֭ER((/z[SQd{<(xE=5袊@Ī]SVilI8I_ۏ=nUMCORMYIB H-n:|P12?ۈ| >)G|65+q0|9X\ۢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm}|cH[yd SQ~M.[XO[xD~v?^rk_5C[?rva[SOmPfv2K,s1xQH5u:vo(q/@!'pw=mM;2T}&u?`2x}'Ǣ1v `Q vɧ8д8Xm#(`7(6񟅕t`}'&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V#$%l hm.)C E"Y_ \}V4 t7$ *NTw$ XKcf詽.J1JP =SlMGUx@|b$m b27 ($Z<(@OUU,]?50/Xr=\3}Ap˺s?zZ4 }e(#]LeeLd$z' OHzΟsd4h۷uQ_k !ң$D?4Sm} ?fѿ>/*M|)C6TrsvKͨ[nm'~P1}+/_ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`U4+![tLJϩk/_ ͣ|_U*Wĺ1bpzW.nR3R@P2I< G\KZl*6H}\O~${LGˏTRAHȢ]!N\k?s$b GXAdQs[tQCw)Vt9NY''%~[n?5U5 > JE6da$RvN:2ǒ=$A + Ꚏj2B:^dn!t0t1` d">hasR B{;-#hLX`GIi IcFzұ5 7Q ?fѿ>/*W.nX_SmŸ3hG+5-[[b5۫ۉUBp8qVk _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOoxxC6 fM ?\۬;kwYy :4~?vBz>ժ%ep7 8|ֵlHmc™$rI'I$rI$M=QH /z[!Y<,q70wݸڧ(EPEPU5Ga3‹71Q<)պ$Hc$9¢D'O4ߢ~u | ^l|.zh_eeLc8FqV`^wgYEeYZe̙ qN9< &ũE}l.ZeL䏕cEC:&:k}̻;8{,ﭮ&m2D# ʜ4YnƟH'1I#5d`pA=ޝiZǙ^X>:ɳ9vF0/QP[^^e"V_*@\ddk'_ ͣ|_U`nU4SOՠikU4,*Hލ?Th}6m%"8ʒ38f+ _ ͣ|_Uii@אli-Y6Tǽ;4*:qJi,LAW;q0k7_ ͣ|_U`nU!44P[I`$b w D:qJi,LAW;q0iYnŸ3hZP}ƚu(o ne1\;Nxf*ڦoJkh DE[Nqr+7_ ͣ|_U`nU&4}4jS_[G`U\]<"*wDڦoJkh DE[Nqr( ?fѿ>/*4}4jS_[G`U\]<"*wE誚0,,K5-q gYSm⨳`nU5 SO`Y+k8Y,3,j[,@ڍCT}Jm%83x tV&Fֹ~u | ^l|.zfUGZҴ/OSsu~1n#8T8 ,ҰH6gbpI=fwiAy}mo5liVPNXj:֕z/3{qG(5sf]FrFpd}~'v:us&vCn큓O‹05hj|ZZl2զQ+I`Ե>f]FrFpd}Jʳ?t˫3.wl rx{oڡg}gonxLfVU?]ZǍw:dd!5i 7X&U\ g IHދ04|lw fS*/@wVӮ]-ncgr22 fDէh4cOW{Gmr0\)'#zwC%Ͷ>pW#Y=?5mb H&bI' >]6(@QEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEpengcheng.mind-test/docs/feature_cards/assets/inference/inference_process.jpg000066400000000000000000003506171471400134000301220ustar00rootroot00000000000000JFIF``C     C   " }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ?S((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((_jVdbK˨m#' ?5Gzw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƝ4I]Q5=ݎZբ(((((dGoK,j2>I4 YBi?Ѣ$)e 4IRhFI' &AK/O +;M'_G$O,!?ƀ4hI4 YBi?Ѣ$)e 4IRhFI' &AK/O +;M'_G$O,!?ƀ4hI4 YBi?Ѣ$)e 4IRhFI' &AK/O +;M'_G$O,!?ƀ4hI4 YBi?Ѣ$)e 4IRhFI' &AK/O *[/bmx0a ( ( ( ( ( ( +&źp]:}4ak+bv_0_G'~Jؿݢ; :WQ ߆aҿ6/*7h/N5CTwt +  ål_Uk+bv_0_G'~Jؿݢ; :WQ ߆aҿ6/*7h/N5CTwt +  ål_Uk+bv_0_G'~Jؿݢ; :WQ ߆aҿ6/*7h/N5CTwt +  ål_Uk+bv_0_G'~Jؿݢ; :WQ ߆aҿ6/*7hh|iV(u2YQ/#$j٠(((((((((]{ÿx;+'0N@i({(ڟ$Bkk5PD4;v94,= RG_Tð|K$5x]r<"0 |JlhEQEQEQEQEQEWX&xV֡|-M&fkk{QdWg_+xtʳC?DW~?7~ ywRu€5,?d}Ÿ<8_iyZJ$ܒj2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP\~x瀕X`5f2 p:u Z.jWu_N+.$ 图]`RHKQ&ixoa=E>r *( ( ( ( ( ( ( ( ( cO{_N9iu5?rh(((((((|YKx_Vn.Y.OkX^]8)('NN2|f|H-אiv1yWw8H+0x=Bo(.|9m8_8Lln(8$mE|QxVǯ+Kllg]cn1iɂI1*[T8iB/m;mmV-qrk~I?>n? |yuZVQCe%qȻz0GҺz߈b[ύ3=l,%ѵE=ĂO6M 1{}'{_׾ 7DKNK[4IRgFٓM4敖 j __#*|e|;Ay-^kCffZ< v1$W|g\!~ź?tv᷸k_ [R>/LԵ ib2k:W$ܘ *]A?lZ_fhƑ`" mZyQ l1z֛۽JM~}v#yEBwb/=U犢$R>PC93Ҽ-1=|qMYw]ۿЉ\Mc~:*[GxwZwuƝa9amv +^Bw\ҏK& i)r/_-?l߆0][G7%'U$S]iMQm%55oy~0 )?ц)y 2ZomWCbM،qYc۶U>| ssh|_qclX[ۜL/|7_*kWjŮqkqlXxBm, ӌiyyg=Z,LR4%g&Nn5mGvy*,}v;#V0IX@A'D':?d|<1x@[cZ[ǪN,R&:&ѵOG?F]+{5>nۦk<kK*X##'x >Q(hJQA~ h[ezGs75 o|__j?htrZe5_j; _ugw?v}W/xJ_Yi^":rk%6Ww/1![i\G|oxHnl4R1)#t3$ ?j?~)~5|a,Q};CU񌳪Z4ecfc?oοh.5=vIw&h@REYkʦ1rYr?hM__C ?/P$P$PH 38 ??E>$|D񩻷?|g'Ԡ 4e_n$WUJ׺"7QEIAEPEPEPEPEPEP H?u&?D>ػ4t|v!GI;OYif>&\c 7RE&t)AF(+,T6(cSۿp?O#nj&?? +,T6(cSۿp?O#n¼g~>5x'ĺZ9u=wfdvF'cN@=HӪW秆l+3o k4oo:G{[JT4Iv#Io6k. #Z^:WGxk:[=Z1g`p~PsG9ׇ/#eRMlr>uYUzW kGĨu?"Q=ZEqk@-|ˤd"ѳsJ75n]Q٧ zO oM-)[Ɗx|ƹ鞕濰ßo`!Oum4O>D uFI>e>k6Ǿ)NԴ HfU d)Zu2?I%;~MQ7koknè1۟AhzoAiCo\W~,m[I-|9 AV>4hm蒀Y9A_|Qׂ,,o~fy."2HCn. Ú~/|7M ESu3Ğ5P l*23xZxx: p R p<:kYo%IKy8nm{}mEsFPFoXbJˈcfb2@]~\P|!o?.mm>Tt{E̲xjA`ysѳ } g~uk_ƟޜxVeGEe|E$>T,~4S-gbϗ}OR>'Im폇5ˍJ̞6x7P-xAܞK /uj;U^P#o]|?~5xhzֺ6%ټZ[Ƕ³\V*J0g? |-? ~~ ֧;GAu }F73#= ;gMQ݃r[ߢ龾ZVQzk]?F) ǚφ;Z |Fռ|]|8Y$_5A񖜍xDY Ѻ#GY$8-~"-jmkoO?> %zßŏ^-j%$UeH|tHmmVbk`5/¾~ _kk|d`bђ]ss]uImk K>| xu1NԾ"kv4 KF2wJ;˟:˩:'Ŀ?0gu-Ib-Tb^/(5Qc;JJ)_#R]Ci6zmo4q RH7pLu;bZN|7ҭuG#k^Gmm{b%$54}Ky[])}˚4[Ixq]"h꾥zE x$rz t_~'|6FpUvMsOG7|B[,<d=CZE.2eM댞ku ꩩjތ׾ZSp-G`]Enݟ_~𳿲?._19Zԥ Wq\}j߁Bq gs1G ;ѯ*Sx2—~9ð ꍨ?fŷo}k/C+q|b=<|,nwЕ'ߍ+k껔ݚ][󾞌J߇w]׹W-~>xFt9/'cy|;3]g4WWM*Q _7=@Ex5~DE_CWWM*Q _7=@Ex5~DE_CWWM*S&x`IOj y!2s?QOmԿk~W\w:_|<.^lnU /rMpGS'U;mhn"N$GS ((((ZK3Dh\5-p_~\W%>&'ھ]GW2mG[>gvJ㒫YU4$|u̫6t)E_2bg[i#?'V-Ojxn1򴻍R%v@Ar#yo28xK~~|C{_k)c'PĖ6q9S?> -.oƿڏAZ1E$|HKKJsZޗug}S1W][.'u?Is¿ tx_J$/5aiH$ w !XҼR iյ~%x{YҡK;g68U}K)y~,|{]2Z~Ǎ֯WƱG{:Ȼx9|^{'O vgt]Fl ]q$gh|1EsOkBٿu>ѓO59iok"gH eTPHZk޵^s)Ԯe"DkvM.3_5o 2|P{&~j)ŤQ}to,v`+ zҭ{wgjFkͽĿk-1πz7.t~Q]99~qcUnu\x~-t "Y-QD3Ӟ~#O^&Փ0jFtn8#׆,-o4km{QVmn,B %vp3~ឡo[~j%99E_Wz5@+`((( ~.ػuWWh??](ܨ((((((((+u= ;׹W{wKr((((((( .!K9d." ^&#0*H2i=Ʒ>7<Xo??WiPKXIn]$Q-C&WgK?z.h-`<G9췢&&-_\> |Gog>hۉ-iY7Nkg?+xSߍa]o ‘[j0-B^2HϽ}o_b4τ 6MyEw2㿙0vϾs^/WuOj:%ԭN!o Ќ@#iwڸgoY[k~8??"Cn|?{,[HPd@2)S 75#/ޕ{? xRD,6WĚOt>[ӤXP>\ |+9?d~_t?~m,+mJ@i|,4$OGv/>Tj:)y&KrIcn85蟱g<WZnmn|35 A2vPdr.߯ޗn{tjKKt_iuXNÛMq[QX{ F"w`pAXuXI,+U.5%/[QxuPn.YUVEQVt]r\wWk_-7RYXpA51Ʃor;8+ߍߵ7ƽZt_ l4 ]EBXd%ƽ? X5y~(x :fZ>#,Fr>n꿱Q^?׵v_}eKj-'s[\'ey[M(KDE;M +DZ\uީ.h]4ws[Ew,* l*|ɴ74Zg߳v}>l5B$Aq;w.Wv7%֡o+ַOl|KS7gំ쵹+v9QJ7Vu0\G0!ػ4| !;I(((>2fLj/ C::EgnԸW 6 b㇅_iVz.kNª} ^K$?iܨ_ |Ix?UC&fz[fpR|Guğ 5M-W$KH&fWW(u!,'s5x/?e˿j/fޙ<D+XLۏB8 .MuKRM}ڿ#}s_┟ ;x,{x~=;lfn#UV%MCV /E7-?i6OOeIF ^[qngOx?B\sK[-# 8?W:ׁuxqxsTk{%.nUe2 jIrVjkL)++[|ӼM[Ukx,˹396ki{qKxSx> ꚑ1qCek?e.?ukȕئI_&}6v!mGs)̲/b2iCӢ//|Wlao迳ō7Y^/ kPKK¡ XAi |y/~$|9a?GsBF湧Ǣ{pIXaO8 wFfgQ{nqH5O|r>#MI4QyO3'v1U =BKw}OX_j:TbM>[{[UݕŤp",7$~~{~6Լuyqx^hl{mkyiʧs_B[5ߎύ]wITq$`$R 4n2~e GBEdxgh>"4h|S|M[ǿ1=U@]άO$ .e}6~w{j{Ǖ}Knk/>w; u %tZ$WveIRkSߌtc޵MSZtAtXNJmT˰m7τ^8ѥյˍs^lgQ$房p>T9~|Ÿ?þBhCr - ďref\8`SWJ: )|%i_K]/˵,xоg/fx KA+fA C+ ټKrĒ q^[ƿ4?qϐk^!%ֱjgy(bd[r<` 1q_O~#.<- Ų4ݖkep]9>{?*V;cI9mʡëCAOKO-H~YG>< BGAd]wCMB%Qr7+?e㖓cKv~ 䯄`dSɼD8ҭ|^~4jk_|}i-eڼ+A&"A'+=;Mt]>1 +QEEP?G7եo-zzW-;! N/ٗ¾,~ Ȋ ;O5UDQO!\N{T mq8ɯ- kZsBF^IFд!vh(ݜl8 %#G-'?_z0۔\=K*v FPe.@qf>-uuuCw ʁF2;.tM:Rn,-g, k!V ppy7]IuAG}(nOSƙi :?į\ç#]sM3N4 *Gw[z]nlyex$I )WpH# A:/ ٛ]'MK P,(Xm泏^ۯ.'e5.?55ػUr w'} z{Q@Q@Q@Q@6Ef[c@`3N<Y׏\0oxkC}TGmQ%^:XG>>;\׸׆('q@EPEPEPEP^y=.{xzH[X|]9hvJUyC/߁5mDkM&(( \d`VUbS,g7^r;'4 &ƺWTu|D'q ^8I )  cW$m?~M`o;;lyy۳+_~Ǻ8_ݎ}:ՄLo8h2! KdKy!5 /ė7oo-u67B28?NKnvק,Wc.XΣ%߄/j__.'"yvF=0q\t_ KɦUyLn#}am#)K, [xVYcWTeiUN$ppsp1F4xᶟ1C=At0+AUc&{kK> ּ|ׅlc Y$Ρ I}C冏;#q^ sx7x~ uD{tS`  w~ELj~2b[ ETxvVxfE$t%Gt6Ι(,o_|Yic-užʹv50ʳ40j?~wvwKKMW/o\|ǿ ך+O xeb0i~Pஹ_N|^xo7w>jc"fE+ſ`r$fڕuixc8?¤_pEWڜx/rܺ.k~n ߈Z 7yg;w_-\vyh~ο}jv_| sW"_N(?hmیdyRw1V{?´!žRma4%f'vͷLk6cZi%{+`((( ~.ػuWWh??](ܨ((((((((+~+EW=E Tu/xvW3Z\Z?${Y#nd{bxz'O #C 2`2A +,~ly|+C&Vi_&xCyf`iY)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}KQ5'Ǿ>3|dKg[@p ׺ژ-A}Ujw|9<+Zmz1%L|V~7 Lſش#zlskf#;p +((((¿kYxXx[]L6F)'c[#c1^[qo*O$X2#%PGsk:4R2IYOH x63l~xM@OƶjiQ̓GOi+~JKW /?GiO |*~ܨ /)A/_/Կe?%r7쿴R}¯_~=ʊ߲ҟUK_S_ _*+~JKW /?GiO |*~ܨ /)A/_/Կe?%r7쿴R}¯_~=ʊ߲ҟUK_S_ _*+~JKW /?GiO |*~ܨ /)A/_/Կe?%r7쿴R}¯_~=ʊ߲ҟUK_S_ _*+~JKW /?GiO |*~ܨ /)A/_/Կe?%r7쿴R}¯_~=ʼM~6:izm3VWsԚz\,o|g%Dt|E|9>[?i^'xB zײxG:71xwt:N֖v j0=9$I$@QEQEQEQEQE߳]~wşځ6cC ΟL1\G<th8ȯy6?MwkvmxGp6Z)IbZS0yI1|?Hix|kV8[ _C4;KJ Kܨ ?dUfҟ2|*R*+iO>?uٿ Կ:=ʊSO_O_Go)C'¯'/Nr77W'QJ Kܨ ?dUfҟ2|*R*+iO>?uٿ Կ:=ʊSO_O_Go)C'¯'/Nr77W'QJ Kܨ ?dUfҟ2|*R*+iO>?uٿ Կ:=ʊSO_O_Go)C'¯'/Nr77W'QJ Kܨ ?dUfҟ2|*R*+iO>?uٿ Կ:=ʼҼT'}+S*%iycEVn'GS_Pޣ#'S|?Zk<+bz|? ;b5mo叹5@1ǻ `0?EPⒼV{Kmkl\\xfX4S!Qprs0r\ƺ`fv'7:4vE.!ʆ 4JW>,٤SA^"Ѿo-_KykSR&;Cil$6b< K+mzƋϋ6 Хli=ƞҕy hʳ0%VlJ+WVՕϾ|eKZ}Ǻ-׆Ѵzwv%y3my=L!PgK[ʿOڷ,mG̕eWfdL*ݝP+[>|RO 4(#ZIE<~T1;{L2 |3R b7xOEs 9(!q=;E:z.>?P\yu鉒Y7.!Sd -z <*7b3@HvgHtˆմ-%Uc8]c(@sߊZUosm%? /w5Y> ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((+$|+o-nV-m$N|+zp~}嗯@AƐ|BN {FIqi7䋈$)DC]r4wx+.-`ԥ 4ڥڭ]OUY&o.Peů 6P6Zڌ͢m5vky@leXxuRs֒9o~Ğ7`Ռ-r .BkJ*6xv?ų.^kEӦk H"c$T+ CcQ+km&eG vO3[s$}E ],yj?,}Z}3Eh0-01RIw|,pv[vK@\_ ;-;!UulodHfbe,~o+$@ϮJ>蚺c$b9$\MQxaJ5mswq&W,zܞp?/~!Hw1Gijϐׄ(f@C q&_7-{Cu(QEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEG"jʣUoKտԿ"zeO տԿ"eO տԿ"zeO տԿ"eO տԿ"z{5tkMZTtf r;zv^'LՉ,IQy.81Vj7 z@E zz$Q zz$WQ@C zz$R+|8S YsR&+'f_UWK]&?oxgg*}zK"4O-PcMEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEU s^Ӽ3jz:~n"OrHu$957 0):l2FChT=e-J !Q ox {WCx[^9^*?᡼-@+᡼-@W׿ g?atzeW׿ ho +?@kG=2 h taT-qGf.|]-3n-n4VIbR|U?xA ]4*_{%櫈'WntWCx[^9^*?᡼-@s{9+:nL !Q ox {sVGQ^g ox {sT{9+ (3sTCx[^9^*=WCx[^9^*z~Оd\W:*Nq&E8{9atzMgݤ]Z\F<.9UAuOY((((((((((((f W(JլbKXSN-)bZcQ^M2s9/ gſ]<}>Uz?|[EUQ gſ]<}>Uz?|[EUQ gſ]<}>Uz?|[EUQ gſ]<}>UzoţkT<[E|=ʪ+׋OOUK gſ]<}>Uzoţk_^-?]<>Uz?|[EUS[ߋGOUJzU0=foS,*9.?=foS,*9.?Ey7)t{TœgOU@|N߀_xSMujy yd)ȦD߲ώ|m7U0=fu9.?9.?=boS,*9.?`z kŧk3.>Ey;|h*_-*((9.?򪑾xX/o]*|[㧏JWϋx|?ʪ?LVϋx|?ʪ?LVϋx|?ʪ?LV⼯ϋx|?ʪxI^^?#n+zҐzי~ΚƱ|5^^ԬAM]E Rwk :q|QEQEQEQE5}w-u G<4Kgخ *|WGp:ZWT7[7oTPʿ_n_o/o_UQ@*}չ~C?᾿ܿh!}}UE| V@ r?U7[7oTPʿ_n_o/o_UU[ NUY,༎)^ Ep#t$XGPA>_r?}չ~CW V@ (_o/oG7[7o᾿ܿh!}_n_>Ur?}չ~CW V@ (_o/oG7[7o᾿ܿh!}_n_wXՙ*f8z-_\\Gi <8G$)ui>k5;=5 [˄0%aq܅bd(᾿ܿh!}_n_>Ur?}չ~CWתοVJxW\M֧oZ%7PȭHq!`}VUxm( ( ( ( ( ( ($|'dh [{HXH x4Fv1\o>:_#ⶋRKM6~Hl[Dly69k,ɧ躷k/xz>t[_;ےw e?v+W~3u| '^/ NSK[tu<Fӆ'q]q7s9|9#x,5[Z+[8.J*#ikG_>u'۫jpZ7ra)P}+j|H<%F*@iHn%M͕ƙ=ŸI[fb-E#Gx6BVg1-gmgQle7mcRg;G9Yn|ajhw1R͵,QpEŬrgB[)GMFI Ym&] Ǭ GMJKe=y3yR"(+'?uxC1ykVе+t3EQa)Iy5=PxzMKhqNs na6(Cmȱ]I#O|(=jz_Zމx MbbmUc¹y3 i-4ڟ ZK&=io+>o ]S}ڸ1 u!n8u;M]?I𽖌׶V1xHY'ްI[DBr#͝i|1OxBϖ=>OW(yde8jHp3y<]w\x?z,Vo Gw@bAШXqRQ'[f)0usstzcqmbƻbyrUlpiڝv7p]CPH9bpHlqx^)'R?E/)o<1evƚ$ģLVYe;DʬLPn@o/KZ I5ln춓c'2y HLj'ῌ/kMs3kQHܫMgl`i և%-'<-gFhwVz {q9XB.d8߂>4Q IS#mKЮn(Q7mRVhdE?|7ڶ[NK 4_.#ށ3U2K0_۟D?\Z/n5B+4LGSKe d@/jMwChz k3)$DLѲ=,{Uco\ׄ[h_UJ~zhibH. Q8p/W}AմWIԵMNgG-M)5I%F0*+֑ 5K='~/4ZV[8&YɔLr^o_ 5x]JM/NT D3ܛ2"eIJ s9|9|i8;w:mBu[b[nyq+U>]3,7.6.\բ.XUb'Pa?DӼEG_ZW!avݼ"OWO07n,U j:^u}{MbHtb &Gy{#E<>rIGu'~#fy7MŸ .tO n?b}F{%ZHHkb ˢk⇉xA%tFn#I,#C3YE`"9"Տ,.ޙnSTQ;s֓ {.<$M1HX_DQY<2$7$nYO {c>z|=ߌ_ ?xv>m"ׂkdJ 31cXs"E[dfPXn&`ndRrBbyGcBE(( : yg$Wş5_O_?CתEPEPEPEPʿT1? ӥ}U_*Q&3NUQEV&KgW~U@oX~$x4tE 5ӨsoKoG2|㿌߱<)g'GJRHԖ bM",ޭ\%vZ7"|w^Vk>1Yx|-?khoh{leuFe~nGQxr{KgOhIs}x{V)FkRCyU2VvfUܠoGh3cxVU1E%,B+x ]̌ Qݹq1-k3_J#/:TJ/N+)umFV݂BN*׷wweZ+Brۙ+M6}mJ蚿i+o)I&kb,2b=XbB˿uXbo\M{ƺ~71٫_jzV5f̲0< *>62k^,s+3GrU}BJe_?g/˫h^n|;ksy)nXz8"j4YEl[i^jԮO{YmukȱEs ȄdA<Fk/V> ymcrx:G+d2-rbY-,Q,w.Tnhz~;?}/* ˑ2vgGSѫx2c⅗-no%&My%O6F%Uu8Z/ۿk+]_}>߈~k_?"iv"Xk:lIU\XA.1Lc=W,؂ocGH;Nqxz:Fԓ]寇mtXLl76H mmV-G×/%%Gk%Ū5V0grDOjN6N[z_}M~#޻!{i+S"Ad_sO# ^F0j퍧 x'M焾Z6>;;Yb& (} ]ݿŸڪ/dGzTJb$7> g ~"KW+$Z-$ޡݝYUL(1a*n^ӷEoe =\}۳O/W nT 7DeCRFto2de w$_?G>!FЛ~<𭇈$0%7I֤ Km9bI0&];1 |?x q &þ04~YJftH)C na8' o|{:.~l[MJ j.m<ӵ[ bKdHBhiY_ B1X6~?e%šƚjl]9*@昺 $(,9هگqS/4OMue5ݳ.%#,\+_|-ٳt:Ωί}`kKY/Ir/2"H"m˂6Fy߲SmKh\IǣKy߾Kx'<'x4Wfo/G'Ӥ{v4nRW- m/ |P{xwH%nj]1mR1yƝo_ᇎ| uj6(c.9I3ò1g7<7y\DWƧ Z5YZb+@R5KKO<ձn:o뽖w;>1|`u/BӠ5}.#yw(kufjfXF'yys|Ok^Po#w4bYyTE$~{= طj2 f' i竾we뮧tV4 (((?5ǎ;VY"7RRCW캶ȋ^> ˡ|@~xKWa]Zj,vLl4$Gxɒ~ 3 ouԡ."O2JOFw6T9͇hAj)LLY<;fqB w1儣5sO߯W66^*U R~16)eᾣh{c<2Pʫ"o7 Źw{KxGŚt}[懦]jx'QYgh {dѨ 8v\5}⯀ɨE DvZo5K 8")(Prh|1i׫]i$sk:i#7SI!Y ;2mf~?rygo_'<ן7kl'vyݻq Ys~uXޭMz5"P[gt+6Wݭ=DqԗWnvqzoA[hWOV;|yZTF-!HѤ[*7xy^iɿ;oU~lu;3]BVƟg}i0wNRim僶XH,_0|?=KQ4 Ϩ 7k%S,FC1nW$(aVڲK7\(Q@A|ep\l"wU|L⧢vwWVu'(ʮ⽺r<GÿYWWaTW9?|p u@EyG#?;]Q(>TW9?|p u@yg#?;]Q(>T=URzo i<\-?64z6#J#[g s<9?|p uE S˿EAG>8Awf2s{˪?EAZ#UY Q8AwώP|.,Zrx,rٯ/s{˪?EAꎭ 7dҏ(*ߚ߆ TLu*ZʿEAG>8AwR=W8漳ώP|.s{˪},OTώP|.s{˪VW9?|p uLT^[(>T9?MOւ(k?EAG>8Awm9 8RqG>8AwώP|.e(s{˪?EA=^G>8AwώP|.(s{˪?EA=^ו9?C/~6o{Q:~?$Wş5|Ïdjz:Σ6j7miy}=ㄅ\vk((((bKJUɉMg-+(((F(_g~syq}u,ٽ۝(㾖kaE ]* .xRMINј]m5B ( D\ߢWj{}Rΰ5sam 6_,љ4oV`m$um/:>1MҬtM6m&0DCT6軎1_+i?h |5π->,Ƴɮ8Ф) ӂ۞F1*,VC-/?ōwsx-<+-֚- #Xo$\xZ|Y5.Wv&OxoZg9m&;Ir2IEேk˦G:?tf7YYHAbFq(|3%HncJdR9 9sƚ~x+O|[h7n{Ȣ8m6a'Wr|S_IOSh`>5#x Q[ Z'v{ ~+|)o|5ڑ<74K!6#wIHW_i'ǛAUώ'5GG÷5o ܚEmVV Ŕ*r+BԵُ4g/_{eYPA6]_hzlzV4ZH.DJ :’צQES᧾Y7viV| %LóO|o),? K?;^Eg O2o sEh}daV:,>(t-;RI^6p9'*]C|ۃFFȪ+ PEyqWX5ƥus]Iu; 6^{Xd(PNNpŀK mmkvײ/~ =sρO|o),zexvS᧾Y7viV| %LóO|o),? K?;^Eg O39EgRY?᧾Y7v2/g =sρ>\)6_3I J!G 3pz4Qxv隆/xR[[Nh,H:d EL3Š(((((((((((LIK@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@SW|߳?1Rz8կ*s G/[ִf?-֝R[t0EE*q |ی]>7i?7;/:^֡)XaDr:,AUߊu|nZBR9!H*ot/>;ybqJc#Ițla`TyyBT~Z|}DG./gϧ>Kvo|~;.8>#j6zwLvΰ@EŴ1vEe7_źOzWlb~_Pa J$1*N7 s߇mO %u&ϻQ^O܈m\(t\V0W[ys+}ZTjsѹ;{¿~6|L?t<볩/mu%<Ģ \gzU5Ipq1Q⯤UJ-//DiJOwy~ |}i(|m|5[ >MB|A)FbGB ;gC}6/>PPX_,Rkw4x[VkVxdR\FT;W?kOhvm-`,.x8>씶i|zdki˾)~n ;f5xM֣gJX5$b2&IuCyW ?cwIxwßfY-pXKe Y,r(1hb6ꊘ-ti?MϚ,7-Sw Y4oxVt Ci]>kq(Ie7m?6p՛j_S#Et @ڎyFUP[$9#覮}Ǘ_sϜ~ I~M> km'xmʉ7 D}._QE(((+_n77޾`_,#z((((((((((((r<GÿYציxN[ǟ(wK5( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( >)xYejI-oʶngc7,x|ԓ9$'$x SW ZI>7<wOO<)(ww}j?$D]^I>7<wOO<)(ww}j?$D]^I>7<wOO<)(>#DGǗ4?::j 6vo4vG 2=c(VYFo+xc^jɩM:Hyt㹐 AtȎde!0 OI>7<OO<) r#÷j{2I>7>~><5|n{) z_sz\ZZU֧s.oqrJw(g۴oPNJ_h<'xúzGn #)YXVRL+OO<)>#D20\ܜ:^e_[/@+i׺kt)ho^^<-R]7>:NJukgk6$`H݈PYFq g/kk9SHT<{X%EWcд%SF7Wo|om7xu`J!5x&h>|>4Ha)hORy=jS|+]h8|P0m?j7[Hę9 5T|8֦ZIcTDSG gW"$lAp5?$|?E//K]5/KᛯmmXWZ v$*xKnA>*xs]tK;]Q֡OkgHXhĊ3,ǺDʩ5@bIǗ?O|?|5lxDŽ| ?^SG[tFY6UN#ޅŷFM%k,8vacWdsmJv9H7=I5ljQEzeαpPt:D$*1hFʒ9PpqqSm7¿7=xO|$ѼB4=G?t@[F/H I-^7gCSn ki<$pA Q@] ''I?5z|ny.MG$x SW@Q ''I?5z|ny.MG$x SW@Q ''I?5z|ny.MQ@O>'W8#^Mn8:_SL(MOK7Q$wٹILq Q~Jʿf'Z~,ԇQU((((S |i&iԼ}X<.T>x\aC Ex:bgSk(_uD+$TWq134î?f/&_?&>U\~_LMًWOI?G:bgSk(_uD+$TWq134î?f/&_?&>U\~_LMًWOI?G:bgSk(_uD+$TWq134î?f/&_?&>U\~_LMًWOI?G:bgSk(_uD+$TWq134î?f/&_?&>U\~_LMًWOI?G:bgSk(_uD+$TWq135_~v׆_WMmwN>vX/L@ ( ( ( (K'D#uC6MM,{&5Q$M&ݐmSD[?Ώ[^)- gWߗޅs(3"9tOon:=\+mx7ASD[?Ώf=2?[^)- gG-?ٿ/L׊xtY k?Es,oAs(3"9tOon:=\+mx7ASD[?Ώf=2?[^)- gG-?ٿ/La׊xtYec>k犤>mLNR4aOY0r?/+mx7ASD[?Ώf=2?[^)- gG-?ٿ/L׊xtY k?Es,oAs(3"9tOon:=\+mx7ASD[?Ώf=2?[^)- gG-?ٿ/L׊xtY k?Es,oAs(3"9tOon:=\+mx7Au|qmVXUݔ:ς{kmt%dU擋Js*QEQEQEQEQEQEQE??| QfZ?/9\; ZJZ(((((:xOŚf(v3ne0ە]]͹@9׋o.|3|s`/<'ۨŞxZ6GQW̐6 >Wo|iZ;o Յvnw4N2X 0_Exீ]:^OkZ^N4iMJB (xVPa)7[[~h$F*cDe\MYn|8㉵=GPi6kSI^ /.eɑif#l%^X*5|( NjK*FgF\kj1OxYKKoE: h-ZKeBI!AȥP V6<}mݍI=I-ЖoO[RtYu-GV]NI˴{}~'x_uدM_FmW+}amihv O[k2 鼹Z[[kRNӍ^ [{ кՇ۞v5߇oc-;ܕ6`/!UH8;3uv_oR}ݭlWX8$e&۩Oh1 q 'F^}WO bL\6v2ZmIqٴ~Z$eaxZNJ|3k{CqoԩW<3E1yJt_N  u+ s͍R6GڀP%=]Ϝo5>>]k^+W52I. ItKEaK{wy)o ̮ѯ7,Mφ|?;/eu +ᖡbFRtîC)!'np|'𞽭iZi6X.|LT2P? xW^uH{oPk G^8 ,bsr@T_Iamekׇ. N[d/gyӴQ3 b5X㌢G.I;\SG]Y<#Պ"Onx#5 O,O[XNfih((((()ө(f'Z~,ԇQU*|Hu Z((((((((((((((((((((((((((+-[0~-cuuj&zeQY((((."yH`K$Q@$Zk}77Q\$XH 8 ({|7!mt/aӬ"yuq+$q8hH#uH\%3D+i)Vqj:k$ӾȤ]Er 1d9]KQ^_,ixQUml6L.݊3pTX|8=3nwW d5m'DԬgQv{\dxE."swUVzi'4+QEQEQE:(zey"cw?Z/]OL+1Q@Q@Q@Q@Q@Q@Q@y'ǝ?_:-wA֡GI%写=7 4FrstPGx8PSqx_;V<=Z?qx_;V<=Z?qx_;V<=Z?qx_;V<=Z?qx_;V<=Z?qx_;V<=ZF!^?ֽ^!^?֗=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(ücCk^ ֭=k௏t"{ۦdAm$U5B͵Usַgܓn>;tS~k_Mj8=qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(x=Zx=Zz(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;~xGO9tP~Κ.5u"AԯuwU:u4]w02Io*h+h ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ?_?|L:zeyů qZ' EVc ( ( (>FҵCoEe׈cjZqIa:.Ti4#/,ӊx]K0]ڬwYխdhe' ̘>xȯyoƫ5n4U`6ZfrOKZFVU.orvm-k[%Eu>=mR\B0(bz)SMo_o"JXHs9l.!#s=}S_ -~ Ҵm@ȠRsԨMM+FO5/p©-ڞc8'M뿗JmmvS_OϷ ׭x.<=a-χ@[fM]^FK^+h^|H;CϏugc'Y/aT5MH'fYs)? 'MեP(I$yT IQh<%nbҼ/]M IJSFB+cݛ/K| 񿅬Uf $&iV%?#ovW 0|Yf>x'Q'u|9IoQ$#@_RR$F68/,o%q(ĺx[_eUY/Mw 0bxJ+kNeݷ;%/]]jՎ)ZĶlEݶrn1JF,o?3 oh&jsHiFx֣U?2,YܣhK{IҼ3KIll"ex'"P2JSs̿IbQ#,) #__/D)7$s~?so|-F:Qi9^m3iVsZE%$^Εote?btiP5Kq4H_$;6Ox^_h/!+$Lki"۵7IIsxB/ IΏCm&dd́I3k(+y/Շ-]{h|_ڒÞ3mbMKŚ5݀ko&/fDtXfc ~{0ջ7o⯷ژ|kxPҵ˘ ҴЫZe"FV&KM2+;xf(b F 8~RK:"*+᤼#@n|ETIxG_?֨% !Q %|@s/BZK? _G4 ψ =j_i/+^W7>"*+᤼#@n|ETIxG_?֨% !Q %|@s/BZK? _G4 ψ =j_i/+^W7>"*+᤼#@n|ETIxG_?׆#8J"*?᤼#@n|ETTWIxG_? !PQ^K %|@s/BK? _@Ey/4 ψ i/+z^W7>"*?᤼#@n|ETTWIxG_? !PQ^K %|@s/BK? _@Ey/4 ψ i/+z!`:oi/+EqHxL++;  #hׁ-ss|Gz-ĺt[wg=,<#2$$N2I@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@yg iKFn!_6x!N7p2k誋w< !Q ox {(;?jy47 5B !WQES᡼-@W׿ 2/g ox {sUQxv?ho +?@kG47 5BLóCx[^9^*?᡼-@ze^5< !Q ox {(W׿ ho +?@k^Eg O3sUV?k3jwrZ xc| cFu>ǐb)#}ӏV2?Qkt^5 !Q ox {(W׿ ho +?@k^Eg O3sTCx[^9^*(;?jy47 5B !WQES᡼-@W׿ 2/g ox {sUQxv?ho +?@kG47 5BLóCx[^9^*?᡼-@ze^5< !Sg.bs<6WyCk%EƱy<,JZ9 ^EJH(QEQEQEQEQEQEQE??| QfZ'o_9\=ZAK@dc'##\|R]c.dyz +~RNvTT1}9~ҷJUuQk?<#KtO PI$l$0:*8NTr \k^24iuootʤv)c BhzSW|l\>h=rzr%I?xF_]KV4Ş H2@,IhK1v%%~BbÂ=1T4i :<iTI-,,9ϳRGFJլ0]Iq.)CT^:dC+,s^F"wVRABDO!ޖQX4'll\ot!bP۬$eJFu;l>,vXVn1<RI9鎴XikWJ_>m sOb+Mr3,n>`U= CiV]a4{dib ޸Tgw5ѳiO"Z+gOֆ_x_Ack6$"7p8l:ִ.m:,侘Z,v3쌜o}`sXEcgOmaĽ:p,|&fUUK K~-?Pk5![KmgB4-JzWY~5mFO{FU+;o,ڨ$bH{vZt_mY7[u9C&&Q 'X\"kkubymj#V!˝Ryvklh@^Ip9bĜ׷&EyH'~(x/K` Nc uXBWDZ!+_S $Ry+؊LGKEd}wc ׶{>oAaƛoۭSUtD2Oyy2 +vb@=3@VEjmw\SȆu"@7k)i _}Jem-f Ϙ#dQY:~<o}\B_[̯0 !r8e@֛{MFxht2,)`ǂ=O[UadhՖ@ 0GjͷFy^3kqG=ΝYP#WsEyq(+PxƧqݶow aa07&E;%%gp>dľ+-oz7 J?uݾ'ɃDb+K$${{Ɵ5;OT]K-2m~6D{$H!=6GYC$T#u_/i&떲^;ݤfG< X BW!Sg~&wbg5xT̑CqZkO-PL |ұX& 7|qx÷EYZG%"0ao1o5E;OJ4,F2,,m)fY$.xg?:Zi7k'L֯9Y AsH>;χ汦Z7 l^K )&GHڻA*|Ӿ,o?~ ZY(l$mݡg*āoG~(Fh˷©nz܋?+̼K?KԾk~-tYL3ΤTK5̲2Z5PJl<ۤ0p*4\.р?N1L#V΅#/-+X]]c쐙|/V )r:݋k k,n?bn1Zi-$RDM!RO,!|VShvAe 71[S7^W/4o4֦Ka!@.dm͙l``ſ>?|XCeZn H..;i.+[pd~ jg7$u'Lm&Z{ ,w)?sڬOB__Z2=> 32I*+\&1𞱣ko j-qmiRO2({9%{ q\ B6zxU/m}RY:ׇ,@ƓvXFvyh!~X^ aEiqivRZZ#o:"Wm?1ug}si_+X@|ȴY`]]'FeWBV)@> #'_Y

=/丆9Ԡq euE=es>!j~^ƾ`w S3:!eT*6O2ċ/7կW/ _Jzeui{&bb{kvi2DsF>A'<ٓ={Qu\k:h k6P$H' aq ģ $}X:x<x.U5khwrx]ԭ4mZ;Fճg<ۘeBLK+ݟ+4m6tӵ6=2P?=䚵'R1;Ɠ,f:?_^"?xM? ,Ml_%Q!p7 _i_E@ sޥ~w'> O\i_ZO89y{5B'V+譞d;œ_k~::ּOm=5?X[[)Ng;TLOu)y6wZotioQyח \OncQ)xO:$^%s.$pJ%FѹC۱[ᔹ o>/ӴV)-nug{kK,z ^z5e\e/3A9<-iͧhZL- 2Th9θeYc| %[nD76@jdEV{/;Eχ?úWVu>}kgs{fRf0l4Lga]Z½ |au˯ \hm& _ݣlK]cnDjQʅO'cGϬ|EE/_xkPK=7m"ǘ`2J!(\/~ɞoW@Jbpۇ#ʞXuȑzRvɔÝWj 4wƚ?[Ax;Mw gm;eQsO7,{ܶlGk:O&ׅ|U8+Wmk(X,VJЉ^䤞`PekZt.xN7he>p#-Ŗ'%VT*q7_G5hڦꏨ 2r-/.P*Un'V7/Z_X-jVB[iLI#kiz]#o-lc▊i>%\h3;\*,f(,*nWzƬk76-ݮ+x3öNwn)-/x?89jzJw|ǓjZ : 0׿6g W?_ECS(3y|sɴ<[9^-Ey3υ+/M&hww+|-A_m>׿6CW?㏈?oU4r +X).f,0Y͇_?^<[9^˷~Zx^|Uw jq[[_(;p(;j{>׿6g W?_ECS(3y|sɴ<[9^-Ey3υ+/M&hww+|-A_m>׿6CQ^g o {hy|sɴZ5=2?g W?_G3υ+/M!W<[9^?|-A_mw OL& o {h;jze>׿6=\\xnoo5e{O}BK}bU啙ܬqf$&kQEŠ((((((('o_9\=׫Wd++GQKH)h(IOE>E@SIORB)QEQEQEQER-֒E;WKE%Š(EPEPrg5mvN((()ө~?NY^^U4:sg:zQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQE~-cuuj?_?|L:i O=2(QEQEQEQ\oo w,6ڪgY(] vU2M:j+|iwm2ָ ;V($pl S!X[_&}4uIK?pf_̷B%3d4rF^6Pɢ~Exi#NjK)BKy:N[[wJ2#g PZO~9Wk㫯 λci:E71"YaS0"$6e޿ vo~]7=ʊߊ#m5]/*η9>dGۿc~1ŸeMb .u[t4D8bIU)|¿W(pnz5𽞻YxQoyoego;9d/ 5ȻXyHV1䫳}4ƺjZ':K737IEջ~ “}rW]Gx5Z׊$O ]^kv|Yak/ fq+ uDmK?% rBK.A%v2֭W_ފ~~_#x7n_>9kZxR]^[w qڬxW]uMLg:MAaTDG!W W/|ݿ̉KJO >_ŌeH˰%Ξ#15Oiۥkr;/=yJFGaN?s~>ExoG~ K nNc0kH徳IV3bgf7U&B>ʆg'6Bgߗiߟ@7WxjP^t :Ouw/6ok8,X@6w ~Ծ#\֛2N,˱I}># 3( Rg|C^*IBiK)r\x.Jd7|Vjm͟7q&en€Q\uOZf. 麔PL|M=7RoyOCz o:VRJ$//%ɱW䨆Ow͕郞Tl.l/ỏ;-ޛqK.CGt,GbGa1,$~!Z73CUŬ6d%u8h\Vxiho8#}_Z×"_.lI6 7L;~"xo`h?Vޛ@/< =~\}~(F=Cw&ԡt Oa\]]͆Sڗ[/Z_tN87iMO#"1RSo;Z=kx:{ fd>#hKWUaw͕}ޫ |csg xInۏ\e ;t$`LnawxWgcK_Z<$#R,%3` >|-~6Fo><9!^ַQ|Tjas|oޠ̲o_W7o}~B?DyY[8NoiqMNg@l<8gص r`-$;&|͝_[نikK_^𿆴J8wiwg]O#poTpϳ!_\^gV)IYK)|\xۗV9 7|ī [!7V>ϊduӦ%VPźC;̒?g̲Soh֣u[mCWU H$K LO7:h\gu/A2 \B\yVg*Iv2oԋ{O/Ch!goˏoug?(uh:ICO'75KS,vL ;?_te kMvox{n$>wIߔpVZ9<{qyo ⤔-$//%ɱWQ uF+u~$"ӼQiz֚]پH-#ϖ($.P?x4f4`#LQEQEQEQEQES[S[,j9RFViuHuZ((((((((((((((((((((((((((+-[0~-cuuj&zeQY(((+ᛟxGPgӡ{j6ROF\cI՘kTxV7j^|Qcex_Yaw}4Gr$ D0lACGE/~w6?7:(?bc:Z5wrHESKo6VOG?|/R<_M7P&gZ *H]J!X1De߃7S6sxŌ u]/!IMtuS!8dPU{K^]ooofkLdյ//Ɵs(ܧ2{stQ\iڽ}VicVBt ?}zנPo}G}|crj Ia_;?YQ&yuy, J,hUڌ:/_6Ru^3KԼOsccٌanm![,ybZo&m}E}nx3~Tzfjw0^h^s3ݙ I׼ }><;ojD T-=zt iokG7M=#XoE.z}Ƴ-:y' }y,l{(etUKP}nM#/m HIYD`+Aȫ4QtV]{ɟc|H[wSޭ#W0,B5%ۋ-AY%B%6(@q 8Uς< xB@lofh4&74.*b[kr|'}u}^?zg4K'Q&~:#˳\8L$j<ŚAݑxui4;#ett%[i23dU}__p) (((+gu_Q?DW__EVc ( ( ( ( ( ( ( 򟌟Qs^^SJ7hEA0b8Qҧg@w_F<5675%ajMwWK2/uWp4  8x\Xns5K{ ;{ן^ (+d+?vMouF jgүcqhƻmI2+;kn\D2$< =/K-Rƞ&}.L͆vͩ}O ^-2 gΐҹ^|S^Լ%ƫov7k6 +lrP/]~ޗ]+\G1^^j_giL[[u\2D# OžKx=MOLJt/_D-bsk4Xf$а76B |-)i h^#R~ }sGk>׵]:Vr[!7AdL;LAaHJHCHŸu]^A?/"oɩY[Si͕A;)"o/>#twĿSVHJ,rEkq+1k+mzj(k m2-7L]e4vtMdVMEZeVVB-^+⧃|A_g6K,"!OW}6V{gEFyOPgMt5Im%i' &In$T*E4D$O+?k2&#I!fxc Ѱu φj~}o0&=As>*+H3w߇> ׏m'5r+Vyg;K9!flsCvW7.5"HwO@%b+W'yQ(n_ѡ mMONl 5V4WGԦHQuA~T7V;̪2ivy3UEM״,;KD@jQܒG|1U֓7I]sPM>Y%>tDIgz{6ƕ`/%2pR6jtgyȬπp)@)=ºm%I3p J [o >)kx>KKx^R,}QuӞdQX:ĭnrӀ؄o1t>3J٢^k5|?e[S/}pGR ILgxgcwc]uT%L׶Zk2dF-;Zr PDg=sj(0\o RhM:maNI`\;%k]ʹ|7Tֵjↁu+Cɐ7Pby򯘓77@e/TB:튖 ( ( ( ( ( kt4kt4埳O_?Cתו~?NY^@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@r_>'6]sT棢V:-h6۷q e\=A hN3K->9-¥O*y*_ih?/VWQG~_r g ?Z|sZ+(K->9-Q܂Ǚ¥O?RO Aeze{G |SE?T)Y^E},y*_ih?/VWQG~_r g ?Z|sZ+(K->9-Q܂Ǚ¥O?RO Aeze{G |SE?/¾6o _ ?LдNCZ[F9-Ag9/xfG;X_T)YG*_ih?Lh</VQ ?Z|sZ++(?/3K->9-¥O(AcRO Ae|SE?2=X?T)YG*_ih?Lh</VQ ?Z|sZ++(?/3K->9-¥O(AcRO Ae|SE?2=X?T)Y]G|;z->w{5c[)2L 1]%V QP0((((((()%WW?k5??| Qf=TT AxT<g 6^׭-=Nn. 74i0 ?,{{ς<)m/όtĚYi [ܤf8VQ0z7ƞr< W&~r|?^w{nߟ+th?<>|c#Ӽ{Xjpymo®]6 w@]K (sC}ÿKu=Ǟ4wzvo<vj}3um>QmuG[ yN2lC(zVesz] y]x6Y6cs(. G1q0ʩix.~;߅,&z:ͻX!1ѳ$Ci=dQT_,߅<-};Zcq*]9q?Tq&Л<#χt]O^t-/Yze棬[[Tm3%]!ܙt#/+x'¾&3xA|GG5Ha&10e]`@Y(d|q֘''C:姊L _I``{կ|RgEkEvxMI8g(\d*ԫpwso&ԃOQנo_izƧc]X~өռǵ|#b)s`b6?j;_u/^x_Xj6cDž]+2Ͻ./5|?#axf·q[%Ġ)³*@TG4|>~jz<1?jf dvȇrá]nP~\H1J t 04j>_شKT ˭c8wsp`<5|>vjZ<3z&Kux"*6 lw!<:sU|XOQxc[񎁣xR1 -#P᷻#1ͽ2,6bGhN3&C8p"v_?ϛ;AQ֋U *S{m#6rS zXaFsj08my-'Z? z~u;7Zxlc*<#b9X B_4MS%S=7Ro4[kSR5e%È^_1פ2/˜Ա/=h;y{EQEQEQEQEQES[S[,j9RFViuHuZ((((((((((((((((((((((((((GmoBӵz-\xdG"]4F@$g]T0W1*קE'QW~;i,'m~;i,'mvecϪhwDÿXO?wDÿXO?__9M5ߎxw [G5ߎxw [GV7}+o>VOa?+hOa?+hϿ} 4W~;i,'m~;i,'mX῜[M<;M<;+>a7T_+]]ec>s+ok4k4oW}SE| w&V w&V0Ϫk̼?'-|;y5ߎxw [\ƟC^@<:ښF}ǕYe7gsہ/9;oW}E| w&V w&V0ϪhwDÿXO?wDÿXO?__9M5ߎxw [G5ߎxw [GV7}+o>VOa?+hOa?+hϿ} 4W~;i,'m~;i,'mX῜[M<;M<;+>a7T_+]]ec>s+ok4k4oW}SE| w&V w&V0ϪhwDÿXOڡ~1΅ 3eIw8l!5.T `@q֗V3}+Ʊf}qEWwQ@Q@Q@Q@Q@Q@yOO(r{YV)%WW?k4ꢖREPEPEPEPEPEPEPEPMt|/ U^SM0 t?XWK.qn)e毞ys-բ8uQEQEQEQEQEQEQEQERfZ( ( ( ( kt4kt4埳O_?Cתו~?NY^@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@~m|!OY脯ZC$_ _]O|sCu/SOOn [+'"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ [^(+k?Mn[ŝϝ0KŮL 烐?A^]:o?&xkKjZ( ( ( ( kt4kt4埳O_?Cתו~?NY^@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@~m|!OY脯ZC$_ _]O|sCuxFOn_6[\dm}>jEFaEPEPEPEPU-;d+K/l,Fpq55ͥ]˫4jɦMxn Em=sZďxU~*D쮯-1VqY S Ũ7ӹ ;N_5[Rԭneic,I $$kڤ> ֯A^1#y¥ Fpa>@߈t=cͥxX{"wubK l`4)KDin\뮾{?wŚg:|ҙ-dOʹBCJAA+˾^Aq/MY%! r|ylb T 'SFT{8Z -5e f' J}~0xW<fZZlI6BB0snlܸXӾ/xN[9u?OF~w}nYA@$;W0 3^ST}'Ǻ\ Fn4}f{62*,G, I)Az. \6 5RHre J qSTIBiiy俙SW{3cP<#k=Mdi|J\Zk(3RAGn5h4'}0WV$A(umWK{?J![Kˋ}>r.KA6t|97Y}5ukip֠0`PXCZ.˿ϥ#iaA/GI2o,mW#y 7;@Bۃ\L69d/⻁.YC '9mcz φ;j7~$sdd扦E0!A%Y^գU.i+GgۢTv dSi8,3^vM]ʧF澺n~Wu}f9"VWRFhцTFkv_M֋TKk[q92eh uzX6ǝ;)Gkz_@* ( ( ( ȷya_NW5%E rH|mr_!뭮KD=XSשKEW'EPEPEPEPEPEP^SJ7kի~2FczI}?Jxdc':?O˭rv</ş uߏ<1k[͵6l72s*@TxWz.x:A2Pmgp%M!ܙ$ލ/еr.붶nP7`W!zv|Q=~jWN[eO%YK. {PşxSg5:?5#5R/.Hq3o}̬a__gLs"?4|:sqkexk7ͽ WUw\*0Ѽc#|}ڤv\#MN0Hꗇ~6|=񆓬zŖ^>DžlZXc`{9u|l{6~<ůo6zͲ \F<PsW I#o=Yxe |Em5&PԬnY%d#*Ўؠcğ?Ku={Ǟb闺m WF;*mOXO|MxcX񎃤xQ1 -#P!{ˉ{ee3.Zn6(r8?NM?}keai$EmgwD3:c$+PW&z{R"q~bv|)t W h1=;VA=ݮV.\; wM_^Ӵ B]\ij.'X&vTsҶ):7O0nʹnc@~6|=񖓭j~?F:ޝA<61vŖ~>υ[kz^m$+6?Ⱦ 2by#h翯\gjXxůt]BM-Mg*))SrGFB:v&~P bO_ƅMV-5D!K dH>h&Gc/><_|A5NkWjqؕ\pZ ĚWֵ+-.;H侸XIm 33z֓ާH۸LŏOg57u/(Yi )&m﹕p9a/-ot< 1E8xm|w;>nc=3X]?RK9 a?9 >5,ZU u a"7Wn 1H }⟃𾷧}Sӵ'2w6=Z]l2-OOfdxT FYhZ}Σaai Ouu"Q,* s O1g֫Yw>[]kjco+lUIVck֟y^xYk6cs(. G9q8AQ] [w"NciwU.a,-ơowyqK<̩h %wצ^ʹ1_| D;<:>W>6cMHĶZFCߘ8q3o}F߼oZoq ŋ%R+DqsصcZWd6".dh;ʳhH,nGQc4 G g#Cm-|e{Ny^w6۷S/ W h1u_`ҴR mU&`p$zgiqjWJ[8;2xa=1\^Ꮕb]s7tA{lo- E3F*e ~6<񖓭j~?F:֝A<61ryȉ1s`|Ynu>Ŗ^>DžlZY[c`{9uGč>[FMLz-7ihـl2g8#ֵ`,.K2yu H) XF]T@6v8Hrg7>.-|+}9&mP\r6qH v|j{/ExgC5>צ^ʹ1_| D;<:78'Aa5C_] ]$(,a"c)|Oi|c$ԌBHuH`#1ͽ2;|m{ _^*7zr_[|$KıaV2.Q~*Y[|d&o.yj ε9%r쑮!f/6Q_[i 9>N%0<tLk>bg]VԜw8Qmv1@u4w\P2Z*1׃xiө3?f'Z~,ԇQU*j9RFV ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( C$_ _I? e*/kܿm߈S5myskcbF7D7WKc桿_Mj_G- |E֥sG{9g[Er_4Z# CG?5=}\- |E֥_Mj_G4{fuW% CG?5=4Z#ٝm_Mj_G- |E֥sG{9g[Er_4Z# CG?5=} ZH:uG@@mJ7 3ӭkW% CG?5=4Z#ԧ]GRֳ:+ϟԿZ? K깣^}å\EiwAV{D(fXN3& i æ*ʳ:y)SOAޱ?hh/&/GW伖aך$YdxsQ.ł_# ﵽ C{;S{3{M𮋢 i=BdKmjm7(P0v)o/jWP^i7W0.b{dwI@P$RFC7pұhh/&/G>~"kRz~-M΂EӴYml-mwXaQ$g9veNIޙ4>-6; VkURHe*a_Mj_G- |E֥{i9jve|;|Wt}2AU)#UFž {S6{ /4fRNSϟԿZ? Kyi5eo{{wJ͙=*Թcnc[j3ڴ+ϟԿZ? KԢ#oV\- |E֥_Mj_O=ϳ:+ϟԿZ? Kpsζhh/&/G>~"kRz9=3/Z? KϟԿhg>k_"9\ CG?5=/O [}SԮ;^W;ߩW%{I? {/'/CӗO^-Q_Q@Q@Q@Q@Q@Q@yOO(r{YV)%WW?k4ꢠz۟^?/§^N9 q\u^@x—<zdֶvNp$O3qn[7~ߋ5}L⽸[WX:IZ؜RZp7^\ּC˭N˾Ylɒ-ְjˇqIe?>E$i H S$ˆ Z1 Q֦JGscǚ|C/~!x.B;u[Μn#,<֟ϒM6KmQ]>ϩ*~ˤWf6 DF4Ʉ1Mb2*@Tl2*HcOkڱu[z5 Yx(edEXź2N?|J>UT 7OxO7 .dyb=VAXt0\n 5^쏁J$#P?hmki~ROԡ^ -OswH2:r;`+C6.xM񍅯IҼSioo!FZ"[+{~}1W&5[HaèK ";q2f}; r 5jn]gݿ;=2q5YbA,H v 6?^3x_xIxoEav-..EKO5a%a56%^5S |,K_/4Mk:O:Wj:E]yzboqf ZWT)~x/յG^eܗE(bœVx<(ח=ݥl ŮoeڏG|-]x~ǚϥZ9,;{R(eH*ZIfbHi:Gƹk^,CKsi?.E–zv*Z!iw|ne$`*}+# #P{6Kq/X*X/qW| ce=/#yy<R9{[_h%Ӹ[R Go= :C?t ?ᆣ[UH-nms5pM<Gb$'ڭᾛχ -"-esJޡ2-k'2_|+Z}Bީax4qɭIF].U\J U~[o"|-: & ͞g#.2R['xGŞtio-l^k!=ԑrFO[4IU٤ꬅr prs)DSk w !:E-ZD@w] dfx V|ljPIoGo h>h:Ecw$yhm'c{4rduH-|o5ψ]xOXMkM5Zke{۝>r¾s4c};Xڕ׮Cmun.hr\+Aێ;GCZЮMͫL$CiDʸ b\OV_%]^F2Jp·4.+=Y|AxMm;N׊i  tĝ#w/ zå- uN-/aOk(q(P)-kr~%? }:M&;/gXdHzs̘!n lciv{W&Xj)=ΙhY&Y"1I"ч[>]Ƒ|-mg|x.!<|1zt?gth!vCE:(J$}w!<MeP΢v!yvKL;y{@Q:o?&xkK I3~y ؙ.kӮK)T103ϯLi n5[@ь6ͽq]KEoREƱr?mR嶏\;MBIoxɵitWdL7Js]dR?mߴcukV]9u+#,l-ϖ}6"Ƌ>YCz*5$Yuu-Z|w4Hin5;nJE` qb}ARxZ <"vHQ$mҭH*Ɨ)7®w|R'|/Ũ[dHxan(?>&<}nmIl oZXݘ.},b0A+Rd ş@إk ,o5g꺝*۪:K6+&rJ$_Dqίn+/oɟk{mJ7y[0݂2+>ņOOеmD?ƫ;Smv76pyW0 ,eJe(E4n5K餸T|IX((((((((+?x_+GLOi?Pj|P>z~먢((((((((+$ƿ]mr_!eJ?Ih((((((+~2FczyOO(r{YU>Oo"9=y f?5O'>Wx?_n_: kƫys˶}B{X. !hWڿ*uuh*fMnu y30{c,?|mỽ7Of^m{m꺊>dvnAd ʲ$%^&R|L '__EoxrP%đ PA;@ JMoHe|+}OΎO]H}Jyr- `McfY.cH7>&sj)mJ ?vw4mGk3 #?'+_g|>U_ޅ薟<u<|^y,47k9c>}C8+ZRWy5H>/5e-y [PM~vw%KHqwLU/b 5)\J~Q5%k!?i4#KO hv,ѧ67 XTfُ>L^rHh|7 C-{w|Ar2G]H#ݴ:B_Ǻu(◈-|AhR[Aui&K0*&% oY\|.Y_Ĉ6~h4uv ,Hmt"$Q.W-82@~<Լ#gZ5}SĒ_iFx3a`d7v F72{Q,oXS}Fx4{ /2GopB.K81n~:~ - L#O'Vc˞[MGiysq]ks+BMJ8fK?α ¨Ž8 INF@ EӖ(ԀcA֮~(xwYP;gVwκk'好2^뮮]o XRe Qs<\J)?z3m@LQ۶}֩(|-fxFΫIumv)s#aP7 t+O{m#FWTi<'),H ֗s'}u-jwxaĒ¶ڳ5^seHYfO׷X> McÒFm4uV&1]* m RBT5} > x:ZaU(ıPybF9/PִkKYB8ܬpA#?h~w|Kl|E&'Ӵvy/p?n'dUcWu'iĿ ;?.-gBHSRYcn- fa'1AGe|_}ɬ%̺ \c "5Bɴ,l|Ǎ4=/1xQa Neh*#:#XNGwmki]i jEE.<ۓ[iۍ+}'O?4C"|2/cN5 $ydm9$"sg̹vhE[[3 "fGa+Ci1Xd*~L3nvס٠cL_ |#ռ[.|>fg}i^ūf!U=B;XSgR8|e_>JҴyud}rE5Ynϵ +#a kKݹ-c=?Ct4yg$Wş5_O_?CתEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEP__+_!+6'W,BWpS;mQ_d|QEQEQEQEQEQEQEQEW#Z'4W]_0'=O<7yȩ4pLYۼ$N GVROW3'R3EOQErQ@Q@Q@Q@Q@Q@Q@Q@r_!뭮KD=LiO^-Q_Q@Q@Q@Q@Q@Q@yOO(r{YV)%WW?k4ꢢ>` gcRZ? xoƿD ^ֵj׺5mn+U%p2vS;*Wݟ@/< =~\}~~%vPxjP^t :Ouw/6ok8,X@6w ~Ծ#\֛2N,˱I}># 3( @ O^x[qx% iC+,rlUwIߔ[-[RJLWYM˗ثy|Ow͕蘒Tl.l-ỏ3-ޛqK.CGt,GrGa1,xUmh~ޣVwZBHwPj^6l/9څ=9r%Tjt˹SSVB>Wݟ@/< =~\}~~%vR@y>"x߅~!xY߇tMGVm?SKw{FҢI ؤpoD9a$W5_%o zQgLIqwԛ_I4#Jv;~Pga&gHfSژ {=煼7P-4|Eq%|y.Mg%D2}lq|C6~NJ䝖M%VP×! ##qٖZ+m;DCFhwj#RDλ/6ֽxiho6/jpȗ˶[9RM .N߽]-?7vT6?>^y{?N6~Jv}9MCx{A8O~=ܿ8پ٬Xcg|;:(M_^𿆴N8wiwg]O#poT@eվ!ۋ xn/$m4|Eq%|y.Mgʈd7|^nOq6\2$r4wBw Dvn;2Lj_Vv.o :mGqk!a)wy;_q>l/}Eqzh|MAۘݨ_k×"_.lI6 7L;~"xqj|?eCj۳h!goˏo֊4;mJk}PgŔ>Ok7Y}{u8ݦ[%v7<`]!Rcc(վ!ۋ xn/$m4|Eq%|y.Mgʈd7|X _kaK8^K Eu=l;Venk5z995%PxNvQlWNvZ+\n(ӂm?+]"luFmƭ_j#-3!T bzS Ѿ"x^þ"7=x"U<J6,e2{qG=օ=?t?AcH[j կm7rVI$:PI;FI=< xI^ZkH$+p3D%wE&oS]]XoWsgtkNFu6,^j .JU$W[?_t7ÚridR] H."!`[daS,X@ө~?NY^^U4:sg:zQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEI? eIkkD%}wU>_ ~no-owWr<7R"e`7Ƈ쫒ܿmCU?x ]K(_爿Կ"(`K~?"—R?W/)u/H9c=_爿Կ"U?x ]K+Xi>W/)u/H~?"—Rh=O9/U?x ]K(_爿Կ"(`K~?"—R?W/)u/H9c=_爿Կ"U?x ]K+Xi>W/)u/H~?"—Rh=O9/U?x ]K*/6cfAk @K"Mw4RM}r_$Q GKEuSa{Ig% GKE$W[E}¯_R_G*,IZ ,Uq9ZԿgxVIfO7܀P@%5iKw-ܷ2zu"嶞}w4v%^]>7#/V6R@KBHG *8$23#+÷zbV~4}as7/xU2 Cg$񎕭[>I!W.A_򞇃kғ~kz_e [eAX۶1m6 1 D * yr\TVk_ t[6k+\m> M&25! []_|4? =@6 Oq:Ȣ- FyΓbKZ$ mfDetr<_r),q-:~dۻ_zҭIy-R |I+tRHc.<6ri0-粄RaAaYa1Kd_]kؤ8R7;>C@ /;Y|;)$/rd{'ْ̧*#N 7>TJmsynYJu[KGO ٹ#>\ wzG]*I[4@ .2JO|W1/SxVS|;{u}bɨZɧiU#B= ,mãʍS@@_B(I?k .˻9j7z$Ѵ=Ρyb3j,Gb|凔$#pk5% Mn_ xFXGg+IVz4o'tW*Rv # ;JtˍE;L9#vr2IZϖ2~iܮhr[o>[~?_kR4V k2_Vi"张xKyd])6gz$)#P&2U7q^r3x}2ž&{L;-|)!1I@Q^^..ʼn](9((/-oWӕu|/nzj~4_~[\$5%zk$ƿD5~E 8QEQEQEQEQEQEWd++Gj򟌟Qs@)i-QEQEQEQEQEQEQEQE`zw;xwZnYMZ"nR,W*pqrusߥz~8!?#AqCP5G?k '׫@Q  @ÏO??^EyG3_/?>f_}z~8!?#AqCP5G?k '׫@Q  @ÏO??^EyG3_/?>f_}z~8!?#AqCP࿃~wjzu=N-5~Vh5Ti`޵1h?SQ@Q@Q@Q@Q@5u5٧I֯c?!kkʿf'Z~,ԇQU(((((((((((((((((((((((((6'W,BW-~m|!OY脯>w9!:k#'7/okOx]|LyuD]$GQԂ{}>f-'Er_k?xiG!Z"o"Qw|?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4D+=8մf-m4+[i-0'&8xĹ٤PNiGQ\!Z"o"QVCƛȕW}ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:ȷya_NW4/%lxcx_G[n/?}5]l$Y^YUQv8 wݔmsV/Ik`K%{I? {*?Ih((((((+~2FczyOO(r{YU ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( kt4kt4埳O_?Cתו~?NY^@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@~m|!OY脯ZC$_ _]O|sCuQE}EPEPEPEPEPEPEPEP\%kğ~]ur:gKcS⇯aG]EVAEPEPEPEPEPEPEPEP\$5%zk$ƿS/SשKEW'EPEPEPEPEPEP^%GxD?> k>#,4 4tnutR9 gU<=+wj1O~(^?OQckԣ@Gzw@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?㴣$>1ckjl"~1ci?~d?v>36S+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETXcٰO_,q+jdc ?俲e|&}CMg.mo-$Y"'ׯ$GRCR5+H9((((((((((((((((((((((((((ͯ?I`K/K__+_!+O~|ζ+񮱪ix~HU^ۼd|xcvy+[ s;5f???&;J\?:+ ~~M71w 6ʿ#/'?|wCh?:+ ~~M71w 6ʿ#/'?|wCh?:+ ~~M71w 6ʿ#/'?|wCh?:+ ~~M71w 6ʿ#'DhzMCIF#HXefi@?|wCkZGĝO52+h3y)†t_pdsLjiJMcT{7=俳|wCh ~~M˾7*o俳|wCh ~~MrhK71w 6'.*o俳|wCh ~~MrhK71w 6'.*o俳|wCh ~~MrhK71w 6'.*o俳|wCh ~~MrhK71w 6xZ׼>n5^{yg# `hlΆ/Ik`K%{I? {|,t5~E 8QEQEQEQEQEQEQE5/RրE"n?JZ(((((((((())hssqk嬖5+)Vtrd0|?cMmz~w<g_>KXmP54X>tZ[bN_b9=4%Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@RdPIK@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@~m|!OY脯ZC$_ _]O|sCuxFOn_6[\dm}>jEFaEPE6I(݂"1|`XjOiVeŭ/@b[xS sU!ߙq𫝝/ГRZӖ15͝$2&rbïE+2Gd/#󝔲 $O>xYocj4z5+{]O=YKy.;Efe@WcAMi=qC%庫n$jYs3N3va(2D5{W6гJSj:eF9TM.ћKwp]+2v +!I|7WJ&6R&< Z0.{?&q<zJ(<,-H˗sѨ<ī]J@LTdGN8I'ף~K(u]z7t9,[vZ׼A1obny^ּu:Gh7 ys}ne5cpSN?Yk__W]?+ɬ&jv77ޛq}KMAd=Inwv8k!oz%?hӚ—9]S#*n8}j?ԺK+ͼq/WfD@Kx֚D+#f.F9{ךiߴvu&UR8wLc5Gl?vw*ı>G{_N9լ})EWqQE|/nzkKwsS:/'/C[\$5%z%S(OŠ((((((G||C5׌&\[˷AwW*m{uyGo(r{Y/M;۟ XWn/2w?rmߊqbKE y(mLQ d+bjo3COD]Y^jCgm<3Ƒs %)v#eW+X'p;eG|C/aƫ5i-=}Bקz8- Bg &{cxÕɾ4 X|OuU/+ͣk"YiC\,Ā- rQ aR?dh?T?;%?yQ^{-ii]׈8]:"Uǝq9Px}Kö 7zmˢ˰7?1ROq7ky2#^R?;%?z2VG'3yӧ"]i_%i+]TDhjKs| 𮬉`{+;8SMSR[YU,[xΚ7eg ;#WUZ?x{^R(= {5^)?">k[6i%֋jHo(`d K(C7Ea]x_ 0n7ho⸷@Db%>rfL nu>QDw?? {5_0ꟴ?M tH^^38!902FNp=;BH5;䷊?e 7 A"\FJz eG>T(= {5\~᷈~1j#5MSH.J6F4C"ɾsȬw(lIEơ C׼ u%|Cs!kٓufYU[r6B\b2#d6xZ[O)ZBx`GŒk%_i:GK' .6æO4#"i򶹎²2*i?;%?O ? {5Y:0|9XlƇ-t[TӤF.$Xcs+  x['\i^TԼu%;5;K6㑷o t_:t 5b{h]7bK;GW'ȑ~*+Us?~^Q ?? :=wǞ6ˁv-4Ѿk2E_I4ʲn5pC?'OE.ak [xR[%ź]Osp--+q=H0΁@4C>QD{{5G2#Z0;1h4O h/j7~u401W3 n!!  C?T(=SLW7ĚnK[-mi= \:ohc88#|Hoψ=%֡G{g֩q40acE;Q' cT݀~{5G2#^VU/CzZN?-1$jI+昁_)yTԲ~%潼5?[#@o >애;uH&1,QKP!DwK?>jo|🏾k} \Zk:x&i"iz2rFUix? x୞o6^/mJK@e.U7 \ =C/aƨQD{{5^q5/$>#t]vkxFuo_HK9I$w1Ď$D,+!MV&u?|?^R?w{5\q M_oj [+]y:${C$8 OƏ_o/]>iVV:De $J3!t۶wvK!?'2#}0kبyS=w_ׯsxq\^=5O5ҵBQk>D$ѧV,#rƳ?~"uANjn b_ofeqU]>9dFv%:I$$F3M7k5ﯙO5yxKhе}M.|MυI2.cs4bd@9$o*Vvᴉ.K7@|cq 8[4WE:J~Unq?,dvPf:l/#*1EfT l$<GZv&MVtŶ+0nt7'iS(H?5{L2>vzmiN[_}GwZe_E6eŝZqxWQ-?{JjXY|?öo.bb;{'|}ۭD_Gg)8# +'75_K/.^[y_ZÍktKEu6!s$"8A^8-w GZ7Lu]DĖŬkLI82B^0z`{1 N+<M7-oFך֧[I -+,B{c 'ވ$,JƬ|[ז+y#zEhv\H8 xg(XT*׷^!wComk ^&Œzm8#`ydo43b; #k6; OG6MޞAzEuʲv3(${]M<޶{ºVIX8 qIN>wI#a)šu;XHnau'OF$0 pAf:k5Rqw;S; 'V唓+yY/t*cM@m(ݬ諒Ծk^FZ.i !Ť_G6ČĹl4͓} EJL>$Z/33jCt3EtyW[ѭ0Y'hxʉ xcwoZx5OX7k}>n[VX9p@OInXx#koa}eWϰywl7`gvϗw3ӊEqQ@r_ [ޯ+k_"9\hN/Ik`K%{I? {|,)k%+p(((((((% _W?k5| Qf6~)YxP.xgmݼ֚[l⸍ nhZH UkK|_+ABռat䶺7;[H&YbJZIX' xTG n%<6&t5i6$KygC1=ˀ S%ur|2׵_8=jZxVZ KwVZL̍#!"Fy*+dON Goo!wcc5)aukF#B*#G38pqQJ~4}[']Y5iggm{7qrpV?5?4a*xIxMw}#Mud`sHd%ih}/߳λ-1xW5χhjܵ*d'jJd 14z?Ÿxu]WVִ__C5ٖ4nsE'ϙ#~~$7zw\"٬"Q8T#';v=On/hM]ß0>. izZ`ԦH<1VpD9bebƞ2xHww7˙ ^ VO@1ak* o1,;-O<_|%zkz?ƺՕu<$.R;urZ2v1 8[o*? :/ϯ_\s-Z<Uˆ'Tg y n#1KT3|Q/bޙ _i$KWvP|Ăp++[_|?dåvrAEd<K.%Fa7'R[GzEjf8<% pp+W~˗ƽx"VO{ ]ޡNAwT/:2E&p+)- zK3o x+G xD--u[( F<*+73ϓ>7ݪi^KgºA54` -b"X[yuxDSyY1̢:MuMxMG%>(2v1neLVIcrp64rW>ߋ^,ּ1/S5m+HKM+SZ:ige:p4;] Xe$t%at6t3hz5X5ͬZ>1mir 7c ~Ù*tk{[+bZYC|ړG=6rMnfDflcĬDhO[#s<#>D<xEs> N[{H~CpJQFǜ}ѾmIaVIvT]FLh;СKT67vj)5p>Lw웫÷_iƿA񅞓M_oDʕd WD86szυ|%x֭ JFX'i)mI1(zg?f~ڷϥrAmB`M.tA dG,"\K 'ѽD%Ն `2>*wnm=VٟCjmE2 +v,H`WtU.zZ|K|4Χ?ͯYJ(8Z) orF+,xM|7vWVz)=x?$HPd k(k;_I٪Agrt64Yk;m4(|yy,$}c+?ፆ[xvGo<CneMuS+7J5g٦Ӿ,^:sqm5HmwI l h00= .~>qk:ޥ0b`Ks$ !AdMM~Qyg$Wş5?Oc?C׫PEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEP__+_!+6'W,BWpS;ms>5MBoY]Z^{ۇ%V7rGtWصsw9//й('*?wBrr_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOUs-o[F43K8@̪[h A] [[NWVH+$ƿ]mr_!O^-Q_Q@Q@Q@Q@Q@Q@q>_aca]m7SӮ zOm$r.ce#v쨠$m._Vo?'3_/?>^f_}~8!z(?#AqC5G?(k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqC5G?(k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqC5G?(k 'm6xH?ˬx3_o[Q@5|*)Vܦi,-?kZZ\ҹ@=??G^*e 9)k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqC5G?(k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqC5G?(k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqCoՃ [( yq=z|;&°B 6 .]]u+I<<#K,b~nEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQE\|5 7ekV@$J-VH=j+crpI߹ŊCjݏϟ'x_~BwAW~/3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~BwV]'OW&.mmo걇ePY®IE~QI՚"GZ| +p(((((((((((((((4u8RC~'|Swz~YtK@x,`ZIQAl+?^0O[|>iK=$+{c4W }̺%=Xx{a')~3-}E| f{u9 ;@DV'uMCAԬKd[+eX'| Umsӽx)?svS6O<{u?AqοN_|vO pF,ziD:TbU!#$Fs~A1|3>#Hk7cqx3US0t>3W`ǿW_?LCE)?svS6|zUA:PWQ|4ߢ|Oq+{smBX ܃׉\k =C@z9UmݧЦL" =抭\ygLHՕԀAx#:U((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((+Z|?zd4Y.:yqFٲ9)޶+?jeu;dz!yc"4|)-񿉭xĩҲͶht}ĉHG+J]U F6n&Mm"+*t#a4((69>$;m2?+ԝ]G;q:_ԞgXh6HTZY&1VbL\`AմMYŠCy=?Cx\縞~c} s|<%^I!1s˪*$ aːUEUǗ! @Tp@8mڣ!Cc'`}x'_Ɗv)Rʒ3Hr{0Gi('37#χ vhNgdR {*H>*σèjW҅Tn"={d8#EQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEU=B4"yȥ28eo?5rs7RᶾDvi? WHzڳw~RI@gXĜܖb 7{qt0]H]cLP$I V Ac5Mg'?_\PgEq~4?wWoƟ'ٌ3('' FGaרâحضI4q A ʈ$ 6i,?[Fߍ?O+i%`z^t~ Zt1[(P*. r 7K,:$/+ q;an,[?i,?[Fߍ?O+jG<ߏ_Ɗvi,?[Fߍ?O+hDY\򶍿?x8o/4iV_M?Cu42 \}^1;Ur%ƭXNQ2/b_[gcZk)om+72]cr-kk#V  /k^""xm6P[ K]i %ÄwSʤp#|ж'䃷6w9bI9===SPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPpengcheng.mind-test/docs/model_cards/000077500000000000000000000000001471400134000201155ustar00rootroot00000000000000pengcheng.mind-test/docs/model_cards/bert.md000066400000000000000000000131151471400134000213740ustar00rootroot00000000000000# BERT ## 模型描述 BERT:全名`Bidirectional Encoder Representations from Transformers`模型是谷歌在2018年基于Wiki数据集训练的Transformer模型。 [论文](https://arxiv.org/abs/1810.04805)J Devlin，et al., Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019 ## 预训练数据集下载 1. 从[zhwiki](https://dumps.wikimedia.org/zhwiki/)或[enwiki](https://dumps.wikimedia.org/enwiki/)中下载数据集。 2. 使用[WikiExtractor](https://github.com/attardi/wikiextractor)提取和整理数据集中的文本，执行命令如下： ```shell pip install wikiextractor python -m wikiextractor.WikiExtractor -o -b ``` ### 数据处理 #### TFRecord类型BERT预训练数据 用户可以参考[BERT](https://github.com/google-research/bert#pre-training-with-bert)代码仓中的create_pretraining_data.py文件， 进行`TFRecord`格式文件的生成， 如果出现下述报错 ```bash AttributeError: module 'tokenization' has no attribute 'FullTokenizer' ``` 请安装`bert-tensorflow`。注意，用户需要同时下载对应的`vocab.txt`文件。 ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) 示例命令如下，将会执行一个12层的BERT模型训练 ```shell python run_mindformer.py --config configs/bert/run_bert_base_uncased.yaml --run_mode train \ --device_target Ascend \ --train_dataset_dir /your_path/wiki_data ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import BertForPreTraining, BertConfig BertForPreTraining.show_support_list() # 输出： # - support list of BertForPreTraining is: # - ['bert_base_uncased'] # - ------------------------------------- # 模型标志加载模型 model = BertForPreTraining.from_pretrained("bert_base_uncased") #模型配置加载模型 config = BertConfig.from_pretrained("bert_base_uncased") # {'model_config': {'use_one_hot_embeddings': False, 'num_labels': 1, 'dropout_prob': 0.1, # 'batch_size': 128, seq_length: 128, vocab_size: 30522, embedding_size: 768, num_layers: 12, # num_heads: 12, expand_ratio: 4, hidden_act: "gelu", post_layernorm_residual: True, # hidden_dropout_prob: 0.1, attention_probs_dropout_prob: 0.1, max_position_embeddings: 512, # type_vocab_size: 2, initializer_range: 0.02, use_relative_positions: False, # use_past: False, checkpoint_name_or_path: "bert_base_uncased"}} model = BertForPreTraining(config) ``` - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task='fill_mask', model='bert_base_uncased', train_dataset='/your_path/wiki_data') trainer.train() # 开启预训练 ``` ### 多卡训练 - 单机8卡数据并行训练BERT-base模型 ```bash RANK_SIZE=$1 HOSTFILE=$2 mpirun --allow-run-as-root -n $RANK_SIZE --hostfile $HOSTFILE \ --output-filename run_distributed_train_bert \ -x NCCL_IB_HCA -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x NCCL_SOCKET_IFNAME -n $RANK_SIZE \ python run_mindformer.py --config ./configs/bert/run_bert_base_uncased.yaml --use_parallel True --run_mode train > distribute_train_gpu_log.txt 2>&1 & ``` 其中各个参数的含义： - RANK_SIZE：总共使用的卡的数量，采用单机8卡训练时，设为8 - HOSTFILE：一个文本文件，格式如下 ```text 10.1.2.3 slots=8 ``` 表示节点ip为10.1.2.3的服务器拥有8张设备卡。用户应该将自己的实际IP替换掉10.1.2.3。 日志会重定向到`distribute_train_gpu_log.txt`中。可以通过`tail -f distribute_train_gpu_log.txt`的 命令及时刷新日志。注意此时8张卡的日志都会输出到上述的文件中，造成重复输出。用户在如下的位置查看每卡的输出 ```bash tail -f run_distributed_train_bert/1/rank.0/stdout ``` 要完成上述训练只需输入命令 ```bash bash scripts/examples/masked_language_modeling/bert_pretrain_distributed_gpu.sh RANK_SIZE hostfile ``` 即可。 #### 计算Loss ```python from mindformers import BertForPreTraining, BertTokenizer from mindspore import Tensor import mindspore.common.dtype as mstype model = BertForPreTraining.from_pretrained('bert_base_uncased') tokenizer = BertTokenizer.from_pretrained('bert_base_uncased') data = tokenizer("Paris is the [MASK] of France.", max_length=128, padding="max_length") input_ids = Tensor([data['input_ids']], mstype.int32) attention_mask = Tensor([data['attention_mask']], mstype.int32) token_type_ids = Tensor([data['token_type_ids']], mstype.int32) masked_lm_positions = Tensor([[4]], mstype.int32) next_sentence_labels = Tensor([[1]], mstype.int32) masked_lm_weights = Tensor([[1]], mstype.int32) masked_lm_ids = Tensor([[3007]], mstype.int32) output = model(input_ids, attention_mask, token_type_ids, next_sentence_labels, masked_lm_positions, masked_lm_ids, masked_lm_weights) print(output) #[0.6706] ``` ## 模型权重 本仓库中的`bert_base_uncased`来自于HuggingFace的[`bert_base_uncased`](https://huggingface.co/bert-base-uncased), 基于下述的步骤获取： 1. 从上述的链接中下载`bert_base_uncased`的HuggingFace权重，文件名为`pytorch_model.bin` 2. 执行转换脚本，得到转换后的输出文件`mindspore_t5.ckpt` ```shell python mindformers/models/bert/convert_weight.py --layers 12 --torch_path pytorch_model.bin --mindspore_path ./mindspore_bert.ckpt ``` pengcheng.mind-test/docs/model_cards/blip2.md000066400000000000000000000413601471400134000214530ustar00rootroot00000000000000# BLIP2 ## 模型描述 BLIP-2: 全名`Bootstrapping Language-Image Pre-training - 2`模型是2023 年 Salesforce提出的一种多模态模型，它从现成的冻结预训练图像编码器 (ViT)和冻结的大型语言模型 (LLM)中引导视觉语言预训练 (contrastive_language_image_pretrain), 中间添加一个轻量级的 Querying Transformer 弥补了模态 gap, 该 Transformer 分两个阶段进行预训练: - 第一阶段从冻结图像编码器引导视觉-语言表示学习，强制 Q-Former 学习与文本最相关的视觉表示。 - 第二阶段基于冻结的语言模型引导从视觉到语言的生成学习，将Q-Former的输出连接到冻结的LLM，并对Q-Former进行训练，使其输出视觉表示能够被LLM解释。 [论文](https://arxiv.org/pdf/2301.12597.pdf) Junnan Li，et al., BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, 2023 ## 预训练数据集下载 1\. coco2014数据集: ```shell ├── annotations │   ├── coco_karpathy_test.json │   ├── coco_karpathy_train.json │   └── coco_karpathy_val.json │ └── images   ├── test2014   ├── test2015   ├── train2014   └── val2014 ``` 可通过LAVIS github库提供的链接下载annotation_files ([链接](https://github.com/salesforce/LAVIS/blob/main/lavis/configs/datasets/coco/defaults_cap.yaml)) 和 images ([链接](https://github.com/salesforce/LAVIS/blob/main/lavis/datasets/download_scripts/download_coco.py))。 2\. Visual Genome数据集: ```shell ├── annotations │   └── vg_caption.json └── images └── VG_100K ``` 同上，可通过LAVIS github库提供的链接下载annotation_files ([链接](https://github.com/salesforce/LAVIS/blob/main/lavis/configs/datasets/vg/defaults_caption.yaml)) 和 images ([链接](https://github.com/salesforce/LAVIS/blob/main/lavis/datasets/download_scripts/download_vg.py))。 ## 关键配置项说明 ```yaml train_dataset: text_transforms: type: CaptionTransform prompt: "" # 二阶段中语言模型添加的prompt，目前仅支持固定prompt max_length: 32 # token ids的最大长度 tokenizer: type: LlamaTokenizer max_length: 32 # token ids的最大长度 vocab_file: "" # 词表文件路径 model: model_config: type: Blip2Config max_txt_len: 32 # token ids的最大长度，需要与train_dataset.text_transforms.max_length保持一致 checkpoint_name_or_path: "" # 模型的预训练权重，在二阶段训练时，可在此处配置一阶段训练得到的权重 prompt: False # 二阶段训练中，是否使用prompt prompt_length: 0 # 二阶段训练中，使用的固定prompt的token长度，具体值根据tokenizer及train_dataset.text_transforms.prompt得到 # 以上两项配置与train_dataset.text_transforms.prompt需要对应 vision_config: # ImageEncoder的相关配置 type: ViTConfig image_size: 224 # 输入图像大小 checkpoint_name_or_path: "vit_g_p16" # vit的权重文件 qformer_config: type: QformerConfig query_length: 32 # query数目 text_config: # 二阶段语言模型相关配置 type: LlamaConfig seq_length: 160 # 语言模型的输入seq_length大小，在训练时该值要等于qformer的query数目+语言模型输入文字token id长度 # 即 seq_length=model.model_config.qformer_config.query_length + model.model_config.max_txt_len checkpoint_name_or_path: "" # 语言模型权重文件 arch: type: XXXX # 推理任务时需要配置 processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # 输入图像大小 tokenizer: type: LlamaTokenizer max_length: 128 vocab_file: "" # 词表文件路径 ``` ## BLIP-2一阶段 ### 快速使用 #### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](https://gitee.com/mindspore/transformer/blob/master/README.md#%E6%96%B9%E5%BC%8F%E4%B8%80clone-%E5%B7%A5%E7%A8%8B%E4%BB%A3%E7%A0%81) 示例命令如下，将会执行一个39层ViT和18层Attention Layer的Qformer的一阶段预训练: ```shell # pretrain python run_mindformer.py --config configs/blip2/run_blip2_pretrain_stage_1.yaml --run_mode train \ --device_target Ascend \ --train_dataset_dir [PATH_TO_DATASET] # evaluate python run_mindformer.py --config configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yaml --run_mode eval \ --device_target Ascend \ --eval_dataset_dir [PATH_TO_DATASET] # predict (单图片) python run_mindformer.py --config configs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml --run_mode predict \ --device_target Ascend \ --predict_data [PATH_TO_IMAGE] # predict (数据集) python run_mindformer.py --config configs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml --run_mode predict \ --device_target Ascend \ --eval_dataset_dir [PATH_TO_DATASET] ``` #### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import Blip2Qformer, Blip2Config Blip2Qformer.show_support_list() # 输出： # support list of Blip2Qformer is: # ['blip2_stage1_vit_g', 'blip2_stage1_classification'] # ------------------------------------- # 模型标志加载模型 model = Blip2Qformer.from_pretrained('blip2_stage1_vit_g') #模型配置加载模型 config = Blip2Config.from_pretrained("blip2_stage1_vit_g") config # {'img_size': 224, # 'num_query_token': 32, # 'cross_attention_freq': 2, # 'drop_path_rate': 0, # 'use_grad_checkpoint': False, # 'vit_model': 'vit_g_p16', # 'layernorm_dtype': 'float32', # 'softmax_dtype': 'float32', # 'model_type': 'blip2', # 'batch_size': 8, # 'embed_dim': 256, # 'vocab_size': 21129, # ... # 'is_training': True} model = Blip2Qformer(config) ``` - Trainer接口开启训练： ```python from mindformers.dataset.dataloader.multi_image_cap_dataloader import MultiImgCapDataLoader from mindformers.trainer import Trainer # 初始化图像-文本数据集 dataset_dir = "/data" annotation_files = [ "vg/annotations/vg_caption.json", "coco2014/coco/annotations/coco_karpathy_train.json" ] image_dirs = [ "vg/images", "coco2014/coco/images" ] train_dataset = MultiImgCapDataLoader(dataset_dir=dataset_dir, annotation_files=annotation_files, image_dirs = image_dirs, stage="train") # blip2一阶段初始化预训练任务 trainer = Trainer(task='contrastive_language_image_pretrain', model='blip2_stage1_vit_g', train_dataset=train_dataset) # 开启预训练 trainer.train() ``` - Trainer接口开启评估: ```python from mindformers.dataset.dataloader.multi_image_cap_dataloader import MultiImgCapDataLoader from mindformers.trainer import Trainer # 初始化图像-文本数据集 dataset_dir: "/data" annotation_files: [ "flickr30k/annotations/test.json" ] image_dirs: [ "flickr30k/images" ] eval_dataset = MultiImgCapDataLoader(dataset_dir=dataset_dir, annotation_files=annotation_files, image_dirs = image_dirs, stage="eval") # 初始化评估任务 trainer = Trainer(task='blip2_retireval', model='blip2_stage1_vit_g', eval_dataset=eval_dataset) # 开启评估 trainer.eval() ``` - Trainer接口开启推理: ```python from mindformers.tools.image_tools import load_image from mindformers import Trainer cls_trainer = Trainer(task='zero_shot_image_classification', model='blip2_stage1_classification', candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) # 加载输入，一张太阳花图片 input_data = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 加载指定的权重以完成推理 predict_result = cls_trainer.predict(input_data=input_data, predict_checkpoint='your_path_to/blip2_pretrained.ckpt') print(predict_result) # 输出 # output result is: [[{'score': 0.99999976, 'label': 'sunflower'}]] # output result is saved at: zero_shot_image_classification_result.txt # .........Predict Over!............. ``` - pipeline推理: ```python from mindformers.tools.image_tools import load_image from mindformers.pipeline import ZeroShotImageClassificationPipeline # 初始化pipeline classifier = ZeroShotImageClassificationPipeline( model='blip2_stage1_classification', candidate_labels=["sunflower", "tree", "dog", "cat", "toy"], hypothesis_template="This is a photo of {}." ) # 太阳花图片 img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 可批次处理，输入图片数组 classifier(img) # 输出 # [[{'score': 0.99999714, 'label': 'sunflower'}, # {'score': 1.315181e-06, 'label': 'tree'}, # {'score': 7.0368844e-07, 'label': 'toy'}, # {'score': 4.7594781e-07, 'label': 'dog'}, # {'score': 3.93686e-07, 'label': 'cat'}]] ``` #### 多卡训练 - 参考 [快速启动指导](https://gitee.com/mindspore/mindformers#%E6%96%B9%E5%BC%8F%E4%B8%80%E4%BD%BF%E7%94%A8%E5%B7%B2%E6%9C%89%E8%84%9A%E6%9C%AC%E5%90%AF%E5%8A%A8) 中的多卡启动部分，blip2的任务配置文件默认为单卡启动，如需多卡启动，注意将相关配置文件中的 ```yaml use_parallel: False ``` 改为 ```yaml use_parallel: True ``` blip2现支持数据并行 (data parallel) 和模型并行 (model parallel)，需要注意根据卡数和并行模式正确配置`data_parallel` 和`model_parallel`的值 (dp * mp = 卡数)。 ##### 评估一阶段Loss ```python from mindformers import Blip2Qformer, Blip2Processor from mindformers.tools.image_tools import load_image from mindspore import Tensor # 加载模型 model = Blip2Qformer.from_pretrained('blip2_stage1_vit_g') # 加载处理器 processor = Blip2Processor.from_pretrained('blip2_stage1_vit_g') image_processor = processor.image_processor tokenizer = processor.tokenizer # 加载输入，一张太阳花图片 image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") text = ["This is a picture of sunflower."] # 输入预处理 image_input = image_processor(image) text_ids = tokenizer(text, max_length=32, padding='max_length') input_ids = Tensor(text_ids['input_ids'], ms.int32) # 放入模型计算loss output = model(image_input, input_ids, return_tuple=True) print(output) # (Tensor(shape=[], dtype=Float32, value= 4.45203), -- overall_loss = loss_itc + loss_itm + loss_lm # Tensor(shape=[], dtype=Float32, value= 0), -- loss_itc # Tensor(shape=[], dtype=Float32, value= 0.681192), -- loss_itm # Tensor(shape=[], dtype=Float32, value= 3.77084)) -- loss_lm ``` ### 模型权重 本仓库中的`blip2_stage1_classification`来自于LAVIS的一阶段预训练权重[`blip2_stage1_pretrained`](https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/blip2_pretrained.pth), 基于下述的步骤获取： 1. 从上述的链接中下载`blip2_stage1_pretrained`的pytorch权重，文件名为`blip2_pretrained.pth` 2. 执行转换脚本，得到转换后的输出文件`blip2_stage1_pretrained.ckpt` ```shell python mindformers/models/blip2/convert_weight.py --torch_path blip2_pretrained.pth --mindspore_path ./blip2_stage1_pretrained.ckpt ``` ## BLIP-2二阶段 BLIP-2第二阶段是通过冻结的语言模型引导从视觉到语言的生成学习，将Q-Former的输出连接到冻结的LLM，并对Q-Former进行训练，使其输出视觉表示能够被LLM解释。 当前BLIP-2二阶段支持使用接入Llama-7b及baichuan-7b的推理任务，并暂未提供预训练模型，以下推理任务中，未加载准确的预训练权重，不保证生成结果正确。 ### 快速使用 #### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](https://gitee.com/mindspore/transformer/blob/master/README.md#%E6%96%B9%E5%BC%8F%E4%B8%80clone-%E5%B7%A5%E7%A8%8B%E4%BB%A3%E7%A0%81) 示例命令如下，将会执行一个39层ViT, 18层Attention Layer的Qformer以及Llama-7b的二阶段推理，**Note:** 当前二阶段仅支持推理任务，并暂不提供预训练模型: ```shell # predict (单图片) 当前二阶段未提供预训练模型，该预测代码仅展示使用方式，不保证预测结果 python run_mindformer.py --config configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml --run_mode predict \ --device_target Ascend \ --predict_data [PATH_TO_IMAGE] ``` **注：** 当二阶段使用baichuan_7b作为LLM模型时进行推理时，需要自行在配置文件`configs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yaml`中 配置对应的baichuan-7b权重文件及词表文件，如何配置可参考上文中关键配置项说明章节； 权重文件和词表文件的获取及转换参考[baichuan_7b](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan/baichuan.md). #### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import Blip2ImageToTextGeneration, Blip2Config Blip2ImageToTextGeneration.show_support_list() # 输出： # support list of Blip2ImageToTextGeneration is: # ['itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b'] # ------------------------------------- # 初始化模型方式1：直接通过预训练加载 model = Blip2ImageToTextGeneration.from_pretrained('itt_blip2_stage2_vit_g_baichuan_7b') # 初始化模型方式2：通过配置文件初始化模型 config = Blip2Config.from_pretrained("itt_blip2_stage2_vit_g_baichuan_7b") # 可通过config配置更改相应的配置 config.model.model_config.text_config.checkpoint_name_or_path = "/path/to/the/llm_checkpoint" # 修改语言模型权重路径 model = Blip2ImageToTextGeneration(config) ``` - Trainer接口开启推理: ```python from mindformers.tools.image_tools import load_image from mindformers import Blip2Config, Blip2ImageToTextGeneration, Trainer config = Blip2Config.from_pretrained("itt_blip2_stage2_vit_g_baichuan_7b") # 可通过config配置更改相应的配置 config.text_config.checkpoint_name_or_path = "/path/to/the/llm_checkpoint" # 修改语言模型权重路径 config.checkpoint_name_or_path = "/path/to/the/1-st_pretrain_checkpoint" # 修改语言模型权重路径 model = Blip2ImageToTextGeneration(config) generate_trainer = Trainer(task='image_to_text_generation', model=model, model_name='itt_blip2_stage2_vit_g_baichuan_7b') # 加载输入，一张太阳花图片 input_data = load_image( "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 加载指定的权重以完成推理 predict_result = generate_trainer.predict(input_data=input_data, hypothesis_template="###问题：\n详细描述图片内容。\n\n###答案：") print(predict_result) # 输出 # output result is: ['###问题：\n详细描述图片内容。\n\n###答案：这是一张图片，上面有一朵黄色的花，花朵周围有许多小黄花，它们在阳光下闪闪发光。'] # output result is saved at: zero_shot_image_to_text_generation_result.txt # .........Predict Over!............. ``` - pipeline推理: ```python from mindformers.tools.image_tools import load_image from mindformers import Blip2Config, Blip2ImageToTextGeneration, pipeline config = Blip2Config.from_pretrained("itt_blip2_stage2_vit_g_baichuan_7b") # 可通过config配置更改相应的配置 config.text_config.checkpoint_name_or_path = "/path/to/the/llm_checkpoint" # 修改语言模型权重路径 config.checkpoint_name_or_path = "/path/to/the/1-st_pretrain_checkpoint" # 修改语言模型权重路径 model = Blip2ImageToTextGeneration(config) # 初始化pipeline generator = pipeline("image_to_text_generation", model=model) # 太阳花图片 img = load_image( "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 可批次处理，输入图片数组 pipeline_result = generator(img, hypothesis_template='###问题：\n详细描述图片内容。\n\n###答案：') print(pipeline_result) # ['###问题：\n详细描述图片内容。\n\n###答案：这是一张图片，上面有一朵黄色的花，花朵周围有许多小黄花，它们在阳光下闪闪发光。'] ```pengcheng.mind-test/docs/model_cards/bloom.md000066400000000000000000000670441471400134000215620ustar00rootroot00000000000000# Bloom ## 1. 简介 ### 1.1 模型描述 Bloom (BigScience Large Open-science Open-access Multilingual) 是一个开源的开放权限的自回归大语言模型(LLM)，用于对用自然语言表达的多种下游任务进行文本生成。Bloom系列模型涵盖从560M到176B的多种规模，其中176B千亿级参数的大模型的预训练基于工业级的计算机集群，在46种语言和13种编程语言的文本生成中达到比拟人类写作的SOTA效果。对于训练数据集中没有显式包括的下游任务，Bloom也可以通过指令的方式，给出令人满意的zero-shot回答。 ### 1.2 仓库介绍 `Bloom` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/bloom` ```text bloom ├── __init__.py ├── bloom_tokenizer.py # tokenizer ├── bloom_config.py # 模型配置项 ├── bloom.py # 模型实现 └── layers.py # bloom 层定义 └── convert_weight.py # 将huggingface ckpt转成mindfomer ckpt ``` 2. 模型配置：`configs/bloom` ```text bloom ├── run_bloom_560m.yaml # 560m 用于推理 ├── run_bloom_7.1b.yaml # 7.1b 用于8卡训练 ├── run_bloom_65b.yaml # 65b 用于96卡训练 └── run_bloom_176b.yaml # 176b 用于128卡训练 ``` 其中Bloom_7.1b可在单机单卡上推理，在单机8卡上训练；Bloom-65B训练至少96卡；Bloom_176B训练至少128卡。 ### 1.3 环境要求 - 硬件：Ascend 910A - MindSpore：2.0.0 - MindFormers版本：dev --- ## 2. 前期准备 ### 2.1 数据集制作 这里以Alpaca为例，数据大概21MB,用于调试。 首先去官方下载[alpaca_data.json文件](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) 然后调用`mindformers/tools/dataset_preprocess/bloom/make_mindrecord.py`脚本将json转换成mindrecord文件。 ```bash python mindformers/tools/dataset_preprocess/bloom/make_mindrecord.py --input_dataset_file=XXX/alpaca_data.json --output_path=XXX --N=51200 ``` 其中`--N=51200`表示将json中的52002条数据中的前51200转换成mindrecord(推荐)，`--N=-1`将转换全部json中的数据. 在执行此脚本时，对于每个prompt如下操作将被执行： - 将问题和回答按照模板制作成prompt text; - 使用BloomTokenizer将prompt从text转成token ids; - 添加eos_token_id直到seq_length。 执行文本后，`--output_path`目录下将生成mindrecord文件。 ### 2.2 CheckPoint转换（选读） 作为参考，这里描述CheckPoint在HuggingFace和MindSpore间的转换，在不同分布式策略间的转换。Bloom_7.1bB的推理、预训练、finetune对这部分没有依赖，可以直接跳到下一章。 #### 2.2.1 在HuggingFace和MindSpore间的转换 Mindformers可以直接通过高级接口下载转换好的560M和7.1B两种规模的ckpt,无需手动转换。其中560m的原始Checkpoint来自于huggingface的[Bloomz-560m](https://huggingface.co/bigscience/bloomz-560m)；7.1B的原始Checkpoint来自于huggingface的[Bloomz-7B1-mt](https://huggingface.co/bigscience/bloomz-7b1-mt). higgingface到mindformers的CheckPoint转换由一下命令完成。 ```bash cd mindformers/models/bloom python convert_weight.py --n_head=xx --hidden_size=xx --torch_path=path_to_hf_bin_file_or_folder --mindspore_path=output_path ``` 其中`--n_head=xx --hidden_size=xx`根据模型定义，bloom_560m的分别为16/1024; bloom_7.1b的分别为32/4096. #### 2.2.2 在不同分布式策略间的转换 通常训练采用分布式训练，而推理采用单机单卡。所以训练后涉及ckpt从分布式策略到单机策略的切换。 策略文件会在每次训练前的图编译环节生成，保存在`mindformers/scripts/mf_parallel0/ckpt_strategy.ckpt`。在转换前，需要将每个机器上的每个rank的`ckpt_strategy.ckpt`拷贝到相同路径下， 如`./src_strategy_dir`或`./dst_strategy_dir`，再将他们合并。 ```python import mindspore as ms # src_strategy_dir/stra0.ckpt, src_strategy_dir/stra1.ckpt ... src_strategy_dir/stra127.ckpt ms.merge_pipeline_strategys("./src_strategy_dir", "./src_strategy.ckpt") # dst_strategy_dir/stra0.ckpt, dst_strategy_dir/stra1.ckpt ... dst_strategy_dir/stra127.ckpt ms.merge_pipeline_strategys("./dst_strategy_dir", "./dst_strategy.ckpt") ``` 然后将checkpoint从src_strategy转换成dst_strategy。如果src_strategy或dst_strategy策略为单机单卡而非分布式，则不需要提供策略文件。 ```python import mindspore as ms # 如果src或dst策略为单机单卡而非分布式，则不需要提供策略文件。 # src_checkpoints_dir/rank_0/xxx.ckpt, src_checkpoints_dir/rank_1/xxx.ckpt ... ms.transform_checkpoints( src_checkpoints_dir, dst_checkpoints_dir, ckpt_prefix = "new_", src_strategy_file = "./src_strategy_file.ckpt", dst_strategy_file = "./dst_strategy_file.ckpt") ``` --- ## 3. Bloom推理 这里以Bloom_560m和bloom_7.1b为例，介绍bloom推理。 ### 3.1 基于Pipeline推理 可以通过一下两种方法实列化Bloom pipeline：`pipeline`接口或AutoClass的`from_pretrain`接口。两者都会自动从云上下载预训练的模型，存储路径：`mindformers/checkpoint_download/bloom` - 调用pipeline接口: ```python from mindformers import pipeline bloom_ppl = pipeline(task='text_generation', model='bloom_560m', max_length=256) ``` - 调用AutoClass的from_pretrain接口： ```python from mindformers import AutoModel, AutoTokenizer, TextGenerationPipeline model = AutoModel.from_pretrained("bloom_560m") tokenizer = AutoTokenizer.from_pretrained("bloom_560m") bloom_ppl = TextGenerationPipeline(model, tokenizer, max_length=256) ``` 实例化`bloom_ppl`之后，可以将问题传入。 ``` python result = bloom_ppl([ "what color is the sky?", "Translate to English: Je t’aime." ]) print(result) # expect print result # {'text_generation_text': ['what color is the sky? blue']}, {'text_generation_text': ['Translate to English: Je t’aime. I love you.']}] ``` ### 3.2 基于API接口推理 推理也可以用更底层的MindSpore API. ```python import numpy as np import mindspore as ms from mindformers import AutoTokenizer from mindformers.models.bloom import BloomConfig, BloomLMHeadModel ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=0) # ############################## # # bloom_560m config # CKPT_FILE="bloom_560m" # SEQ_LENGTH = 256 # config = BloomConfig( # param_init_type="float16", # embedding_init_type="float16", # checkpoint_name_or_path=CKPT_FILE, # max_decode_length=SEQ_LENGTH, # seq_length=SEQ_LENGTH, # hidden_size=1024, # num_layers=24, # num_heads=16, # hidden_dropout_rate=0.0, # attention_dropout_rate=0.0, # batch_size = 1, # use_past = True # # ) # ############################## # 7B CKPT_FILE = "bloom_7.1b" # CKPT_FILE also takes absolute path to ckpt file, e.g. # "/home/xxx/mindformers/checkpoint_download/bloom/bloom_7.1b.ckpt" SEQ_LENGTH = 256 config = BloomConfig( param_init_type="float16", embedding_init_type="float16", checkpoint_name_or_path=CKPT_FILE, max_decode_length=SEQ_LENGTH, seq_length=SEQ_LENGTH, hidden_size=4096, num_layers=30, num_heads=32, hidden_dropout_rate=0.0, attention_dropout_rate=0.0, batch_size = 1, use_past = True ) def chat(): tokenizer = AutoTokenizer.from_pretrained("bloom_560m") model = BloomLMHeadModel(config) model.set_train(False) question_list = [ "what color is the sky?", "Translate to English: Je t’aime.", ] while True: if question_list: question = question_list.pop(0) else: question = input("please input your question: ") inputs = tokenizer.encode(question) inputs = np.array([inputs]).astype(np.int32) # add batch dim outputs = model.generate(inputs, max_length=None, do_sample=False, eos_token_id=2) outputs = outputs[0] # remove batch dim print(tokenizer.decode(outputs)) if __name__ == "__main__": chat() ``` - Bloom_560m的预期输出为: - what color is the sky?_**blue**_ - Translate to English: Je t’aime. _**I love you.**_ - Bloom_7.1B的预期输出为: - what color is the sky?_**blue**_ - Translate to English: Je t’aime. _**I love you.**_ --- ## 4. 单机8卡训练或微调 我们以单机8卡训练/微调Bloom_7.1b为例。默认配置为 | | SEQ | DP | MP | PP | uB | num_uB | GBS | |------------|:----:|:--:|:--:|:--:|:--:|:------:|:---:| | Bloom_7.1B | 2048 | 1 | 4 | 2 | 4 | 2 | 8 | 在YAML配置文件中，具体设置变量如下 - __DP__: 数据并行维度, `parallel_config.data_parallel = 1`; - __MP__: 模型并行维度, `parallel_config.model_parallel = 4`; - __PP__: 流水线并行维度, `parallel_config.pipeline_stage = 2`; - __uB__: 每个microBatch大小, `runner_config.batch_size = 4`; - __num_uB__: microBatch的数量, `parallel_config.micro_batch_num = 2`, 常稳训练时建议用16; - __GBS__: Global Batch Size = num_uB x uB x DP. ### 4.1 生成HCCL文件 运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； ```shell # step1：机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" ``` 若使用ModelArts的notebook环境，可从 `/user/config/jobstart_hccl.json` 路径下直接获取rank table，无需手动生成。 RANK_TABLE_FILE 单机8卡参考样例: ```json { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 4.2 启动预训练或微调 通过`/configs/bloom/run_bloom_7.1b.yaml`中的`model:checkpoint_name_or_path:`字段来控制是否加载CKPT. 请安如下方式区分预训练或者微调： | | pretrain | finetune | |--------------------------------------------: |:----------:|:----------:| | `load_checkpoint` | "" | "xxx/bloom_7.1b.ckpt'"| | `train_dataset:data_loader:dataset_dir` | PRETRAIN_DATASET | FINETUNE_DATASET | | `parallel_config.micro_batch_num` | 16 | 16 | | `runner_config.epochs` | 1 | 3 | | `lr_schedule.learning_rate` | 0.00001 | 0.000001 | | `lr_schedule.lr_end` | 0.000001 | 0.000001 | | `lr_schedule.warmup_steps` | 1000 | 0 | | `lr_schedule.total_steps` | -1 | -1 | | `callbacks.save_checkpoint_steps` | 1000 | 400 | | 初始loss | 12.xx | 3.xx | 其中`PRETRAIN_DATASET`和`FINETUNE_DATASET`都可以用alpaca_2049调试。 ```bash cd scripts # pretrain bash run_distribute.sh RANK_TABLE_FILE ../configs/bloom/run_bloom_7.1b.yaml [0,8] train 8 # finetune bash run_distribute.sh RANK_TABLE_FILE ../configs/bloom/run_bloom_7.1b.yaml [0,8] finetune 8 ``` 其中RANK_TABLE_FILE为上一步生成的rank table文件。执行后，相关日志输出在`mindformers/output/log/`训练过程中保存的ckpt存在`mindformers/output/checkpoint`目录下。可以通过`tail -f ../output/log/rank_7/mindformers.log`来查看当前的训练情况。参考收敛曲线如下图所示。 >默认情况下，图编译大约1.5小时，51200条alpaca数据集微调大约4小时/epoch. ![输入图片说明](https://foruda.gitee.com/images/1689316828651589261/75882b0b_11484396.png) ### 4.3 微调后对话效果 在`mindformers/scripts`路径下执行以下脚本`combine_ckpt.py`.这个脚本会 - 对strategy进行合并 - 清理微调ckpt文件中的优化器状态 - 合并微调ckpt文件用于单机推理 ```python # combine_ckpt.py import os import mindspore as ms CKPT_SUFFIX = "300_8" # 300(sink number) * 8 (sink size) = 2400 step CLEANED_CKPT_DIR = "../output/checkpoint_cleaned" COMBINED_CKPT_DIR = "../output/checkpoint_combined" COMBINED_STGY = "../output/strategy/ckpt_strategy.ckpt" # combine straegies ms.merge_pipeline_strategys("../output/strategy", COMBINED_STGY) # clean ckpt by removing optimizer states for rank_id in range(8): input_file_name = f"../output/checkpoint/rank_{rank_id}/mindformers_rank_{rank_id}-{CKPT_SUFFIX}.ckpt" params = ms.load_checkpoint(input_file_name) new_params = [{"name": key, "data": val} for key, val in params.items() if not ("accu_grads" in key or "adam_" in key) ] save_path = os.path.join(CLEANED_CKPT_DIR, f"rank_{rank_id}") os.makedirs(save_path, exist_ok=True) ms.save_checkpoint(new_params, f"{save_path}/cleaned.ckpt") print(f"saved {save_path}") # combine ckpt ms.transform_checkpoints(CLEANED_CKPT_DIR, COMBINED_CKPT_DIR, ckpt_prefix = "combined_", src_strategy_file = COMBINED_STGY) ``` 然后执行以下脚本进行新的对话。相比与3.2章的脚本，这里有三个改动： - `CKPT_FILE`改成新生成的ckpt文件 - 对问题使用了Alpaca数据集相同prompt模板 - 模型embedding_init_type改成FP32因为SFT用的是FP32 > 以下脚本针对Alpaca数据集的prompt模板。如果使用其他数据集微调，请更换对应模板。 ```python import numpy as np import mindspore as ms from mindformers import AutoTokenizer from mindformers.models.bloom import BloomConfig, BloomLMHeadModel ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=0) alpaca_prompt = ( "Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Response:\n") # 7B CKPT_FILE = "xxx/mindformers/output/checkpoint_combined/rank_0/combined_0.ckpt" SEQ_LENGTH = 1024 config = BloomConfig( param_init_type="float16", embedding_init_type="float32", checkpoint_name_or_path=CKPT_FILE, max_decode_length=SEQ_LENGTH, seq_length=SEQ_LENGTH, hidden_size=4096, num_layers=30, num_heads=32, hidden_dropout_rate=0.0, attention_dropout_rate=0.0, batch_size = 1, use_past = True ) def chat(): tokenizer = AutoTokenizer.from_pretrained("bloom_560m") model = BloomLMHeadModel(config) model.set_train(False) question_list = [ "why the earth is unique?", "why the sky is blue?", "write a job application for a data scientist and explain your related work experience." ] while True: if question_list: question = question_list.pop(0) else: question = input("please input your question: ") question = alpaca_prompt.format_map({"instruction":question}) inputs = tokenizer.encode(question) inputs = np.array([inputs]).astype(np.int32) # add batch dim outputs = model.generate(inputs, max_length=None, do_sample=False, eos_token_id=2) outputs = outputs[0] # remove batch dim print(tokenizer.decode(outputs)) if __name__ == "__main__": chat() ``` 预期的对话效果大致为下表所示 | | Before | After | |:--------------------------------------------------------------------------------------:|:-----------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | why the sky is blue? | light from the sun is scattered | The sky is blue because of the presence of water droplets in the atmosphere. These droplets reflect light back to the sky, causing the sky to appear blue. | | what would be the best way to travel from San Fransisco to New York? | take a flight | The best way to travel from San Francisco to New York is by taking the flight. The flight is the fastest and most convenient way to travel from San Francisco to New York. | | write a job application for a data scientist and explain your related work experience. | | Dear Employer, I am writing to apply for the position of Data Scientist. I have over 5 years of experience in data science and machine learning, and I am excited to join your team. I have experience in supervised and unsupervised machine learning algorithms, data visualization, and data cleaning. I am also proficient in Python, R, and SQL. I am looking forward to discussing my qualifications further and hearing from you soon. Sincerely, [Your Name] | | why the earth is unique? | it is the only planet with a liquid surface | The Earth is unique because it is the only planet with a liquid surface, a magnetic field, and a protective atmosphere. It is also the only planet with a life-supporting atmosphere and a diverse and abundant life-supporting ecosystem. | ## 5. 多机多卡的训练 这里以12机96卡训练65B为例。默认配置为 | | SEQ | DP | MP | PP | uB | num_uB | GBS | |------------|:----:|:--:|:--:|:--:|:--:|:------:|:---:| | Bloom_7.1B | 2048 | 2 | 4 | 12 | 1 | 48 | 96 | 在YAML配置文件中，具体设置变量如下 - __DP__: 数据并行维度, `parallel_config.data_parallel = 2`; - __MP__: 模型并行维度, `parallel_config.model_parallel = 4`; - __PP__: 流水线并行维度, `parallel_config.pipeline_stage = 12`; - __uB__: 每个microBatch大小, `runner_config.batch_size = 1`; - __num_uB__: microBatch的数量, `parallel_config.micro_batch_num = 48`, 常稳训练时建议用80; - __GBS__: Global Batch Size=num_uB x uB x DP. ### 5.1 生成HCCL文件 - step1: 参考单机多卡启动方式，在每台机器上运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； - step2: 将每台机器上生成的RANK_TABLE_FILE拷贝到一起，执行merge_hccl.py脚本将不同机器上生成的RANK_TABLE_FILE文件中的hccl*.json进行合并，包括server_list合并，server_count设为机器数，rank_id顺序增加，生成合并后的RANK_TABLE_FILE文件， - step3: 拷贝到所有机器中，保证不同机器上的RANK_TABLE_FILE相同； ```shell # step1：在每个机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" # step2：运行如下命令，合并每个机器上的RANK_TABLE_FILE文件。 python ./mindformers/tools/merge_hccl.py hccl*.json # step3：将step2得到的合并后的RANK_TABLE_FILE文件分别复制到所有的机器上。 ``` ### 5.2 启动预训练或微调 在每台机器上启动`bash run_distribute.sh`。 ```bash # launch ranks in the 0th server cd scripts bash run_distribute.sh $RANK_TABLE_FILE ../configs/bloom/run_bloom_65b.yaml [0,8] train 96 # launch ranks in the 1-11 server via ssh for idx in {1..11} do let rank_start=8*$idx let rank_end=$rank_start+8 ssh ${IP_LIST[$idx]} "cd scripts; bash run_distribute.sh $RANK_TABLE_FILE ../configs/bloom/run_bloom_65b.yaml [$rank_start,$rank_end] train 96" done ``` 其中 - `RANK_TABLE_FILE`为上一步汇总并分发的总rank table文件； - `IP_LIST`为12台服务器的IP地址。如192.168.0.[0-11] ```bash IP_LIST=("192.168.0.0", "192.168.0.1", ..., "192.168.0.11") ``` --- ## 附录A 更多推理技巧 ### A.1 模型并行推理（以1机8卡推理Bloom_7.1B为例） 这里我们以1机器8卡推理bloom_7.1B为例。涉及两个文件`chat.py`和`run_chat.py`。 ```text /SOME/PATH/ ├── chat.py # 负责定义一个并行进程 └── run_chat.py # 负责多次执行chat.py并拉起分布式 ``` 加载ckpt有两种方式, 由`run_chat.py` 中的`DISTRIBUTED_CKPT_PATH`变量来控制。这里默认使用`DISTRIBUTED_CKPT_PATH=""`代表的`Load then Shard`的方式加载ckpt. | 分布式加载ckpt的方法 | Load then Shard | Shard then Load | |-----------------------:|:-----------------|:-----------------| | DISTRIBUTED_CKPT_PATH= | "" | "/path/to/distributed/ckpt/" | |说明| 先加载全量ckpt，然后切分模型。|先切分模型，然后加载分布式ckpt。| || 不用预先按照策略切分ckpt，推理时可以灵活调整策略。|需要确定推理的分布式策略，并按照策略预先切分ckpt。 | || 对host内存的占用较高。| 对host内存的占用较低。| |适用| 适用于较小模型，如`560m`，`7.1b`。|适用于较大模型，如`65b`, `176b`。 | ``` python # >>> `chat.py`文件 import os import time import numpy as np import mindspore as ms from mindspore.train import Model from mindspore import load_checkpoint, load_param_into_net from mindspore.parallel import set_algo_parameters from mindspore.parallel._cost_model_context import _set_multi_subgraphs from mindformers import pipeline from mindformers import BloomLMHeadModel, BloomConfig, AutoTokenizer from mindformers import init_context from mindformers.modules import TransformerOpParallelConfig from mindformers.trainer.utils import get_last_checkpoint from mindformers.tools import logger SEQ_LENGTH = 256 DISTRIBUTED_CKPT_PATH = os.getenv("DISTRIBUTED_CKPT_PATH", "") # set context context_config = {"device_target": "Ascend", "mode": 0, "max_device_memory": "31GB"} parallel_context_config = {"parallel_mode": 1, "gradients_mean": False, "full_batch": True} rank_id, device_num = init_context(use_parallel=True, context_config=context_config, parallel_config=parallel_context_config) set_algo_parameters(elementwise_op_strategy_follow=True, fully_use_devices=True) _set_multi_subgraphs() # config blooom 7.1b config = BloomConfig( embedding_init_type="float32" if DISTRIBUTED_CKPT_PATH else "float16", checkpoint_name_or_path="" if DISTRIBUTED_CKPT_PATH else "bloom_7.1b", seq_length=SEQ_LENGTH, hidden_size=4096, num_layers=30, num_heads=32, hidden_dropout_rate=0.0, attention_dropout_rate=0.0, top_k=1, top_p=1, do_sample=True, parallel_config=TransformerOpParallelConfig( data_parallel=1, model_parallel=8, pipeline_stage=1 ) ) def chat(): # init bloom tokenizer = AutoTokenizer.from_pretrained("bloom_560m") bloom = BloomLMHeadModel(config) bloom.set_train(False) if DISTRIBUTED_CKPT_PATH: # find the sharded ckpt path for this rank ckpt_path = os.path.join(DISTRIBUTED_CKPT_PATH, "rank_{}".format(rank_id)) ckpt_path = get_last_checkpoint(ckpt_path) logger.info("ckpt path: %s", str(ckpt_path)) # shard bloom and load sharded ckpt m = Model(bloom) m.infer_predict_layout(ms.Tensor(np.ones(shape=(1, SEQ_LENGTH)), ms.int32)) checkpoint_dict = load_checkpoint(ckpt_path) not_load_network_params = load_param_into_net(bloom, checkpoint_dict) logger.info("Network parameters are not loaded: %s", str(not_load_network_params)) question_list = [ "请问为什么说地球是独一无二的？", "Translate to English: Je t’aime.", ] for question in question_list: inputs = tokenizer.encode(question) inputs = np.array([inputs]).astype(np.int32) # add batch dim outputs = bloom.generate(inputs, max_length=None, do_sample=False, eos_token_id=2) outputs = outputs[0] # remove batch dim print(tokenizer.decode(outputs)) if __name__ == "__main__": chat() ``` ```bash # >>> `run_chat.sh`文件 # define variable export RANK_SIZE=8 export RANK_TABLE_FILE="../hccl_8p.json" # <<< change to yours # distributed ckpt path to load after sharding model. # use "" if load full ckpt before sharding model. export DISTRIBUTED_CKPT_PATH="" export START_RANK=0 # this server start rank export END_RANK=8 # this server end rank # run for((i=${START_RANK}; i<${END_RANK}; i++)) do export RANK_ID=$i export DEVICE_ID=$((i-START_RANK)) echo "Start distribute running for rank $RANK_ID, device $DEVICE_ID" python3 ./chat.py &> minformers_$RANK_ID.log & done ``` 使用一下命令拉起分布式推理: ```bash bash run_chat.sh ``` 日志可以通过`tail -f mindformers_0.log`查看。预期结果与单机单卡`bloom_7.1b`推理相同： - 请问为什么说地球是独一无二的？ _**地球是太阳系中唯一有生物的地方**_ - Translate to English: Je t’aime. _**I love you.**_ ## 附录B BELLE [BELLE](https://github.com/LianjiaTech/BELLE)（Be Everyone's Large Language model Engine）是一个旨在促进中文对话大模型开源社区发展的组织。BELLE-7B是基于Bloomz-7B-mt，使用中文问答数据集微调出来开源的中文对话模型。根据微调所使用的中文数据大小分为0.2M, 0.6M, 1M, 2M四个权重。 微调的模板为 > Human: {input} \n\nAssistant:{output} 原始的开源BELLE的数据集和权重可以通过以下链接获得 | | 文件 | 链接 | |----------| --- |--------------------------------------------------------| | 2M SFT数据集 | train_2M_CN.json | https://huggingface.co/BelleGroup/BELLE-7B-2M | | 2M 模型权重 | pytorch_model.bin | https://huggingface.co/datasets/BelleGroup/train_2M_CN | 数据集和权重的转换参考2.1章和2.2章，命令如下： ```bash # 数据集转换 python mindformers/tools/dataset_preprocess/bloom/make_mindrecord.py --input_dataset_file=XXX/train_2M_CN.json --output_path=XXX # 权重转换 cd mindformers/models/bloom python convert_weight.py --n_head=32 --hidden_size=4096 --torch_path=xxx/pytorch_model.bin --mindspore_path=output_path ``` 数据集和权重的转换到mindspore后，可以按照Bloom的方式进行推理和微调。 pengcheng.mind-test/docs/model_cards/clip.md000066400000000000000000000163321471400134000213730ustar00rootroot00000000000000# CLIP ## 模型描述 CLIP (Contrastive Lanuguage-Image Pre-Training)：是一种基于图文对进行训练的transformer模型，在预训练完成以后，任意给定一张图片，它可以在不用微调的情况下，完成对图片的零样本分类。 [论文](https://arxiv.org/abs/2103.00020) Alec Radford, Jong Wook Kim, et al., Learning Transferable Visual Models From Natural Language Supervision, 2021. 注：CLIP训练代码未开源，故MindFormers提供训练pretrain、finetune功能，但不不保证精度，目前仅对zero shot图片分类精度做了对齐。 ## 数据集准备 ### 预训练使用数据集：Flickr8k([链接](https://pan.baidu.com/s/1LRlQUL1MRipPL4MLOdExzg)，密码: s4be) - 数据集大小：2.2G，共8000张彩色图像，每张图像都与五个不同的标题配对，这些标题提供了对图片中物体和事件的内容描述 - 训练集：6000张图像 - 验证集：1000张图像 - 测试集：1000张图像 - 数据格式：RGB ```bash 数据集目录格式 └─Flickr8k ├─Flickr8k_Dataset | └─Flickr8k_Dataset └─Flickr8k_text ├─Flickr8k.devImages.txt ├─Flickr8k.testImages.txt ├─Flickr8k.trainImages.txt └─Flickr8k.token.txt ``` ### 零样本下游任务使用的数据集：[Cifar100](http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz) - 数据集大小：161M，共60000张图片，100个类别 - 训练集：50000张图片 - 测试集：10000张图片 - 数据格式：二进制文件 ```bash 数据集目录格式 └─cifar-100-python ├─meta ├─test └─train ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 当前clip多卡精度有异常，仅支持单卡，后续版本会修复 ```shell # pretrain python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml --run_mode train --train_dataset_dir [DATASET_PATH] # evaluate python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml --run_mode eval --eval_dataset_dir [DATASET_PATH] # predict python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml --run_mode predict --predict_data [PATH_TO_IMAGE] ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import CLIPModel, CLIPConfig CLIPModel.show_support_list() # 输出： # - support list of CLIPModel is: # - ['clip_vit_b_32', 'clip_vit_B_16', 'clip_vit_l_14', 'clip_vit_l_14@336'] # - ------------------------------------- # 模型标志加载模型 model = CLIPModel.from_pretrained("clip_vit_b_32") #模型配置加载模型 config = CLIPConfig.from_pretrained("clip_vit_b_32") # {'text_config': {'hidden_size': 512, 'vocab_size': 49408, 'max_position_embeddings': 77, # 'num_hidden_layers': 12}, 'vision_config': {'hidden_size': 768, 'image_size': 224, 'patch_size': 32, # 'num_hidden_layers': 12}, 'projection_dim': 512, 'ratio': 64, 'checkpoint_name_or_path': 'clip_vit_b_32', # 'dtype': 'float16'} model = CLIPModel(config) ``` - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 初始化预训练任务 trainer = Trainer(task='contrastive_language_image_pretrain', model='clip_vit_b_32', train_dataset='./Flickr8k') trainer.train() # 开启预训练 #初始化零样本图像分类下游任务 trainer = Trainer(task='zero_shot_image_classification', model='clip_vit_b_32', eval_dataset='./cifar-100-python') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1: 使用训练好的权重进行评估和推理 trainer.evaluate(eval_checkpoint=True) predict_result = trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式2: 从obs下载训练好的权重并进行评估和推理 trainer.evaluate() #下载权重进行评估 predict_result = trainer.predict(input_data=img, top_k=3) #下载权重进行推理 print(predict_result) ``` - pipeline接口开启快速推理 ```python from mindformers import pipeline from mindformers.tools.image_tools import load_image classifier = pipeline("zero_shot_image_classification", model="clip_vit_b_32", candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") classifier(img) # 输出 # [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, # {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, # {'score': 2.396818e-06, 'label': 'cat'}]] ``` ## 模型性能 | model | task_type | model_Type | datasets | Top1-accuracy | log | example | |:-----:|:------------------------------:|:-----------------------------------------------------------------------------:|:--------:|:---------------------------------------:|:---:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | clip | pretrained | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | flickr8k | \ | \ | pretrain [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/contrastive_language_image_pretrain/clip_vit_b_32_pretrain_on_flickr8k.sh) | \| | clip | zero_shot_image_classification | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | cifar100 | 57.24%
61.41%
69.67%
68.19% | \ | eval [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/zero_shot_image_classification/clip_vit_b_32_eval_on_cifar100.sh)
predict [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/zero_shot_image_classification/clip_vit_b_32_predict_on_cifar100.sh) | ## 模型权重 本仓库中的`clip_vit_b_32`来自于openai/clip的[`ViT-B/32`](https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt), 基于下述的步骤获取： 1. 从上述的链接中下载`ViT-B/32`的模型权重 2. 执行转换脚本，得到转换后的输出文件`clip_vit_b_32.ckpt` 其余参数获取方式相同 ```shell python mindformers/models/clip/convert_weight.py --torch_path "PATH OF ViT-B/32.pt" --mindspore_path "SAVE PATH OF clip_vit_b_32.ckpt" ```pengcheng.mind-test/docs/model_cards/glm.md000066400000000000000000001033341471400134000212220ustar00rootroot00000000000000# ChatGLM6B ## 模型描述 ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 [General Language Model (GLM)](https://github.com/THUDM/GLM) 架构，具有 62 亿参数。ChatGLM-6B 使用了和 ChatGPT 相似的技术，针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练，辅以监督微调、反馈自助、人类反馈强化学习等技术的加持，62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答，更多信息请参考清华的[博客](https://chatglm.cn/blog)。在此仓中，提供ChatGLM6B的推理和微调能力。 ## 仓库介绍 `chatGLM6B` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/glm` ```bash glm ├── __init__.py ├── attention.py # 自注意力 ├── chatglm_6b_tokenizer.py # tokenizer ├── glm_config.py # 模型配置项 ├── glm.py # 模型实现 └── layers.py # glm 层定义 ``` 2. 模型配置：`configs/glm` ```bash glm ├── run_glm_6b_fintune.yaml # 全量微调启动配置 ├── run_glm_6b_lora.yaml # lora低参微调启动配置 ├── run_glm_6b_infer.yaml # 推理启动配置 └── run_glm_6b_lora_infer.yaml # lora模型推理启动配置 ``` ## 环境要求 - 硬件：Ascend 910A - MindSpore：2.0.0rc1 / 1.10.1 - MindFormers版本：dev 推理可在单机单卡上完成部署 全量微调训练需要最少单机8卡，Lora微调训练最少需要1卡 ## ChatGLM6B推理 > 需开发者提前pip安装。具体接口说明请参[API接口](https://gitee.com/mindspore/transformer/wikis/API/) ### AutoClass推理 可以使用AutoClass接口，通过模型名称获取相应的模型/tokenizer实例，并自动下载并加载权重 `from_pretrained()` 接口会自动从云上下载预训练的模型，存储路径：`mindformers/checkpoint_download/glm` 首次运行pipeline推理时需要进行模型编译，需等待一段时间 ```python >>> from mindformers import AutoModel, AutoTokenizer, TextGenerationPipeline >>> model = AutoModel.from_pretrained("glm_6b_chat") >>> tokenizer = AutoTokenizer.from_pretrained("glm_6b") >>> pipeline = TextGenerationPipeline(model, tokenizer, max_length=2048) >>> pipeline("你好") [{'text_generation_text': ['你好 你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。']}] ``` > 注：`AutoModel.from_pretrained()` 接口当前支持 `glm_6b` 和 `glm_6b_chat` 两类模型，前者为通用模型，后者具备推理加速特性，仅用于推理，两者共享权重，在推理场景下建议使用后者，以获得更快的推理体验 ### pipeline推理 也可以不实例化构造模型，直接通过指定任务模型与模型名的方式进行pipeline的构造 pipeline中，也可以使用 `glm_6b_chat` 模型加速推理 ```python >>> from mindformers import pipeline >>> task_pipeline = pipeline(task='text_generation', model='glm_6b_chat', max_length=2048) >>> task_pipeline('你好') [{'text_generation_text': ['你好 你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。']}] ``` ### 基于API接口的推理 可使用如下`chat_glm.py`脚本： ```python import time import mindspore as ms import numpy as np from mindformers.models.glm import GLMConfig, GLMChatModel from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer from mindformers.models.glm.glm_processor import process_response config = GLMConfig( position_encoding_2d=True, use_past=True, is_sample_acceleration=True, ) def chat_glm(): ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=7) model = GLMChatModel(config) ms.load_checkpoint("./checkpoint_download/glm/glm_6b.ckpt", model) tokenizer = ChatGLMTokenizer('./checkpoint_download/glm/ice_text.model') prompts = ["你好", "请介绍一下华为", "用python写一个快排"] history = [] for query in prompts: input_ids = tokenizer(query)['input_ids'] start_time = time.time() outputs = model.generate(input_ids, max_length=config.max_decode_length, do_sample=False) end_time = time.time() print(f'generate speed: {outputs[0].shape[0]/(end_time-start_time):.2f} tokens/s') response = tokenizer.decode(outputs) response = process_response(response[0]) print(response) if __name__ == "__main__": chat_glm() ``` ## 微调 下面以 [ADGEN](https://aclanthology.org/D19-1321.pdf) (广告生成) 数据集为例介绍代码的使用方法 ### 数据处理（在线加载与离线生成二选一，优先推荐在线加载方式） ADGEN 数据集任务为根据输入（content）生成一段广告词（summary）。数据集可选离线生成 `Mindrecord` 或者实时生成两种方式，两种方式选其一即可。 ```json { "content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳", "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点，穿在身上有着很好的显瘦效果。领口装饰了一个可爱的抽绳，漂亮的绳结展现出了十足的个性，配合时尚的泡泡袖型，尽显女性甜美可爱的气息。" } ``` 从 [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) 或者 [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) 下载处理好的 ADGEN 数据集，将解压后的 `AdvertiseGen` 任意目录下 #### 1. 在线加载 将任务配置文件 `configs/glm/run_glm_6b_*.yaml` 中的 `==== dataset config ====` 部分中的 `dataset_dir` 指向 `*.json` 文件，`vocab_file` 指向词表文件，**跳过** “2. 离线生成” 步骤。 #### 2. 离线生成 将任务配置文件 `configs/glm/run_glm_6b_*.yaml` 中的 `==== dataset config ====` 部分替换成： ```yaml train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset ``` 使用 `mindformers/tools/dataset_preprocess/glm/adgen_dataset.py` 脚本将数据集处理成mindrecord格式。 执行命令生成训练数据集： ```bash python adgen_dataset.py \ --input_file /path/to/AdvertiseGen/train.json \ --vocab_file /path/to/ice_text.model\ --output_file /path/to/AdvertiseGen/train_0604_128.mindrecord \ --max_source_length 64 \ --max_target_length 64 \ --mode train ``` 执行命令生成评估数据集： ```bash python adgen_dataset.py \ --input_file /path/to/AdvertiseGen/dev.json \ --vocab_file /path/to/ice_text.model \ --output_file /path/to/AdvertiseGen/eval_0604_256.mindrecord \ --max_source_length 256 \ --max_target_length 256 \ --mode eval ``` ### 生成HCCL文件 运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； ```shell # step1：机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" ``` > 注：若使用ModelArts的notebook环境，可从 `/user/config/jobstart_hccl.json` 路径下直接获取rank table，无需手动生成 RANK_TABLE_FILE 单机8卡参考样例: ```json { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 全参微调 #### run_mindformers脚本启动全参微调 全参微调使用 `configs/glm/run_glm_6b_finetune.yaml` 配置文件，配置文件中定义了微调所需的各配置项 修改数据集/模型权重配置路径： - 数据集：修改 `mindformers/configs/glm/run_glm_6b_finetune.yaml` 脚本中`train_dataset` 的 `dataset_dir` 为前文生成的数据集路径。 - 加载预训练模型权重：修改 `mindformers/configs/glm/run_glm_6b_finetune.yaml` 脚本中的 `load_checkpoint` 为预训练模型权重路径。 启动全参微调脚本： ```shell cd scripts # Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] bash run_distribute.sh /path/to/hccl_8p_01234567_127.0.1.1.json ../configs/glm/run_glm_6b_finetune.yaml '[0,8]' finetune # 将此处rank_table_file替换为实际路径 ``` 参数说明 ```text RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的glm/run_glm_6b.yaml配置文件 DEVICE_RANGE: 为单机分布式卡的范围，如 '[0,8]' 为8卡分布式，不包含8本身 RUN_STATUS: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` > 注：由于GLM6B的模型较大，无法在单卡上运行，此处仅提供分布式启动脚本 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint #### Trainer高阶接口启动全参微调 下面提供一个使用高阶接口进行GLM模型开发的样例脚本 `task.py`，用户可参照以下步骤熟悉如何使用高阶接口进行GLM模型的训练开发 ```python import argparse from mindformers import Trainer, TrainingArguments from mindformers import init_context, ContextConfig, ParallelContextConfig def context_init(use_parallel=False, optimizer_parallel=False): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, enable_parallel_optimizer=optimizer_parallel, full_batch=True) rank_id, device_num = init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) def main(use_parallel=False, run_mode='train', task='text_generation', model_type='glm_6b', checkpoint_path='./glm_6b.ckpt', train_dataset='./train', eval_dataset='./eval', predict_data='你好', batch_size=4, dp=1, mp=1, pp=1, micro_size=1, op=False): if use_parallel.lower() == "true": use_parallel = True else: use_parallel = False # 环境初始化 context_init(use_parallel, op) # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=1, batch_size=batch_size, learning_rate=5e-5, warmup_steps=100, sink_mode=True, sink_size=4) # 定义任务，预先准备好相应数据集 task = Trainer(task=task, model=model_type, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) task.set_parallel_config(data_parallel=dp, model_parallel=mp, pipeline_stage=pp, optimizer_shard=op, micro_batch_num=micro_size) if run_mode == 'train': # 训练 task.train() elif run_mode == 'finetune': # 微调 task.finetune(checkpoint_path) elif run_mode == 'eval': # 评估 task.evaluate(checkpoint_path) elif run_mode == 'predict': # 推理，仅支持单卡推理 assert use_parallel == False, "only support predict under stand_alone mode." result = task.predict(input_data=predict_data) print(result) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--run_mode', default='train', required=True, help='set run mode for model.') parser.add_argument('--use_parallel', default=False, help='open parallel for model.') parser.add_argument('--task', default='text_generation', required=True, help='set task type.') parser.add_argument('--model_type', default='glm_6b', required=True, help='set model type.') parser.add_argument('--checkpoint_path', default=None, help='set checkpoint path.') parser.add_argument('--train_dataset', default=None, help='set train dataset.') parser.add_argument('--eval_dataset', default=None, help='set eval dataset.') parser.add_argument('--batch_size', default=4, help='batch size of dataset.') parser.add_argument('--data_parallel', default=1, type=int,help='set data parallel number. Default: None') parser.add_argument('--model_parallel', default=1, type=int, help='set model parallel number. Default: None') parser.add_argument('--pipeline_parallel', default=1, type=int, help='set pipeline parallel number. Default: None') parser.add_argument('--micro_size', default=1, type=int, help='set micro batch number. Default: None') parser.add_argument('--optimizer_parallel', default=False, type=bool, help='whether use optimizer parallel. Default: None') args = parser.parse_args() print(args) main(run_mode=args.run_mode, task=args.task, use_parallel=args.use_parallel, model_type=args.model_type, checkpoint_path=args.checkpoint_path, train_dataset=args.train_dataset, eval_dataset=args.eval_dataset, batch_size=int(args.batch_size), dp=args.data_parallel, mp=args.model_parallel, pp=args.pipeline_parallel, micro_size=args.micro_size, op=args.optimizer_parallel) ``` 因GLM模型过大，**无法在单卡上启动训练**，因此需要**通过分布式脚本拉起多卡训练任务** 在此提供 `run_distribute_single_node.sh` 单机多卡标准启动脚本，用户可用其拉起分布式训练 ```bash #!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 4 ] then echo "Usage Help: bash run_distribute_single_node.sh [EXECUTE_ORDER] [RANK_TABLE_PATH] [DEVICE_RANGE] [RANK_SIZE] For Multiple Devices In Single Machine" exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } EXECUTE_ORDER=$1 RANK_TABLE_PATH=$(check_real_path $2) DEVICE_RANGE=$3 DEVICE_RANGE_LEN=${#DEVICE_RANGE} DEVICE_RANGE=${DEVICE_RANGE:1:DEVICE_RANGE_LEN-2} PREFIX=${DEVICE_RANGE%%","*} INDEX=${#PREFIX} START_DEVICE=${DEVICE_RANGE:0:INDEX} END_DEVICE=${DEVICE_RANGE:INDEX+1:DEVICE_RANGE_LEN-INDEX} if [ ! -f $RANK_TABLE_PATH ] then echo "error: RANK_TABLE_FILE=$RANK_TABLE_PATH is not a file" exit 1 fi if [[ ! $START_DEVICE =~ ^[0-9]+$ ]]; then echo "error: start_device=$START_DEVICE is not a number" exit 1 fi if [[ ! $END_DEVICE =~ ^[0-9]+$ ]]; then echo "error: end_device=$END_DEVICE is not a number" exit 1 fi ulimit -u unlimited export RANK_SIZE=$4 export RANK_TABLE_FILE=$RANK_TABLE_PATH shopt -s extglob for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export DEVICE_ID=${i} export RANK_ID=$((i-START_DEVICE)) mkdir -p ./output/log/rank_$RANK_ID echo "start training for rank $RANK_ID, device $DEVICE_ID" $EXECUTE_ORDER &> ./output/log/rank_$RANK_ID/mindformer.log & done shopt -u extglob ``` 全参微调分布式拉起命令(8卡)： ```bash bash run_distribute_single_node.sh "python task.py --task text_generation --model_type glm_6b --checkpoint_path ./glm_6b.ckpt --train_dataset ./train --run_mode finetune --use_parallel True --data_parallel 1 --model_parallel 8" /path/to/hccl_8p_xxx.json '[0,8]' 8 ``` 参数含义: - `"python task.py --task text_generation --model_type glm_6b --checkpoint_path ./glm_6b.ckpt --train_dataset ./train --run_mode finetune --use_parallel True --data_parallel 1 --model_parallel 8"`: 需执行的命令，此处完整输入task.py的启动命令 python task.py 各项参数含义： - `task`: 需运行的训练任务，此处为 `text_generation` 文本生成任务 - `model_type`: 模型类型，此处选择 `glm_6b` 模型 - `checkpoint_path`: 权重路径，此处替换为实际需加载的权重路径 - `train_dataset`: 训练数据集路径，替换为实际路径 - `run_mode`: 启动模式，train——训练，finetune——微调，eval——评估，predict——推理，此处选择 `finetune` - `use_parallel`: 是否使用多卡并行训练，此处为 `True` - `data_parallel`: 数据并行数，此处为1表示不开启 - `model_parallel`: 模型并行数，此处为8表示8卡并行 bash 脚本其余参数： - `/path/to/hccl_4p_xxx.json`: rank table file路径，替换为之前准备的rank table file的实际路径 - `'[0,8]'`: 占用的卡范围，0包含，8不包含，表示使用 `0~7` 8张卡并行训练 - `8`: rank size，一共使用了多少张卡，此处为8 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint ### LoRA低参微调 全参微调能够在微调数据集上取得良好效果，但存在遗忘预训练知识的现象 因此推荐使用低参微调算法，冻结原模型权重，仅在小规模参数量上进行训练，在微调数据集上取得良好效果的同时，缓解模型遗忘现象 #### run_mindformers脚本启动LoRA低参微调 使用LoRA算法进行低参微调时，使用 `configs/glm/run_glm_6b_lora.yaml` 配置文件，该配置文件包含了lora低参微调算法所需的配置项 修改数据集/模型权重配置路径： - 数据集：修改 `mindformers/configs/glm/run_glm_6b_lora.yaml` 脚本中`train_dataset` 的 `dataset_dir` 为前文生成的数据集路径。 - 加载预训练模型权重：修改 `mindformers/configs/glm/run_glm_6b_lora.yaml` 脚本中的 `load_checkpoint` 为预训练模型权重路径。 #### 启动LoRA低参微调脚本(1卡)： 执行命令： ```shell cd scripts # Usage Help: bash run_standalone.sh [CONFIG_PATH] [DEVICE_ID] [RUN_STATUS] bash run_standalone.sh ../configs/glm/run_glm_6b_lora.yaml 0 finetune ``` 训练的log日志路径：mindformers/scripts/mf_standalone/ checkpoint存储路径：mindformers/scripts/mf_standalone/output/checkpoint #### 启动LoRA低参微调脚本(4卡)： > 注：如果需要进行多卡训练，则需要对`glm/run_glm_6b_lora.yaml`配置文件对应参数进行修改，以4卡为例，需要重新生成4卡的HCCL文件： ```shell data_parallel: 4 ``` ```shell cd scripts # Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] bash run_distribute.sh /path/to/hccl_4_0123_xxx.json ../configs/glm/run_glm_6b_lora.yaml '[0,4]' finetune # 将此处rank_table_file替换为实际路径 ``` 参数说明： 对比全参微调启动方式，仅将 `CONFIG_PATH` 项修改为configs文件夹下面的 `glm/run_glm_6b_lora.yaml` 配置文件，表示使用该接口进行 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint #### Trainer高阶接口启动LoRA低参微调 可复用全参微调部分所提供的 `task.py` 和 `run_distribute_single_node.sh` 脚本 4卡分布式启动命令： ```bash bash run_distribute_single_node.sh "python task.py --task text_generation --model_type glm_6b_lora --checkpoint_path ./glm_6b.ckpt --train_dataset ./train --run_mode finetune --use_parallel True --data_parallel 4 --model_parallel 1" /path/to/hccl_4p_xxx.json '[0,4]' 4 ``` 参数说明：对比全参微调启动，仅改动以下几点： - `model_type`: 指定模型类型为 `glm_6b_lora`，表示使用低参微调算法 - `data_parallel`: 4卡启动，数据并行改为4 - `/path/to/hccl_4p_xxx.json`: 使用4卡的rank_table_file - `'[0,4]' 4`: 使用0~3共4卡 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint 1卡启动命令： ```shell python task.py --task text_generation --model_type glm_6b_lora --checkpoint_path ./glm_6b.ckpt --train_dataset ./train --run_mode finetune --use_parallel False --data_parallel 1 --model_parallel 1 ``` ### 多机多卡微调训练 多机多卡启动 首先在每台机器上运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； 将不同机器上生成的RANK_TABLE_FILE文件中的server_list合并，server_count设为机器数，rank_id顺序增加，并保证不同机器上的RANK_TABLE_FILE相同； 在多机上同时拉起任务，拉起方式为 cd scripts bash run_distribute.sh RANK_TABLE_FILE CONFIG_PATH DEVICE_RANGE RUN_MODE RANK_SIZE #### 参数说明 - RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 - CONFIG_PATH: 为configs文件夹下面的gpt2/run_gpt2*.yaml配置文件 - DEVICE_RANGE: 为单机分布式卡的范围, 如[0,8]为8卡分布式，不包含8本身 - RUN_MODE: 为任务运行状态，支持关键字 train 预训练、predict（文本生成预测） - RANK_SIZE: 总运行卡数 #### 4机32卡参考RANK_TABLE_FILE样例 ```text { "version": "1.0", "server_count": "4", "server_list": [ { "server_id": "10.155.111.140", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" }, { "server_id": "10.155.111.141", "device": [ {"device_id": "0","device_ip": "192.1.27.8","rank_id": "8"}, {"device_id": "1","device_ip": "192.2.27.8","rank_id": "9"}, {"device_id": "2","device_ip": "192.3.27.8","rank_id": "10"}, {"device_id": "3","device_ip": "192.4.27.8","rank_id": "11"}, {"device_id": "4","device_ip": "192.1.27.9","rank_id": "12"}, {"device_id": "5","device_ip": "192.2.27.9","rank_id": "13"}, {"device_id": "6","device_ip": "192.3.27.9","rank_id": "14"}, {"device_id": "7","device_ip": "192.4.27.9","rank_id": "15"}], "host_nic_ip": "reserve" }, { "server_id": "10.155.111.142", "device": [ {"device_id": "0","device_ip": "192.1.27.10","rank_id": "16"}, {"device_id": "1","device_ip": "192.2.27.10","rank_id": "17"}, {"device_id": "2","device_ip": "192.3.27.10","rank_id": "18"}, {"device_id": "3","device_ip": "192.4.27.10","rank_id": "19"}, {"device_id": "4","device_ip": "192.1.27.11","rank_id": "20"}, {"device_id": "5","device_ip": "192.2.27.11","rank_id": "21"}, {"device_id": "6","device_ip": "192.3.27.11","rank_id": "22"}, {"device_id": "7","device_ip": "192.4.27.11","rank_id": "23"}], "host_nic_ip": "reserve" }, { "server_id": "10.155.111.143", "device": [ {"device_id": "0","device_ip": "192.1.27.12","rank_id": "24"}, {"device_id": "1","device_ip": "192.2.27.12","rank_id": "25"}, {"device_id": "2","device_ip": "192.3.27.12","rank_id": "26"}, {"device_id": "3","device_ip": "192.4.27.12","rank_id": "27"}, {"device_id": "4","device_ip": "192.1.27.13","rank_id": "28"}, {"device_id": "5","device_ip": "192.2.27.13","rank_id": "29"}, {"device_id": "6","device_ip": "192.3.27.13","rank_id": "30"}, {"device_id": "7","device_ip": "192.4.27.13","rank_id": "31"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` #### 任务拉起命令示例 ```shell # 第一台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/glm/run_glm_6b_lora.yaml [0,8] train 32 # 第二台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/glm/run_glm_6b_lora.yaml [8,16] train 32 # 第三台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/glm/run_glm_6b_lora.yaml [16,24] train 32 # 第四台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/glm/run_glm_6b_lora.yaml [24,32] train 32 ``` ### 微调后推理 #### 推理样例脚本 下面提供一个模型推理样例脚本 `infer.py` ```python import time import mindspore as ms import numpy as np import argparse from mindformers.models.glm import GLMConfig, GLMChatModel, GLMChatModelWithLora from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer from mindformers.models.glm.glm_processor import process_response from mindformers.pet.pet_config import LoraConfig parser = argparse.ArgumentParser() parser.add_argument('--seq_length', default=1024, type=int, help='Which device to run service.') parser.add_argument('--device_id', default=0, type=int, help='Which device to run service.') parser.add_argument('--checkpoint_path', type=str, default='/path/chatglm6b.ckpt', help='Checkpoint file to load on.') parser.add_argument('--vocab_path', type=str, default='/path/ice_text.model', help='Vocab file to load on.') parser.add_argument('--is_lora', type=str, default='false',help='Whether is lora model.') args = parser.parse_args() if args.is_lora.lower() == "true": is_lora = True else: is_lora = False config = GLMConfig( position_encoding_2d=True, use_past=True, is_sample_acceleration=True, ) pet_config = LoraConfig( lora_rank=8, lora_alpha=32, lora_dropout=0.1 ) def chat_glm(): ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=args.device_id) if is_lora: config.pet_config = pet_config model = GLMChatModelWithLora(config) else: model = GLMChatModel(config) ms.load_checkpoint(args.checkpoint_path, model) tokenizer = ChatGLMTokenizer(args.vocab_path) inputs = ["你好", "请介绍一下华为", "用Python写一个快排", "类型#上衣*材质#牛仔布*颜色#白色*风格#简约*图案#刺绣*衣样式#外套*衣款式#破洞"] for query in inputs: input_ids = tokenizer(query)['input_ids'] start_time = time.time() outputs = model.generate(input_ids, max_length=config.max_decode_length, do_sample=False) end_time = time.time() print(f'generate speed: {outputs[0].shape[0]/(end_time-start_time):.2f} tokens/s') response = tokenizer.decode(outputs) response = process_response(response[0]) print(response) if __name__ == "__main__": chat_glm() ``` #### 运行命令 ```shell python infer.py --seq_length 1024 --device_id 0 --checkpoint_path /path/chatglm6b.ckpt --vocab_path /path/ice_text.model --is_lora True ``` 参数说明： - `seq_length`: 用于指定推理输入长度 - `device_id`: 指定推理在那张设备运行 - `checkpoint_path`: 指定训练出来的模型文件路径用于推理 - `vocab_path`: 模型词表 - `is_lora`: 用于区分是否是lora模型，设置为true表示为lora微调训练模型 ## 评估 ### 模型权重文件合一 微调所得到的权重文件为根据模型切分策略切分后的权重，我们需要手动将切分权重合一，以用于评估和推理 1. 获取模型切分策略文件： 在执行全参微调脚本时，模型完成编译后，将会在运行路径下，生成名为 `ckpt_strategy.ckpt` 的切分策略文件，该文件将用于第二步模型合成 2. MindSpore提供了根据切分策略转换模型权重切分的接口，[mindspore.transform_checkpoints](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.transform_checkpoints.html)，执行以下python脚本，将8份模型文件合成一份 ```python from mindspore import transform_checkpoints transform_checkpoints( src_checkpoints_dir="./output/checkpoint/", # 原切分权重文件夹 dst_checkpoints_dir="./target_checkpoint/", # 目标路径 ckpt_prefix="glm-6b", # .ckpt文件前缀名 src_strategy_file="ckpt_stragery.ckpt", # 步骤1中的切分策略文件路径 dst_strategy_file=None # None表示不切分，权重合一 ) ``` > 注：`transform_checkpoints` 接口当前仅mindspore 2.0以上版本支持，如当前硬件环境只支持2.0以下版本，可以新建conda环境安装mindspore 2.0的cpu版本以执行该脚本 ### 使用全参微调权重 #### run_mindformers启动eval 使用全参微调权重时，启动如下shell脚本，执行单卡评估 配置文件选择 `configs/glm/run_glm_6b_infer.yaml` glm模型推理配置，此配置下评估速度更快 ```bash python run_mindformer.py --config configs/glm/run_glm_6b_infer.yaml --run_mode eval --load_checkpoint /path/to/glm_6b.ckpt --eval_dataset_dir /path/to/data/AdvertiseGen/ --device_id 0 ``` > 注：使用离线生成数据方式时，将 `eval_dataset_dir` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 各项参数： - `config`: 指定用于评估的配置文件名称，此处为`configs/glm/run_glm_6b_infer.yaml` - `run_mode`: 指定执行模式，此为`eval`，表示为评估模式 - `load_checkpoint`: 指定要加载的checkpoint路径，此处为`/path/to/glm_6b.ckpt`，替换为需加载的权重的真实路径 - `eval_dataset_dir`: 评估数据集的路径 - `device_id`: 指定要使用的设备编号（从0开始） 评估完成后会打印评估指标 `bleu-4`、`rouge-1`、`rouge-2`、`rouge-l` > 注：由于默认评估指标的获取方式为生成完整文本后与预期文本做比较，评估速度将受限于模型大小与文本生成速度，评估流程可能较为缓慢 #### Trainer高阶接口启动eval 仍然可复用 `task.py` 脚本，启动命令： ```bash python task.py --task text_generation --model_type glm_6b_chat --checkpoint_path /path/to/glm_6b.ckpt --eval_dataset /path/to/data/AdvertiseGen/ --run_mode eval --batch_size 1 ``` > 1. 当前评估时，batch_size需为1，否则评估速度下降严重 > 2. 使用离线生成数据方式时，将 `eval_dataset` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 ### 使用LoRA低参微调权重 #### run_mindformers启动lora eval 使用LoRA低参微调权重时，启动如下shell脚本，执行单卡评估 配置文件选择 `configs/glm/run_glm_6b_lora_infer.yaml` glm_lora模型推理配置，此配置可用于lora模型，并且评估速度更快 ```bash python run_mindformer.py --config configs/glm/run_glm_6b_lora_infer.yaml --run_mode eval --load_checkpoint /path/to/glm_6b_lora.ckpt --eval_dataset_dir /path/to/data/AdvertiseGen/ --device_id 0 ``` 各项参数同上，路径需替换为实际路径 > 使用离线生成数据方式时，将 `eval_dataset_dir` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 #### Trainer高阶接口启动lora eval 仍然可复用 `task.py` 脚本，启动命令： ```bash python task.py --task text_generation --model_type glm_6b_lora_chat --checkpoint_path /path/to/glm_6b_lora.ckpt --eval_dataset /path/to/data/AdvertiseGen/ --run_mode eval --batch_size 1 ``` > 1. 当前评估时，batch_size需为1，否则评估速度下降严重 > 2. 使用离线生成数据方式时，将 `eval_dataset_dir` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 ## 模型权重转化 本仓库中的`glm`来自于HuggingFace的[chatglm-6b](https://huggingface.co/THUDM/chatglm-6b)，基于下述的步骤获取： 1. 克隆chatglm-6b代码仓，下载分布式的模型文件。 ```shell git lfs install git clone https://huggingface.co/THUDM/chatglm-6b ``` 2. 执行 python 脚本，合并模型权重。 ```python from transformers import AutoModel import torch as pt pt_ckpt_path="Your chatglm-6b path" model = AutoModel.from_pretrained(pt_ckpt_path, trust_remote_code=True).half() pt_pth_path = "pt_glm_6b.pth" pt.save(model.state_dict(), pt_pth_path) ``` 3. 执行转换脚本，得到转换后的输出文件`ms_glm_6b.ckpt`。 ```shell python mindformers/models/glm/convert_weight.py --pt_ckpt_path "replace your ptroch pth path" --ms_ckpt_path ./ms_glm_6b.ckpt ``` pengcheng.mind-test/docs/model_cards/glm2.md000066400000000000000000000435741471400134000213150ustar00rootroot00000000000000# ChatGLM2-6B ## 模型描述 ChatGLM**2**-6B 是开源中英双语对话模型 [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B) 的第二代版本，在保留了初代模型对话流畅、部署门槛较低等众多优秀特性的基础之上，ChatGLM**2**-6B引入了新特征：**更强大的性能**、**更长的上下文**、**更高效的推理**、**更开放的协议**。 ## 仓库介绍 `chatGLM2-6B` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/glm2` ```bash glm2 ├── __init__.py ├── glm2.py # 模型实现 ├── glm2_config.py # 模型配置项 ├── glm2_modules.py # 模组实现 ├── glm2_tokenizer.py # tokenizer └── glm2_transformer.py # transformer层实现 ``` 2. 模型配置：`configs/glm2` ```bash glm2 ├── run_glm2_6b_fintune.yaml # 全量微调启动配置 └── run_glm2_6b_lora.yaml # lora低参微调启动配置 ``` ## 环境要求 - 硬件：Ascend 910A - MindSpore：2.0 推理可在单机单卡上完成部署 全量微调训练需要最少单机8卡，Lora微调训练最少需要1卡 ## 基线 测试环境同上述环境要求 ### 性能 | | data parallel | model parallel | pipeline parallel | batch size | sink size | sequence length | accumulate | per step time (ms) | tokens/s/p | 优化器并行 | 重计算 | Memory (GB) | | -------- | ------------- | -------------- | ----------------- | ---------- | --------- | --------------- | ---------- | ------------------ | ----------- | ---------- | ------ | ----------- | | 全量微调 | 8 | 1 | 1 | 8 | 4 | 193 | 1 | 1894 | 815.2059134 | True | True | 25.2 | | LoRA微调 | 4 | 1 | 1 | 8 | 4 | 193 | 1 | 476 | 3243.697479 | False | False | 22.38 | ### 评估指标 | | rouge-1 | rouge-2 | rouge-l | bleu-4 | | -------- | ------------------ | ----------------- |--------------------| ----------------- | | 全量微调 | 30.784298224299064 | 7.073415046728972 | 24.773958598130843 | 7.466147757009345 | | LoRA微调 | 31.05639289719626 | 7.1753861682243 | 24.229674859813084 | 7.229435140186916 | ## ChatGLM2-6B推理 > 需开发者提前pip安装。具体接口说明请参[API接口](https://gitee.com/mindspore/transformer/wikis/API/) ### AutoClass推理 可以使用AutoClass接口，通过模型名称获取相应的模型/tokenizer实例，并自动下载并加载权重 `from_pretrained()` 接口会自动从云上下载预训练的模型，存储路径：`mindformers/checkpoint_download/glm2` 首次运行pipeline推理时需要进行模型编译，需等待一段时间 ```python from mindformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("glm2_6b") model = AutoModel.from_pretrained("glm2_6b") query = "你好" prompted_inputs = tokenizer.build_prompt(query) input_tokens = tokenizer([prompted_inputs]) outputs = model.generate(input_tokens["input_ids"], max_length=100) response = tokenizer.decode(outputs)[0] print(response) ``` ### pipeline推理 也可以不实例化构造模型，直接通过指定任务模型与模型名的方式进行pipeline的构造 ```python >>> from mindformers import pipeline, TextGenerationPipeline >>> task_pipeline = pipeline(task='text_generation', model='glm2_6b', max_length=2048) >>> task_pipeline('你好') [{'text_generation_text': ['你好，我是 ChatGLM2-6B， 一个人工智能助手。我背后使用的模型是 GLM2-6B， 是一种大型语言模型， 具有超过 2000 亿参数，支持多种任务。']}] >>> pipeline = TextGenerationPipeline(model='glm2_6b', max_length=2048) >>> pipeline("你好") [{'text_generation_text': ['你好，我是 ChatGLM2-6B， 一个人工智能助手。我背后使用的模型是 GLM2-6B， 是一种大型语言模型， 具有超过 2000 亿参数，支持多种任务。']}] ``` ## 微调 下面以 [ADGEN](https://aclanthology.org/D19-1321.pdf) (广告生成) 数据集为例介绍代码的使用方法 ### 数据处理 ADGEN 数据集任务为根据输入（content）生成一段广告词（summary）。 ```json { "content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳", "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点，穿在身上有着很好的显瘦效果。领口装饰了一个可爱的抽绳，漂亮的绳结展现出了十足的个性，配合时尚的泡泡袖型，尽显女性甜美可爱的气息。" } ``` 从 [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) 或者 [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) 下载处理好的 ADGEN 数据集，目录结构为 ```shell AdvertiseGen ├── train.json └── dev.json ``` 将任务配置文件 `configs/glm2/run_glm2_6b_*.yaml` 中的 `==== dataset config ====` 部分替换成： ```yaml train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/train.json" shuffle: True phase: "train" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" input_columns: ["input_ids", "labels"] max_source_length: 64 max_target_length: 128 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/dev.json" shuffle: False phase: "eval" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset ``` ### 生成HCCL文件 运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； ```shell # step1：机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" ``` > 注：若使用ModelArts的notebook环境，可从 `/user/config/jobstart_hccl.json` 路径下直接获取rank table，无需手动生成 RANK_TABLE_FILE 单机8卡参考样例: ```json { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 全参微调 #### run_mindformers脚本启动全参微调 全参微调使用 `configs/glm2/run_glm2_6b.yaml` 配置文件，配置文件中定义了微调所需的各配置项 修改数据集/模型权重配置路径： - 数据集：修改 `mindformers/configs/glm2/run_glm2_6b.yaml` 脚本中`train_dataset` 的 `dataset_dir` 为前文生成的数据集路径。 - 加载预训练模型权重：修改 `mindformers/configs/glm2/run_glm2_6b.yaml` 脚本中的 `load_checkpoint` 为预训练模型权重路径。 启动全参微调脚本： ```shell cd scripts # Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] bash run_distribute.sh /path/to/hccl_8p_01234567_127.0.1.1.json ../configs/glm2/run_glm2_6b.yaml '[0,8]' finetune # 将此处rank_table_file替换为实际路径 ``` 参数说明 ```text RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的glm2/run_glm2_6b.yaml配置文件 DEVICE_RANGE: 为单机分布式卡的范围，如 '[0,8]' 为8卡分布式，不包含8本身 RUN_STATUS: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` > 注：由于GLM2_6B的模型较大，无法在单卡上运行，此处仅提供分布式启动脚本 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint ### LoRA低参微调 全参微调能够在微调数据集上取得良好效果，但存在遗忘预训练知识的现象 因此推荐使用低参微调算法，冻结原模型权重，仅在小规模参数量上进行训练，在微调数据集上取得良好效果的同时，缓解模型遗忘现象 #### run_mindformers脚本启动LoRA低参微调 使用LoRA算法进行低参微调时，使用 `configs/glm2/run_glm2_6b_lora.yaml` 配置文件，该配置文件包含了lora低参微调算法所需的配置项 修改数据集/模型权重配置路径： - 数据集：修改 `mindformers/configs/glm2/run_glm2_6b_lora.yaml` 脚本中`train_dataset` 的 `dataset_dir` 为前文生成的数据集路径。 - 加载预训练模型权重：修改 `mindformers/configs/glm2/run_glm2_6b_lora.yaml` 脚本中的 `load_checkpoint` 为预训练模型权重路径。 #### 启动LoRA低参微调脚本(1卡)： 执行命令： ```shell cd scripts # Usage Help: bash run_stanalone.sh [CONFIG_PATH] [DEVICE_ID] [RUN_STATUS] bash run_standalone.sh ../configs/glm2/run_glm2_6b_lora.yaml 0 finetune ``` 训练的log日志路径：mindformers/scripts/mf_standalone/ checkpoint存储路径：mindformers/scripts/mf_standalone/output/checkpoint #### Trainer高阶接口启动LoRA低参微调 示例脚本如下，需要指定训练数据集路径和微调权重。 ```python from mindformers import Trainer trainer = Trainer(task="text_generation", model="glm2_6b", pet_method="lora", train_dataset="/path/to/AdvertiseGen/train.json") trainer.finetune(finetune_checkpoint="glm2_6b") ``` ### 微调后推理 #### 推理样例脚本 下面提供一个模型推理样例脚本 `infer.py` ```python from mindformers import AutoConfig, AutoModel, AutoTokenizer import mindspore as ms ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=0) config = AutoConfig.from_pretrained("glm2_6b") config.checkpoint_name_or_path = "/path/to/glm2_6b_finetune.ckpt" model = AutoModel.from_config(config) tokenizer = AutoTokenizer.from_pretrained("glm2_6b") inputs = tokenizer(tokenizer.build_prompt("你好"))["input_ids"] print(inputs) print(tokenizer.decode(inputs)) outputs = model.generate(inputs, max_length=128) print(tokenizer.decode(outputs)) inputs = tokenizer(tokenizer.build_prompt("请介绍一下华为"))["input_ids"] print(inputs) outputs = model.generate(inputs, max_length=128) print(tokenizer.decode(outputs)) inputs = tokenizer(tokenizer.build_prompt("晚上睡不着应该怎么办"))["input_ids"] print(inputs) outputs = model.generate(inputs, max_length=128) print(tokenizer.decode(outputs)) inputs = tokenizer(tokenizer.build_prompt("类型#上衣*材质#牛仔布*颜色#白色*风格#简约*图案#刺绣*衣样式#外套*衣款式#破洞"))["input_ids"] print(inputs) outputs = model.generate(inputs, max_length=128) print(tokenizer.decode(outputs)) ``` ## 评估 ### 模型权重文件合一 微调所得到的权重文件为根据模型切分策略切分后的权重，我们需要手动将切分权重合一，以用于评估和推理 1. 获取模型切分策略文件： 在执行全参微调脚本时，模型完成编译后，将会在运行路径下，生成名为 `ckpt_strategy.ckpt` 的切分策略文件，该文件将用于第二步模型合成 2. MindSpore提供了根据切分策略转换模型权重切分的接口，[mindspore.transform_checkpoints](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.transform_checkpoints.html)，执行以下python脚本，将8份模型文件合成一份 ```python from mindspore import transform_checkpoints transform_checkpoints( src_checkpoints_dir="./output/checkpoint/", # 原切分权重文件夹 dst_checkpoints_dir="./target_checkpoint/", # 目标路径 ckpt_prefix="glm2-6b", # .ckpt文件前缀名 src_strategy_file="ckpt_stragery.ckpt", # 步骤1中的切分策略文件路径 dst_strategy_file=None # None表示不切分，权重合一 ) ``` > 注：`transform_checkpoints` 接口当前仅mindspore 2.0以上版本支持，如当前硬件环境只支持2.0以下版本，可以新建conda环境安装mindspore 2.0的cpu版本以执行该脚本 ### 使用全参微调权重 #### run_mindformers启动eval 使用全参微调权重时，启动如下shell脚本，执行单卡评估 配置文件选择 `configs/glm2/run_glm2_6b.yaml` glm2模型推理配置，修改其中`model`字段下`model_config`中`use_past: True`开启增量推理使评估速度更快 ```bash python run_mindformer.py --config configs/glm2/run_glm2_6b.yaml --run_mode eval --load_checkpoint /path/to/glm2_6b_finetune.ckpt --eval_dataset_dir /path/to/data/AdvertiseGen/ --device_id 0 ``` > 注：使用离线生成数据方式时，将 `eval_dataset_dir` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 各项参数： - `config`: 指定用于评估的配置文件名称，此处为`configs/glm2/run_glm2_6b.yaml` - `run_mode`: 指定执行模式，此为`eval`，表示为评估模式 - `load_checkpoint`: 指定要加载的checkpoint路径，此处为`/path/to/glm2_6b_finetune.ckpt`，替换为需加载的权重的真实路径 - `eval_dataset_dir`: 评估数据集的路径 - `device_id`: 指定要使用的设备编号（从0开始） 评估完成后会打印评估指标 `bleu-4`、`rouge-1`、`rouge-2`、`rouge-l` > 注：由于默认评估指标的获取方式为生成完整文本后与预期文本做比较，评估速度将受限于模型大小与文本生成速度，评估流程可能较为缓慢 #### Trainer高阶接口启动eval 与上文类似： ```bash from mindformers import Trainer, ChatGLM2Config, ChatGLM2ForConditionalGeneration # 开启增量推理使评估速度更快 config = ChatGLM2Config(use_past=True) model = ChatGLM2ForConditionalGeneration(config) trainer = Trainer(task="text_generation", model=model, eval_dataset="/path/to/AdvertiseGen/dev.json") trainer.evaluate(eval_checkpoint="/path/to/glm2_6b_finetune.ckpt") ``` ### 使用LoRA低参微调权重 #### run_mindformers启动lora eval 使用LoRA低参微调权重时，启动如下shell脚本，执行单卡评估 配置文件选择 `configs/glm2/run_glm2_6b_lora.yaml` glm2_lora模型推理配置，此配置可用于lora模型，修改其中`model`字段下`model_config`中`use_past: True`开启增量推理使评估速度更快 ```bash python run_mindformer.py --config configs/glm2/run_glm2_6b_lora.yaml --run_mode eval --load_checkpoint /path/to/glm2_6b_lora.ckpt --eval_dataset_dir /path/to/data/AdvertiseGen/ --device_id 0 ``` 各项参数同上，路径需替换为实际路径 #### Trainer高阶接口启动lora eval 与上文类似： ```bash from mindformers import Trainer, ChatGLM2Config, ChatGLM2WithLora from mindformers.pet.pet_config import LoraConfig # 开启增量推理使评估速度更快 config = ChatGLM2Config(use_past=True) config.pet_config = LoraConfig() model = ChatGLM2WithLora(config) trainer = Trainer(task="text_generation", model=model, eval_dataset="/path/to/AdvertiseGen/dev.json") trainer.evaluate(eval_checkpoint="/path/to/glm2_6b_lora.ckpt") ``` ## 模型权重转化 本仓库中的`glm2`来自于HuggingFace的 [ChatGLM2-6B](https://huggingface.co/THUDM/chatglm2-6b)，基于下述的步骤获取： 1. 克隆chatglm2-6b代码仓，下载分布式的模型文件。 ```shell git lfs install git clone https://huggingface.co/THUDM/chatglm2-6b ``` 2. 执行 python 脚本，合并模型权重。 ```python from transformers import AutoTokenizer, AutoModel import torch tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True) model = AutoModel.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True) with open("pt_model_arch.txt", "w") as fp: print(model, file=fp, flush=True) with open("pt_ckpt.txt", "w") as fp: for name, param in model.named_parameters(): fp.write(f"{name} {param.shape} {param.dtype}\n") torch.save(model.state_dict(), "glm2_6b.pth") ``` 3. 执行转换脚本，得到转换后的输出文件`glm2_6b.ckpt`。 ```python import mindspore as ms import torch as pt from tqdm import tqdm pt_ckpt_path = "glm2_6b.pth" pt_param = pt.load(pt_ckpt_path) type_map = {"torch.float16": "ms.float16", "torch.float32": "ms.float32"} ms_param = [] with open("check_pt_ckpt.txt", "w") as fp: for k, v in tqdm(pt_param.items()): if "word_embeddings.weight" in k: k = k.replace("word_embeddings.weight", "embedding_table") fp.write(f"{k} {v.shape} {v.dtype}\n") ms_param.append({"name": k, "data": ms.Tensor(v.numpy())}) ms.save_checkpoint(ms_param, "glm2_6b.ckpt") ``` pengcheng.mind-test/docs/model_cards/gpt2.md000066400000000000000000001062701471400134000213210ustar00rootroot00000000000000# GPT2 ## 模型描述 GPT-2由OpenAI于2019年发布。GPT-2模型是继承于GPT模型，GPT-2是一个非常庞大的语言模型，它主要是用于预测下一个单词。按照参数量的大小，原生GPT-2模型可分为small（124M）、medium（355M）、large（774M）、xlarge（1.5B），但在此仓中，基于GPT2扩展了13B，52B等规格。 [论文](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)A Radford，et al., Language Models are Unsupervised Multitask Learners, 2019 ## 模型性能 ```txt Mindspore: 2.0.0rc1 Ascend: 910A ``` | config | task | Datasets | metric | score | train performance | predict performance | | :---------------------------------: | :-----------------: | :--------------------------------: | :------: | :---------------------------------: | :---------------: | :--------------------------------------: | | [gpt2](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2.yaml) | text_generation | wikitext2 | ppl | 22.11 | 1265 tokens/s | 4.66/11.37 tokens/s(use past True/False) | | [gpt2_lora](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2_lora.yaml) | text_generation | wikitext2 | - | - | 1428 tokens/s | - | | [gpt2_txtcls](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2_txtcls.yaml) | text_classification | SST-2
IMDB
AGNews
COLA | accuracy | 0.908
0.934
0.941
0.693 | - | - | ## 仓库介绍 1、模型具体实现：`mindformers/models/gpt2` `gpt2`基于`mindformers`实现，主要涉及的文件有： ```bash model ├── __init__.py ├── convert_weight.py # 权重转换脚本 ├── gpt2.py # 模型实现 ├── gpt2_config.py # 模型配置项 ├── gpt2_processor.py # gpt2预处理 ├── gpt2_tokenizer.py # tokenizer └── gpt2_modules.py # transformer层实现 ``` 2、模型配置：`configs/gpt2` ```bash gpt2 ├── run_gpt2.yaml # gpt2 small模型启动配置 ├── run_gpt2_13b.yaml # gpt 13b模型启动配置 ├── run_gpt2_52b.yaml # gpt 52b模型启动配置 ├── run_gpt2_lora.yaml # gpt2 small lora低参微调启动配置 ├── run_gpt2_txtcls.yaml # gpt2 small文本分类模型启动配置 ├── run_gpt2_xl.yaml # gpt2 xlarge模型启动配置 └── run_model_xl_lora.yaml # gpt2 xlarge lora低参微调启动配置 ``` 3、预处理脚本和任务启动脚本：`mindformers/tools/dataset_preprocess/gpt2` ```bash model ├── txtcls_dataset_to_mindrecord.py # 文本分类数据集预处理 └── wikitext2_data_process.py # wikitext2数据集预处理 ``` ## 前期准备 ### [mindformers安装](https://gitee.com/mindspore/mindformers/tree/dev#%E4%BA%8Cmindformers%E5%AE%89%E8%A3%85) ### 生成RANK_TABLE_FILE(多卡运行必备环节) 运行`mindformers/tools/hccl_tools.py`生成RANK_TABLE_FILE的json文件 ```bash # 运行如下命令，生成当前机器的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" ``` **注：若使用ModelArts的notebook环境，可从 `/user/config/jobstart_hccl.json` 路径下直接获取rank table，无需手动生成** RANK_TABLE_FILE 单机8卡参考样例: **其中`server_id`是机器ip地址** ```json { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 多机RANK_TABLE_FILE合并(多机多卡必备环节) - step 1. 首先根据上章节内容，在每个机器上生成各自的`RANK_TABLE_FILE`文件，然后将不同机器上生成的`RANK_TABLE_FILE`文件全部拷贝到同一台机器上。 - step 2. 运行mindformers/tools/merge_hccl.py将不同机器上生成的`RANK_TABLE_FILE`文件合并 ```bash # 运行如下命令，合并每个机器上的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/merge_hccl.py hccl*.json ``` - step 3. 将合并后的`RANK_TABLE_FILE`文件拷贝到所有机器中，保证不同机器上的`RANK_TABLE_FILE`相同。 RANK_TABLE_FILE 双机16卡参考样例: 以下`server_id`为机器ip地址，不同机器需设置不同的值 ```json { "version": "1.0", "server_count": "2", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ { "device_id": "0", "device_ip": "192.168.0.0", "rank_id": "0" }, { "device_id": "1", "device_ip": "192.168.1.0", "rank_id": "1" }, { "device_id": "2", "device_ip": "192.168.2.0", "rank_id": "2" }, { "device_id": "3", "device_ip": "192.168.3.0", "rank_id": "3" }, { "device_id": "4", "device_ip": "192.168.0.1", "rank_id": "4" }, { "device_id": "5", "device_ip": "192.168.1.1", "rank_id": "5" }, { "device_id": "6", "device_ip": "192.168.2.1", "rank_id": "6" }, { "device_id": "7", "device_ip": "192.168.3.1", "rank_id": "7" } ], "host_nic_ip": "reserve" }, { "server_id": "xx.xx.xx.xx", "device": [ { "device_id": "0", "device_ip": "192.168.0.1", "rank_id": "8" }, { "device_id": "1", "device_ip": "192.168.1.1", "rank_id": "9" }, { "device_id": "2", "device_ip": "192.168.2.1", "rank_id": "10" }, { "device_id": "3", "device_ip": "192.168.3.1", "rank_id": "11" }, { "device_id": "4", "device_ip": "192.168.0.2", "rank_id": "12" }, { "device_id": "5", "device_ip": "192.168.1.2", "rank_id": "13" }, { "device_id": "6", "device_ip": "192.168.2.2", "rank_id": "14" }, { "device_id": "7", "device_ip": "192.168.3.2", "rank_id": "15" } ], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 模型权重下载与转换 作为参考，这里描述CheckPoint在HuggingFace或者官方开源github仓库和MindSpore间的转换，在不同分布式策略间的转换。 Huggingface权重： [gpt2 small](https://huggingface.co/gpt2/resolve/main/pytorch_model.bin) [gpt2 xlarge](https://huggingface.co/gpt2-xl/resolve/main/pytorch_model.bin) [gpt2 13b](https://huggingface.co/cerebras/Cerebras-GPT-13B/tree/main) 其中，13b的权重需要将上述链接下的`pytorch_model-00001-of-00002.bin`、`pytorch_model-00002-of-00002.bin`、`pytorch_model.bin.index.json `、`config.json`下载并存到一个文件夹`torch_weights`中，然后使用如下命令将Huggingface的权重进行合并 ```python from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("torch_weights") model.save_pretrained("gpt_13b.bin", max_shard_size="60GB") ``` 权重转换： ```bash # gpt2 small python mindformers/models/gpt2/convert_weight.py --layers 12 --torch_path gpt2_small.bin --mindspore_path ./gpt2_small.ckpt # gpt2 xlarge python mindformers/models/gpt2/convert_weight.py --layers 48 --torch_path gpt2_small.bin --mindspore_path ./gpt2_xlarge.ckpt # gpt2 13b python mindformers/models/gpt2/convert_weight.py --layers 40 --torch_path gpt_13b.bin --mindspore_path ./gpt_13b.ckpt ``` 另，`mindformers`已经提供转换好的权重（其中lora权重为mindformers训练得到，非Huggingface官方权重转化得到）： [gpt2 small](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2.ckpt) [gpt2 small lora](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_lora.ckpt) [gpt2 xlarge](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_xl.ckpt) [gpt2 xlarge lora](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_xl_lora.ckpt) [gpt2 13b](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_13b.ckpt) ### [模型权重切分与合并](https://gitee.com/mindspore/mindformers/blob/dev/docs/feature_cards/Transform_Ckpt.md) 从hugging face或官方github仓库转换而来的权重通常是单卡权重，基于该权重进行多卡微调，评测，推理，涉及ckpt从单机策略到分布式策略的切换。 通常训练采用分布式训练，基于该权重进行评测，推理多采用单卡，涉及ckpt从分布式策略到单机策略的切换。 以上涉及到ckpt的单卡，多卡转换，详细教程请参考特性文档模型[权重切分与合并](https://gitee.com/mindspore/mindformers/blob/dev/docs/feature_cards/Transform_Ckpt.md) ## 基于API的快速使用 ### 基于AutoClass的使用 可以使用AutoClass接口，通过模型名称获取相应的model/preprocess/tokenizer等实例，并自动下载并加载权重 `from_pretrained()` 接口会自动从云上下载预训练的模型，存储路径：`mindformers/checkpoint_download/gpt2` ```python # 以gpt2 small为例 import mindspore from mindformers import AutoModel, AutoTokenizer # 指定图模式，指定使用训练卡id mindspore.set_context(mode=0, device_id=0) tokenizer = AutoTokenizer.from_pretrained("gpt2") model = AutoModel.from_pretrained("gpt2") model.set_train(False) inputs = tokenizer("An increasing sequence: one,") outputs = model.generate(inputs["input_ids"], max_length=20, do_sample=20) response = tokenizer.decode(outputs)[0] print(response) # An increasing sequence: one, two, three, four, five, six, seven, eight, ``` ### 基于Trainer的训练，微调，评测，推理 ```python # 以gpt2 small为例 import mindspore from mindformers.trainer import Trainer # 指定图模式，指定使用训练卡id mindspore.set_context(mode=0, device_id=0) # 初始化预训练任务 trainer = Trainer(task='text_generation', model='gpt2', train_dataset='path/to/train_dataset', eval_dataset='path/to/eval_dataset') # 开启预训练 trainer.train() # 开启全量微调 trainer.finetune() # 开启评测 trainer.evaluate() # 开启推理 predict_result = trainer.predict(input_data="An increasing sequence: one,", do_sample=False, max_length=20) print(predict_result) # output result is: [{'text_generation_text': ['An increasing sequence: one, two, three, four, five, six, seven, eight,']}] # Lora微调 trainer = Trainer(task="text_generation", model="gpt2", pet_method="lora", train_dataset="path/to/train_dataset") trainer.finetune(finetune_checkpoint="gpt2") ``` ### 基于Pipeline的推理 ```python # 以gpt2 small为例 import mindspore from mindformers.pipeline import pipeline # 指定图模式，指定使用训练卡id mindspore.set_context(mode=0, device_id=0) pipeline_task = pipeline(task="text_generation", model="gpt2") pipeline_result = pipeline_task("An increasing sequence: one,", do_sample=False, max_length=20) print(pipeline_result) # [{'text_generation_text': ['An increasing sequence: one, two, three, four, five, six, seven, eight,']}] ``` **注：**要提高推理速度，可在对应模型配置文件中进行如下配置，设置增量推理`use_past`为True。 ## 预训练 ### 数据集准备-预训练 以Wikitext2数据集为例 1、数据集下载：[WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip) 2、词表下载：[vocab.json](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/vocab.json)，[merges.txt](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/merges.txt) 3、参考[wikitext-2处理脚本](https://gitee.com/mindspore/mindformers/blob/931cf93045473d5827ee26638d83fabe94058d28/mindformers/tools/dataset_preprocess/gpt2/wikitext2_data_process.py#)，将数据处理成Mindrecord格式。注：训练数据处理时，长度应等于模型接收长度加一 ```bash # 训练 python mindformers/tools/dataset_preprocess/gpt2/wikitext2_data_process.py \ --input_file ./wikitext-2/wiki.train.tokens \ --output_file ./wikitext-2.train.mindrecord \ --max_length 1025 ``` ### 脚本启动 #### 单卡训练 - python启动 ```bash # dataset_dir可指定文件目录或文件路径，指定文件路径时，读取单文件， # 指定目录时，读取目录下所有以字符串mindrecord结尾的数据文件 python run_mindformer.py --config configs/gpt2/run_gpt2.yaml \ --run_mode train \ --train_dataset_dir ./wikitext-2.train.mindrecord ``` - bash启动 ```bash cd scripts bash run_standalone.sh ../configs/gpt2/run_gpt2.yaml [DEVICE_ID] train ``` #### 多卡训练 多卡运行需要RANK_FILE_TABLE，请参考前期准备-[生成RANK_TABLE_FILE](#生成rank_table_file多卡运行必备环节) - 单机多卡 请提前在yaml文件中修改相应的分布式配置，配置参考[并行配置](https://gitee.com/mindspore/mindformers/blob/dev/docs/readthedocs/source_zh_cn/docs/design/Parallel_Design.md#config-%E5%B9%B6%E8%A1%8C%E9%85%8D%E7%BD%AE) **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** ```bash cd scripts bash run_distribute.sh RANK_TABLE_FILE ../configs/gpt2/run_gpt2.yaml [0,8] train ``` - 多机多卡 在每台机器上启动`bash run_distribute.sh`。 请提前在yaml文件中修改相应的分布式配置，配置参考[并行配置](https://gitee.com/mindspore/mindformers/blob/dev/docs/readthedocs/source_zh_cn/docs/design/Parallel_Design.md#config-%E5%B9%B6%E8%A1%8C%E9%85%8D%E7%BD%AE) **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** ```bash server_count=12 device_num=8*$server_count # launch ranks in the 0th server cd scripts bash run_distribute.sh $RANK_TABLE_FILE path/to/config.yaml [0,8] train $device_num # launch ranks in the 1-11 server via ssh for idx in {1..11} do let rank_start=8*$idx let rank_end=$rank_start+8 ssh ${IP_LIST[$idx]} "cd scripts; bash run_distribute.sh $RANK_TABLE_FILE path/to/config.yaml [$rank_start,$rank_end] train $device_num" done ``` 其中 - `RANK_TABLE_FILE`为上一步汇总并分发的总rank table文件； - `IP_LIST`为12台服务器的IP地址。如192.168.0.[0-11] ```bash IP_LIST=("192.168.0.0", "192.168.0.1", ..., "192.168.0.11") ``` ## 微调 ### 数据集准备-微调数据集 - [参考预训练数据集制作](#数据集准备-预训练) ### 全参微调 #### 单卡微调 - python启动 ```bash python run_mindformer.py --config configs/gpt2/run_gpt2.yaml \ --run_mode finetune \ --train_dataset_dir ./wikitext-2.train.mindrecord ``` - bash启动 **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** ```bash cd scripts bash run_standalone.sh ../configs/gpt2/run_gpt2.yaml [DEVICE_ID] finetune ``` #### 多卡微调 多卡运行需要RANK_FILE_TABLE，请参考前期准备-[生成RANK_TABLE_FILE](#生成rank_table_file多卡运行必备环节) - 单机多卡 **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** ```bash cd scripts bash run_distribute.sh RANK_TABLE_FILE ../configs/gpt2/run_gpt2.yaml [0,8] finetune 8 ``` 多机多卡运行需要合并不同机器的RANK_FILE_TABLE，参考前期准备-[多机RANK_TABLE_FILE合并](#多机rank_table_file合并多机多卡必备环节) - 多机多卡 **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** 在每台机器上启动`bash run_distribute.sh`。 ```bash server_count=12 device_num=8*$server_count # launch ranks in the 0th server cd scripts bash run_distribute.sh $RANK_TABLE_FILE path/to/config.yaml [0,8] finetune $device_num # launch ranks in the 1-11 server via ssh for idx in {1..11} do let rank_start=8*$idx let rank_end=$rank_start+8 ssh ${IP_LIST[$idx]} "cd scripts; bash run_distribute.sh $RANK_TABLE_FILE path/to/config.yaml [$rank_start,$rank_end] finetune $device_num" done ``` 其中 - `RANK_TABLE_FILE`为上一步汇总并分发的总rank table文件； - `IP_LIST`为12台服务器的IP地址。如192.168.0.[0-11] ```bash IP_LIST=("192.168.0.0", "192.168.0.1", ..., "192.168.0.11") ``` ### Lora微调 #### 单卡微调 ```bash python run_mindformer.py --config configs/gpt2/run_gpt2_lora.yaml --run_mode finetune ``` ```bash cd scripts bash run_standalone.sh ../configs/gpt2/run_gpt2_lora.yaml [DEVICE_ID] finetune ``` #### 多卡微调 多卡运行需要RANK_FILE_TABLE，请参考前期准备-[生成RANK_TABLE_FILE](#生成rank_table_file多卡运行必备环节) - 单机多卡 ```bash cd scripts bash run_distribute.sh RANK_TABLE_FILE path/to/config_lora.yaml [0,8] finetune 8 ``` 多机多卡运行需要合并不同机器的RANK_FILE_TABLE，参考前期准备-[多机RANK_TABLE_FILE合并](#多机生成rank_table_file合并多机多卡必备环节) - 多机多卡 在每台机器上启动`bash run_distribute.sh`。 ```bash server_count=12 device_num=8*$server_count # launch ranks in the 0th server cd scripts bash run_distribute.sh $RANK_TABLE_FILE path/to/config_lora.yaml [0,8] finetune $device_num # launch ranks in the 1-11 server via ssh for idx in {1..11} do let rank_start=8*$idx let rank_end=$rank_start+8 ssh ${IP_LIST[$idx]} "cd scripts; bash run_distribute.sh $RANK_TABLE_FILE path/to/config_lora.yaml [$rank_start,$rank_end] finetune $device_num" done ``` 其中 - `RANK_TABLE_FILE`为上一步汇总并分发的总rank table文件； - `IP_LIST`为12台服务器的IP地址。如192.168.0.[0-11] ```bash IP_LIST=("192.168.0.0", "192.168.0.1", ..., "192.168.0.11") ``` ## 评测 GPT2支持文本生成和文本分类两个任务的评测。 - 文本生成： - 获取数据集： - [WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip)是从维基百科上经过验证的优质文章集中提取的超过1亿个token的集合。 - 处理数据成mindrecord格式 - WikiText2： ```bash cd mindformers/tools/dataset_preprocess/gpt2 python wikitext2_data_process.py --input_file {your_path/wiki.valid.tokens} \ --output_file {your_path/wikitext-2.valid.mindrecord} ``` - 开启评测： - WikiText2 ```bash python run_mindformer.py --config configs/gpt2/run_gpt2.yaml \ --eval_dataset_dir {your_path/wikitext-2.valid.mindrecord} \ --run_mode eval \ --epochs 1 # gpt2: PerplexityMetric: {'PerplexityMetric': {'loss': 3.24, 'PPL': 25.55} # gpt2_13b(需替换yaml文件): PerplexityMetric: {'PerplexityMetric': {'loss': 2.35, 'PPL': 10.49} ``` - 文本分类： - 获取数据集: - [SST-2数据集](https://dl.fbaipublicfiles.com/glue/data/SST-2.zip)数据集包含电影评论中的句子和它们情感的人类注释。类别分为两类正面情感（positive，样本标签对应为1）和负面情感（negative，样本标签对应为0） - [IMDB数据集](https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews)影评数据集，包含5万条IMDB影评，评论的情绪是二元的，专门用于情绪分析。 - [AG-News数据集](http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)数据集包含496,835条来自AG新闻语料库4大类别超过2000个新闻源的新闻文章。 - [COLA数据集](https://nyu-mll.github.io/CoLA/)数据集来自语言理论的书籍和期刊，每个句子被标注为是否合乎语法的单词序列。 - 处理数据成mindrecord格式 ```bash # 因评测前需要微调模型，所以需要生成训练/评测数据集。注：生成的数据集文件需以.mindrecord结尾 cd mindformers/tools/dataset_preprocess/gpt2 python txtcls_dataset_to_mindrecord.py --dataset_name {select one from ['cola', 'sst_2', 'ag_news', 'imdb']} --input_file {your_path/train.tsv} \ --output_file {your_path/dataset_name.train.mindrecord} python txtcls_dataset_to_mindrecord.py --dataset_name {the same as above} --input_file {your_path/dev.tsv} \ --output_file {your_path/dataset_name.dev.mindrecord} ``` - 开启微调：因为原始权重中不包含隐向量向类别映射的参数，所以无法进行zero-shot，评测前需要事先进行微调。 ```bash # 运行前请确保run_gpt2_txtcls.yaml中的model.model_config.num_labels准确，具体的， # sst2/cola/imdb: num_labels = 2, agnews: num_labels = 4 python run_mindformer.py --config configs/gpt2/run_gpt2_txtcls.yaml \ --train_dataset_dir {your_path/dataset_name.train.mindrecord} \ --run_mode finetune ``` - 开启评测：评测指标为ACC ```bash # 运行前请确保run_gpt2_txtcls.yaml中的model.model_config.num_labels准确，具体的， # sst2/cola/imdb: num_labels = 2, agnews: num_labels = 4 python run_mindformer.py --config configs/gpt2/run_gpt2_txtcls.yaml \ --eval_dataset_dir {your_path/dataset_name.dev.mindrecord} \ --run_mode eval \ --epochs 1 # ACC: COLA-0.693, SST-2-0.908, IMDB-0.934, AG-News-0.941 ``` ## 推理 ### 基于pipeline的推理 Pipeline接口进行推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline("text_generation", model='gpt2', max_length=20) pipeline_result = pipeline_task("I love Beijing, because", do_sample=False) print(pipeline_result) # [{'text_generation_text': ["I love Beijing, because it's a beautiful city. It's a beautiful city. It's a"]}] ``` 以下为基于pipeline接口的自定义推理脚本，支持多卡多batch推理。 ```python # predict_custom.py 文件 import os import argparse import numpy as np import mindspore as ms from mindspore.train import Model from mindspore import load_checkpoint, load_param_into_net from mindformers import AutoConfig, AutoTokenizer, AutoModel, pipeline from mindformers import init_context, ContextConfig, ParallelContextConfig from mindformers.trainer.utils import get_last_checkpoint from mindformers.tools.utils import str2bool def context_init(use_parallel=False, device_id=0): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=device_id) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, full_batch=True) init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) def main(use_parallel=False, device_id=0, checkpoint_path="", use_past=True, do_sample=False, max_decode_length=30): """main function.""" # 初始化单卡/多卡环境 context_init(use_parallel, device_id) # 多batch输入 inputs = ["I love Beijing, because", "An increasing sequence: one,"] # set model config model_config = AutoConfig.from_pretrained("gpt2") model_config.use_past = use_past model_config.do_sample = do_sample model_config.max_decode_length = max_decode_length if checkpoint_path and not use_parallel: model_config.checkpoint_name_or_path = checkpoint_path print(f"config is: {model_config}") # build tokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") # build model from config network = AutoModel.from_config(model_config) # if use parallel, load distributed checkpoints if use_parallel: # find the sharded ckpt path for this rank ckpt_path = os.path.join(checkpoint_path, "rank_{}".format(os.getenv("RANK_ID", "0"))) ckpt_path = get_last_checkpoint(ckpt_path) print("ckpt path: %s", str(ckpt_path)) # shard gpt2 and load sharded ckpt model = Model(network) model.infer_predict_layout(ms.Tensor(np.ones(shape=(1, model_config.seq_length)), ms.int32)) checkpoint_dict = load_checkpoint(ckpt_path) not_load_network_params = load_param_into_net(model, checkpoint_dict) print("Network parameters are not loaded: %s", str(not_load_network_params)) text_generation_pipeline = pipeline(task="text_generation", model=network, tokenizer=tokenizer) outputs = text_generation_pipeline(inputs) for output in outputs: print(output) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--use_parallel', default=False, type=str2bool, help='whether use parallel.') parser.add_argument('--device_id', default=0, type=int, help='set device id.') parser.add_argument('--checkpoint_path', default='', type=str, help='set checkpoint path.') parser.add_argument('--use_past', default=True, type=str2bool, help='whether use past.') parser.add_argument('--do_sample', default=False, type=str2bool, help='whether enable do_sample.') parser.add_argument('--max_decode_length', default=30, type=int, help='the length of generated text.') args = parser.parse_args() main(args.use_parallel, args.device_id, args.checkpoint_path, args.use_past, args.do_sample, args.max_decode_length) # {'text_generation_text': ["I love Beijing, because it's a beautiful city. It's a beautiful city. It's a beautiful city. It's a beautiful city. It"]} # {'text_generation_text': ['An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen,']} ``` 以下为多卡运行自定义多batch推理的脚本 ```bash # >>> `run_predict.sh`文件 CHECKPOINT_PATH=$2 export RANK_TABLE_FILE=$1 # define variable export RANK_SIZE=8 export START_RANK=0 # this server start rank export END_RANK=8 # this server end rank # run for((i=${START_RANK}; i<${END_RANK}; i++)) do export RANK_ID=$i export DEVICE_ID=$((i-START_RANK)) echo "Start distribute running for rank $RANK_ID, device $DEVICE_ID" python3 ./predict_custom.py --use_parallel True --checkpoint_path CHECKPOINT_PATH &> minformers_$RANK_ID.log & done ``` #### 单卡pipeline推理 ```bash python predict_custom.py ``` #### 多卡pipeline推理 ```bash bash run_predict.sh RANK_TABLE_FILE path/to/gpt2_shard_checkpoint_dir ``` ### 基于generate的推理 Generate接口进行推理 ```python # 使用AutoModel.from_pretrained实例化模型 from mindformers import AutoModel, AutoTokenizer model = AutoModel.from_pretrained("gpt2") tokenizer = AutoTokenizer.from_pretrained("gpt2") input_ids = tokenizer("I love Beijing, because")["input_ids"] generated_ids = model.generate(input_ids, do_sample=False, max_length=30) generate_result = tokenizer.decode(generated_ids) print(generate_result) # ["I love Beijing, because it's a beautiful city. It's a beautiful city. It's a beautiful city. It's a beautiful city. It"] ``` ```python # 使用AutoModel.from_config实例化模型 from mindformers import AutoModel, AutoTokenizer, AutoConfig # 可对model_config进行修改，如model_config.do_sample = False model_config = AutoConfig.from_pretrained("gpt2") model = AutoModel.from_config(model_config) tokenizer = AutoTokenizer.from_pretrained("gpt2") input_ids = tokenizer("I love Beijing, because")["input_ids"] generate_result = model.generate(input_ids, do_sample=False, max_length=30) print(generate_result) # ["I love Beijing, because it's a beautiful city. It's a beautiful city. It's a beautiful city. It's a beautiful city. It"] ``` 以下为基于model.generate接口的自定义推理脚本，支持多卡多batch推理。 ```python # predict_custom.py 文件 import os import argparse import numpy as np import mindspore as ms from mindspore.train import Model from mindspore import load_checkpoint, load_param_into_net from mindformers import AutoConfig, AutoTokenizer, AutoModel from mindformers import init_context, ContextConfig, ParallelContextConfig from mindformers.trainer.utils import get_last_checkpoint from mindformers.tools.utils import str2bool def context_init(use_parallel=False, device_id=0): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=device_id) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, full_batch=True) init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) def main(use_parallel=False, device_id=0, checkpoint_path="", use_past=True, do_sample=False, max_decode_length=30): """main function.""" # 初始化单卡/多卡环境 context_init(use_parallel, device_id) # 多batch输入 inputs = ["I love Beijing, because", "An increasing sequence: one,"] # set model config model_config = AutoConfig.from_pretrained("gpt2") model_config.batch_size = len(inputs) model_config.use_past = use_past model_config.do_sample = do_sample model_config.max_decode_length = max_decode_length if checkpoint_path and not use_parallel: model_config.checkpoint_name_or_path = checkpoint_path print(f"config is: {model_config}") # build tokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") # build model from config model = AutoModel.from_config(model_config) # if use parallel, load distributed checkpoints if use_parallel: # find the sharded ckpt path for this rank ckpt_path = os.path.join(checkpoint_path, "rank_{}".format(os.getenv("RANK_ID", "0"))) ckpt_path = get_last_checkpoint(ckpt_path) print("ckpt path: %s", str(ckpt_path)) # shard gpt2 and load sharded ckpt model = Model(model) model.infer_predict_layout(ms.Tensor(np.ones(shape=(1, model_config.seq_length)), ms.int32)) checkpoint_dict = load_checkpoint(ckpt_path) not_load_network_params = load_param_into_net(model, checkpoint_dict) print("Network parameters are not loaded: %s", str(not_load_network_params)) inputs_ids = tokenizer(inputs, max_length=model_config.max_decode_length, padding="max_length")["input_ids"] outputs = model.generate(inputs_ids, max_length=model_config.max_decode_length) for output in outputs: print(tokenizer.decode(output)) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--use_parallel', default=False, type=str2bool, help='whether use parallel.') parser.add_argument('--device_id', default=0, type=int, help='set device id.') parser.add_argument('--checkpoint_path', default='', type=str, help='set checkpoint path.') parser.add_argument('--use_past', default=True, type=str2bool, help='whether use past.') parser.add_argument('--do_sample', default=False, type=str2bool, help='whether enable do_sample.') parser.add_argument('--max_decode_length', default=30, type=int, help='the length of generated text.') args = parser.parse_args() main(args.use_parallel, args.device_id, args.checkpoint_path, args.use_past, args.do_sample, args.max_decode_length) # I love Beijing, because it's a beautiful city. It's a beautiful city. It's a beautiful city. It's a beautiful city. It # An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, ``` 以下为多卡运行自定义多batch推理的脚本 ```bash # >>> `run_predict.sh`文件 CHECKPOINT_PATH=$2 export RANK_TABLE_FILE=$1 # define variable export RANK_SIZE=8 export START_RANK=0 # this server start rank export END_RANK=8 # this server end rank # run for((i=${START_RANK}; i<${END_RANK}; i++)) do export RANK_ID=$i export DEVICE_ID=$((i-START_RANK)) echo "Start distribute running for rank $RANK_ID, device $DEVICE_ID" python3 ./predict_custom.py --use_parallel True --checkpoint_path CHECKPOINT_PATH &> minformers_$RANK_ID.log & done ``` #### 单卡generate推理 ```bash python predict_custom.py ``` #### 多卡generate推理 ```bash bash run_predict.sh RANK_TABLE_FILE path/to/pangualpha_2_6b_shard_checkpoint_dir ``` ### 脚本启动 #### 单卡推理 ```bash python run_mindformer.py --config configs/pangualpha/run_pangualpha_2_6b.yaml --run_mode predict --predict_data 上联：欢天喜地度佳节 下联： --use_parallel False # output result is: [{'text_generation_text': ['上联:欢天喜地度佳节 下联:笑逐颜开迎佳期 横批:幸福快乐']}] ``` **注**：要提高推理速度，可在对应模型配置文件中进行如下配置，设置增量推理`use_past`为True。 ```yaml # model config use_past: True # 开启增量推理 use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "gpt2" repetition_penalty: 1 max_decode_length: 1024 top_k: 3 top_p: 1 do_sample: True ```pengcheng.mind-test/docs/model_cards/index.rst000066400000000000000000000101561471400134000217610ustar00rootroot00000000000000.. role:: raw-html-m2r(raw) :format: html 模型 ========================== **official** .. toctree:: :glob: :maxdepth: 1 bert bloom clip glm gpt2 llama mae pangualpha swin t5 vit 此处给出了MindFormers套件中支持的任务名称和模型名称，用于高阶开发时的索引名 .. list-table:: :header-rows: 1 * - 模型 - 任务（task name） - 模型（model name） * - `BERT `_ - masked_language_modeling\ :raw-html-m2r:`
`\ `text_classification <../task_cards/text_classification.html>`_\ :raw-html-m2r:`
`\ `token_classification <../task_cards/token_classification.html>`_\ :raw-html-m2r:`
`\ `question_answering <../task_cards/question_answering.html>`_ - bert_base_uncased :raw-html-m2r:`
`\ txtcls_bert_base_uncased\ :raw-html-m2r:`
`\ txtcls_bert_base_uncased_mnli :raw-html-m2r:`
`\ tokcls_bert_base_chinese\ :raw-html-m2r:`
`\ tokcls_bert_base_chinese_cluener :raw-html-m2r:`
`\ qa_bert_base_uncased\ :raw-html-m2r:`
`\ qa_bert_base_chinese_uncased * - `T5 `_ - translation - t5_small * - `GPT2 `_ - `text_generation <../task_cards/text_generation.html>`_ - gpt2_small :raw-html-m2r:`
`\ gpt2_13b :raw-html-m2r:`
`\ gpt2_52b * - `PanGuAlpha `_ - `text_generation <../task_cards/text_generation.html>`_ - pangualpha_2_6_b\ :raw-html-m2r:`
`\ pangualpha_13b * - `GLM `_ - `text_generation <../task_cards/text_generation.html>`_ - glm_6b\ :raw-html-m2r:`
`\ glm_6b_lora * - `GLM2 `_ - `text_generation <../task_cards/text_generation.html>`_ - glm2_6b\ :raw-html-m2r:`
`\ glm2_6b_lora * - `LLama `_ - `text_generation <../task_cards/text_generation.html>`_ - llama_7b :raw-html-m2r:`
`\ llama_13b :raw-html-m2r:`
`\ llama_65b :raw-html-m2r:`
`\ llama_7b_lora * - `Bloom `_ - `text_generation <../task_cards/text_generation.html>`_ - bloom_560m\ :raw-html-m2r:`
`\ bloom_7.1b :raw-html-m2r:`
`\ bloom_65b\ :raw-html-m2r:`
`\ bloom_176b * - `MAE `_ - masked_image_modeling - mae_vit_base_p16 * - `VIT `_ - `image_classification <../task_cards/image_classification.html>`_ - vit_base_p16 * - `Swin `_ - `image_classification <../task_cards/image_classification.html>`_ - swin_base_p4w7 * - `CLIP `_ - `contrastive_language_image_pretrain <../task_cards/contrastive_language_image_pretrain.html>`_\ :raw-html-m2r:`
`\ `zero_shot_image_classification <../task_cards/zero_shot_image_classification.html>`_ - clip_vit_b_32\ :raw-html-m2r:`
`\ clip_vit_b_16 :raw-html-m2r:`
`\ clip_vit_l_14\ :raw-html-m2r:`
`\ clip_vit_l_14@336 * - `BLIP2 `_ - `contrastive_language_image_pretrain <../task_cards/contrastive_language_image_pretrain.html>`_\ :raw-html-m2r:`
`\ `zero_shot_image_classification <../task_cards/zero_shot_image_classification.html>`_ - blip2_vit_g **research** .. toctree:: :glob: :maxdepth: 1 ../../research/baichuan/baichuan ../../research/baichuan2/baichuan2 ../../research/internlm/internlm ../../research/ziya/ziya 此处给出了MindFormers套件中支持的任务名称和模型名称，用于高阶开发时的索引名 .. list-table:: :header-rows: 1 * - 模型 - 任务（task name） - 模型（model name） * - `baichuan <../../research/baichuan/baichuan.html>`_ - `text_generation <../task_cards/text_generation.html>`_ - baichuan_7b\ :raw-html-m2r:`
`\ baichuan_13b * - `baichuan2 <../../research/baichuan2/baichuan2.html>`_ - `text_generation <../task_cards/text_generation.html>`_ - baichuan2_7b\ :raw-html-m2r:`
`\ baichuan2_13b * - `internlm <../../research/internlm/internlm.html>`_ - `text_generation <../task_cards/text_generation.html>`_ - internlm_7b :raw-html-m2r:`
`\ internlm_7b_lora * - `ziya <../../research/ziya/ziya.html>`_ - `text_generation <../task_cards/text_generation.html>`_ - ziya_13b pengcheng.mind-test/docs/model_cards/llama.md000066400000000000000000000545411471400134000215360ustar00rootroot00000000000000# LLaMA ## 模型描述 LLaMA是由Meta于2023年发布。LLaMA模型是类GPT模型，是一个生成式的语言模型，主要是用于预测下一个单词。LLaMA目前按照参数量，目前有四个版本：LLaMA-7B（7B）、LLaMA-13B（13B）、LLaMA-33B（33B）以及LLaMA-65B（65B），目前在本仓库中，支持了7B，13B和65B三个规格的模型，权重文件来源于OpenLLaMA。 [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) ``` text @article{touvron2023llama, title={LLaMA: Open and Efficient Foundation Language Models}, author={Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth{\'e}e and Rozi{\`e}re, Baptiste and Goyal, Naman and Hambro, Eric and Azhar, Faisal and Rodriguez, Aurelien and Joulin, Armand and Grave, Edouard and Lample, Guillaume}, journal={arXiv preprint arXiv:2302.13971}, year={2023} } ``` ## 代码结构介绍 `LLaMA` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/llama` ```bash llama ├── __init__.py ├── convert_weight.py # 权重转换脚本 ├── llama.py # 模型实现 ├── llama_config.py # 模型配置项 ├── llama_layer.py # llama网络层定义 ├── llama_processor.py # llama预处理 ├── llama_tokenizer.py # tokenizer └── llama_transformer.py # transformer层实现 ``` 2. 模型配置：`configs/llama` ```bash llama ├── run_llama_7b.yaml # 7b模型全量微调启动配置 ├── run_llama_7b_lora.yaml # 7b lora低参微调启动配置 ├── run_llama_13b.yaml # 13b全量微调启动配置 └── run_llama_65b.yaml # 65b全量微调启动配置 ``` ## 环境要求 - 硬件：Ascend 910A - MindSpore：2.0.0 / 1.10.1 - MindFormers版本：dev 注：推理可在单机单卡上完成部署；全量微调至少需要单机8卡，Lora微调至少需要单卡。 ## 权重转换与权重合并 ### 开源预训练权重转换 从huggingface下载英文预训练权重（权重来源于OpenLLaMA）： - [llama-7b](https://huggingface.co/openlm-research/open_llama_7b) - [llama-13b](https://huggingface.co/openlm-research/open_llama_13b) 注意：65B权重OpenLLaMA未提供，如有需要，请开发者自行解决。 下载完成后，运行如下转换脚本，将huggingface的权重转换为完整的ckpt权重。 ```shell python mindformers/models/llama/convert_weight.py \ --torch_ckpt_dir TORCH_CKPT_DIR \ --mindspore_ckpt_path {path}/MS_CKPT_NAME ``` ```text # 参数说明 torch_ckpt_dir: huggingface权重保存目录路径 mindspore_ckpt_path: 权重保存文件名，可以指定自定义保存路径 ``` ### 分布式训练/微调权重合并 分布式训练/微调后所得到的权重文件为根据策略切分后的权重，需要手动将切分权重合一，以用于评估和推理。 - step 1. 获取模型切分策略文件： 在执行微调脚本时，模型完成编译后，将会在`output/strategy`路径下生成各卡的切分策略文件，用于权重合并。 > 注：lora微调时需要确认配置文件`parallel context config`中`only_trainable_params`为`False`，以获取所有参数完整策略。 - step 2. 运行`mindformers/tools/transform_ckpt.py`脚本进行多卡权重合并： ```shell python transform_ckpt.py \ --src_ckpt_strategy {path}/output/strategy/ \ --src_ckpt_dir {path}/output/checkpoint/ \ --dst_ckpt_dir {path}/target_checkpoint/ \ --prefix llama_7b ``` ```text # 参数说明 src_ckpt_strategy: 步骤1中的切分策略文件路径 src_ckpt_dir: 原切分权重文件夹 dst_ckpt_dir: 目标路径 prefix: ckpt文件前缀名 ``` > 注：`transform_checkpoints` 接口当前仅mindspore 2.0以上版本支持，如当前硬件环境只支持2.0以下版本，可以新建conda环境安装mindspore 2.0的cpu版本以执行该脚本 ## 推理 ### llama推理 以llama-7b模型为例，利用预训练权重进行推理。 #### pipeline推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline(task="text_generation", model="llama_7b", max_length=50) pipeline_result = pipeline_task("I love Beijing, because", top_k=3) print(pipeline_result) # output: # [{'text_generation_text': ['I love Beijing, because it’s a city that’s constantly changing. It’s a city that’s constantly evolving. It’s a city that’s constantly reinventing itself. And I think that’s what makes it']}] ``` #### 基于API接口的推理 ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task="text_generation", model="llama_7b", train_dataset="{dataset file path}") # 方式1：从obs下载训练好的权重并进行推理 res = trainer.predict(input_data="I love Beijing, because") # 方式2：用户自行指定权重路径并进行推理 res = trainer.predict(input_data="I love Beijing, because", predict_checkpoint="{checkpoint file path}") ``` ### lora推理 可直接使用以下两种方式，利用lora微调后的权重进行推理，运行脚本时会自动拉取本仓库提供的lora权重文件。 #### pipeline推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline("text_generation", model="llama_7b_lora", max_length=20) pipeline_result = pipeline_task("I love Beijing, because", top_k=3) print(pipeline_result) ``` #### 基于API接口的推理 ```python from mindformers import Trainer import mindspore as ms cls_trainer = Trainer(task="text_generation", # 已支持的任务名 model="llama_7b", pet_method="lora") # 已支持的模型名 # 根据alpaca数据集的prompt模板，在instruction处填入自定义指令 input_data = "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{}\n\n### Response:".format("Tell me about alpacas.") # 方式1： 传入lora微调后的权重进行推理 lora_ckpt = "./llama_7b_lora.ckpt" predict_result = cls_trainer.predict(input_data=input_data, predict_checkpoint=lora_ckpt) # 方式2： 从obs下载训练好的权重进行推理 predict_result = cls_trainer.predict(input_data=input_data) print(predict_result[0]["text_generation_text"][0]) # output: # Alpacas are a species of South American camelid. They are domesticated animals that are raised for their wool, meat, and milk. Alpacas are gentle, docile animals that are very friendly and easy to care for. They are also very intelligent and can be trained to perform certain tasks. Alpacas are very social animals and live in herds of up to 20 individuals. They are also very vocal and can make a variety of sounds, including a loud, high-pitched bark. ``` #### lora推理结果示例 **Instruction**: Tell me about the president of Mexico in 2019. **Output**: The president of Mexico in 2019 is Andrés Manuel López Obrador. He was elected in 2018 and is the first left-wing president of Mexico in over 70 years. He is known for his populist policies and his focus on improving the lives of the poor and marginalized. He has also been criticized for his lack of transparency and his authoritarian style of leadership. --- **Instruction**: List all Canadian provinces in alphabetical order. **Output**: Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Nova Scotia, Ontario, Prince Edward Island, Quebec, Saskatchewan, and Yukon. --- **Instruction**: Write a Python program that prints the first 10 Fibonacci numbers. **Output**: ```python def fibonacci(n): if n == 0: return 0 elif n == 1: return 1 else: return fibonacci(n-1) + fibonacci(n-2) print(fibonacci(10)) ``` --- **Instruction**: Write a program that prints the numbers from 1 to 100. But for multiples of three print 'Fizz' instead of the number and for the multiples of five print 'Buzz'. For numbers which are multiples of both three and five print 'FizzBuzz'. **Output**: ```python def fizzbuzz(n): if n % 3 == 0 and n % 5 == 0: print('FizzBuzz') elif n % 3 == 0: print('Fizz') elif n % 5 == 0: print('Buzz') else: print(n) ``` --- **Instruction**: Translate the sentence 'I have no mouth but I must scream' into Spanish. **Output**: No tengo boca pero tengo que gritar. --- ## 预训练 ### 数据集准备 以Wikitext2数据集为例: - 数据集下载：[WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip) - 分词模型下载：例如下载huggingface的[tokenizer.model](https://huggingface.co/openlm-research/open_llama_7b/blob/main/tokenizer.model) - 使用以下预处理脚本生成mindrecord训练数据 ```bash # 使用tools/dataset_preprocess/llama/llama_preprocess.py进行数据预处理+Mindrecord数据生成 python llama_preprocess.py \ --dataset_type wiki \ --input_glob /{path}/wiki.train.tokens \ --model_file /{path}/tokenizer.model \ --seq_length 2048 \ --output_file /{path}/wiki2048.mindrecord ``` ### 脚本启动（LLaMA-7B为例） - 用户需要首先clone整个仓库，请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动)完成启动准备工作。 #### 单机多卡启动 - step 1. 在仓库主目录下，运行`mindformers/tools/hccl_tools.py`生成`RANK_TABLE_FILE`。 ```shell # 以八卡运行为例，生成0~7卡的hccl json文件,不包含8本身. python ./mindformers/tools/hccl_tools.py --device_num [0,8] ``` ```python # RANK_TABLE_FILE 参考样例 # 单机8卡 { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "10.155.111.140", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` - step 2. 修改模型对应的配置文件。 在模型对应的配置文件`configs/llama/run_llama_{7/13/65}b.yaml`中，用户可自行修改模型、训练相关参数，并通过`train_dataset`的`dataset_dir`参数，指定训练数据集的路径。 配置文件中各参数含义详见[Config配置说明文档](https://gitee.com/mindspore/mindformers/blob/master/configs/README.md)。 - step3：进入`scripts`文件夹，启动运行脚本，进行8卡分布式运行。 ```shell cd scripts bash run_distribute.sh hccl_xxxx.json ../configs/llama/run_llama_7b.yaml [0,8] train ``` ```text # 脚本启动格式： bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_MODE] # 参数说明 RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的llama/run_llama_7b.yaml配置文件 DEVICE_RANGE: 为单机分布式卡的范围，如[0,8]为8卡分布式，不包含8本身 RUN_MODE: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` #### 多机多卡启动 - step 1. 首先参考单机多卡启动方式，在每台机器上运行`mindformers/tools/hccl_tools.py`生成`RANK_TABLE_FILE`的json文件。 ```shell # 在每个机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/hccl_tools.py --device_num [0,8] ``` - step 2. 合并每台机器上生成的`RANK_TABLE_FILE`。 将不同机器上生成的`RANK_TABLE_FILE`文件拷贝到一起，执行`merge_hccl.py`脚本进行合并，包括server_list合并，`server_count`设为机器数，`rank_id`顺序增加。 ```shell # 运行如下命令，合并每个机器上的RANK_TABLE_FILE文件。 python ./mindformers/tools/merge_hccl.py hccl*.json ``` - step 3. 将合并后的`RANK_TABLE_FILE`文件拷贝到所有机器中，保证不同机器上的`RANK_TABLE_FILE`相同。 - step 4. 根据服务器节点数等信息，修改相应的配置。 ```shell # 以llama-13b模型两机训练为例，默认配置2机16卡，如果节点数有变，需要修改相应的配置。 # 配置文件路径：../configs/llama/run_llama_13b.yaml parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 ``` - step 5. 执行运行脚本。 在多机上同时拉起任务，每台机器拉起方式参考单机多卡启动方式。需注意，多机多卡的拉起方式，相对于单机多卡，多了一个总卡数`[RANK_SIZE]`的入参。 ```shell # 第一台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/llama/run_llama_7b.yaml [0,8] train 16 # 第二台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the second device} ../configs/llama/run_llama_7b.yaml [8,16] train 16 ``` ## 微调 ### 数据集准备 目前提供alpaca数据集的预处理脚本用于全参微调/lora微调任务。 数据集下载链接如下： - [alpaca_data](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) alpaca数据集原始格式样例： ```text # alpaca examples: { "instruction": "Describe a time when you had to make a difficult decision.", "input": "", "output": "I had to make a difficult decision when I was working as a project manager at a construction company. I was in charge of a project that needed to be completed by a certain date in order to meet the client\u2019s expectations. However, due to unexpected delays, we were not able to meet the deadline and so I had to make a difficult decision. I decided to extend the deadline, but I had to stretch the team\u2019s resources even further and increase the budget. Although it was a risky decision, I ultimately decided to go ahead with it to ensure that the project was completed on time and that the client\u2019s expectations were met. The project was eventually successfully completed and this was seen as a testament to my leadership and decision-making abilities." }, { "instruction": "Identify the odd one out.", "input": "Twitter, Instagram, Telegram", "output": "Telegram" }, ``` - step 1. 执行`alpaca_converter.py`，使用fastchat工具添加prompts模板，将原始数据集转换为多轮对话格式。 ``` bash # 脚本路径：tools/dataset_preprocess/llama/alpaca_converter.py # 执行转换脚本 python alpaca_converter.py \ --data_path /{path}/alpaca_data.json \ --output_path /{path}/alpaca-data-conversation.json ``` ```text # 参数说明 data_path: 存放alpaca数据的路径 output_path: 输出转换后对话格式的数据路径 ``` 转换后格式样例： ```text { "id": "1", "conversations": [ { "from": "human", "value": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGive three tips for staying healthy.\n\n### Response:" }, { "from": "gpt", "value": "1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule." } ] }, ``` - step 2. 执行`llama_preprocess.py`，进行数据预处理、Mindrecord数据生成，将带有prompt模板的数据转换为mindrecord格式。 ```bash # 脚本路径：tools/dataset_preprocess/llama/llama_preprocess.py # 由于此工具依赖fschat工具包解析prompt模板，请提前安装fschat >= 0.2.13 python = 3.9 python llama_preprocess.py \ --dataset_type qa \ --input_glob /{path}/alpaca-data-conversation.json \ --model_file /{path}/tokenizer.model \ --seq_length 2048 \ --output_file /{path}/alpaca-fastchat2048.mindrecord ``` ### 全参微调 以llama7b为例 - step 1. 修改`config/llama/run_llama_7b.yaml`中训练数据集路径为微调数据集路径，并在`input_columns`中添加`labels`。 ```python train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "/{path}/alpaca-fastchat2048.mindrecord" shuffle: True input_columns: ["input_ids", "labels"] ``` - step 2. 修改训练时学习率和优化器参数，与预训练不同，微调学习率配置如下： ```python # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.999 eps: 1.e-8 learning_rate: 1.e-5 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset ``` - step 3. 添加预训练权重路径，修改配置文件中的`load_checkpoint`，配置预训练权重路径。 - step 4. 启动微调任务，llama-7b模型以单机八卡为例进行微调，命令如下： ```shell cd scripts bash run_distribute.sh [RANK_TABLE_FILE] ../configs/llama/run_llama_7b.yaml [0,8] finetune ``` 多机多卡微调任务启动参考[预训练章节](#jump)，添加预训练权重，修改启动脚本中的`RUN_MODE`为`finetune`即可。 ### lora微调 目前lora微调适配了llama_7b模型，并给出了默认配置文件`config/llama/run_llama_7b_lora.yaml`。 #### 脚本启动 - step 1. 修改配置文件，参考全参微调修改训练数据集路径与预训练权重路径。 - step 2. 启动lora微调任务。 注：llama_7b_lora模型支持单卡启动，需将配置文件中的`use_parallel`参数置为`False`。 ```shell cd scripts # 单卡启动 bash run_standalone.sh ../configs/llama/run_llama_7b_lora.yaml [DEVICE_ID] finetune # 多卡启动（以单机八卡为例） bash run_distribute.sh [RANK_TABLE_FILE] ../configs/llama/run_llama_7b_lora.yaml [0,8] finetune ``` #### API高阶接口启动 lora微调支持使用高阶接口启动单卡微调任务，示例代码如下： ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task='text_generation', model='llama_7b', pet_method='lora', train_dataset="{dataset file path}") # 调用finetune接口进行微调 trainer.finetune(finetune_checkpoint="{checkpoint file path}") ``` ## 评测 Llama当前支持的评测任务如下： | 任务类型 | 评测指标 | 数据集 | |:----:|:----------:|:---------:| | 文本生成 | Perplexity | WikiText2 | | 阅读理解 | Em/F1 | SQuAD 1.1 | - 文本生成： step 1. 获取数据集 [WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip)是从维基百科上经过验证的优质文章集中提取的超过1亿个token的集合。 step 2. 处理数据成mindrecord格式 ```bash # 使用tools/dataset_preprocess/llama/llama_preprocess.py进行数据预处理+Mindrecord数据生成 python llama_preprocess.py \ --dataset_type wiki \ --input_glob /{path}/wiki.valid.tokens \ --model_file /{path}/tokenizer.model \ --seq_length 2047 \ --output_file /{path}/wiki2048.mindrecord ``` step 3. 开启评测，指标为PPL ```bash python run_mindformer.py \ --config configs/llama/run_llama_7b.yaml \ --eval_dataset_dir /{path}/wiki2048.mindrecord \ --run_mode eval \ --load_checkpoint /{path}/llama_7b.ckpt \ --epochs 1 \ --use_parallel False \ --device_id 0 # PerplexityMetric = {'PerplexityMetric': {'loss': 2.1142693907022476, 'PPL': 8.283531529594038}} ``` - 阅读理解： step 1. 获取数据集 [SQuAD 1.1](https://data.deepai.org/squad1.1.zip)包含针对500+文章的10万+问答对,是一个阅读理解数据集，由维基百科文章上提出的问题组成，其中每个问题的答案都是相应文章中的一段文本。 step 2. 处理数据成mindrecord格式 ```bash # 使用tools/dataset_preprocess/llama/squad_data_process.py进行数据预处理+Mindrecord数据生成 python squad_data_process.py \ --input_file /{path}/squad/dev-v1.1.json \ --output_file /{path}/squad2048.mindrecord \ --mode eval \ --max_length 2048 \ --tokenizer_type "llama_7b" ``` 预处理后数据格式举例： ```text Read the passage and answer the question below. ### Instruction: The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated the Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by defeating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made eight appearances in the Super Bowl. ### Input: Which Carolina Panthers player was named Most Valuable Player? ### Response: Cam Newton ``` step 3. 修改配置文件，eval_dataset的input_columns中增加`labels`，修改metric类型为`EmF1Metric` ```python # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids", "labels"] # 增加"labels" # metric metric: type: EmF1Metric # metric type设为EmF1Metric ``` 此外，要提高推理速度，可以进行如下配置，设置增量推理`use_past`，并限制生成最大长度`max_new_tokens`。 ```python # model config use_past: True # 开启增量推理 pretrain_seqlen: 2048 extend_method: "None" offset: 0 checkpoint_name_or_path: "llama_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False max_new_tokens: 20 #设置最大生成长度 ``` step 4. 开启评测，指标为`Em/F1` ```bash python run_mindformer.py \ --config configs/llama/run_llama_7b.yaml \ --eval_dataset_dir /{path}/squad2048.mindrecord \ --run_mode eval \ --load_checkpoint /{path}/llama_7b.ckpt \ --epochs 1 \ --batch_size 1 \ --use_parallel False \ --device_id 0 # F1 score: 48.48954955952303, Em score: 26.850507982583455, total_count: 2067 ``` pengcheng.mind-test/docs/model_cards/llama2.md000066400000000000000000000446011471400134000216140ustar00rootroot00000000000000# Llama 2 ## 模型描述 Llama 2，是Meta基于LLaMA 1的更新版本，基于新的公开可用数据混合进行训练，同时将预训练语料库的大小增加了40%，最后将模型的上下文长度翻倍（由2048提高到4096），并采用了分组查询注意力机制。Llama 2模型是类GPT模型，是一个生成式的语言模型，主要是用于预测下一个单词。Llama 2按照参数量，目前有三个版本：Llama 2-7B（7B）、Llama 2-13B（13B）、Llama 2-70B（70B），本仓库已全部支持三版权重，权重文件来源于MetaLLama2。Llama 2 的7B和13B 模型结构与LLaMA 1一致，70B 则加入分组查询注意力（GQA）。 [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf%C3%82%C2%A0) ``` text @article{touvron2023llama, title={Llama 2: Open foundation and fine-tuned chat models}, author={Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others}, journal={arXiv preprint arXiv:2307.09288}, year={2023} } ``` ## 代码结构介绍 `Llama 2` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/llama` ```bash llama ├── __init__.py ├── convert_weight.py # 权重转换脚本 ├── llama.py # 模型实现 ├── llama_config.py # 模型配置项 ├── llama_layer.py # llama网络层定义 ├── llama_processor.py # llama预处理 ├── llama_tokenizer.py # tokenizer └── llama_transformer.py # transformer层实现 ``` 2. 模型配置：`configs/llama2` ```bash llama ├── run_llama2_7b.yaml # 7b模型全量微调启动配置 ├── run_llama2_13b.yaml # 13b全量微调启动配置 └── run_llama2_70b.yaml # 70b全量微调启动配置 ``` ## 环境要求 - 硬件：Ascend 910A/910B - MindSpore：2.0 / 2.1 - MindFormers版本：dev 注：推理可在单机单卡上完成部署；全量微调至少需要单机8卡。 ## 权重转换与权重合并 ### 开源预训练权重转换 从huggingface下载英文预训练权重（权重来源于MetaLLama2）： - [llama2-7b](https://huggingface.co/meta-llama/Llama-2-7b) - [llama2-13b](https://huggingface.co/meta-llama/Llama-2-13b) - [llama2-70b](https://huggingface.co/meta-llama/Llama-2-70b) 注：Llama 2的所有权重都需要向Meta提交[申请](https://ai.meta.com/resources/models-and-libraries/llama-downloads)，如有需要，请开发者自行申请。 下载完成后，运行如下转换脚本，将huggingface的权重转换为完整的ckpt权重。 ```shell python mindformers/models/llama/convert_weight.py \ --torch_ckpt_dir TORCH_CKPT_DIR \ --mindspore_ckpt_path {path}/MS_CKPT_NAME ``` ```text # 参数说明 torch_ckpt_dir: huggingface权重保存目录路径 mindspore_ckpt_path: 权重保存文件名，可以指定自定义保存路径 ``` ### 分布式训练/微调权重合并 分布式训练/微调后所得到的权重文件为根据策略切分后的权重，需要手动将切分权重合一，以用于评估和推理。 - step 1. 获取模型切分策略文件： 在执行微调脚本时，模型完成编译后，将会在`output/strategy`路径下生成各卡的切分策略文件，用于权重合并。 - step 2. 运行`mindformers/tools/transform_ckpt.py`脚本进行多卡权重合并： ```shell python transform_ckpt.py \ --src_ckpt_strategy {path}/output/strategy/ \ --src_ckpt_dir {path}/output/checkpoint/ \ --dst_ckpt_dir {path}/target_checkpoint/ \ --prefix llama2_7b ``` ```text # 参数说明 src_ckpt_strategy: 步骤1中的切分策略文件路径 src_ckpt_dir: 原切分权重文件夹 dst_ckpt_dir: 目标路径 prefix: ckpt文件前缀名 ``` > 注：`transform_checkpoints` 接口当前仅mindspore 2.0以上版本支持，如当前硬件环境只支持2.0以下版本，可以新建conda环境安装mindspore 2.0的cpu版本以执行该脚本 ## 推理 ### Llama 2推理 以llama2-7b模型为例，利用预训练权重进行推理。 #### pipeline推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline(task="text_generation", model="llama2_7b", max_length=50) pipeline_result = pipeline_task("I love Beijing, because", top_k=3) print(pipeline_result) # output: # [{'text_generation_text': ['I love Beijing, because it’s a city that’s constantly changing. It’s a city that’s constantly evolving. It’s a city that’s constantly reinventing itself. And I think that’s what makes it']}] ``` #### 基于API接口的推理 ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task="text_generation", model="llama2_7b", train_dataset="{dataset file path}") # 方式1：从obs下载训练好的权重并进行推理 res = trainer.predict(input_data="I love Beijing, because") # 方式2：用户自行指定权重路径并进行推理 res = trainer.predict(input_data="I love Beijing, because", predict_checkpoint="{checkpoint file path}") ``` ## 预训练 ### 数据集准备 以Wikitext2数据集为例: - 数据集下载：[WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip) - 分词模型下载：例如下载申请通过后huggingface里对应Files 中的tokenizer.model - 使用以下预处理脚本生成mindrecord训练数据 ```bash # 使用tools/dataset_preprocess/llama/llama_preprocess.py进行数据预处理+Mindrecord数据生成 python llama_preprocess.py \ --dataset_type wiki \ --input_glob /{path}/wiki.train.tokens \ --model_file /{path}/tokenizer.model \ --seq_length 4096 \ --output_file /{path}/wiki4096.mindrecord ``` ### 脚本启动（Llama 2-7B为例） - 用户需要首先clone整个仓库，请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动)完成启动准备工作。 #### 单机多卡启动 - step 1. 在仓库主目录下，运行`mindformers/tools/hccl_tools.py`生成`RANK_TABLE_FILE`。 ```shell # 以八卡运行为例，生成0~7卡的hccl json文件,不包含8本身. python ./mindformers/tools/hccl_tools.py --device_num [0,8] ``` ```python # RANK_TABLE_FILE 参考样例 # 单机8卡 { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "10.155.111.140", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` - step 2. 修改模型对应的配置文件。 在模型对应的配置文件`configs/llama2/run_llama2_{7/13/70}b.yaml`中，用户可自行修改模型、训练相关参数，并通过`train_dataset`的`dataset_dir`参数，指定训练数据集的路径。 配置文件中各参数含义详见[Config配置说明文档](https://gitee.com/mindspore/mindformers/blob/master/configs/README.md)。 - step3：进入`scripts`文件夹，启动运行脚本，进行8卡分布式运行。 ```shell cd scripts bash run_distribute.sh hccl_xxxx.json ../configs/llama2/run_llama2_7b.yaml [0,8] train ``` ```text # 脚本启动格式： bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_MODE] # 参数说明 RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的llama2/run_llama2_7b.yaml配置文件 DEVICE_RANGE: 为单机分布式卡的范围，如[0,8]为8卡分布式，不包含8本身 RUN_MODE: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` #### 多机多卡启动 - step 1. 首先参考单机多卡启动方式，在每台机器上运行`mindformers/tools/hccl_tools.py`生成`RANK_TABLE_FILE`的json文件。 ```shell # 在每个机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/hccl_tools.py --device_num [0,8] ``` - step 2. 合并每台机器上生成的`RANK_TABLE_FILE`。 将不同机器上生成的`RANK_TABLE_FILE`文件拷贝到一起，执行`merge_hccl.py`脚本进行合并，包括server_list合并，`server_count`设为机器数，`rank_id`顺序增加。 ```shell # 运行如下命令，合并每个机器上的RANK_TABLE_FILE文件。 python ./mindformers/tools/merge_hccl.py hccl*.json ``` - step 3. 将合并后的`RANK_TABLE_FILE`文件拷贝到所有机器中，保证不同机器上的`RANK_TABLE_FILE`相同。 - step 4. 根据服务器节点数等信息，修改相应的配置。 ```shell # 以llama2-13b模型两机训练为例，默认配置2机16卡，如果节点数有变，需要修改相应的配置。 # 配置文件路径：../configs/llama2/run_llama2_13b.yaml parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 ``` - step 5. 执行运行脚本。 在多机上同时拉起任务，每台机器拉起方式参考单机多卡启动方式。需注意，多机多卡的拉起方式，相对于单机多卡，多了一个总卡数`[RANK_SIZE]`的入参。 ```shell # 第一台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/llama2/run_llama2_13b.yaml [0,8] train 16 # 第二台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the second device} ../configs/llama2/run_llama2_13b.yaml [8,16] train 16 ``` ## 微调 ### 数据集准备 目前提供alpaca数据集的预处理脚本用于全参微调任务。 数据集下载链接如下： - [alpaca_data](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) alpaca数据集原始格式样例： ```text # alpaca examples: { "instruction": "Describe a time when you had to make a difficult decision.", "input": "", "output": "I had to make a difficult decision when I was working as a project manager at a construction company. I was in charge of a project that needed to be completed by a certain date in order to meet the client\u2019s expectations. However, due to unexpected delays, we were not able to meet the deadline and so I had to make a difficult decision. I decided to extend the deadline, but I had to stretch the team\u2019s resources even further and increase the budget. Although it was a risky decision, I ultimately decided to go ahead with it to ensure that the project was completed on time and that the client\u2019s expectations were met. The project was eventually successfully completed and this was seen as a testament to my leadership and decision-making abilities." }, { "instruction": "Identify the odd one out.", "input": "Twitter, Instagram, Telegram", "output": "Telegram" }, ``` - step 1. 执行`alpaca_converter.py`，使用fastchat工具添加prompts模板，将原始数据集转换为多轮对话格式。 ``` bash # 脚本路径：tools/dataset_preprocess/llama/alpaca_converter.py # 执行转换脚本 python alpaca_converter.py \ --data_path /{path}/alpaca_data.json \ --output_path /{path}/alpaca-data-conversation.json ``` ```text # 参数说明 data_path: 存放alpaca数据的路径 output_path: 输出转换后对话格式的数据路径 ``` 转换后格式样例： ```text { "id": "1", "conversations": [ { "from": "human", "value": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGive three tips for staying healthy.\n\n### Response:" }, { "from": "gpt", "value": "1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule." } ] }, ``` - step 2. 执行`llama_preprocess.py`，进行数据预处理、Mindrecord数据生成，将带有prompt模板的数据转换为mindrecord格式。 ```bash # 脚本路径：tools/dataset_preprocess/llama/llama_preprocess.py # 由于此工具依赖fschat工具包解析prompt模板，请提前安装fschat >= 0.2.13 python = 3.9 python llama_preprocess.py \ --dataset_type qa \ --input_glob /{path}/alpaca-data-conversation.json \ --model_file /{path}/tokenizer.model \ --seq_length 2048 \ --output_file /{path}/alpaca-fastchat2048.mindrecord ``` ### 全参微调 以llama2 7b为例 - step 1. 修改`config/llama2/run_llama2_7b.yaml`中训练数据集路径为微调数据集路径，并在`input_columns`中添加`labels`。 ```python train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "/{path}/alpaca-fastchat2048.mindrecord" shuffle: True input_columns: ["input_ids", "labels"] ``` - step 2. 修改训练时学习率和优化器参数，与预训练不同，微调学习率配置如下： ```python # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.999 eps: 1.e-8 learning_rate: 1.e-5 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset ``` - step 3. 修改seq_length，seq_length配置如下： ```python # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 ``` ​ 注意：alpaca数据集最长不超过2048，因此seq_length采用2048即可。 - step 3. 添加预训练权重路径，修改配置文件中的`load_checkpoint`，配置预训练权重路径。 - step 4. 启动微调任务，llama2-7b模型以单机八卡为例进行微调，命令如下： ```shell cd scripts bash run_distribute.sh [RANK_TABLE_FILE] ../configs/llama2/run_llama2_7b.yaml [0,8] finetune ``` 多机多卡微调任务启动参考[预训练章节](#jump)，添加预训练权重，修改启动脚本中的`RUN_MODE`为`finetune`即可。 ## 评测 Llama 2当前支持的评测任务如下： | 任务类型 | 评测指标 | 数据集 | | :------: | :--------: | :-------: | | 文本生成 | Perplexity | WikiText2 | | 阅读理解 | Em/F1 | SQuAD 1.1 | - 文本生成： step 1. 获取数据集 [WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip)是从维基百科上经过验证的优质文章集中提取的超过1亿个token的集合。 step 2. 处理数据成mindrecord格式 ```bash # 使用tools/dataset_preprocess/llama/llama_preprocess.py进行数据预处理+Mindrecord数据生成 python llama_preprocess.py \ --dataset_type wiki \ --input_glob /{path}/wiki.valid.tokens \ --model_file /{path}/tokenizer.model \ --seq_length 4095 \ --output_file /{path}/wiki4096.mindrecord ``` step 3. 开启评测，指标为PPL ```bash python run_mindformer.py \ --config configs/llama2/run_llama2_7b.yaml \ --eval_dataset_dir /{path}/wiki4096.mindrecord \ --run_mode eval \ --load_checkpoint /{path}/llama2_7b.ckpt \ --epochs 1 \ --use_parallel False \ --device_id 0 # PerplexityMetric = {'PerplexityMetric': {'loss': 2.1142693907022476, 'PPL': 8.283531529594038}} ``` - 阅读理解： step 1. 获取数据集 [SQuAD 1.1](https://data.deepai.org/squad1.1.zip)包含针对500+文章的10万+问答对,是一个阅读理解数据集，由维基百科文章上提出的问题组成，其中每个问题的答案都是相应文章中的一段文本。 step 2. 处理数据成mindrecord格式 ```bash # 使用tools/dataset_preprocess/llama/squad_data_process.py进行数据预处理+Mindrecord数据生成 python squad_data_process.py \ --input_file /{path}/squad/dev-v1.1.json \ --output_file /{path}/squad2048.mindrecord \ --mode eval \ --max_length 2048 \ --tokenizer_type "llama2_7b" ``` 预处理后数据格式举例： ```text Read the passage and answer the question below. ### Instruction: The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated the Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by defeating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made eight appearances in the Super Bowl. ### Input: Which Carolina Panthers player was named Most Valuable Player? ### Response: Cam Newton ``` step 3. 修改配置文件，eval_dataset的input_columns中增加`labels`，修改metric类型为`EmF1Metric`，修改seq_length为`2048` ```python # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids", "labels"] # 增加"labels" # metric metric: type: EmF1Metric # metric type设为EmF1Metric # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 ``` 此外，要提高推理速度，可以进行如下配置，设置增量推理`use_past`，并限制生成最大长度`max_new_tokens`。 ```python # model config use_past: True # 开启增量推理 pretrain_seqlen: 4096 extend_method: "None" offset: 0 checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False max_new_tokens: 20 #设置最大生成长度 ``` step 4. 开启评测，指标为`Em/F1` ```bash python run_mindformer.py \ --config configs/llama2/run_llama2_7b.yaml \ --eval_dataset_dir /{path}/squad2048.mindrecord \ --run_mode eval \ --load_checkpoint /{path}/llama2_7b.ckpt \ --epochs 1 \ --batch_size 1 \ --use_parallel False \ --device_id 0 # F1 score: 48.48954955952303, Em score: 26.850507982583455, total_count: 2067 ``` pengcheng.mind-test/docs/model_cards/mae.md000066400000000000000000000110631471400134000212020ustar00rootroot00000000000000# MAE ## 模型描述 MAE是一种基于MIM（Masked Imange Modeling）的无监督学习方法。 MAE由何凯明团队提出，将NLP领域大获成功的自监督预训练模式用在了计算机视觉任务上，效果拔群，在NLP和CV两大领域间架起了一座更简便的桥梁。 [论文](https://gitee.com/link?target=https%3A%2F%2Farxiv.org%2Fabs%2F2111.06377): He, Kaiming et al. “Masked Autoencoders Are Scalable Vision Learners.” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 15979-15988. ## 数据集准备 使用的数据集：[ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```text 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 ```shell # pretrain python run_mindformer.py --config ./configs/mae/run_mae_vit_base_p16_224_800ep.yaml --run_mode train # predict python run_mindformer.py --config ./configs/mae/run_mae_vit_base_p16_224_800ep.yaml --run_mode predict --predict_data [PATH_TO_IMAGE] ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import ViTMAEForPreTraining, ViTMAEConfig ViTMAEForPreTraining.show_support_list() # 输出： # - support list of ViTMAEForPreTraining is: # - ['mae_vit_base_p16'] # - ------------------------------------- # 模型标志加载模型 model = ViTMAEForPreTraining.from_pretrained("mae_vit_base_p16") #模型配置加载模型 config = ViTMAEConfig.from_pretrained("mae_vit_base_p16") # {'decoder_dim': 512, 'patch_size': 16, 'in_chans': 3, 'embed_dim': 768, 'depth': 12, # ..., 'decoder_embed_dim': 512, 'norm_pixel_loss': True, 'window_size': None} model = ViTMAEForPreTraining(config) ``` - Trainer接口开启训练/推理： ```python from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 初始化任务 mae_trainer = Trainer( task='masked_image_modeling', model='mae_vit_base_p16', train_dataset="imageNet-1k/train") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1: 重头开始训练，并使用训练好的权重进行推理 mae_trainer.train() # 开启训练 predict_result = mae_trainer.predict(predict_checkpoint=True, input_data=img) print(predict_result) # 方式3： 从obs下载训练好的权重并进行推理 predict_result = mae_trainer.predict(input_data=img) print(predict_result) ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("masked_image_modeling", model='mae_vit_base_p16') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img) ``` Trainer和pipeline接口默认支持的task和model关键入参 | task（string） | model（string） | |:---------------------:|:----------------:| | masked_image_modeling | mae_vit_base_p16 | ## 模型性能 | model | type | pretrain | Datasets | Top1-Accuracy | Log | pretrain_config | finetune_config | |:-----:|:----------------:|:--------------------:|:-----------:|:-------------:|:---:|:--------------------------------------------------:|:-----------------------------------------------------:| | mae | mae_vit_base_p16 | [mae_vit_base_p16]() | ImageNet-1K | 83.71% | \ | [link](../../configs/mae/run_mae_vit_base_p16_224_800ep.yaml) | [link](../../configs/vit/run_vit_base_p16_100ep.yaml) | ## 模型权重 本仓库中的`mae_vit_base_p16`来自于facebookresearch/mae的[`ViT-Base`](https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base.pth), 基于下述的步骤获取： 1. 从上述的链接中下载`ViT-Base`的模型权重 2. 执行转换脚本，得到转换后的输出文件`mae_vit_base_p16.ckpt` ```python python mindformers/models/mae/convert_weight.py --torch_path "PATH OF ViT-Base.pth" --mindspore_path "SAVE PATH OF mae_vit_base_p16.ckpt" ```pengcheng.mind-test/docs/model_cards/pangualpha.md000066400000000000000000000250601471400134000225620ustar00rootroot00000000000000# PanguAlpha ## 模型描述 「鹏程·盘古α」由以鹏城实验室为首的技术团队联合攻关，首次基于“鹏城云脑Ⅱ”和国产MindSpore框架的自动混合并行模式实现在2048卡算力集群上的大规模分布式训练，训练出业界首个2000亿参数以中文为核心的预训练生成语言模型。鹏程·盘古α预训练模型支持丰富的场景应用，在知识问答、知识检索、知识推理、阅读理解等文本生成领域表现突出，具备很强的小样本学习能力。 [论文](https://arxiv.org/abs/2104.12369)J Wei Zeng, Xiaozhe Ren, Teng Su，et al., PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation, 2021 ## 数据集准备 以悟道数据集为例 - 数据集下载：[悟道数据集](https://data.baai.ac.cn/details/WuDaoCorporaText#a2) - 词表下载：[model.vocab](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PanGu-Alpha/src/branch/master/tokenizer/vocab.model) - 参考[ModelZoo](https://gitee.com/mindspore/models/tree/master/official/nlp/Pangu_alpha#%E6%95%B0%E6%8D%AE%E9%9B%86%E7%94%9F%E6%88%90)，将数据处理成Mindrecord格式。注：训练数据处理时，长度应等于模型接收长度加一 ```bash # 数据预处理示例代码，代码来源于ModelZoo # 生成Mindrecord数据，其中output_file需以字符串mindrecord结尾 python -m preprocess.py --input_glob 'data/*.txt' --tokenizer jieba --eot 40000 --data_column_name input_ids --seq_length 1025 ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) 示例命令如下，将会执行2.6b大小的pangualpha模型训练 #### 单卡启动 ```shell python run_mindformer.py --config configs/pangualpha/run_pangualpha_2_6b.yaml \ --run_mode train \ --device_target Ascend \ --train_dataset_dir /your_path/wudao-mindrecord ``` 其中`device_target`根据用户的运行设备不同，可选`GPU/Ascend`。另，模型和训练等相关配置可在`configs/pangualpha`目录下的yaml文件中配置。 #### 单机多卡启动 - 运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； ```shell # step1：机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" # step2：# 执行运行脚本：8卡分布式运行， DEVICE_RANGE = [0, 8]， 不包含8本身。 cd scripts bash run_distribute.sh RANK_TABLE_FILE CONFIG_PATH DEVICE_RANGE RUN_STATUS ``` ```python # RANK_TABLE_FILE 参考样例 # 单机8卡 { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "10.155.111.140", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ```text # 参数说明 RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的pangualpha/run_pangualpha_*.yaml配置文件 DEVICE_ID: 为设备卡，范围为0~7 DEVICE_RANGE: 为单机分布式卡的范围，如[0,8]为8卡分布式，不包含8本身 RUN_STATUS: 为任务运行状态，支持关键字 train\finetune\predict ``` 其中，模型和训练等相关配置可在`configs/pangualpha`目录下的yaml文件中配置，如数据集路径，可在`configs/pangualpha/run_pangualpha_*.yaml`中配置`dataset_dir`参数。 #### 多机多卡启动 - 首先参考单机多卡启动方式，在每台机器上运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； - 执行merge_hccl.py脚本将不同机器上生成的RANK_TABLE_FILE文件中的hccl*.json进行合并，包括server_list合并，server_count设为机器数，rank_id顺序增加，并保证不同机器上的RANK_TABLE_FILE相同； - 在多机上同时拉起任务，每台机器拉起方式参考单机多卡启动方式，需注意的是，多机多卡的拉起方式，相对于单机多卡，多了一个总卡数`[RANK_SIZE]`的入参。 ```shell # step1：在每个机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" # step2：运行如下命令，合并每个机器上的RANK_TABLE_FILE文件。 python ./mindformers/tools/merge_hccl.py hccl*.json # step3：将step2得到的合并后的RANK_TABLE_FILE文件分别复制到所有的机器上。 # step4：根据服务器节点数等信息，修改相应的配置 ''' 以pangualpha-13b模型两机训练为例，默认配置2机16卡，如果节点数有变，需要修改相应的配置。配置文件在../configs/pangualpha/run_pangualpha_13b.yaml parallel_config: data_parallel: 4 model_parallel: 4 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 ''' # step5：执行运行脚本 # 第一台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/pangualpha/run_pangualpha_13b.yaml [0,8] train 32 # 第二台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the second device} ../configs/pangualpha/run_pangualpha_13b.yaml [8,16] train 32 # 第三台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the third device} ../configs/pangualpha/run_pangualpha_13b.yaml [16,24] train 32 # 第四台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the forth device} ../configs/pangualpha/run_pangualpha_13b.yaml [24,32] train 32 ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import PanguAlphaHeadModel, PanguAlphaTokenizer model = PanguAlphaHeadModel.from_pretrained('pangualpha_2_6b') model.set_train(False) tokenizer = PanguAlphaTokenizer.from_pretrained('pangualpha_2_6b') inputs = tokenizer(["今天天气很好"], padding='max_length', max_length=model.config.seq_length, return_tensors='ms') output = model(inputs["input_ids"]) print(output) # 计算输出的logits model.set_train(True) inputs = tokenizer(["今天天气很好"], padding='max_length', max_length=model.config.seq_length+1, return_tensors='ms') output = model(inputs["input_ids"]) print(output) # 计算loss ``` - Trainer接口开启训练/推理： ```python from mindformers.trainer import Trainer # 方式1: 开启训练，并使用训练完成后的权重进行推理 trainer = Trainer(task='text_generation', model='pangualpha_2_6b', train_dataset="your data file path") trainer.train() res = trainer.predict(predict_checkpoint=True, input_data="我喜欢北京，因为") # 方式2： 从obs下载训练好的权重并进行推理 trainer = Trainer(task='text_generation', model='pangualpha_2_6b') res = trainer.predict(input_data="我喜欢北京，因为") ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline("text_generation", model='pangualpha_2_6b', max_length=50) pipeline_result = pipeline_task("我喜欢北京，因为", top_k=3) print(pipeline_result) ``` **注：**，要提高推理速度，可在`config`yaml文件中进行如下配置，设置增量推理`use_past`为True。 ```python # model config use_past: True # 开启增量推理 use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_2_6b" repetition_penalty: 1 max_decode_length: 1024 top_k: 3 top_p: 1 do_sample: False ``` ## 模型权重下载 [盘古Alpha权重下载](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PanGu-Alpha) ## 评测 盘古α支持TNEWS和CMRC2018两个中文数据集的评测。 - 文本分类： - 获取数据集: - [TNEWS数据集](https://storage.googleapis.com/cluebenchmark/tasks/tnews_public.zip)自今日头条的新闻版块，共提取了15个类别的新闻，包括旅游，教育，金融，军事等。 - 处理数据成mindrecord格式 - TNEWS： ```bash # 注：生成的数据集文件需以.mindrecord结尾 cd mindformers/tools/dataset_preprocess/pangualpha python tnews_data_process.py --input_file {your_path/dev.json} \ --label_file {your_path/labels.json} \ --output_file {your_path/tnews.mindrecord} ``` - 开启评测：评测指标为ACC - TNEWS ```bash python run_mindformers.py --config configs/pangualpha/run_pangualpha_2_6b_prompt_txtcls.yaml \ --eval_dataset_dir {your_path/tnews.mindrecord} \ --run_mode eval # ACC: 0.646, total_acc_num: 6458, total_num: 10000 ``` - 阅读理解： - 获取数据集: - [CMRC2018数据集](https://storage.googleapis.com/cluebenchmark/tasks/cmrc2018_public.zip)是用于中文机器阅读理解的片段抽取任务(Span-Extraction)的数据，这个数据集由近20000个真实的问题组成，这些问题由人类专家在维基百科的段落中注释。 - 处理数据成mindrecord格式 - CMRC2018： ```bash # 注：生成的数据集文件需以.mindrecord结尾 cd mindformers/tools/dataset_preprocess/pangualpha python wikitext2_data_process.py --train_file {your_path/train.json} \ --dev_file {your_path/dev.json} \ --output_file {your_path/cmrc2018.mindrecord} ``` - 开启评测：评测指标为Em/F1 - CMRC2018 ```bash python run_mindformers.py --config configs/pangualpha/run_pangualpha_2_6b_em_f1.yaml \ --eval_dataset_dir {your_path/cmrc2018.mindrecord} \ --run_mode eval # F1 score: 21.12, Em score: 2.10, total_count: 1806 ``` pengcheng.mind-test/docs/model_cards/swin.md000066400000000000000000000132241471400134000214210ustar00rootroot00000000000000# SWIN ## 模型描述 swin：全名swin transformer，是一个基于Transformer在视觉领域有着SOTA表现的深度学习模型。比起VIT拥有更好的性能和精度。 [论文](https://arxiv.org/abs/2103.14030) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo, 2021 ## 数据集准备 使用的数据集：[ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```bash 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 ```shell # pretrain python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode train --train_dataset_dir [DATASET_PATH] # evaluate python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode eval --eval_dataset_dir [DATASET_PATH] # predict python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode predict --predict_data [PATH_TO_IMAGE] ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import SwinForImageClassification, SwinConfig SwinForImageClassification.show_support_list() # 输出： # - support list of SwinForImageClassification is: # - ['swin_base_p4w7'] # - ------------------------------------- # 模型标志加载模型 model = SwinForImageClassification.from_pretrained("swin_base_p4w7") #模型配置加载模型 config = SwinConfig.from_pretrained("swin_base_p4w7") # {'batch_size': 128, 'image_size': 224, 'patch_size': 4, 'num_labels': 1000, 'num_channels': 3, # 'embed_dim': 128, 'depths': [2, 2, 18, 2], 'num_heads': [4, 8, 16, 32], # 'checkpoint_name_or_path': 'swin_base_p4w7'} model = SwinForImageClassification(config) ``` - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 初始化任务 swin_trainer = Trainer( task='image_classification', model='swin_base_p4w7', train_dataset="imageNet-1k/train", eval_dataset="imageNet-1k/val") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1：开启训练，并使用训练好的权重进行eval和推理 swin_trainer.train() swin_trainer.evaluate(eval_checkpoint=True) predict_result = swin_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式2： 从obs下载训练好的权重并进行eval和推理 swin_trainer.evaluate() # 下载权重进行评估 predict_result = swin_trainer.predict(input_data=img, top_k=3) # 下载权重进行推理 print(predict_result) # 输出 # - mindformers - INFO - output result is: [[{'score': 0.89573187, 'label': 'daisy'}, # {'score': 0.005366202, 'label': 'bee'}, {'score': 0.0013296203, 'label': 'fly'}]] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("image_classification", model='swin_base_p4w7') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img, top_k=3) print(pipeline_result) # 输出 # [[{'score': 0.89573187, 'label': 'daisy'}, {'score': 0.005366202, 'label': 'bee'}, # {'score': 0.0013296203, 'label': 'fly'}]] ``` Trainer和pipeline接口默认支持的task和model关键入参 | task（string） | model（string） | |:--------------:| :-------------: | | image_classification | swin_base_p4w7 | ## 模型性能 | model | type | pretrain | Datasets | Top1-Accuracy | Log | pretrain_config | finetune_config | |:-----:|:--------------:|:--------:| :----: |:-------------:| :---: |:---------------:|:--------------------------------------------------------:| | swin | swin_base_p4w7 | \ | ImageNet-1K | 83.44% | \ | \ | [link](../../configs/swin/run_swin_base_p4w7_100ep.yaml) | ## 模型权重 本仓库中的`swin_base_p4w7`来自于MicroSoft的[`Swin-Transformer`](https://github.com/microsoft/Swin-Transformer), 基于下述的步骤获取： 1. 从上述的链接中下载[`swin_base_p4w7`](https://pan.baidu.com/s/16bqCTEc70nC_isSsgBSaqQ?pwd=swin)的官方权重，文件名为`swin_base_patch4_window7_224.pth` 2. 执行转换脚本，得到转换后的输出文件`swin_base_p4w7.ckpt` ```shell python mindformers/models/swin/convert_weight.py --torch_path swin_base_patch4_window7_224.pth --mindspore_path swin_base_p4w7.ckpt --is_pretrain False ``` 如需转换官方simmim的预训练权重进行finetune，则执行如下步骤： 1. [从simmim官网](https://github.com/microsoft/SimMIM)提供的google网盘下载[`simmim_swin_192`](https://drive.google.com/file/d/1Wcbr66JL26FF30Kip9fZa_0lXrDAKP-d/view?usp=sharing)的官方权重，文件名为`simmim_pretrain__swin_base__img192_window6__100ep.pth` 2. 执行转换脚本，得到转换后的输出文件`simmim_swin_p4w6.ckpt` ```shell python mindformers/models/swin/convert_weight.py --torch_path simmim_pretrain__swin_base__img192_window6__100ep.pth --mindspore_path simmim_swin_p4w6.ckpt --is_pretrain True ```pengcheng.mind-test/docs/model_cards/t5.md000066400000000000000000000101341471400134000207660ustar00rootroot00000000000000# T5 ## 模型描述 T5:全名`Text-to-Text Transfer Transformer`模型是谷歌在2019年基于C4数据集训练的Transformer模型。 [论文](https://arxiv.org/abs/1910.10683)C Raffel，N Shazeer，A Roberts，K Lee，S Narang，M Matena，Y Zhou，W Li，PJ Liu, 2020 ## 数据集准备 使用的数据集：[WMT16](https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz) 对应的文件路径如下： ```bash └── wmt_en_ro ├── test.source ├── test.target ├── train.source ├── train.target ├── val.source └── val.target ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) 示例命令如下，将会执行一个只有1层的T5模型训练 ```shell python run_mindformer.py --config configs/t5/run_t5_tiny_on_wmt16.yaml --run_mode train \ --device_target Ascend \ --train_dataset_dir /your_path/wmt_en_ro ``` 其中`device_target`根据用户的运行设备不同，可选`GPU/Ascend/CPU`。`config`的入参还可以为`configs/t5/run_t5_small.yaml`，在 这个配置下将会加载`t5_small`的权重并且开始执行微调。 ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) #### Model调用接口 - 模型计算Loss ```python from mindformers import T5ForConditionalGeneration, T5Tokenizer model = T5ForConditionalGeneration.from_pretrained('t5_small') tokenizer = T5Tokenizer.from_pretrained('t5_small') src_output = tokenizer(["hello world"], padding='max_length', max_length=model.config.seq_length, return_tensors='ms') model_input = tokenizer(["So happy to see you!"], padding='max_length', max_length=model.config.max_decode_length, return_tensors='ms')["input_ids"] input_ids = src_output['input_ids'] attention_mask = src_output['attention_mask'] output = model(input_ids, attention_mask, model_input) print(output) # [5.64458] ``` - 推理 执行下述的命令，可以自动云上拉取`t5_small`模型并且进行推理。 ```python from mindformers import T5ForConditionalGeneration, T5Tokenizer t5 = T5ForConditionalGeneration.from_pretrained("t5_small") tokenizer = T5Tokenizer.from_pretrained("t5_small") words = tokenizer("translate the English to the Romanian: UN Chief Says There Is No Military " "Solution in Syria")['input_ids'] output = t5.generate(words, do_sample=False) output = tokenizer.decode(output, skip_special_tokens=True) print(output) # "eful ONU declară că nu există o soluţie militară în Siri" ``` - Trainer接口开启训练/预测： ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task='translation', model='t5_small', train_dataset="your data file path") # 方式1: 开启训练，并使用训练好的权重进行推理 trainer.train() res = trainer.predict(predict_checkpoint=True, input_data="translate the English to Romanian: a good boy!") print(res) #[{'translation_text': ['un băiat bun!']}] # 方式2： 从obs下载训练好的权重并进行推理 res = trainer.predict(input_data="translate the English to Romanian: a good boy!") print(res) #[{'translation_text': ['un băiat bun!']}] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline("translation", model='t5_small') pipeline_result = pipeline_task("translate the English to Romanian: a good boy!", top_k=3) print(pipeline_result) #[{'translation_text': ['un băiat bun!']}] ``` ## 模型权重 本仓库中的`t5_small`来自于HuggingFace的[`t5_small`](https://huggingface.co/t5-small), 基于下述的步骤获取： 1. 从上述的链接中下载`t5_small`的HuggingFace权重，文件名为`pytorch_model.bin` 2. 执行转换脚本，得到转换后的输出文件`mindspore_t5.ckpt` ```shell python mindformers/models/t5/convert_weight.py --layers 6 --torch_path pytorch_model.bin --mindspore_path ./mindspore_t5.ckpt ``` pengcheng.mind-test/docs/model_cards/vit.md000066400000000000000000000124171471400134000212460ustar00rootroot00000000000000# ViT ## 模型描述 vit：全名vision transformer，不同于传统的基于CNN的网络结果，是基于transformer结构的cv网络，2021年谷歌研究发表网络，在大数据集上表现了非常强的泛化能力。大数据任务（如clip）基于该结构能有良好的效果。mindformers提供的Vit权重及精度均是是基于MAE预训练ImageNet-1K数据集进行微调得到。 [论文](https://gitee.com/link?target=https%3A%2F%2Farxiv.org%2Fabs%2F2010.11929): Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. 2021. ## 数据集准备 使用的数据集：[ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```text 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 ```shell # pretrain python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode train # evaluate python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode eval --eval_dataset_dir [DATASET_PATH] # predict python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode predict --predict_data [PATH_TO_IMAGE] ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import ViTForImageClassification, ViTConfig ViTForImageClassification.show_support_list() # 输出： # - support list of ViTForImageClassification is: # - ['vit_base_p16'] # - ------------------------------------- # 模型标志加载模型 model = ViTForImageClassification.from_pretrained("vit_base_p16") #模型配置加载模型 config = ViTConfig.from_pretrained("vit_base_p16") # {'patch_size': 16, 'in_chans': 3, 'embed_dim': 768, 'depth': 12, 'num_heads': 12, 'mlp_ratio': 4, # ..., 'batch_size': 32, 'image_size': 224, 'num_classes': 1000} model = ViTForImageClassification(config) ``` - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 初始化任务 vit_trainer = Trainer( task='image_classification', model='vit_base_p16', train_dataset="imageNet-1k/train", eval_dataset="imageNet-1k/val") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 vit_trainer.train(resume_or_finetune_from_checkpoint="mae_vit_base_p16", do_finetune=True) vit_trainer.evaluate(eval_checkpoint=True) predict_result = vit_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式2: 重头开始训练，并使用训练好的权重进行eval和推理 vit_trainer.train() vit_trainer.evaluate(eval_checkpoint=True) predict_result = vit_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式3： 从obs下载训练好的权重并进行eval和推理 vit_trainer.evaluate() predict_result = vit_trainer.predict(input_data=img, top_k=3) print(predict_result) ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("image_classification", model='vit_base_p16') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img, top_k=3) ``` Trainer和pipeline接口默认支持的task和model关键入参 | task（string） | model（string） | |:--------------------:|:-------------:| | image_classification | vit_base_p16 | ## 模型性能 | model | type | pretrain | Datasets | Top1-Accuracy | Log | pretrain_config | finetune_config | |:-----:|:------------:|:--------------------:|:-----------:|:-------------:|:---:|:--------------------------------------------------:|:-----------------------------------------------------:| | vit | vit_base_p16 | [mae_vit_base_p16]() | ImageNet-1K | 83.71% | \ | [link](../../configs/mae/run_mae_vit_base_p16_224_800ep.yaml) | [link](../../configs/vit/run_vit_base_p16_100ep.yaml) | ## 模型权重 本仓库中的`vit_base_p16`来自于facebookresearch/mae的[`ViT-Base`](https://dl.fbaipublicfiles.com/mae/finetune/mae_finetuned_vit_base.pth), 基于下述的步骤获取： 1. 从上述的链接中下载`ViT-Base`的模型权重 2. 执行转换脚本，得到转换后的输出文件`vit_base_p16.ckpt` ```shell python mindformers/models/vit/convert_weight.py --torch_path "PATH OF ViT-Base.pth" --mindspore_path "SAVE PATH OF vit_base_p16.ckpt" ```pengcheng.mind-test/docs/model_support_list.md000066400000000000000000000736561471400134000221330ustar00rootroot00000000000000# 模型支持列表 ## NLP ### masked_language_modeling | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :--------------------------------------: | :-----------------: | :------------------: | :-----------------: | :--------------------------------------------------------------------------------------------------------------------: | | [bert_base_uncased](model_cards/bert.md) | wiki | - | - | [run_bert_base_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bert/run_bert_base_uncased.yaml) | ### [text_classification](task_cards/text_classification.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :-----------------------------------------------------------------------------------------------------------------------------------: | :-----------------: | :----------------------: | :-----------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [txtcls_bert_base_uncased](task_cards/text_classification.md)
[txtcls_bert_base_uncased_mnli](task_cards/text_classification.md) | Mnli
Mnli | Entity F1
Entity F1 | -
84.80% | [run_txtcls_bert_base_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/txtcls/run_txtcls_bert_base_uncased.yaml)
[run_txtcls_bert_base_uncased_mnli.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml) | ### [token_classification](task_cards/token_classification.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :----------------------------------------------------------------------------------------------------------------------------------------: | :------------------: | :----------------------: | :-----------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [tokcls_bert_base_chinese](task_cards/token_classification.md)
[tokcls_bert_base_chinese_cluener](task_cards/token_classification.md) | CLUENER
CLUENER | Entity F1
Entity F1 | -
0.7905 | [run_tokcls_bert_base_chinese.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/tokcls/run_tokcls_bert_base_chinese.yaml)
[run_tokcls_bert_base_chinese_cluener.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/tokcls/run_tokcls_bert_base_chinese_cluener.yaml) | ### [question_answering](task_cards/question_answering.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :----------------------------------------------------------------------------------------------------------------------------: | :------------------------: | :------------------: | :------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [qa_bert_base_uncased](task_cards/question_answering.md)
[qa_bert_base_chinese_uncased](task_cards/question_answering.md) | SQuAD v1.1
SQuAD v1.1 | EM / F1
EM / F1 | 80.74 / 88.33
- | [run_qa_bert_base_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/qa/run_qa_bert_base_uncased.yaml)
[run_qa_bert_base_chinese_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/qa/run_qa_bert_base_chinese_uncased.yaml) | ### translation | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :---------------------------: | :-----------------: | :------------------: | :-----------------: | :------------------------------------------------------------------------------------------------------------------: | | [t5_small](model_cards/t5.md) | WMT16 | - | - | [run_t5_small_on_wmt16.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/t5/run_t5_small_on_wmt16.yaml) | ### [text_generation](task_cards/text_generation.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :----------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------: | :------------------------------------------: | :----------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [bloom_560m](model_cards/bloom.md)
[bloom_7.1b](model_cards/bloom.md)
[bloom_65b](model_cards/bloom.md)
[bloom_176b](model_cards/bloom.md) | alpaca
alpaca
alpaca
alpaca | -
-
-
- | -
-
-
- | [run_bloom_560m.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bloom/run_bloom_560m.yaml)
[run_bloom_7.1b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bloom/run_bloom_7.1b.yaml)
[run_bloom_65b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bloom/run_bloom_65b.yaml)
[run_bloom_176b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bloom/run_bloom_176b.yaml) | | [glm_6b](model_cards/glm.md)
[glm_6b_lora](model_cards/glm.md) | ADGEN
ADGEN | BLEU-4 / Rouge-1 / Rouge-2 / Rouge-l
- | 8.42 / 31.75 / 7.98 / 25.28
- | [run_glm_6b_finetune.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/glm/run_glm_6b_finetune.yaml)
[run_glm_6b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/glm/run_glm_6b_lora.yaml) | | [glm2_6b](model_cards/glm2.md)
[glm2_6b_lora](model_cards/glm2.md) | ADGEN
ADGEN | BLEU-4 / Rouge-1 / Rouge-2 / Rouge-l
- | 7.47 / 30.78 / 7.07 / 24.77
7.23 / 31.06 / 7.18 / 24.23 | [run_glm2_6b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/glm2/run_glm2_6b.yaml)
[run_glm2_6b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/glm2/run_glm2_6b_lora.yaml) | | [gpt2_small](model_cards/gpt2.md)
[gpt2_13b](model_cards/gpt2.md)
[gpt2_52b](model_cards/gpt2.md) | wikitext-2
wikitext-2
wikitext-2 | -
-
- | -
-
- | [run_gpt2.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2.yaml)
[run_gpt2_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2_13b.yaml)
[run_gpt2_52b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2_52b.yaml) | | [llama_7b](model_cards/llama.md)
[llama_13b](model_cards/llama.md)
[llama_65b](model_cards/llama.md)
[llama_7b_lora](model_cards/llama.md) | alpaca
alpaca
alpaca
- | -
-
-
- | -
-
-
- | [run_llama_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_7b.yaml)
[run_llama_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_13b.yaml)
[run_llama_65b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_65b.yaml)
[run_llama_7b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_7b_lora.yaml) | | [llama_7b](model_cards/llama.md)
[llama_13b](model_cards/llama.md)
[llama_65b](model_cards/llama.md)
[llama_7b_lora](model_cards/llama.md) | alpaca
alpaca
alpaca
- | -
-
-
- | -
-
-
- | [run_llama_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_7b.yaml)
[run_llama_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_13b.yaml)
[run_llama_65b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_65b.yaml)
[run_llama_7b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_7b_lora.yaml) | | [pangualpha_2_6_b](model_cards/pangualpha.md)
[pangualpha_13b](model_cards/pangualpha.md) | 悟道数据集
悟道数据集 | -
- | -
- | [run_pangualpha_2_6b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/pangualpha/run_pangualpha_2_6b.yaml)
[run_pangualpha_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/pangualpha/run_pangualpha_13b.yaml) | | [baichuan_7b](../research/baichuan/baichuan.md)
[baichuan_13b](../research/baichuan/baichuan.md) | -
- | -
- | -
- | [run_baichuan_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan/run_baichuan_7b.yaml)
[run_baichuan_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan/run_baichuan_13b.yaml) | | [baichuan2_7b](../research/baichuan2/baichuan2.md)
[baichuan2_13b](../research/baichuan2/baichuan2.md) | -
- | -
- | -
- | [run_baichuan2_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan2/run_baichuan2_7b.yaml)
[run_baichuan2_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan2/run_baichuan2_13b.yaml) | | [internlm_7b](../research/internlm/internlm.md)
[internlm_7b_lora](../research/internlm/internlm.md) | wikitext-2
alpaca | -
- | -
- | [run_internlm_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/internlm/run_internlm_7b.yaml)
[run_internlm_7b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/internlm/run_internlm_7b_lora.yaml) | | [ziya_13b](../research/ziya/ziya.md) | -
- | -
- | -
- | [run_ziya_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan/run_ziya_13b.yaml) | ## CV ### masked_image_modeling | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :------------------------------------: | :-----------------: | :------------------: | :-----------------: | :-------------------------------------------------------------------------------------------------------------------------------------: | | [mae_vit_base_p16](model_cards/mae.md) | ImageNet-1k | - | - | [run_mae_vit_base_p16_224_800ep.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/mae/run_mae_vit_base_p16_224_800ep.yaml) | ### [image_classification](task_cards/image_classification.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :-----------------------------------: | :-----------------: | :------------------: | :-----------------: | :----------------------------------------------------------------------------------------------------------------------------------: | | [vit_base_p16](model_cards/vit.md) | ImageNet-1k | Accuracy | 83.71% | [run_vit_base_p16_224_100ep.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/vit/run_vit_base_p16_224_100ep.yaml) | | [swin_base_p4w7](model_cards/swin.md) | ImageNet-1k | Accuracy | 83.44% | [run_swin_base_p4w7_224_100ep.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/swin/run_swin_base_p4w7_224_100ep.yaml) | ## Multi-Modal ### [zero_shot_image_classification](task_cards/zero_shot_image_classification.md) (by [contrastive_language_image_pretrain](task_cards/contrastive_language_image_pretrain.md)) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :---------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------: | :---------------------------------------------------------: | :----------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [clip_vit_b_32](model_cards/clip.md)
[clip_vit_b_16](model_cards/clip.md)
[clip_vit_l_14](model_cards/clip.md)
[clip_vit_l_14@336](model_cards/clip.md) | Cifar100
Cifar100
Cifar100
Cifar100 | Accuracy
Accuracy
Accuracy
Accuracy | 57.24%
61.41%
69.67%
68.19% | [run_clip_vit_b_32_pretrain_flickr8k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml)
[run_clip_vit_b_16_pretrain_flickr8k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/clip/run_clip_vit_b_16_pretrain_flickr8k.yaml)
[run_clip_vit_l_14_pretrain_flickr8k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/clip/run_clip_vit_l_14_pretrain_flickr8k.yaml)
[run_clip_vit_l_14@336_pretrain_flickr8k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/clip/run_clip_vit_l_14@336_pretrain_flickr8k.yaml) | | [blip2_vit_g](model_cards/blip2.md) | -
flickr30k
- | -
ITM
- | -
-
- | [run_blip2_vit_g_qformer_pretrain.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/blip2/run_blip2_vit_g_qformer_pretrain.yaml)
[run_blip2_vit_g_retrieval_flickr30k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/blip2/run_blip2_vit_g_retrieval_flickr30k.yaml)
[run_blip2_vit_g_zero_shot_image_classification_cifar100.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/blip2/run_blip2_vit_g_zero_shot_image_classification_cifar100.yaml) | ## 模型能力支持度 ### 核心关键模型能力一览表 | 关键模型 | 并行模式 | 数据并行 | 优化器并行 | 模型并行 | 流水并行 | 多副本并行 | 预训练 | 微调 | 评估 | 推理 | | :------: | :------------------------------: | :------: | :--------: | :------: | :------: | :--------: | ------ | :----------------: | :------------: | ---: | | Bloom | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调 | 不支持 | 推理 | | GLM | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调，Lora微调 | Blue/Rouge评估 | 推理 | | GLM2 | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调，Lora微调 | Blue/Rouge评估 | 推理 | | GPT | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调 | PPL评估 | 推理 | | LLaMa | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调，Lora微调 | PPL评估 | 推理 | | LLaMa2 | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调 | PPL评估 | 推理 | | PanGu | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调 | PPL评估 | 推理 | ### Research模型支持情况一览表 | 模型 | 任务（task name） | 模型（model name） | | :--------------------------------------------: | :----------------------------------------------------------------------------------------------------: | :------------------------------ | | [Baichuan](../research/baichuan/baichuan.md) | [text_generation](https://gitee.com/mindspore/mindformers/blob/dev/docs/task_cards/text_generation.md) | baichuan_7b
baichuan_13b | | [Baichuan2](../research/baichuan2/baichuan2.md) | [text_generation](https://gitee.com/mindspore/mindformers/blob/dev/docs/task_cards/text_generation.md) | baichuan2_7b
baichuan2_13b | | [Internlm](../research/internlm/internlm.md) | [text_generation](https://gitee.com/mindspore/mindformers/blob/dev/docs/task_cards/text_generation.md) | InternLM-7B | | [ziya](../research/ziya/ziya.md) | [text_generation](https://gitee.com/mindspore/mindformers/blob/dev/docs/task_cards/text_generation.md) | ziya-13B | ### Text Generator支持度表 | model | 模型文档链接 | 增量推理 | 流式推理 | | :---------: | :-------------------------------------------: | :------: | :------: | | bloom | [link](../model_cards/bloom.md) | √ | √ | | GLM | [link](../model_cards/glm.md) | √ | √ | | GLM2 | [link](../model_cards/glm2.md) | √ | √ | | GPT | [link](../model_cards/gpt2.md) | √ | √ | | llama | [link](../model_cards/llama.md) | √ | √ | | llama2 | [link](../model_cards/llama.md) | √ | √ | | pangu-alpha | [link](../model_cards/pangualpha.md) | √ | √ | | T5 | [link](../model_cards/t5.md) | × | √ | | research | research | research | research | | baichuan | [link](../../research/baichuan/baichuan.md) | √ | √ | | baichuan2 | [link](../../research/baichuan2/baichuan2.md) | √ | √ | | internlm | [link](../../research/internlm/internlm.md) | √ | √ | | ziya | [link](../../research/ziya/ziya.md) | √ | √ | ### 边训练边评估支持度表 | 模型 | 评估指标 | 可用Model.eval完成评估 | 是否支持 | 数据并行模式 | 半自动并行模式 | | ----------- | :-------------------: | :--------------------: | -------- | ------------ | :---------------: | | bert | - | - | - | - | - | | blip2 | - | - | - | - | - | | bloom | - | - | - | - | - | | clip | - | - | - | - | - | | filip | - | - | - | - | - | | glm | Rouge，Bleu | 否 | 否 | × | × | | gpt2 | PPL | 是 | 是 | √ | √ | | llama | PPL | 是 | 是 | √ | √（7b 至少8卡） | | llama2 | PPL | 是 | 是 | √ | √（7b 至少8卡） | | MAE | 暂缺 | - | - | - | - | | pangu alpha | PPL | 是 | 是 | √ | √ | | qa-bert | f1, precision, recall | 是 | 是 | √ | × | | swin | Accuracy | 是 | 是 | √ | × | | t5 | 暂缺 | - | - | - | - | | tokcls-bert | f1, precision, recall | 是 | 是 | √ | × | | txtcls-bert | Accuracy | 是 | 是 | √ | × | | vit | Accuracy | 是 | 是 | √ | × | | research | research | research | research | research | research | | baichuan | PPL | 是 | 是 | √ | √（7b 至少8卡） | | baichuan2 | PPL | 是 | 是 | √ | √（7b 至少8卡） | | internlm | PPL | 是 | 是 | √ | √（7b 至少8卡） | | ziya | PPL | 是 | 是 | √ | √（13b 至少16卡） | ### 微调支持列表 | 模型 | 微调算法 | 运行模式 | | :------------------------------: | :------: | :---------------------: | | [GPT2](../model_cards/gpt2.md) | Lora | finetune、eval、predict | | [LLama](../model_cards/llama.md) | Lora | finetune、eval、predict | | [GLM](../model_cards/glm.md) | Lora | finetune、eval、predict | | [GLM2](../model_cards/glm2.md) | Lora | finetune、eval、predict | ### Chat Web支持列表 | 模型 | 规格 | 分词器 | 增量推理 | | ----- | ------------- | ------------- | -------- | | GLM | glm_6b | glm_6b | 支持 | | GLM2 | glm2_6b | glm2_6b | 支持 | | BLOOM | bloom_7.1b | bloom_7.1b | 支持 | | LLAMA | llama_7b_lora | llama_7b_lora | 支持 | ### 其余库上模型分布式支持情况一览表 | 模型 | 并行模式 | 数据并行 | 优化器并行 | 模型并行 | 流水并行 | 多副本并行 | | ----- | ------------- | -------- | ---------- | -------- | -------- | ---------- | | Bert | data_parallel | 是 | 是 | 否 | 否 | 否 | | BLIP2 | data_parallel | 是 | 是 | 否 | 否 | 否 | | CLIP | data_parallel | 是 | 是 | 否 | 否 | 否 | | MAE | data_parallel | 是 | 是 | 否 | 否 | 否 | | Swin | data_parallel | 是 | 是 | 否 | 否 | 否 | | T5 | data_parallel | 是 | 是 | 否 | 否 | 否 | | VIT | data_parallel | 是 | 是 | 否 | 否 | 否 | pengcheng.mind-test/docs/readthedocs/000077500000000000000000000000001471400134000201265ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/Makefile000066400000000000000000000012121471400134000215620ustar00rootroot00000000000000# Minimal makefile for Sphinx documentation # # You can set these variables from the command line, and also # from the environment for the first two. SPHINXOPTS ?= SPHINXBUILD ?= sphinx-build SOURCEDIR = source_zh_cn BUILDDIR = build_zh_cn # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) pengcheng.mind-test/docs/readthedocs/requirements.txt000066400000000000000000000001001471400134000234010ustar00rootroot00000000000000sphinx docutils myst_parser sphinx_rtd_theme == 0.5.2 mindspore pengcheng.mind-test/docs/readthedocs/source_zh_cn/000077500000000000000000000000001471400134000226075ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/_static/000077500000000000000000000000001471400134000242355ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/_static/my_theme.css000066400000000000000000000000661471400134000265600ustar00rootroot00000000000000.wy-nav-content { max-width: 1200px !important; } pengcheng.mind-test/docs/readthedocs/source_zh_cn/_templates/000077500000000000000000000000001471400134000247445ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/_templates/classtemplate.rst000066400000000000000000000007371471400134000303460ustar00rootroot00000000000000.. role:: hidden :class: hidden-section .. currentmodule:: {{ module }} {% if objname in [] %} {{ fullname | underline }} .. autofunction:: {{ fullname }} {% elif objname[0].istitle() %} {{ fullname | underline }} .. autoclass:: {{ name }} :exclude-members: construct :members: {% else %} {{ fullname | underline }} .. autofunction:: {{ fullname }} {% endif %} .. autogenerated from _templates/classtemplate.rst note it does not have :inherited-members: pengcheng.mind-test/docs/readthedocs/source_zh_cn/conf.py000066400000000000000000000121051471400134000241050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ # Configuration file for the Sphinx documentation builder. # # This file only contains a selection of the most common options. For a full # list see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html # -- Path setup -------------------------------------------------------------- # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # """Configuration file for the Sphinx documentation builder.""" import os import shutil from sphinx.util import logging # -- Project information ----------------------------------------------------- project = 'mindformers' # pylint: disable=W0622 copyright = '2023, mindformers contributors' author = 'mindformers contributors' # The full version, including alpha/beta/rc tags release = 'dev' # -- General configuration --------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = ['myst_parser', 'sphinx.ext.autodoc', 'sphinx.ext.viewcode', 'sphinx.ext.autosummary', 'sphinx.ext.intersphinx'] source_suffix = {'.rst': 'restructuredtext', '.md': 'markdown'} # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. exclude_patterns = [] pygments_style = 'sphinx' autodoc_inherit_docstrings = False autosummary_generate = True # -- Options for HTML output ------------------------------------------------- # Reconstruction of sphinx auto generated document translation. language = 'zh_CN' gettext_compact = False # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'sphinx_rtd_theme' html_static_path = ['_static'] # Copy source files of chinese python api from mindformers repository. logger = logging.getLogger(__name__) copy_path = ['../../../README.md', '../../../research', '../../FAQ.md', '../../model_support_list.md', '../../transformer仓Python编程规范.md' '../../aicc_cards', '../../feature_cards', '../../model_cards', '../../task_cards'] # origin path in readthedocs.org is like # /abs/readthedocs.org/user_builds/project_name/checkouts/version/docs/xxx # new path in readthedocs.org is like # /abs/readthedocs.org/user_builds/project_name/checkouts/version/docs/readthedocs/source_zh_cn/(xxx or docs/xxx) # 此处为了在readthedocs中构造出与gitee上同样的目录结构，以达到md文件中相对路径在gitee和readthedos都可以使用的目的 def copy_file_to_docs(ori_name): """copy dir in docs to readthedocs workspace.""" ori_path = os.path.realpath(ori_name) new_name = ori_path.split('/')[-1] if ori_name.endswith('README.md'): new_path = new_name shutil.copy(ori_path, new_path) elif 'research' in ori_name: new_path = new_name shutil.copytree(ori_path, new_path) elif ori_name.endswith('.md'): new_path = os.path.join('docs', new_name) shutil.copy(ori_path, new_path) else: new_path = os.path.join('docs', new_name) shutil.copytree(ori_path, new_path) for file_path in copy_path: copy_file_to_docs(file_path) # split README into 4 parts with open('README.md', 'r') as f: title_list = ['Introduction', 'Install', 'Version_Match', 'Quick_Tour', 'Contribution', 'License'] title_for_index = ['# 一、介绍', '# 二、安装', '# 三、版本配套', '# 四、快速开始', '-', '-'] file_count = 0 fn = None for line in f: if line.startswith('##') and not line.startswith('###'): if fn: fn.close() fn = open(f'{title_list[file_count]}.md', 'w') fn.write(title_for_index[file_count]) file_count += 1 continue if fn: fn.write(line) fn.close() # in _static, set page width 1200 px def setup(app): app.add_css_file('my_theme.css') pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/000077500000000000000000000000001471400134000235375ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/000077500000000000000000000000001471400134000257115ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/README.md000066400000000000000000000123141471400134000271710ustar00rootroot00000000000000# 接口支持度说明 ## Datasets | Dataset API | 说明 | | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | [mindformers.dataset.CausalLanguageModelDataset](mindformers.dataset.CausalLanguageModelDataset) | GPT类模型文本数据集，支持MindRecord，TFRecord，自定义数据集等 | | [mindformers.dataset.ContrastiveLanguageImagePretrainDataset](mindformers.dataset.ContrastiveLanguageImagePretrainDataset) | CLIP类模型预训练图文对数据集，如Flickr8k等 | | [mindformers.dataset.ImageCLSDataset](mindformers.dataset.ImageCLSDataset) | 图片分类数据集，如ImageNet2012等 | | [mindformers.dataset.KeyWordGenDataset](mindformers.dataset.KeyWordGenDataset) | GLM模型文本数据集，支持MindRecord，自定义数据集等，如ADGen | | [mindformers.dataset.MaskLanguageModelDataset](mindformers.dataset.MaskLanguageModelDataset) | Bert类模型文本数据集，仅支持MindRecord，TFRecord | | [mindformers.dataset.MIMDataset](mindformers.dataset.MIMDataset) | Mae等图片数据集，支持MindRecord，ImageNet2012等 | | [mindformers.dataset.TextClassificationDataset](mindformers.dataset.TextClassificationDataset) | 文本分类数据集，仅支持TFRecord | | [mindformers.dataset.TokenClassificationDataset](mindformers.dataset.TokenClassificationDataset) | Token分类文本数据集，支持自定义数据集，如CLUENER | | [mindformers.dataset.ZeroShotImageClassificationDataset](mindformers.dataset.ZeroShotImageClassificationDataset) | 零样本分类图片数据集，支持自定义数据集，如Cifar100 | ## Learning Rate | Learning Rate API | 说明 | | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | | [mindformers.core.lr.ConstantWarmUpLR](mindformers.core.lr.ConstantWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后LR固定为LR×常数 | | [mindformers.core.lr.CosineWithRestartsAndWarmUpLR](mindformers.core.lr.CosineWithRestartsAndWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后进行cos(1)->cos(0) 衰减或周期变化 | | [mindformers.core.lr.CosineWithWarmUpLR](mindformers.core.lr.CosineWithWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后进行consine衰减或周期变化 | | [mindformers.core.lr.LinearWithWarmUpLR](mindformers.core.lr.LinearWithWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后LR从0开始增长到指定LR | | [mindformers.core.lr.PolynomialWithWarmUpLR](mindformers.core.lr.PolynomialWithWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后进行衰减 | ## Loss | Loss API | 说明 | | -------------------------------------------------------------------------------------------- | ------------------------------------ | | [mindformers.core.loss.CrossEntropyLoss](mindformers.core.loss.CrossEntropyLoss) | NLP常用的交叉熵损失函数 | | [mindformers.core.loss.L1Loss](mindformers.core.loss.L1Loss) | 平均绝对误差，SIMMIM所使用的损失函数 | | [mindformers.core.loss.MSELoss](mindformers.core.loss.MSELoss) | 均方误差，Mae所使用的的损失函数 | | [mindformers.core.loss.SoftTargetCrossEntropy](mindformers.core.loss.SoftTargetCrossEntropy) | Swin所使用的的损失函数 | ## Optimizer | Optimizer API | 说明 | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | [mindformers.core.optim.FusedAdamWeightDecay](mindformers.core.optim.FusedAdamWeightDecay) | 权重衰减Adam算法的融合算子 | | [mindformers.core.optim.FP32StateAdamWeightDecay](mindformers.core.optim.FP32StateAdamWeightDecay) | 权重衰减Adam算法，与mindspore中一致，区别为优化器状态改为fp32 | pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.core.rst000066400000000000000000000036021471400134000317200ustar00rootroot00000000000000mindformers.core ================== .. automodule:: mindformers.core mindformers.core ----------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.ClipGradNorm mindformers.core.build_context mindformers.core.init_context mindformers.core.callback -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.callback.CheckpointMointor mindformers.core.callback.MFLossMonitor mindformers.core.callback.ObsMonitor mindformers.core.callback.SummaryMonitor mindformers.core.callback.ProfileMonitor mindformers.core.callback.EvalCallBack mindformers.core.loss -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.loss.CrossEntropyLoss mindformers.core.loss.L1Loss mindformers.core.loss.MSELoss mindformers.core.loss.SoftTargetCrossEntropy mindformers.core.lr -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.lr.ConstantWarmUpLR mindformers.core.lr.CosineWithRestartsAndWarmUpLR mindformers.core.lr.CosineWithWarmUpLR mindformers.core.lr.LinearWithWarmUpLR mindformers.core.lr.PolynomialWithWarmUpLR mindformers.core.metric -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.metric.EntityScore mindformers.core.metric.SQuADMetric mindformers.core.metric.PerplexityMetric mindformers.core.metric.ADGENMetric mindformers.core.optim -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.optim.FusedAdamWeightDecay mindformers.core.optim.FP32StateAdamWeightDecay pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.dataset.rst000066400000000000000000000013011471400134000324070ustar00rootroot00000000000000mindformers.dataset ==================== .. automodule:: mindformers.dataset .. autosummary:: :toctree: dataset :nosignatures: :template: classtemplate.rst mindformers.dataset.BaseDataset mindformers.dataset.CausalLanguageModelDataset mindformers.dataset.ContrastiveLanguageImagePretrainDataset mindformers.dataset.ImageCLSDataset mindformers.dataset.ImageToTextRetrievalDataset mindformers.dataset.KeyWordGenDataset mindformers.dataset.MaskLanguageModelDataset mindformers.dataset.MIMDataset mindformers.dataset.TextClassificationDataset mindformers.dataset.TokenClassificationDataset mindformers.dataset.ZeroShotImageClassificationDataset pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.models.rst000066400000000000000000000102141471400134000322500ustar00rootroot00000000000000mindformers.models =================== .. automodule:: mindformers.models mindformers.models ------------------- .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.BaseConfig mindformers.models.BaseImageProcessor mindformers.models.BaseModel mindformers.models.BaseProcessor mindformers.models.Tokenizer mindformers.models.bert ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.bert.BertConfig mindformers.models.bert.BertModel mindformers.models.bert.BertForMultipleChoice mindformers.models.bert.BertForPreTraining mindformers.models.bert.BertForQuestionAnswering mindformers.models.bert.BertForTokenClassification mindformers.models.bert.BertTokenizer mindformers.models.bert.BertProcessor mindformers.models.t5 ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.t5.T5Config mindformers.models.t5.T5ForConditionalGeneration mindformers.models.t5.T5Processor mindformers.models.t5.T5Tokenizer mindformers.models.clip ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.clip.CLIPConfig mindformers.models.clip.CLIPVisionConfig mindformers.models.clip.CLIPTextConfig mindformers.models.clip.CLIPModel mindformers.models.clip.CLIPTokenizer mindformers.models.clip.CLIPProcessor mindformers.models.clip.CLIPImageProcessor mindformers.models.mae ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.mae.ViTMAEConfig mindformers.models.mae.ViTMAEForPreTraining mindformers.models.mae.ViTMAEProcessor mindformers.models.mae.ViTMAEImageProcessor mindformers.models.swin ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.swin.SwinConfig mindformers.models.swin.SwinModel mindformers.models.swin.SwinForImageClassification mindformers.models.swin.SwinImageProcessor mindformers.models.swin.SwinProcessor mindformers.models.vit ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.vit.ViTConfig mindformers.models.vit.ViTModel mindformers.models.vit.ViTProcessor mindformers.models.vit.ViTImageProcessor mindformers.models.vit.ViTForImageClassification mindformers.models.gpt2 ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.gpt2.GPT2Config mindformers.models.gpt2.GPT2Model mindformers.models.gpt2.GPT2LMHeadModel mindformers.models.gpt2.GPT2Tokenizer mindformers.models.gpt2.GPT2Processor mindformers.models.glm ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.glm.GLMConfig mindformers.models.glm.GLMChatModel mindformers.models.glm.GLMForPreTraining mindformers.models.glm.GLMForPreTrainingWithLora mindformers.models.glm.GLMChatModelWithLora mindformers.models.glm.ChatGLMTokenizer mindformers.models.glm.GLMProcessor mindformers.models.llama ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.llama.LlamaConfig mindformers.models.llama.LlamaModel mindformers.models.llama.LlamaForCausalLM mindformers.models.llama.LlamaForCausalLMWithLora mindformers.models.llama.LlamaTokenizer mindformers.models.llama.LlamaProcessor mindformers.models.bloom ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.bloom.BloomConfig mindformers.models.bloom.BloomModel mindformers.models.bloom.BloomLMHeadModel mindformers.models.bloom.BloomTokenizer mindformers.models.bloom.BloomProcessor pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.modules.rst000066400000000000000000000025201471400134000324360ustar00rootroot00000000000000mindformers.modules ===================== .. automodule:: mindformers.modules mindformers.modules.layers ---------------------------------------------- .. autosummary:: :toctree: modules :nosignatures: :template: classtemplate.rst mindformers.modules.layers.Dropout mindformers.modules.layers.FixedSparseAttention mindformers.modules.layers.LayerNorm mindformers.modules.layers.Linear mindformers.modules.transformer ---------------------------------------------- .. autosummary:: :toctree: modules :nosignatures: :template: classtemplate.rst mindformers.modules.transformer.AttentionMask mindformers.modules.transformer.EmbeddingOpParallelConfig mindformers.modules.transformer.FeedForward mindformers.modules.transformer.MoEConfig mindformers.modules.transformer.MultiHeadAttention mindformers.modules.transformer.OpParallelConfig mindformers.modules.transformer.Transformer mindformers.modules.transformer.TransformerDecoder mindformers.modules.transformer.TransformerDecoderLayer mindformers.modules.transformer.TransformerEncoder mindformers.modules.transformer.TransformerEncoderLayer mindformers.modules.transformer.TransformerOpParallelConfig mindformers.modules.transformer.TransformerRecomputeConfig mindformers.modules.transformer.VocabEmbedding pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.pipeline.rst000066400000000000000000000010511471400134000325710ustar00rootroot00000000000000mindformers.pipeline ===================== .. automodule:: mindformers.pipeline .. autosummary:: :toctree: pipeline :nosignatures: :template: classtemplate.rst mindformers.pipeline.BasePipeline mindformers.pipeline.pipeline mindformers.pipeline.FillMaskPipeline mindformers.pipeline.TranslationPipeline mindformers.pipeline.ImageClassificationPipeline mindformers.pipeline.TextClassificationPipeline mindformers.pipeline.TokenClassificationPipeline mindformers.pipeline.ZeroShotImageClassificationPipeline pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.rst000066400000000000000000000004171471400134000307720ustar00rootroot00000000000000mindformers ================== .. automodule:: mindformers .. autosummary:: :toctree: mindformers :nosignatures: :template: classtemplate.rst mindformers.AutoConfig mindformers.AutoModel mindformers.AutoProcessor mindformers.AutoTokenizer pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.trainer.rst000066400000000000000000000015271471400134000324400ustar00rootroot00000000000000mindformers.trainer ==================== .. automodule:: mindformers.trainer .. autosummary:: :toctree: trainer :nosignatures: :template: classtemplate.rst mindformers.trainer.ContextConfig mindformers.trainer.ParallelContextConfig mindformers.trainer.TrainingArguments mindformers.trainer.Trainer mindformers.trainer.TranslationTrainer mindformers.trainer.CausalLanguageModelingTrainer mindformers.trainer.ContrastiveLanguageImagePretrainTrainer mindformers.trainer.ImageClassificationTrainer mindformers.trainer.MaskedImageModelingTrainer mindformers.trainer.MaskedLanguageModelingTrainer mindformers.trainer.QuestionAnsweringTrainer mindformers.trainer.TextClassificationTrainer mindformers.trainer.TokenClassificationTrainer mindformers.trainer.ZeroShotImageClassificationTrainer pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.wrapper.rst000066400000000000000000000004111471400134000324430ustar00rootroot00000000000000mindformers.wrapper ==================== .. automodule:: mindformers.wrapper .. autosummary:: :toctree: wrapper :nosignatures: :template: classtemplate.rst mindformers.wrapper.MFTrainOneStepCell mindformers.wrapper.MFPipelineWithLossScaleCellpengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/000077500000000000000000000000001471400134000263735ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.core.rst000066400000000000000000000036021471400134000324020ustar00rootroot00000000000000mindformers.core ================== .. automodule:: mindformers.core mindformers.core ----------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.ClipGradNorm mindformers.core.build_context mindformers.core.init_context mindformers.core.callback -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.callback.CheckpointMointor mindformers.core.callback.MFLossMonitor mindformers.core.callback.ObsMonitor mindformers.core.callback.SummaryMonitor mindformers.core.callback.ProfileMonitor mindformers.core.callback.EvalCallBack mindformers.core.loss -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.loss.CrossEntropyLoss mindformers.core.loss.L1Loss mindformers.core.loss.MSELoss mindformers.core.loss.SoftTargetCrossEntropy mindformers.core.lr -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.lr.ConstantWarmUpLR mindformers.core.lr.CosineWithRestartsAndWarmUpLR mindformers.core.lr.CosineWithWarmUpLR mindformers.core.lr.LinearWithWarmUpLR mindformers.core.lr.PolynomialWithWarmUpLR mindformers.core.metric -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.metric.EntityScore mindformers.core.metric.SQuADMetric mindformers.core.metric.PerplexityMetric mindformers.core.metric.ADGENMetric mindformers.core.optim -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.optim.FusedAdamWeightDecay mindformers.core.optim.FP32StateAdamWeightDecay pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.dataset.rst000066400000000000000000000012741471400134000331020ustar00rootroot00000000000000mindformers.dataset ==================== .. automodule:: mindformers.dataset .. autosummary:: :toctree: dataset :nosignatures: :template: classtemplate.rst mindformers.dataset.BaseDataset mindformers.dataset.MIMDataset mindformers.dataset.ContrastiveLanguageImagePretrainDataset mindformers.dataset.ImageCLSDataset mindformers.dataset.ImageToTextRetrievalDataset mindformers.dataset.MaskLanguageModelDataset mindformers.dataset.KeyWordGenDataset mindformers.dataset.CausalLanguageModelDataset mindformers.dataset.TextClassificationDataset mindformers.dataset.TokenClassificationDataset mindformers.dataset.ZeroShotImageClassificationDataset pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.models.rst000066400000000000000000000102141471400134000327320ustar00rootroot00000000000000mindformers.models =================== .. automodule:: mindformers.models mindformers.models ------------------- .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.BaseConfig mindformers.models.BaseImageProcessor mindformers.models.BaseModel mindformers.models.BaseProcessor mindformers.models.Tokenizer mindformers.models.bert ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.bert.BertConfig mindformers.models.bert.BertModel mindformers.models.bert.BertForMultipleChoice mindformers.models.bert.BertForPreTraining mindformers.models.bert.BertForQuestionAnswering mindformers.models.bert.BertForTokenClassification mindformers.models.bert.BertTokenizer mindformers.models.bert.BertProcessor mindformers.models.t5 ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.t5.T5Config mindformers.models.t5.T5ForConditionalGeneration mindformers.models.t5.T5Processor mindformers.models.t5.T5Tokenizer mindformers.models.clip ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.clip.CLIPConfig mindformers.models.clip.CLIPVisionConfig mindformers.models.clip.CLIPTextConfig mindformers.models.clip.CLIPModel mindformers.models.clip.CLIPTokenizer mindformers.models.clip.CLIPProcessor mindformers.models.clip.CLIPImageProcessor mindformers.models.mae ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.mae.ViTMAEConfig mindformers.models.mae.ViTMAEForPreTraining mindformers.models.mae.ViTMAEProcessor mindformers.models.mae.ViTMAEImageProcessor mindformers.models.swin ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.swin.SwinConfig mindformers.models.swin.SwinModel mindformers.models.swin.SwinForImageClassification mindformers.models.swin.SwinImageProcessor mindformers.models.swin.SwinProcessor mindformers.models.vit ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.vit.ViTConfig mindformers.models.vit.ViTModel mindformers.models.vit.ViTProcessor mindformers.models.vit.ViTImageProcessor mindformers.models.vit.ViTForImageClassification mindformers.models.gpt2 ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.gpt2.GPT2Config mindformers.models.gpt2.GPT2Model mindformers.models.gpt2.GPT2LMHeadModel mindformers.models.gpt2.GPT2Tokenizer mindformers.models.gpt2.GPT2Processor mindformers.models.glm ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.glm.GLMConfig mindformers.models.glm.GLMChatModel mindformers.models.glm.GLMForPreTraining mindformers.models.glm.GLMForPreTrainingWithLora mindformers.models.glm.GLMChatModelWithLora mindformers.models.glm.ChatGLMTokenizer mindformers.models.glm.GLMProcessor mindformers.models.llama ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.llama.LlamaConfig mindformers.models.llama.LlamaModel mindformers.models.llama.LlamaForCausalLM mindformers.models.llama.LlamaForCausalLMWithLora mindformers.models.llama.LlamaTokenizer mindformers.models.llama.LlamaProcessor mindformers.models.bloom ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.bloom.BloomConfig mindformers.models.bloom.BloomModel mindformers.models.bloom.BloomLMHeadModel mindformers.models.bloom.BloomTokenizer mindformers.models.bloom.BloomProcessor pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.modules.rst000066400000000000000000000025201471400134000331200ustar00rootroot00000000000000mindformers.modules ===================== .. automodule:: mindformers.modules mindformers.modules.layers ---------------------------------------------- .. autosummary:: :toctree: modules :nosignatures: :template: classtemplate.rst mindformers.modules.layers.Dropout mindformers.modules.layers.FixedSparseAttention mindformers.modules.layers.LayerNorm mindformers.modules.layers.Linear mindformers.modules.transformer ---------------------------------------------- .. autosummary:: :toctree: modules :nosignatures: :template: classtemplate.rst mindformers.modules.transformer.AttentionMask mindformers.modules.transformer.EmbeddingOpParallelConfig mindformers.modules.transformer.FeedForward mindformers.modules.transformer.MoEConfig mindformers.modules.transformer.MultiHeadAttention mindformers.modules.transformer.OpParallelConfig mindformers.modules.transformer.Transformer mindformers.modules.transformer.TransformerDecoder mindformers.modules.transformer.TransformerDecoderLayer mindformers.modules.transformer.TransformerEncoder mindformers.modules.transformer.TransformerEncoderLayer mindformers.modules.transformer.TransformerOpParallelConfig mindformers.modules.transformer.TransformerRecomputeConfig mindformers.modules.transformer.VocabEmbedding pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.pipeline.rst000066400000000000000000000010511471400134000332530ustar00rootroot00000000000000mindformers.pipeline ===================== .. automodule:: mindformers.pipeline .. autosummary:: :toctree: pipeline :nosignatures: :template: classtemplate.rst mindformers.pipeline.BasePipeline mindformers.pipeline.pipeline mindformers.pipeline.FillMaskPipeline mindformers.pipeline.TranslationPipeline mindformers.pipeline.ImageClassificationPipeline mindformers.pipeline.TextClassificationPipeline mindformers.pipeline.TokenClassificationPipeline mindformers.pipeline.ZeroShotImageClassificationPipeline pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.rst000066400000000000000000000004171471400134000314540ustar00rootroot00000000000000mindformers ================== .. automodule:: mindformers .. autosummary:: :toctree: mindformers :nosignatures: :template: classtemplate.rst mindformers.AutoConfig mindformers.AutoModel mindformers.AutoProcessor mindformers.AutoTokenizer pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.trainer.rst000066400000000000000000000015271471400134000331220ustar00rootroot00000000000000mindformers.trainer ==================== .. automodule:: mindformers.trainer .. autosummary:: :toctree: trainer :nosignatures: :template: classtemplate.rst mindformers.trainer.ContextConfig mindformers.trainer.ParallelContextConfig mindformers.trainer.TrainingArguments mindformers.trainer.Trainer mindformers.trainer.TranslationTrainer mindformers.trainer.CausalLanguageModelingTrainer mindformers.trainer.ContrastiveLanguageImagePretrainTrainer mindformers.trainer.ImageClassificationTrainer mindformers.trainer.MaskedImageModelingTrainer mindformers.trainer.MaskedLanguageModelingTrainer mindformers.trainer.QuestionAnsweringTrainer mindformers.trainer.TextClassificationTrainer mindformers.trainer.TokenClassificationTrainer mindformers.trainer.ZeroShotImageClassificationTrainer pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.wrapper.rst000066400000000000000000000004111471400134000331250ustar00rootroot00000000000000mindformers.wrapper ==================== .. automodule:: mindformers.wrapper .. autosummary:: :toctree: wrapper :nosignatures: :template: classtemplate.rst mindformers.wrapper.MFTrainOneStepCell mindformers.wrapper.MFPipelineWithLossScaleCellpengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/000077500000000000000000000000001471400134000250105ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/AutoClass_Design.md000066400000000000000000000022221471400134000305170ustar00rootroot00000000000000# AutoClass MindFormers大模型套件提供了AutoClass类，包含AutoConfig、AutoModel、AutoTokenizer、AutoProcessor4个便捷高阶接口，方便用户调用套件中已封装的API接口。上述4类提供了相应领域模型的ModelConfig、Model、Tokenzier、Processor的实例化功能。 ![输入图片说明](https://foruda.gitee.com/images/1686128219487920380/00f18fec_9324149.png) | AutoClass类 | from_pretrained属性（实例化功能） | from_config属性（实例化功能） | save_pretrained（保存配置功能） | | ------------- | --------------------------------- | ----------------------------- | :-----------------------------: | | AutoConfig | √ | × | √ | | AutoModel | √ | √ | √ | | AutoProcessor | √ | × | √ | | AutoTokenizer | √ | × | √ | pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/Config_Design.md000066400000000000000000000446571471400134000300500ustar00rootroot00000000000000# Config和API组件 ## config概念 config包含模型配置、训练配置、环境配置等信息。启动训练、微调、评估和推理时，会先根据config中的内容进行配置。 ## config设计 ![](https://foruda.gitee.com/images/1691027467343740998/29a1db43_11500692.png) MindformerConfig是Mindformers整体的配置类，里面包含Mindformers中的各种配置。 - run_xxx.yaml中的配置为默认配置。默认情况下，Mindformers根据yaml配置文件创建MindFormerConfig。 - 如果使用run_mindformer.py方式启动，可以通过命令行参数修改配置中的参数。 - 如果使用Trainer高阶接口方式启动，可以通过TrainArguments修改配置中的参数。 MindformerConfig中包含的具体配置可以参考[Config.md](../../../../../configs/README.md) ## API组件注册机制 ### 注册原理图 ![输入图片说明](https://foruda.gitee.com/images/1673344591842027306/a4d581a3_9324149.png "image-20230110151323393.png") ### 注册机制简介 MindFormers套件全仓代码主要基于Python语言和MindSpore AI框架开发，通过设计业界主流的装饰器注册机制和MindSpore AI框架易用性工具，将套件中涉及的API自动注册至registry字典中，从而实现对API接口的灵活调用。 注册代码设计主要包含两大类： 1. MindFormerModuleType 全局注册模块定义类 2. MindFormerRegister API注册和实例化类。 通过该注册机制的设定，可以有效的统一套件中各类API接口的调用机制和开发范式，实现套件模块化设计和灵活配置。 ### MindFormerModuleType MindFormerModuleType类目前共支持全局注册模块共22个，可根据实际模型或者工程需求进行增加或改动，详情请见下表： | MindFormerModuleType | | | | -------------------- | ------------------ | ------------------------------------------------ | | 模块名称 | 模块赋值（String） | 模块定义 | | TRAINER | 'trainer' | 任务流程创建模块，定义训练/微调/评估/推理流程 | | PIPELINE | 'pipeline' | 推理模块，定义任务的推理流程 | | PROCESSOR | 'procrssor' | 处理流模块，定义模型权重预处理和加载流程 | | DATASET | 'dataset' | 数据流模块，定义数据集加载及预处理流程 | | MASK_POLICY | 'mask_policy' | 掩码模块，定义掩码策略 | | DATASET_LOADER | 'dataset_loader' | 数据集加载模块，定义数据集加载策略 | | DATASET_SAMPLER | 'dataset_sampler' | 数据集采样模块，定义数据集采样策略 | | TRANSFORMS | 'transforms' | 数据集增强模块，定义数据集增强策略 | | MODELS | 'models' | 模型网络模块，定义整体网络架构 | | ENCODER | 'encoder' | 模型骨干网络模块，定义网络中encoder模块 | | HEAD | 'head' | 模型head模块，定义网络中head头模块 | | MODULES | 'modules' | 模型组件模块，定义网络组件 | | BASE_LAYER | 'base_layer' | 定义模型组件最小单元模块 | | CORE | 'core' | 定义模型组件最小插件模块 | | LOSS | 'loss' | 损失函数模块，定义损失函数类型 | | LR | 'lr' | 学习策略模块，定义学习策略类型 | | OPTIMIZER | 'optimizer' | 优化器模块，定义优化器类型 | | CONFIG | 'config' | 配置模块，定义模型相关的配置 | | WRAPPER | 'wrapper' | 模型训练封装模块，定义模型前向计算和梯度更新流程 | | METRIC | 'metric' | 模型度量模块，定义模型评估策略 | | CALLBACK | 'callback' | 回调函数模块，定义模型回调类型 | | TOOLS | 'tools' | 通用工具模块，定义模型公用工具 | ### MindFormerRegister MindFormerRegister 类涉及7类属性函数，主要功能包含：1. API 注册 2. API实例化 3. API检查。详情请见下表： | 属性函数 | 入参说明 | 功能说明 | | ------------------------ | ------------------------------------------------------------ | ---------------------------- | | registry | dict类型 | 存储已注册的API | | register | 1. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 2. **alias**(string): 注册API重命名，默认None | 装饰器函数，完成API 注册 | | register_cls | 1. **register_class**(class or function): 注册API 2. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 3. **alias**(string): 注册API重命名，默认None | 注册函数，完成API注册 | | get_instance_from_config | 1. **config**(dict): 配置文件字典 2. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 3. **default_args**(dict): API默认参数，默认None | 通过配置文件获取API实例 | | get_instance | 1. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 2. **class_name**(string): 已注册的API名称，默认None 3. ***args**: 定省参数，已注册API支持参数 4. **kwargs: 不定省参数，已注册API支持参数 | 通过已注册API名称获取API实例 | | get_cls | 1. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 2. **class_name**(string): 已注册的API名称，默认None | 获取已注册的API类 | | is_exist | 1. **module_type**(string): MindFormerModuleType包含的模块名，默认’common’2. class_name(string): 已注册的API名称，默认None | 检查API是否注册 | ### 注册机制使用示例 - API注册Python代码示例：用户可以利用以下两种注册方式将任意API注册至registry字典中。 ```python from mindformers.tools.register import MindFormerRegister, MindFormerModuleType # Type1 装饰器注册自定义API接口 # 此处即可通过装饰MindFormerRegister的register属性完成 YourDefineLRAPI 类的注册，并将其注册到LR模块 @MindFormerRegister.register(MindFormerModuleType.LR) class YourDefineLRAPI(): def __init__(self, *args, **kwargs): Pass # Type2 通过API类进行注册 # 此处即可通过装饰MindFormerRegister的register_cls属性完成 YourDefineAPILR类的注册，并将其注册到LR模块 MindFormerRegister.register_cls(YourDefineLRAPI, MindFormerModuleType.LR) ``` ## API 组件 Build 机制 MindFormers套件在注册机制的基础之上，为每个模块都配备的对应的build函数，方便用户使用统一且通用的build形式来创建不同模块的API实例。然而register和build的形式仍然具有一定的`黑盒`特性，需要大家提前了解MindFormers套件中的注册和API创建原理。 ### build 支持列表 | Build Module | 功能 | | :------------------: | :------------------------------------------------: | | [build_callback]() | 创建回调函数，返回API实例或者包含API实例的列表 | | [build_lr]() | 创建学习率函数，返回API实例 | | [build_metric]() | 创建评估指标函数，返回API实例 | | [build_optim]() | 创建优化器函数，返回API实例 | | [build_loss]() | 创建loss函数，返回API实例 | | [build_dataset]() | 创建数据集，返回MindSpore Dataset实例 | | [build_dataloader]() | 创建数据集加载函数，返回API实例 | | [build_sampler]() | 创建数据集采样函数，返回API实例 | | [build_transforms]() | 创建数据增广函数，返回API实例或者包含API实例的列表 | | [build_mask]() | 创建掩码策略，返回API实例 | | [build_config]() | 创建模型配置函数，返回API实例 | | [build_model]() | 创建模型网络函数，返回API实例 | | [build_processor]() | 创建数据预处理函数（用于推理），返回API实例 | | [build_tokenizer]() | 创建文本的tokenizer函数，返回API实例 | | [build_module]() | 创建网络组件函数，返回API实例 | | [build_layer]() | 创建网络基本构成函数，返回API实例 | | [build_pipeline]() | 创建pipeline推理函数，返回API实例 | | [build_trainer]() | 创建trainer函数，返回API实例 | | [build_wrapper]() | 创建前反向封装函数，返回API实例 | 可以看到，MindFormers套件中目前已经支持20个模块的build，基本覆盖组网过程的各个模块的创建。值得注意的是，我们build函数的设计主要依赖于配置文件（即包含相关模块和相应配置的一个字典文件），因此要充分理解build机制，需要同时学习如何开发配置文件。 ### Build 使用案例 下面为用户以某数据集模块的注册和创建使用为例，详细介绍如何使用我们的Build机制来完成一个数据流的创建。 #### 创建分类任务数据集 ```text Note: 1. MindFormers套件中已经集成的Dataset类均需要依赖配置文件来进行创建，因此使用前可以参考相关配置文件用例来自定义数据任务配置 2. 如果不想使用配置文件来调用Dataset，用户可以自行定义完整的数据集模块并注册，只需保证返回的实例是mindspore的DATASET类型 ``` - Step 1: 开发配置文件 [配置文件链接](https://gitee.com/mindspore/mindformers/blob/dev/configs/mae/run_mae_vit_base_p16_224_800ep.yaml) ```yaml # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 64 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: RandomCropDecodeResize size: 224 scale: [0.2, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: Normalize mean: [123.675, 118.575, 103.53] std: [58.395, 62.22, 57.375] - type: HWC2CHW mask_policy: type: MaeMask input_size: 224 patch_size: 16 mask_ratio: 0.75 input_columns: ["image"] output_columns: ["image", "mask", "ids_restore", "unmask_index"] column_order: ["image", "mask", "ids_restore", "unmask_index"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: MIMDataset dataset_config: *train_dataset ``` - 开发MIMDataset类并注册 [MIMDataset类](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/dataset/mim_dataset.py) Note: 如果用户不想依赖配置文件创建数据任务，这里可以由用户自行填写__new__中的创建过程，并保证最后返回的是mindspore dataset实例即可 ```python """Masked Image Modeling Dataset.""" import os import mindspore from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from mindformers.dataset.dataloader import build_dataset_loader from mindformers.dataset.mask import build_mask from mindformers.dataset.transforms import build_transforms from mindformers.dataset.sampler import build_sampler from mindformers.dataset.base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class MIMDataset(BaseDataset): """ Masked Image Modeling Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MIMDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['masked_image_modeling']['mae_vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/mae.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MIMDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MIMDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Masked Image Modeling Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) mask = build_mask(dataset_config.mask_policy) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) if transforms is not None: for column in dataset_config.input_columns: dataset = get_dataset_map(dataset, input_columns=column, operations=transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if mask is not None: dataset = get_dataset_map(dataset, operations=mask, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.project(columns=dataset_config.output_columns) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset ``` - [Build接口](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/dataset/build_dataset.py)：该接口无需用户开发，以及集成在MindFormers套件中供调用 ```python """Build Dataset API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_dataset( config: dict = None, default_args: dict = None, module_type: str = 'dataset', class_name: str = None, **kwargs): r"""Build dataset For MindFormer. Instantiate the dataset from MindFormerRegister's registry. Args: config (dict): The task dataset's config. Default: None. default_args (dict): The default argument of dataset API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset'. class_name (str): The class name of dataset API. Default: None. Return: The function instance of dataset API. Examples: >>> from mindformers import build_dataset >>> from mindformers.dataset import check_dataset_config >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_class_name = build_dataset(class_name='ImageCLSDataset', dataset_config=config.train_dataset_task) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.DATASET, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) ``` - 创建Dataset实例 通过build_dataset接口，根据dataset的配置文件来创建MIMDataset实例： ```python from mindformers.tools.register import MindFormerConfig from mindformers.dataset import build_dataset, check_dataset_config # Initialize a MindFormerConfig instance with a specific config file of yaml. config = MindFormerConfig("configs/mae/task_config/mim_dataset.yaml") check_dataset_config(config) # 1) use config dict to build dataset dataset_from_config = build_dataset(config.train_dataset_task) # 通过train_dataset_task关键字来创建是MIMDataset API实例 # 2) use class name to build dataset dataset_from_name = build_dataset(class_name='MIMDataset', dataset_config=config.train_dataset_task) # 3) use class to build dataset dataset_from_class = MIMDataset(config.train_dataset_task) # 4) 不使用配置文件创建dataset实例 dataset_from_self_define1 = build_dataset(class_name='MIMDataset') dataset_from_self_define2 = MIMDataset() ``` pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/Parallel_Design.md000066400000000000000000001413511471400134000303640ustar00rootroot00000000000000# Parallel **《并行专题介绍》** ## 前言 ### 分布式并行场景 - **超大规模图像分类与识别** 这类网络往往有一个超大FC层以及loss层。在人脸识别和超大规模图像分类领域，**分类数可能高达千万，FC层参数量能到达到10B级别**。这类网络即使在数万分类这种场景，**模型并行相对数据并行来说性能也能更优，因为数据并行通信的是参数梯度，通信数据量等于参数量，而模型并行通信的是feature map，这类网络feature map的数据量远小于参数的数据量，进而在通信上能够性能更优。** - **高分辨率3D图像处理** 例如医疗领域的3维CT图像处理，典型网络是Unet-3D。这类场景，输入图像往往有亿级像素，模型的Feature Map非常大，叠加上比较深的网络结构，模型也往往是数百GB； - **NLP领域** NLP领域，目前基于Transformer结构的网络是SOTA方法，最典型网络的大网络有GPT2/3、Google翻译、FaceBook翻译等模型。基于Transformer结构的网络，从bert的340M参数，transformer-xl 800M参数，gpt-2 1542M参数，NVIDIA megatron有8B参数、GPT-3的175B参数。往往更大的网络，就需要用更大的数据来喂，训练这类网络，就需要使用**数据并行叠加模型并行的方式，同时处理大规模数据集及大规模参数。** - **推荐网络 Wide&Deep/DeepFM/DCN**等 在推荐领域，特征数量是百亿甚至千亿的规模，会有超大Embedding层，内存开销远超单卡内存，就必须使用模型并行把Embedding参数切分到集群。 总的来说，不同场景的模型，结构各有不同，如下图所示，不同结构的模型，需要不同的并行策略，才能实现性能的最优。图中方块的大小表示layer的内存开销。 ![image.png](https://bbs-img.huaweicloud.com/data/attachment/forum/202011/03/092326vsbttpsalxmf11yl.png) **Transformer一般会采用数据并行叠加模型并行的策略；推荐一般会采用模型并行转数据并行的策略；ReID一般会采用数据并行转模型并行的策略；** ### MindSpore分布式并行模式简介 当前MindSpore提供分布式并行训练的功能，它支持了多种模式包括： | 并行模式 | 配置 | 动态图 | 静态图 | 支持设备 | | ---------- | ------------------ | ------ | ------ | --------------- | | 数据并行 | DATA_PARALLEL | 支持 | 支持 | GPU、Ascend 910 | | 半自动并行 | SEMI_AUTO_PARALLEL | 不支持 | 支持 | GPU、Ascend 910 | | 全自动并行 | AUTO_PARALLEL | 不支持 | 支持 | GPU、Ascend 910 | | 混合并行 | HYBRID_PARALLEL | 不支持 | 支持 | GPU、Ascend 910 | - `DATA_PARALLEL`：**数据并行模式**。用户的网络参数规模在单卡上可以计算的情况下使用。这种模式会在每卡上复制相同的网络参数，训练时输入不同的训练数据，适合大部分用户使用。 - `AUTO_PARALLEL`：**自动并行模式**。融合了数据并行、算子级模型并行的分布式并行模式，可以自动建立代价模型，找到训练时间较短的并行策略，为用户选择合适的并行模式。当前MindSpore支持算子级并行策略的自动搜索，提供了如下的三种不同的策略搜索算法： - `dynamic_programming`：动态规划策略搜索算法。能够搜索出代价模型刻画的最优策略，但在搜索巨大网络模型的并行策略时耗时较长。其代价模型是围绕Ascend 910芯片基于内存的计算开销和通信开销对训练时间建模。 - `recursive_programming`：双递归策略搜索算法。对于巨大网络以及大规模多卡切分能够保证瞬间生成最优策略。其基于符号运算的代价模型可以自由适配不同的加速器集群。 - `sharding_propagation`：**切分策略传播算法**。由配置并行策略的算子向未配置的算子传播并行策略。在传播时，算法会尽量选取引发张量重排布通信最少的策略。关于算子的并行策略配置和张量重排布，可参考这篇[设计文档](https://www.mindspore.cn/docs/zh-CN/r1.9/design/distributed_training_design.html#自动并行原理)。 该模式适用于用户的神经网络在单卡上无法计算，但是不知道如何配置算子策略。用户启动这种模式，MindSpore会自动针对每个算子进行配置策略，**适合想要并行训练但是不知道如何配置策略的用户**。 - `SEMI_AUTO_PARALLEL`：**半自动并行模式**。相较于自动并行，该模式需要用户对算子手动配置切分策略实现并行。 - `HYBRID_PARALLEL`：**混合并行模式**。在MindSpore中特指用户通过手动切分模型实现混合并行的场景。该模式完全由用户自己设计并行训练的逻辑和实现，用户可以自己在网络中定义`AllGather`等通信算子，适合熟悉并行训练的用户。 ## MindSpore 分布式并行设计 ### 设计抽象 MindSpore目前是基于**集合通信**的模式来实现并行，如下图所示，分布式并行问题可以抽象成这样一个问题： **将计算图（Graph）和Tensor切分到集群并行执行，并能保持高性能。** ![image.png](https://bbs-img.huaweicloud.com/data/attachment/forum/202011/03/0924434kcjcswerenfg2yz.png) **从Tensor切分角度看数据并行、模型并行和混合并行。** 如下图所示，**数据并行就是在输入数据Tensor的batch维度切分，模型并行就是对模型参数Tensor的切分，而混合并行就是对输入数据Tensor和模型参数Tensor同时切分**。 ![image.png](https://bbs-img.huaweicloud.com/data/attachment/forum/202011/03/092515mmmwvsrt1iyu1rgi.png) ### 设计目标 MindSpore自动并行的目标是**构建一种易用高效的的分布式并行训练模式，融合了数据并行、模型并行和混合并行，让算法人员不再需要关注算法模型到底需要用哪种模式训练**。主要目标是： - **简化分布式并行编程，串行代码实现分布式训练，对用户屏蔽并行细节，并且保持高性能；** - **计算逻辑上保持和单卡串行流程一致；** - **实现上统一数据并行和模型并行，一套框架支持多种并行模式；** - **结合集群拓扑优化性能；** 实现上，就像前面介绍一下，MindSpore的自动并行分成两部分，半自动并行和全自动并行。 **半自动并行**：主要是在图编译阶段实现了一套算子切分及图切分的框架，把模型切片和设备进行调度绑定。在API这层，把并行逻辑和算法逻辑解耦，变成了一些配置。 **自动并行**：其在半自动并行基础上，构建了一套Cost Model，能够了基于数据量、模型参数量、网络集群拓扑带宽等信息的代价模型，通过一套策略搜索算法，计算是性能最优的切分策略，这样用户就不需要感知切分策略。 ## 数据并行和自动并行 ### 数据并行 数据并行是业界非常常用的并行方式，本章节开始介绍MindSpore中`ParallelMode.DATA_PARALLEL`数据并行的实现原理和使用方式。 #### 数据并行原理 ![数据并行图解](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/data_parallel.png) 1. 环境依赖 每次开始进行并行训练前，通过调用`mindspore.communication.init`接口初始化通信资源，并自动创建全局通信组`WORLD_COMM_GROUP`。 2. 数据分发（Data distribution） 数据并行的核心在于将数据集在样本维度拆分并下发到不同的卡上。在`mindspore.dataset`模块提供的所有数据集加载接口中都有`num_shards`和`shard_id`两个参数，它们用于将数据集拆分为多份并循环采样的方式，采集`batch`大小的数据到各自的卡上，当出现数据量不足的情况时将会从头开始采样。 3. 网络构图 数据并行网络的书写方式与单机网络没有差别，这是因为在正反向传播（Forward propagation & Backward Propagation）过程中各卡的模型间是独立执行的，只是保持了相同的网络结构。唯一需要特别注意的是为了保证各卡间训练同步，相应的网络参数初始化值应当是一致的，在`DATA_PRALLEL`和`HYBRID_PARALLEL`模式下建议通过使能`parameter_broadcast`达到权重广播的目的；在`AUTO_PRALLEL`和`SEMI_AUTO_PARALLEL`模式下，框架内部会自动分析参数的并行度，并设置相应的随机数种子，保证在数据并行维度的设备上参数初始化值一致。 4. 梯度聚合（Gradient aggregation） 数据并行理论上应该实现和单机一致的训练效果，为了保证计算逻辑的一致性，在梯度计算完成后插入`AllReduce`算子实现各卡间的梯度聚合操作。**MindSpore设置了`mean`开关，用户可以选择是否要对求和后的梯度值进行求平均操作，也可以将其视为超参项，打开开关等价于学习率倍数缩小。** 5. 参数更新（Parameter update） 因为引入了梯度聚合操作，所以各卡的模型会以相同的梯度值一起进入参数更新步骤。因此MindSpore实现的是一种同步数据并行训练方式。理论上最终每卡训练出来的模型是相同的，如果网络中含有在样本维度的归约类型操作，网络的输出可能会有所差别，这是由数据并行的切分性质决定的。 #### 数据并行代码 1. 集合通信 - [management.py](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/python/mindspore/communication/management.py)：这个文件中涵盖了集合通信过程中常用的`helper`函数接口，例如获取集群数量和卡的序号等。当在Ascend芯片上执行时，框架会加载环境上的`libhccl.so`库文件，通过它来完成从Python层到底层的通信接口调用。 - [comm_ops.py](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/python/mindspore/ops/operations/comm_ops.py)：MindSpore将支持的集合通信操作都封装为算子的形式放在这个文件下，包括`AllReduce`、`AllGather`、`ReduceScatter`和`Broadcast`等。`PrimitiveWithInfer`中除了定义算子所需属性外，还包括构图过程中输入到输出的`shape`和`dtype`推导。 2. 梯度聚合 - [grad_reducer.py](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/python/mindspore/nn/wrap/grad_reducer.py)：这个文件实现了梯度聚合的过程。对入参`grads`用`HyperMap`展开后插入`AllReduce`算子，这里采用的是全局通信组，用户也可以根据自己网络的需求仿照这个模块进行自定义开发。MindSpore中单机和分布式执行共用一套网络封装接口，在`Cell`内部通过`ParallelMode`来区分是否要对梯度做聚合操作，网络封装接口建议参考`TrainOneStepCell`代码实现。 #### 数据并行案例 在数据并行中，用户定义网络的方式和单机脚本一样，但是在网络定义之前调用[init()](https://www.mindspore.cn/docs/zh-CN/r1.9/api_python/mindspore.communication.html#mindspore.communication.init)去初始化设备通信状态。 ```python import numpy as np import mindspore as ms from mindspore.communication import init from mindspore import ops, nn class DataParallelNet(nn.Cell): def __init__(self): super(DataParallelNet, self).__init__() # 初始化权重 weight_init = np.random.rand(512, 128).astype(np.float32) self.weight = ms.Parameter(ms.Tensor(weight_init)) self.fc = ops.MatMul() self.reduce = ops.ReduceSum() def construct(self, x): x = self.fc(x, self.weight) x = self.reduce(x, -1) return x init() # 设置并行模式为数据并行，其他方式一致 ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.DATA_PARALLEL) net = DataParallelNet() model = ms.Model(net) model.train(*args, **kwargs) ``` ### 自动并行 自动并行作为MindSpore的关键特性，用于实现自动的数据并行加模型并行的混合并行训练方式，旨在帮助用户以单机的脚本表达并行算法逻辑，降低分布式训练难度，提高算法研发效率，同时又能保持训练的高性能。这个小节介绍了在MindSpore中`ParallelMode.AUTO_PARALLEL`自动并行模式及`ParallelMode.SEMI_AUTO_PARALLEL`半自动并行模式是如何工作的。 #### 自动并行原理 下面给出MindSpore自动并行原理设计图： **注意**：图示的Shard Strategy Search中缺少 Sharding Propagation方法，这是目前自动并行推荐的策略自动配置方法 ![自动并行图解](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/auto_parallel.png) 可以看到，图中给出了自动并行实现的关键流程，其中需经过一些关键操作完成整个自动并行的实现，这里先介绍一下基本的概念： - 关键概念 | 关键概念 | 说明 | | :-------------------------------------: | :----------------------------------------------------------: | | ANF（Accelerate Network Forward） Graph | 正向计算图 | | Sub ANF Graph | 正向计算子图 | | Graph Partition | 图切分 | | Layer-wise partition | 按layer切分 | | Pipeline partition | 按stage/block切分 | | Distributed Operator | 分布式算子 | | Distributed Optimizer | 分布式优化器 | | Distributed Auto-Grad | 分布式自动微分 | | Tensor Layout | 张量排布（张量被切分之后在集群上的分布情况） | | Tensor Redistribution | 张量重排 （两种Tensor Layout之间的转换）
当上一个算子的输出切分策略和下一个算子的输入切分策略不匹配时，会触发张量重排，会自动插入一些通信算子以保证计算逻辑的正确性 | | Device Topology | 设备拓扑 | | Group Manager | 群组管理 | | Cost Model | 代价模型 | | Distributed Operator Cost | 计算分布式算子代价（确定分布式算子切分策略） | | Cost Weight Estimator | 估计分布式算子的重要性（估计分布式算子不同切分策略的权重占比） | #### 算子级切分 本小接将会介绍如何基于上述并行原理中的**分布式算子和张量排布模型（Distributed Operator && Tensor Layout）**来实现MindSpore中的算子切分。 计算图中的分布式算子计算流程 - 首先对每个算子的输入tensor按策略进行切分，生成算子输入的tensor layout，然后根据算子的数学定义，推导出输出tensor layout； - 然后再检查前一个算子输出tensor layout和下一个算子的输入tensor layout，如果两种不同，则会插入一个tensor redistribution。 ![image.png](https://bbs-img.huaweicloud.com/data/attachment/forum/202011/03/092548flhx7ubmapjrlnvb.png) **Tensor Layout**表示tensor切分后，tensor切片在集群分布情况。 **Tensor Redistribution**表示两种Tensor Layout之间的转换。 **Note**：这种建模设计充分地表达了张量和设备间的映射关系，用户无需感知模型各切片放到哪个设备上运行，框架会自动调度分配。 - **如何得到每个算子的张量排布模型？？？** 为了得到张量的排布模型，每个算子都具有切分策略（Shard Strategy），它表示算子的各个输入在相应维度的切分情况。 MindSpore中目前支持用户手动对算子进行shard配置或使用全自动并行模式进行自动获取两种形式。 - **切分策略配置原则？？？** - 通常情况下只要满足以2为基、均匀分配的原则，张量的任意维度均可切分； - MindSpore为内置算子均实现了shard属性，用户可以通过operator.shard()来为单算子的输入或者输出配置相应的切分策略； - shard属性的入参：`.shard(in_strategy, out_strategy)`，其中in_strategy，out_strategy分布表示算子的输入、输出的切分策略，且均需以元组形式传入，比如当算子本身只需一个Tensor输入时，`in_strategy=((1, 1, 1),)`，当算子本身需要两个Tensor输入时，`in_strategy=((1, 1, 1), (1, 1, 1))`，以此类推即可。 - 不切分的维度shard策略写1，需要切分的维度能够被切分策略整除(建议切分策略以2为基数，被切分之后的tensor shape仍是2的倍数，可以获得较好的切分性能)； - 如果一个算子同时对输入的数据和模型权重同时切分，假设数据（batch维度）部分切dp份，模型权重部分切mp份，需要满足`dp * mp <= total_device_num` - **算子级切分用例** 以下图为例，这是一个三维矩阵乘（BatchMatMul）操作，它的切分策略由两个元组构成，分别表示`input`和`weight`的切分形式。 ![算子切分定义](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/operator_split.png) ```python from mindspore.ops import operations as P batch_matmul = P.BatchMatMul() # 根据算子要输入的Tensor维度来配置切分策略 # 以MatMul算子为例，其一般有两个输入 input, weight， 假设其维度分别为[batch, height, width] [batch, width, channel] # 因此可以分别为这两个Tensor配置切分策略 # 配置1：不做切分 batch_matmul.shard(((1, 1, 1), (1, 1, 1)),) # 配置2：batch维度切分，切8份，8卡数据并行，只切数据，不切参数，batch要求能够被8整除，建议的数值有8,16,32,64.... batch_matmul.shard(((8, 1, 1), (1, 1, 1)),) # 配置3：混合切分策略，数据和模型参数均切分， 数据切4份（dp），参数切2份（mp）， 其中dp和mp值需遵循dp*mp <= total_device_num的原则 batch_matmul.shard(((4, 1, 1), (1, 1, 2)),) ``` #### 张量排布变换（Tensor Redistribution） 当前一个算子的输出张量模型和后一个算子的输入张量模型不一致时，就需要引入计算、通信操作的方式实现张量排布间的变化。自动并行流程引入了张量重排布算法（Tensor Redistribution），可以推导得到任意排布的张量间通信转换方式。下面三个样例表示公式`Z=(X×W)×V`的并行计算过程，即两个二维矩阵乘操作，体现了不同并行方式间如何转换。 在样例一中，第一个数据并行矩阵乘的输出在行方向上存在切分，而第二个模型并行矩阵乘的输入需要全量张量，框架将会自动插入`AllGather`算子实现排布变换。 ![张量排布变换](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/tensor_redistribution1.png) ```python from mindspore.ops import operations as P # 图1中的Y = X * W 计算 matmul_1 = P.MatMul() # 在X的batch维度切分成4份，W不切分 matmul_1.shard(((4, 1, 1), (1, 1)),) # X --> shape: [64, 196, 3] --> 切分成4份 X1: [16, 196, 3] X2: [16, 196, 3] X3: [16, 196, 3] X4: [16, 196, 3] # W --> shape: [3, 32] # 图1中的Z = Y * V 计算 matmul_2 = P.MatMul() # Y不切分，V在channel维度切分成4份 matmul_2.shard(((1, 1, 1), (1, 4)),) # Y --> shape: [64, 196, 32] # V --> shape: [32, 768] --> channel维度切分成4份 V1: [64, 32, 192] V2: [32, 192] V3: [32, 192] V4: [32, 192] Y = batch_matmul_1(X, W) # 内部并行计算逻辑, X切分后使用Scatter算子进行Tensor分发到不同的设备上并行执行运算，具体分在哪台设备上运算由底层算法分配，上层不感知 # X1 * W --> Y1: shape [16, 196, 32] -- *device1 # X2 * W --> Y2: shape [16, 196, 32] -- *device2 # X3 * W --> Y3: shape [16, 196, 32] -- *device3 # X4 * W --> Y4: shape [16, 196, 32] -- *device4 # Y = [Y_c1, Y_c2, Y_c3, Y_c4]T 此时每个计算结果还保留在各自的设备上， T表示转置 # 由于matmul_2算子的输入Y 是 matmul_1算子的输出，但两者的切分策略不一致 # matmul_1.shard(((4, 1, 1), (1, 1)),) <--> matmul_2.shard(((1, 1, 1), (1, 4)),) # 这就导致matmul_2要求的输入必须是一个完整的数据shape，因此在matmul_2计算前，要对matmul_1的输出做All-Gather，将分散在各个设备上的计算结果收集拼接并同步 Z = matmul_2(Y, V) # 先经过All-Gather之后收集Y1, Y2, Y3, Y4 拼接成完整的数据Y： [64, 196, 32] # 将V在channel维度切分成4份： # V1: [32, 192] # V2: [32, 192] # V3: [32, 192] # V4: [32, 192] # 内部并行计算逻辑，V切分后使用Scatter算子分发Tensor给不同设备并行计算： # Y * V1 --> Z1：shape [64, 196, 192] -- *device1 # Y * V2 --> Z2：shape [64, 196, 192] -- *device2 # Y * V3 --> Z3：shape [64, 196, 192] -- *device3 # Y * V4 --> Z4：shape [64, 196, 192] -- *device4 ``` 在样例二中，第一个模型并行矩阵乘的输出在列方向上存在切分，而第二个数据并行矩阵乘的输入在行方向上存在切分，框架将会自动插入等价于集合通信中`AlltoAll`操作的通信算子实现排布变换。 ![张量排布变换](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/tensor_redistribution2.png) ```python from mindspore.ops import operations as P # 图2中的Y = X * W 计算 matmul_1 = P.MatMul() # X不切分，W在channel维度切分成4份 matmul_1.shard(((1, 1, 1), (1, 4)),) # X --> shape: [64, 196, 3] # W --> shape: [3, 32] --> 在channel处切分： W_c1: [3, 8] W_c2: [3, 8] W_c3: [3, 8] W_c4: [3, 8] # 图2中的Z = Y * V 计算 matmul_2 = P.MatMul() # Y在batch维度切分，V不切分 matmul_2.shard(((4, 1, 1), (1, 1)),) # Y --> shape: [64, 196, 32] --> batch维度切分成4份 V1: [16, 196, 32] V2: [16, 196, 32] V3: [16, 196, 32] V4: [16, 196, 32] # V --> shape: [32, 768] Y = matmul_1(X, W) # 内部并行计算逻辑, 分别在不同的设备上并行执行运算，具体分在哪台设备上运算由底层分配，上层不感知 # X [64, 196, 3] * W_c1 [64, 3, 8] --> Y_c1: shape [64, 196, 8] -- *device1 # X * W_c2 --> Y_c2: shape [64, 196, 8] -- *device2 # X * W_c3 --> Y_c3: shape [64, 196, 8] -- *device3 # X * W_c4 --> Y_c4: shape [64, 196, 8] -- *device4 # Y = [Y_c1, Y_c2, Y_c3, Y_c4] 此时每个计算结果还保留在各自的设备上 # 由于matmul_2算子的输入Y 是 matmul_1算子的输出，但两者的切分策略不一致 # matmul_1.shard(((1, 1, 1), (1, 4)),) <--> matmul_2.shard(((4, 1, 1), (1, 1)),) # 即需要将每个device上的 Tensor Shape [64, 196, 8] --> [16, 196, 32] Tensor Layout转换 # 这就导致matmul_2要求的输入不仅是完整的权重shape，而且需要进一步完成batch维度的切分操作。 # 通常的做法是先All-Gather，将Y转换成一个完整的权重shape: [64, 196, 32]， 然后再执行Scatter操作，将Y在batch维度切分之后分发出去 # 然而All-To-All操作可以很方便且高效的同时完成上述两种操作，在All-Gather的同时进行Scatter，完成上述转换过程！ Z = matmul_2(Y, V) # 先经过All-To-All之后完成Y在batch维度切分成4份并分发给不同的设备： # Y_b1: [16, 196, 32] # Y_b2: [16, 196, 32] # Y_b3: [16, 196, 32] # Y_b4: [16, 196, 32] # 内部并行计算逻辑 # Y_b1 * V --> Z_b1：shape [ , 196, 192] -- *device1 # Y_b2 * V --> Z_b2：shape [64, 196, 192] -- *device2 # Y_b3 * V --> Z_b3：shape [64, 196, 192] -- *device3 # Y_b4 * V --> Z_b4：shape [64, 196, 192] -- *device4 ``` 在样例三中，第一个混合并行矩阵乘的输出切分方式和第二个混合并行矩阵乘的输入切分方式一致，所以不需要引入重排布变换。但由于第二个矩阵乘操作中，两个输入的相关维度存在切分，所以需要插入`AllReduce`算子保证运算正确性。 ![张量排布变换](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/tensor_redistribution3.png) ```python from mindspore.ops import operations as P # 图3中的Y = X * W 计算 matmul_1 = P.MatMul() # X在batch维度切2份，W在channel维度切分成2份 matmul_1.shard(((2, 1, 1), (1, 2)),) # X --> shape: [64, 196, 3] --> 在batch处切分： X1：[32, 196, 3] X2: [32, 196, 3] # W --> shape: [3, 32] --> 在channel处切分： W1: [3, 16] W2: [3, 16] # 图3中的Z = Y * V 计算 matmul_2 = P.MatMul() # Y在batch维度切分2份，V在batch维度切分2份 matmul_2.shard(((2, 1, 1), (2, 1)),) # Y --> shape: [64, 196, 32] --> 在batch处切分： Y1：[32, 196, 32] Y2: [32, 196, 32] # V --> shape: [32, 768] --> 在0维处切分： V1: [16, 768] V2: [16, 768] Y = matmul_1(X, W) # X, W 维度切分之后利用Scatter分发Tensor到不同设备上进行并行计算 # [X1, X2]T * [W1, W2] = [X1*W1, X1*W2 = [Y11, Y12 # X2*W1, X2*W2] Y21, Y22] # X1([32, 196, 3]) * W1(3, 16) = Y11([32, 196, 16]) --> *device1 # X1([32, 196, 3]) * W2(3, 16) = Y12([32, 196, 16]) --> *device2 # X2([32, 196, 3]) * W1(3, 16) = Y21([32, 196, 16]) --> *device3 # X2([32, 196, 3]) * W1(3, 16) = Y22([32, 196, 16]) --> *device4 # 由于matmul_2算子的输入Y 是 matmul_1算子的输出，但两者的切分策略一致 # matmul_1.shard(((2, 1, 1), (1, 2)),) <--> matmul_2.shard(((2, 1, 1), (2, 1)),) # 既不需要在中间插入额外Tensor Redistribution变换，直接可以使用各设备上的值进行运算 Z = matmul_2(Y, V) # 由于对Tensor Y的切分策略相同，不需要再进行Tensor Redistribution，所以此处直接使用每台设备上相应的Y*切片即可 # V在切分之后，分发给不同的设备进行并行计算 [2,2] * [2,1] --> [2,1] # [Y11, Y12 * [V1 = [Y11*V1 + Y12*V2 = [Z1 # Y21, Y22] V2] Y21*V1 + Y22*V2] Z2] # Y11([32, 196, 16]) * V1([16, 768]) = Z11([32, 196, 768]) # Y12([32, 196, 16]) * V2([16, 768]) = Z12([32, 196, 768]) # Y21([32, 196, 16]) * V1([16, 768]) = Z21([32, 196, 768]) # Y22([32, 196, 16]) * V2([16, 768]) = Z22([32, 196, 768]) # Z11 + Z12 = Z1 All-Reduce(SUM) # Z21 + Z22 = Z2 All-Reduce(SUM) # 其中由于Y11*V1、Y12*V2、Y21*V1、Y22*V2分别在不同的设备上，因此要执行加法操作的话，必须要插入All-Reduce算子进行SUM的规约运算 ``` 综上，1、2两点是自动并行实现的基础，总体来说这种分布式表达打破了数据并行和模型并行的边界，轻松实现混合并行。 从脚本层面上，用户仅需构造单机网络，即可表达并行算法逻辑，框架将自动实现对整图切分。 - 切分策略搜索算法 当用户熟悉了算子的切分表达，并手动对算子配置切分策略，这就是`SEMI_AUTO_PARALLEL`半自动并行模式。这种方式对手动调优有帮助，但还是具有一定的调试难度，用户需要掌握并行原理，并根据网络结构、集群拓扑等计算分析得到高性能的并行方案。为了帮助用户加速并行网络训练过程，在半自动并行模式的基础上，`AUTO_PARALLEL`自动并行模式支持并行策略传播（Sharding Propagation），能够有效地降低用户手配算子切分策略的工作量，算法将切分策略由用户配置的算子向未配置的算子传播。 为进一步降低用户手配算子切分策略的工作量，支持切分策略完全自动搜索。为此，围绕硬件平台构建相应的代价函数模型（Cost Model），计算出一定数据量、一定算子在不同切分策略下的计算开销（Computation Cost），内存开销（Memory Cost）及通信开销（Communication Cost）。然后通过动态规划算法（Dynamic Programming）或者递归规划算法（Recursive Programming），以单卡的内存上限为约束条件，高效地搜索出性能较优的切分策略。 策略搜索这一步骤代替了用户手动指定模型切分，在短时间内可以得到较高性能的切分方案，极大降低了并行训练的使用门槛。 - 分布式自动微分 传统的手动模型切分除了需要关注正向网络通信还需要考虑网络反向的并行运算，MindSpore通过将通信操作包装为算子，并利用框架原有的自动微分操作自动生成通信算子反向，所以即便在进行分布式训练时，用户同样只需关注网络的前向传播，真正实现训练的全自动并行。 #### 自动并行代码 1. 张量排布模型 - [tensor_layout](https://gitee.com/mindspore/mindspore/tree/r1.9/mindspore/ccsrc/frontend/parallel/tensor_layout)：这个目录下包含了张量排布模型相关功能的定义及实现。其中`tensor_layout.h`中声明了一个张量排布模型需要具备的成员变量`tensor_map_origin_`，`tensor_shape_`和`device_arrangement_`等。在`tensor_redistribution.h`中声明了实现张量排布间`from_origin_`和`to_origin_`变换的相关方法，将推导得到的重排布操作保存在`operator_list_`中返回，并计算得到重排布所需的通信开销`comm_cost_`, 内存开销`memory_cost_`及计算开销`computation_cost_`。 2. 分布式算子 - [ops_info](https://gitee.com/mindspore/mindspore/tree/r1.9/mindspore/ccsrc/frontend/parallel/ops_info)：这个目录下包含了分布式算子的具体实现。在`operator_info.h`中定义了分布式算子实现的基类`OperatorInfo`，开发一个分布式算子需要继承于这个基类并显式实现相关的虚函数。其中`InferTensorInfo`，`InferTensorMap`和`InferDevMatrixShape`函数定义了推导该算子输入、输出张量排布模型的算法。`InferForwardCommunication`，`InferMirrorOps`等函数定义了切分该算子需要插入的额外计算、通信操作。`CheckStrategy`和`GenerateStrategies`函数定义了算子切分策略校验和生成。根据切分策略`SetCostUnderStrategy`将会产生该策略下分布式算子的并行开销值`operator_cost_`。 3. 策略搜索算法 - [auto_parallel](https://gitee.com/mindspore/mindspore/tree/r1.9/mindspore/ccsrc/frontend/parallel/auto_parallel)：这个目录下实现了切分策略搜索的算法。`graph_costmodel.h`定义了构图信息，其中每个点表示一个算子`OperatorInfo`，有向边`edge_costmodel.h`表示算子的输入输出关系及重排布的代价。`operator_costmodel.h`中定义了每个算子的代价模型，包括计算代价、通信代价和内存代价。`dp_algorithm_costmodel.h`主要描述了动态规划算法的主要流程，由一系列图操作组成。在`costmodel.h`中定义了cost和图操作的数据结构。 4. 设备管理 - [device_manager.h](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/ccsrc/frontend/parallel/device_manager.h)：这个文件实现了集群设备通信组的创建及管理。其中设备矩阵模型由`device_matrix.h`定义，通信域由`group_manager.h`管理。 5. 整图切分 - [step_auto_parallel.h](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/ccsrc/frontend/parallel/step_auto_parallel.h), [step_parallel.h](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/ccsrc/frontend/parallel/step_parallel.h)：这两个文件包含了自动并行流程的核心实现。首先由`step_auto_parallel.h`调用策略搜索流程并产生分布式算子的`OperatorInfo`，然后在`step_parallel.h`中处理算子切分和张量重排布等流程，对单机计算图进行分布式改造。 6. 通信算子反向 - [grad_comm_ops.py](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/python/mindspore/ops/_grad/grad_comm_ops.py)：这个文件定义了`AllReduce`和`AllGather`等通信算子的反向操作。 ### 自动并行案例 MindSpore2.0及之前版本在启动半自动和自动模式进行训练时，**必须**通过`model.train(*args, **kwargs)`接口进行训练，不支持自定义循环进行网络训练。 #### 半自动并行 半自动并行模式相较于自动并行模式，需要用户**手动配置**每个算子的**shard**接口对并行策略进行调优。 以`SemiAutoParallelNet`为例，在半自动并行模式下的脚本代码如下，`MatMul`的切分策略为`((1, 1),(1, 2))`，指定`self.weight`在第二维度上被切分两份。 ```python import numpy as np import mindspore as ms from mindspore.communication import init from mindspore import ops, nn class SemiAutoParallelNet(nn.Cell): def __init__(self): super(SemiAutoParallelNet, self).__init__() # 初始化权重 weight_init = np.random.rand(128, 128).astype(np.float32) self.weight = ms.Parameter(ms.Tensor(weight_init)) self.weight2 = ms.Parameter(ms.Tensor(weight_init)) # 设置切分策略。在construct中fc的输入有两个，第一个输入是x，第二个输入是权重self.weight # 因此shard需要提供一个tuple元组，分别对应每个输入tensor在对应维度的切分份数 # (1,1)表示输入x的每一维度都没有切分 # (1,2)表示在self.weight的第二维度上切成了两份 # 切分的过程是在图编译的过程中，在编译完成后，self.weight的shape就会发生改变 self.fc = ops.MatMul().shard(((1, 1),(1, 2))) self.reduce = ops.ReduceSum() def construct(self, x): # x shape [64, 196, 128] self.weight shape [128, 128] # x 不切分， self.weight在第二个维度切分2份： [w1, w2] # 内部并行计算逻辑： # x([64, 196, 128]) * w1([128, 64]) = x1([64, 196, 64]) --> *device0 # x([64, 196, 128]) * w2([128, 64]) = x2([64, 196, 64]) --> *device1 # 结果：[x1, x2] 此时x1、x2还分布在各自device上 x = self.fc(x, self.weight) # 在construct函数中去初始化并行调用operation算子时，相当于用户没有设置matmul算子的策略。 # 那么默认的策略会自动配置数据并行，即((8, 1), (1, 1))。其中8表示用户此次运行的卡数 # 显然当前算子的策略和上一个算子策略不匹配（上一个算子的输出是这个算子的输入时）：shard(((1, 1),(1, 2))) <--> shard(((8, 1), (1, 1))) # 因此需要引入Tensor Redistribution变换操作: # 将： x1 [64, 196, 64] x2 [64, 196, 64] --> [8, 196, 128 # 8, 196, 128 # ..... ] # 8份 # 显然要完成这部分操作，要同时经过StrideSlice切片和All-Gather算子拼接同步结果，再经由Scatter算子分发重新切分的Tensor数据 # 此时每台设备（8 device）上x的shape均为 [8, 196, 128] x = ops.MatMul()(x, self.weight2) # 内部并行计算逻辑： # x1([8, 196, 128]) * w([128, 128]) = x1([8, 196, 128]) --> *device0 # x2([8, 196, 128]) * w([128, 128]) = x2([8, 196, 128]) --> *device1 # .......8张卡同时运算 --> *device(2~7) x = self.reduce(x, -1) # 由于self.reduce没有配置并行策略，默认的策略为shard(((8, 1, 1),),),因此并行策略与上一个算子策略相同，故不需要插入额外的算子进行重排操作 # 直接可以并行计算得到每台设备的x值 return x init() ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.SEMI_AUTO_PARALLEL) net = SemiAutoParallelNet() model = ms.Model(net) model.train(*args, **kwargs) ``` #### 全自动并行 自动并行模式，融合了数据并行、模型并行及混合并行的分布式并行模式，可以自动建立代价模型，找到训练时间较短的并行策略，为用户选择合适的并行模式。MindSpore提供了如下的三种不同的策略搜索算法： - `dynamic_programming`：动态规划策略搜索算法。能够搜索出代价模型刻画的最优策略，但在搜索巨大网络模型的并行策略时耗时较长。其代价模型是围绕Ascend 910芯片基于内存的计算开销和通信开销对训练时间建模。 - `recursive_programming`：双递归策略搜索算法。对于巨大网络以及大规模多卡切分能够保证瞬间生成最优策略。其基于符号运算的代价模型可以自由适配不同的加速器集群。 - `sharding_propagation`：**切分策略传播算法。由配置并行策略的算子向未配置的算子传播并行策略。在传播时，算法会尽量选取引发张量重排布通信最少的策略。关于算子的并行策略配置和张量重排布，可参考[半自动并行](####半自动并行)。** 用户可以通过如下代码去设置上述的策略搜索算法： ```python import mindspore as ms # 设置动态规划算法进行策略搜索 ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.AUTO_PARALLEL, search_mode="dynamic_programming") # 设置双递归方法进行策略搜索 ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.AUTO_PARALLEL, search_mode="recursive_programming") # 设置切分策略传播算法 ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.AUTO_PARALLEL, search_mode="sharding_propagation") ``` > - 在`sharding_propagation`模式下，算法根据用户设置的`shard`策略传播到整个模型，在`dynamic_programming`模式下，用户设置的`shard`策略也会生效，不会被搜索出来的策略覆盖掉。 > - 在全自动并行模式下，如果需要对某个Cell里的所有算子手动配置数据并行策略，可用Cell.set_data_parallel()统一设置。 #### 混合并行 在MindSpore中特指用户通过手动切分模型实现混合并行的场景，用户可以在网络结构中定义通信算子原语`AllReduce`和`AllGather`等，手动执行并行流程。此时，用户需要自己实现参数的切分，算子切分后的通信等操作。例如下面的代码示例： ```python import numpy as np import mindspore as ms from mindspore.communication import init from mindspore import ops, nn class HybridParallelNet(nn.Cell): def __init__(self): super(HybridParallelNet, self).__init__() # 以下2卡运行的场景为例子，实现分布式矩阵乘法来模拟单卡矩阵乘的结果。 # 即原始的逻辑 # 输入x,weight的shape分别为(32, 512), (512, 128) # 经过计算：matmul(x, weight) # 输出结果shape为(32, 128)的tensor # 下面我们手动实现上面的矩阵乘法逻辑 # 我们需要手动的指定当前权重的切片的shape,我们希望在matmul的相关维度进行切分。相关维度切分的情况下 # 需要对matmul的结果进行AllReduce操作，确保数值和单机的保持一致 # # 分布式逻辑 # 输入x,weight的shape分别为(32, 256), (256, 128) # 经过计算 output = matmul(x, weight) # output = allreduce(output) # 输出结果shape为(32, 128)的tensor weight_init = np.random.rand(256, 128).astype(np.float32) self.weight = ms.Parameter(ms.Tensor(weight_init)) self.fc = ops.MatMul() self.reduce = ops.AllReduce() def construct(self, x): x = self.fc(x, self.weight) x = self.reduce(x) return x init() ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.HYBRID_PARALLEL) net = HybridParallelNet() model = ms.Model(net) model.train(*args, **kwargs) ``` #### 数据导入方式 在并行训练中，支持三种数据的导入方式： - 全量导入。仅在**半自动**和**全自动**并行模式下生效。用户可以通过`set_auto_parallel_context(full_batch=True)`开启。开启全量导入之后，在自动并行流程中认为读入的`batch`是一个网络输入的完整shape。例如，在8卡训练的情况下，假设每张卡`dataset`返回的shape是`[32, 8]`，那么当前一个迭代训练的训练的数据即为`[32, 8]`。因此，**用户需要保证每卡在每轮迭代输入的数据是一致的**。例如，确保每卡数据集的`shuffle`的顺序是一致的。 - 数据并行导入。用户不设置 `full_batch`的情况下，每卡读入的数据是当前训练迭代的一个分片，因此要求每卡读入的数据内容**不一样**。例如8卡训练的情况下，每卡读入数据的`shape`为`[32,8]`，那么当前一个迭代训练的数据总量为`[32*8, 8]`。 - 模型并行导入。模型并行导入的方式主要针对图像领域中图像尺寸太大无法在单卡进行计算时，直接在输入流程上就对图像进行切分。MindSpore在`set_auto_parallel_context`中提供了`dataset_strategy`接口，用户可以通过这个接口配置更加灵活的输入策略。注意，当用户使用此接口时，需要确保`dataset`返回的`tensor`符合对应的切分策略。如下代码所示： ```python import mindspore as ms # 设置输入在第1维度上进行切分， 此时要求用户确保dataset返回的输入在第1维度上进行切分 ms.set_auto_parallel_context(dataset_strategy=((1, 8), (1, 8))) # 相当于设置full_batch=False ms.set_auto_parallel_context(dataset_strategy="data_parallel") # 相当于设置full_batch=True ms.set_auto_parallel_context(dataset_strategy="full_batch") ``` 因此，在用户设置上述的配置之后，需要**手动**设置dataset的获取顺序，确保每卡的数据是期望的。 ### 流水线并行 [MindSpore流水线并行设计原理及使用实践](https://www.mindspore.cn/tutorials/experts/zh-CN/r2.0/parallel/pipeline_parallel.html#) ## MindFormers 并行手册 ### MindFormers 并行设计 MindFormers集成了MindSpore原生的并行能力，在`transformer API`中利用配置化方式对Transformer网络做并行配置。 ![parallel_mindformers](./assets/parallel_mindformers.png) MindFormers支持基于`Transformer API`开发的大模型通过配置化接口进行并行配置，主要提供两种使用方式：1. config配置文件； 2. Trainer高阶接口； ### config 并行配置 配置文件介绍及使用请参见：[MindFormers Config 配置]()，主要用于`run_mindformer.py`脚本启动时使用。 用户可通过提供的并行模块和关键字，完成并行策略的设定： - use_parallel: 是否开启并行 - parallel: 自动并行配置，相关入参支持可参考：[mindspore.set_auto_parallel_context](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_auto_parallel_context.html) - parallel_mode: 并行模式，0-dataset数据并行, 1-semi半自动并行, 2-auto自动并行, 3-hybrid手工实现并行 - gradients_mean: 是否在梯度AllReduce后执行平均算子。通常半自动并行模式下为False，数据并行模式下为True - enable_alltoall: 允许在通信期间生成AllToAll通信算子的开关。通常仅在MOE场景下打开，默认False - full_batch: 在auto_parallel模式下加载整个batch数据集时为True。半自动并行模式通常设置为True，数据并行模式必须设置为False，否则会报错 - search_mode: 策略搜索模式，有三种，分别是recursive_programming，dynamic_programming和sharding_propagation。仅在全自动并行模式下生效，其他模式不生效，实验性接口，谨慎使用 - enable_parallel_optimizer: 数据并行训练时对权重更新计算进行分片。优化器并行开关，在数据并行训练时默认会将模型权重参数切分成device_num份，与parallel_config中optimizer_shard保持一致；半自动并行时默认将模型权重参数切份data_parallel份 - strategy_ckpt_save_file: 保存并行切分策略的路径。 - parallel_config: 并行策略配置，相关入参配置可参考[transformer.TransformerOpParallelConfig](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/modules/transformer/transformer.py#L197) - data_parallel: 数据并行 - model_parallel: 模型并行 - pipeline_stage: 流水线并行 - optimizer_shard: 是否开启优化器切分。优化器并行开关，通常在半自动并行模式下生效，与parallel中的enable_parallel_optimizer保持一致，默认将模型权重参数切份data_parallel份 - micro_batch_num: 流水线并行的微批次大小。pipeline_satge大于1时，开启流水并行时使用，此处需满足micro_batch_num >= pipeline_satge - gradient_aggregation_group: 梯度通信算子融合组的大小 ### Trainer 并行配置 MindFormers Trainer接口介绍请见：[MindFormers Trainer API]() 利用 `Trainer.set_parallel_config(**kwargs)`来为模型设定切分策略：[set_parallel_config](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L690) 使用样例： ```python import argparse from mindformers import Trainer, TrainingArguments from mindformers import init_context, ContextConfig, ParallelContextConfig ############################################# 并行环境初始化 ################################################## def context_init(use_parallel=False, optimizer_parallel=False): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, enable_parallel_optimizer=optimizer_parallel, full_batch=True) rank_id, device_num = init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) ############################################# 并行环境初始化 ################################################## def main(use_parallel=False, run_mode='train', task='text_generation', model_type='gpt2', pet_method='', train_dataset='./train', eval_dataset='./eval', predict_data='hello!', dp=1, mp=1, pp=1, micro_size=1, op=False): # 环境初始化 context_init(use_parallel, op) # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=1, batch_size=2, learning_rate=0.001, warmup_steps=100, sink_mode=True, sink_size=2) # 定义任务，预先准备好相应数据集 task = Trainer(task=task, model=model_type, pet_method=pet_method, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) ############################################# 设定并行策略 ################################################## task.set_parallel_config(data_parallel=dp, model_parallel=mp, pipeline_stage=pp, optimizer_shard=op, micro_batch_num=micro_size) ############################################# 设定并行策略 ################################################## if run_mode == 'train': task.train() elif run_mode == 'finetune': task.finetune() elif run_mode == 'eval': task.evaluate() elif run_mode == 'predict': result = task.predict(input_data=predict_data) print(result) ``` ### 并行注意事项 -`data_parallel` * `model_parallel` * `pipeline_stage` <= `total_device_num`; -`micro_batch_num` >= `pipeline_satge`，`micro_batch_num`的大小通常越高，流水并行的效率就越好，但是往往会导致**编译时间过长和Task Stream超限2016的问题**，请根据实际情况调整该数值； -`model_parallel`的大小通常和`Transformer`中`hidden_size或vocab_size`的大小相关，需要满足`model_parallel` % `hidden_size` == 0,不然会导致相应参数无法切分，从而引入**额外的算子重排，引起性能劣化**； -`pipeline_satge`的大小通常和`Transformer`中`num_layers`的大小相关，建议配比满足`num_layers` % `pipeline_satge` == 0,使得`Transformer`的`Layer`层可以被均匀切分到不同的卡上，如果无法被均匀切分，容易导致**编译时间过长**或者**编译失败** - LLM 模型相关配置说明： - model: 模型配置 - arch: 模型类配置 - type: 模型类 - model_config: 模型参数配置 - type: 模型参数配置类 - checkpoint_name_or_path: 评估时不指定权重，模型默认加载的权重名 *# 以下配置针对大规模语言模型推理* - top_k: 从概率最大的top_k个tokens中采样 - top_p: 从概率最大且概率累计不超过top_p的tokens中采样 - do_sample: 使能top_k或top_p采样，为False时top_k和top_p均重置为1 - use_past: 使能增量推理，为True时为增量推理，否则为自回归推理，使用时请参考[模型支持列表](https://gitee.com/mindspore/mindformers/tree/dev/docs#text-generator) - max_decode_length: 文本生成最大长度（输入长度统计在内） - repetition_penalty: 重复文本惩罚系数，该值不小于1，等于1时不惩罚 pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/Pet_Tuners_Design.md000066400000000000000000000072121471400134000307150ustar00rootroot00000000000000# 低参微调 当前基于Transformer结构的LLMs取得了显著的成就，如何更好地利用已经开源的语言预训练大模型来应用到自己的应用场景中，往往需要使用微调来达到较好的。微调又分为两种：全参微调和低参微调，使用全参微调需要较大的算力，这往往对于普通开发者来说很难获取到足够大的算力资源来进行自己的实验；此时就需要使用低参微调（Parameters Efficient Tuning），只需要训练极少量的参数就可以接近全参微调的效果，以此突破算力限制。 当前，MindFormer是专注于Transformer结构的大模型的开发，对外使能开发者开发大模型，训练大模型；MindPet则是专注于低参微调算法的开发，对外提供低参微调算法接口，使能开发者修改自己开发的模型进行低参微调。但是，以上两个工具包均是使能各自的开发，而对于完整的大模型开发，则需要大模型的开发->大模型训练->低参微调的流程链路，当前存在断点，则希望在本工程中将MindFormer的能力进行拓展，能支持MindFormer开发的大模型自动的链接上MindPet上的低参微调算法，加速大模型的孵化。 ## 设计概述 由于是两个完全独立的开发组件，需要结合两者的功能来完成对于大模型的微调，此时使用适配器设计思路去整合两个部件则是最合适的。由此整体的设计如下图： ![整体设计图](assets/MindPet.png) 在本部分则是实现上图绿色部分的功能，调用微调组件的APIs，针对用户输入的模型进行适配修改，然后返回给用户，用户可以将其送入到MindFormer的Trainer接口中，进行后续的模型微调训练。 ## 接口设计 **PetAdapter** ```python class PetAdapter: r""" PetAdapter is the base class of adapter to modify the pretrained model. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): """Add efficient tuning parameters to ptm.""" raise(NotImplementedError("should implemented by the certain tuning algorithm.")) @classmethod def load_ckpt(cls, model: nn.Cell, config: PetConfig): """Load ckpt of ptm.""" pass @classmethod def get_pretrained_model(cls, config): """ Get pretrained model from config. """ pass @classmethod def freeze_pretrained_model(cls, model, pet_type:PetType): pass ``` PetAdapter是所有微调算法适配器的基类，针对于具体的微调算法，一般只需要实现实现`get_pet_model()`接口，将微调算法的改动添加到用户的模型中。 **PetAdapter.get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None)** ``` text 功能: 获取微调算法适配的预训练模型 参数: - model(nn.Cell)：原预训练模型，支持用户直接输入模型实例，默认是None，当输入是None值，则是从config中由用户输入的模型实例化MindFormer中支持的模型。 - config(PetConfig)：微调算法的配置，包含微调算法的超参或者需要实例化的预训练模型。 返回值： 适配了微调算法的预训练模型(nn.Cell) ``` **PetAdapter.get_pretrained_model(cls, config: PetConfig)** ``` text 功能: 实例化预训练模型，当用户不传入模型实例时 参数: - config：微调算法的配置参数 返回值: 返回预训练模型 ``` **PetAdapter.freeze_pretrained_model(cls, model, pet_type:PetType)** ``` text 功能: 根据微调算法类型冻结预训练模型权重 参数: - model：需要冻结权重的预训练模型 - pet_type： 微调算法类型 ``` pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/Pipeline_Design.md000066400000000000000000000007451471400134000303760ustar00rootroot00000000000000# Pipeline MindFormers大模型套件面向任务设计pipeline推理接口，旨在让用户可以便捷的体验不同AI领域的大模型在线推理服务，当前已集成10+任务的推理流程； ![输入图片说明](https://foruda.gitee.com/images/1673432339378334189/fb24c2fe_9324149.png "image-20230104093648200.png") MindFormers大模型套件为用户提供了pipeline高阶API，支持用户便捷的使用套件中已经集成的任务和模型完成推理流程。 pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/Trainer_Design.md000066400000000000000000000005561471400134000302350ustar00rootroot00000000000000# Trainer - Task Trainer 结构 Task Trainer开发依赖于MindFormers套件中的注册机制，方便开发者使用MindFormers套件提供的各个模块快速完成整网的搭建，各个模块之间可以做到有效的解耦。 ![输入图片说明](https://foruda.gitee.com/images/1673431864815390341/da621a72_9324149.png "image-20230103154930330.png") pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/assets/000077500000000000000000000000001471400134000263125ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/design/assets/MindPet.png000066400000000000000000000332031471400134000303610ustar00rootroot00000000000000PNG  IHDRA:kesRGBgAMA a pHYsod6IDATx^ |T/UQ>j RIX6fl %j^Wn\Zƒm%-vcmMDbh c/6oμ<3g>:ϙ3g&w<9FGG5![orN gΜ?{o:"'dO?RSOaĉ'E%_>:?:dރkJBXqqq@1Fn;}7ݑCqG6{{w#BHMMM#Ȱ:q WO_0KN8`=?8Frjx饗טhٖz}WoZv?͋]C|ǿOV?PrR{y$1ʩG}Ԗaѭ[a}Wb[~}w2ssӝߒXa׭w%9}lY>5S3gQ,KNa6>Y$-zn,@!v̙v\ 1^ #D\ٳjyYhPZZGFFd7<3D >`=c#dGz5-;#~3b}||4cV,@ _x{{e/v߃8]vH&{o۾MYrv$+-̗ ̘N,̀fKW#G^M}CV/O!nL\d昷zkOH^!n}:6?TqƱH_jۼgdKsqYOI];wҫxUIVݢ`['_ ?4782!Oƍ?~\!aq\7a,@vUz Bz(3z'9W|TDmhz?0yR2e豾!ħiwN~' y0ǜ|qĻҶuse*RIѐhŢwg{2iLb󹖲XxzUW=:3l޴~tocmw6M}6Yxӕ>3 h$O{y383W_w,@RkO5>TǜhHh{Sb߁~<5|ribkiNo϶\{'q=vx+],GƇ3e!"g@mNJcݰ F1[(wȟvD~x@w{P\&BSfy6 aZN.Յvл{/nb^SYΛWJX҈U=m]@O]ccb*Z+׋_LDN///}nǾ^;0쾝uLzgPlCqG];Esl;[}0m}s}:FYC933/H>'G|J]@.c'<l=͊rcpe%2L\ Q(,]}TL5z&Oԯv0Wh ZW KQqdɻL"ʷxWtx6>u:Li.mZ|>9rkK> ZM^27,iMM+X˾sOfg϶O}rY-S-1|Aɂ, Kі(vxa&JӳDC4i għȋ0ujzn4zhT.r5@X/~h2Z[Rqy{m}Bs\`߇V}w"Uc[L+>g--Il ߆_&[ӓK+l<~Nߖڴ㪫KDq1ǏuzλoirR`M=^'XC>?mY|M+ yipL>ݔIPQ=,a#ZmOd ӑ`2sSwaeok|ۙ0fFe F鑹~ƑujG۽ !3DA즚66NܱEUMߏQӼEu_V̧ye0rf3Ī):~2RŏWD휯 fLehߺx5/Fs猼mMWha»=sG3mN?o7e!>L+[l9SxשBś8-ߕ_4'Čs&W9,͛U  l'ZEñ*7Ϭh̬h>^?f˕x̖b>b9k]|KSi~(@ҳ}lEl 4jKz)}&(;et&eX迍6=UfI࿢#ع@u;͡7g.#'댏>%L-.Q뵡ޅe%ۏA6O-^ݝ:S[?C[j͓mg(?Ѝ}7v|bQ9'FX!mG oqrr*ػ'GWف?zS4nlRw*oM˃L*1w=hq2L>>);c=YP剢՟.h/wvfLH=o2g1]V;cXlֈ~J({x(qXOiW<8Gs?D *aNJKK˝{(w ￬ U+-7/Pt͞O;ܡm{/n~7$}f3oFy\ck^)Xʯ45}f%2ˆvbǚ5 l>٣}k9F50~ Q\Ww=lj^SF:VKYFPkGR`[C̜fͧrk§-fw2F=؝!>TdW<53=144 sXW_^Yq̶@WՔ<^ڶ{0֢].K4'k^{/)tɢe1>%}_ɶw}:2>\\[b{ml0 To1wN^0Ș^d\gOt{c-:g-Bt\/k9V:`-\1B| 9B,y"Fی]ܷb0?^W]_^" -0˹Wqܰώ? #.=(.q+ϿeKRќH|#%}g#=ljPB?MN7wz}+@Ϟ ޳`vHOHo!![|-uZ}WM!RoW--Vy ;y>4[2<%OEY yH%qfnQ5ȩ#o_ߗџ,C Üo?o#t}x~^VWemqر$[nH7ClJZ?TһPNQDxdM^]/0?K4ز08qH4~?~yH 0>,"nKV0]c Qn֬۴~ k}9g28;8|4NIzj| 1>lq۽)%y㤸pU _=4^˖~"e/ip{*lӓ_*ӺObY8}|*͟O'Nrdܨvg-)(`0dX9vzwO >}ndȹ7Musz 7ܐ6iҍie~vڴirE ÐaJ CP2 T U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P{+uDNWG?d_>~x9)h 999 JH '$I~gdV}蚌!ƷosǾ꒯ʩ@p=>87O<)F6N8B!Üt!ieY5~—oTâF8}dgSYv6N8aH 0'!F],N@ߜ?,; 'l21\;m}}}i-e!8`@xaihq,(ʩͲ{ɂCqƁ v8܌& H@kA"  ƒ s̕\$-'c4FlZ ZYp6N8dcXH[D 6N8d Ba*d Ba*d Ba*d Ba*d Ba*d Ba*d Ba*d r&ZX,O.eV}fx3t:kBJńrź1s~~cg 03.׼ʮqտuScynKa J]M-f4i5Je3(}4;D X*fh4dUV[kG,[)K뢊c-¦-qG悲|ٞW{U<@qO`a'x{bK^q\"z*O:ƕjk5_m%u,nbaX~cccTZ-s59'v0ǩ0JMڔ9 ™ + pE{JfBY^2Hlb\2 T U!@U0P2 T UE&Zm7L( {?{` ɰrYh{" rΚ|ϐW|AZH>Q,='_d cVZǯ+P$q!I =ƺvn6 D8ZjE+0a%.eY ͉.d .oq+ 90'Ioz:ZHR{k:{GK cO{EіZ^BaqcwW3>ַA49xuJж|ٖ(g^v(ZEƿG;;1~k9|~cf}>}֮=|ub/gX֕M{|qD4^d44 ɧ1kٚե}n7V:[ 9ٹiQ,El'ݖ7y{ >9aիseAӛ+g.$Əj!vK ЊX k󴮽ziJbT ꘵V<mk Ii6@u첻%r&UksdamֿrJ6>mPnAqZS]q +3n|wgC%&lwzN.޶l'wSAz^EqjEH9-ۜ-߉89-\șO'2}*ObӬ|!HbԒIΞă*1**棠"Ч߲!+ٖHf ,7 Kӱ񃅜>sec֫X ~⁣iUuN7)ǿ~J=.Z6P9`>NR~~~cg pi1nc٠U;1{ݩI92|֤mGz4ěxP.?H9WS_F=NNl.yRMjlWa; cΊF~LN|Yk, $N {c^G#X%׳S#Y)~c藨;Yx{|!dAižǭYU*K,D eMrѾ(nnZb6Iʉ˱7:::sC >__ &M}5:s6gYZW\mٽea{rxOwd!4bKU .a#ĝ@18=ͬv B cF;M9|g^-wrZV=տh[Tƞg>|Lȝ# rm)h_x.ok>ц=: gSħN14}gV瑯E@2Dގu>"fxdUMZO'sj[7;0q|}6׬3^ㅣ}eSڤr mKKgئhiѥ]-2I6,Jx4إ^w4Q^3gDbz-t*jAŭ1mxϸ]c}edςl~,i{a MO!`b|Ob!ʡ_RZTaD,^3]av;gz ȰxİMsVM2L6d:M۳,4J[X`ccc~(4k#쀱_ߘx;V{ SA ۴%bף0cѫMg>rg|Azc*ʃ +2~j*0J:k4y1.;QRw\o4c@ũІmvܵG^]+*|^l] 5/{ƭ'&|̣6ˀ@1']sK0̜Zvc=t l|پp{ѫ\u J}l(6އ_]xu3t<\!ACŭm,4L*ZNȘG݊CZ<N6 k9$؉wȘU [Ds.[ZceYA^~q]c.o}H2 M!&DT(È]eH%}kQKlyn&Yk־~~vIM˞"  ؖL(È#Ǚ$mh5 |>L GBJ*b倧1tj\`F5VԶs $o/9j(gsbHjh.ee^ŷb'QG[lTV0;a^8Wb.4V(Q‘P0bƒ_جo f/=20UwHPFBx qUriL~AdE`.؋~d85օ#B@ZdrW,,a%9K7'K4FGG=nex^7S^!hed0E*ܾm o]ûzCDv&GO]A81ڿQl1 4nݚH3Vhl:ET559/{P0Ta:҆& !7/./''sJD[g;Yqsap!MK/,d}!2ppiLARbB=TT-OEIEW)%84p )!q0^ӷXxm Z&wg5O85? !go46ui0r1wǚv1\0<58+= m͍Z/}_RgR Mh\WXn"[y-׽&l.hfbmX}Ȭo';P,ˁ@"alq Z%L",67$Ɛ;4.0,ˍD0`/(@x`V-R ֖G]L^ľ-" #HУo7 Bxˁelȅf妱U~>؊JQksuFwڽ;ET3[Qc5J_O}Fz%"E< VMZMM"jMXFsXI]g&mnn ca^10T-]]oTha ?$U!%uN ca\I34X\ԕ6BXu Ř6R0ƒ_O[bjd՟(BM3|fkcռW%DBxˁ ܵ[ ;ǨLVS^yDBvГeM<χYo+4 MBLJ%r _g][A\z1j v͌+mV/[E3/3,4YX do f3=Tu:#:{(QXto_=llc-$<㬚v(3/3b Yyc-D~2 `x!jI@0P2 T U!@UgXEazq)^۔je9l,63gl[$3,sjrW@L똼 3HV%u5^U8~1C:P[0T |Z)',7xV]M%z݉w4jh- aN?jh+}ё]%$WJ- ŤG n\&OLɒzPԕЛMrNӎn=9mļp}M[= cW ֖ц(;^sc5[6ͧT+[iѦ~a`+( E$# ӐU4pѲhn#YŲ\ZHݻWQHB^oB1$ror|E.dN)nr_+۲ܸ~Cѥgߑ5calDmq [to=~1 U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P21峄$Fp|]qƁ /=1$ vD>IlZ ZYp6N8dcrrr㏐ c.lq <0̟[YYPܟ_*P$F`܂Yp6N8dc͛wO>":ZʊKd9"Yv6N8dxr^?K K+:7}s(lq FGGgΜy!9!EyW}xRnNNιxK*^6C`Gz76N~;2,":::Z:Zn\}2_k_?~zŜs͛'F6N8!A!Ta*d Ba*d Ba*d Ba*d Ba*d Ba*dj̙3:$'$`{{n <(@b5kօ rrr뮢"95~;2y=Pjj*9 ###?EIahKtXUUUqq5kn@£?sc?yç?9$-Yd Iϟv Q sLGGGjj* iџ?e"昗^z Iv Bƈ~c7:::sC (:+]A2yhQU!@U0P2 T U!@U0P2 T U!@U,f͚% *:+]A2yhY]pvIVVP")7##j;CyĦfVVP")7##j;CyĦf6gΜjwu,DRboFFv8M2l޼y###{/$Z)Z"Yތhp ɹ165TUUdɒb|m&Qx!>olŗo4X_ᅟ[cGnhtW'NaͪU8v$N֭mOnOOoUlߌCqRpflAGWpǦ#=hF o^Ov|?"g_D;G~4lW?~M}@ ɫpUyšyo:?3 ZDa-b܍<19._k3Gг?d IԌoYC^ Fg{8?d v|tӆFhE&“qKx{<;Ddݷa/FeKwnntQV`/q:!Z~:ugotQVW4#. ъwkpwFzSm .8Μ93nlXGɓ'~=X. xO^qb1@?w`l 8v|ljpiVcǧ\{/tقXm~}4HJ$[ɑ#Gw}ɋ~,C.wyc zG_*'}!J|(>nn9sF}׮]܃puشFi 2i|膵. p73 DW4L\.#H,z9 LF>/ Zq!vz?.?##iy8.;l6n܈~X>߿f2;vPa||)9o:=<OM'ӵZ_7q+ >-EjqIGs+b?((=P/ݻǎ'|5hooG<<~CСCrxnFyOy}Ѳ9?e~muhZ5asq3Wrs'UV:Q(fX,ֽ Hl6"̪P(ZadddA'xdrE[lfchhHa|)~]/wNѫx-ܹG"9_ @(C6T&^D"ᥗ^pMD*Uc||2>D `͘,Yu;vdW󴶶fR)9ƍ1>>.3rrZ*ZHWX,ZΛ4ML' dTj*0*qA slsT.5;嘩AY<է)s)u`0~o^>h&j;v4'bb_˕};QVrݻW.˖S]7~ob79R.?ϚX^8dht1,q42ܹJsNzĹF]u;re_K>/rijYMӜu}+я_W6Ϝ`.ÇN. 4MntPx<`&-针UeYy0 CVOaOUݍn-fp"{_v>qT"8==]rcZ<#%Mv!".gYV9RWhLFԻ{addd^V3Q(yCcM%#fUx\ks?^\0dY:%l:::W $/A4Viv+6,˱`0h4<Զ.bqNv!j188hi[SO_'00 9m```;NDsV_ną\T7/t}}}%*|.WDBri:<[W%@t>ƃî]: ðD>,sBʖr_ɓ'K/J[ơ촼>[O]۶m&{1 \L bvwSO=%rd26kfdU}<8qr͛8Yݻwվ}fO Vgj/j-hy}vcǎe]vUk6k`bb(Lbǎ?N_|}޽_^ `)T +FQ.#75=H&x<jPzS-J)pMgX$ƪ[w)| j%NG?8o>:urv^+q˿KSFEtǎ׾>1atuu`C)r"Q iڢ 1"^mےL&xub{ݒ 6D 1X>|>O{eGXM鑑s&5XB"ӧ/|D[[۬~OLLXV#%·C}׿zJN]k"lUEqPZ=P_T1z0^Bܠ>6^+AreDj[$ h|ffrD"9I;x~P̆aػwc144u B PPV&Ն.3nw^! 'HDVirygWԟO(mId{x^KwrLˈFFjwP߿(+-rR9'1dfs}i[G$ia}uPH~L -GޡU+^W6LR)nf=e@,Ps.D[pەSWp8دZ,cN1O4-u}ݤ@/B^~9D5J_a۷o/i`Q,[n`?' iM]-2L:V> uxrAUA 'Fr9(~vrTJ9rD)ZFtz8rw1eT}uriT> XPDK:l bzF6S !A{b۶m%)oTKٵkFFF~a|+_qںqFDQ|K_SO=o|޽{Vm""ZlA0FjFS b `bzѬq{[n:(@cΝ)SKeΝ/85j`)Ei~iHvV曙pZκ/zQ`fk"bWlIF!""7lhd2 w\d2d2\.D"sNHDDT+lԁip\y&Ȉ$ ڵ ^lq/4Miqh]a&ɤ2N4May9n?|7Qva07@bjnjnʍUD65bFjUZ##"HU6fϘh6B!3Fֿ?@D+ xd;@ ~ཽFĺFFF4-b*4͒٘m8w5a&yU}whB!O3r9B̃PUQu,%~#Kx'b<" ,` ^daq+aմBPg&/u zc#~MFM<G:88P[Rjx,V5 6L4X" -]בJUTf6^iZv"K.DTod^D]r谬KWeԖ#JL&L3J-P(X,Vpz>#:e.F/V"zd2F4EX,B&&#!\L,'FdM!20(p1Q#!Kf4M$ Z'*EKRfh>sQ.ԄB!Km$fT8ɤlTR,-m@D7:b8O4O&̆ZcfjM=M6d?P=Cf60,YglPc0ب1ǃx<^2]]].C&|چ{9e6N8FhW'N6˂@6>vV.sNK7نD qQDi9`U( ,r Ԕ冋CzW}Vb*Ewê-˾~1h@Ξ7N袬Xwlڀ/W'ᶍ768^P烮Cܳgu1 z, W3"Uĭa kJ㿚'O_7"jZ}ѿ~6nXH/]//@Du+.^'-m(NM.Q?6>zZf6݄ntQV+WMǧeOGqkz96)s|6(+~o+x . ъ+UxpL5YukWч^np3ogz8{} v܁'.ъ/"DM7O܎h|L߿Ђ5?ğ3G`&wn[66DKﾇP$!нX=ԁ#xGزfl܊ Q-S„ǧqkC-` 6Ի(NNsZplU~3l;?]$Gx/M;>pҕFhٹe:ن-oƃ;5Yu* 6β .-gDWyq`h!XLDDDu` """+DDDTW]NA~Rh)oc={nV~و1XB{=>}zmܸ--#w """+DDDTW 6lPqDH$f]6P(8H$ZY;ax<fOG$q> ,nUZI$A&[a!,$IbD~Ŧܣ6%r-b[OdE"LLL :.iFFF H& a%EQR)D>T T2=8~^2 RB5lP* |it:`0(,ʼn>yF.tra4 |^PT*xebJpX^].ׂ`ttn[>D"3^E˼"N166VR}RvؘM6!ZN٠rСCwpX.788(~y|w .Ibh9ud)DzxfT@AuA ˅B}v.LӄF$Aoo^ (8\c%+Dve5E>UBbѲiT:Q[uYrd2F4dG  5+׾#^4N3h8.ؘwpp$@$Z)fU6E @&A0\-KѮ`>w.vrrRN4M) QEH$F+GFF,UDCCCBe-˨ksQFnQ g#N$b85~Ty:tXLflBL&IKF$`llzDmwboS(ٷC0Hmת@6j 4ez5ɓZ5`l2dz{{{nyb1*UXDWW|Q ju*Fz1<<\Ҩt @"Ç-U=1tvvZڮ)g-jl%g5x#rի|2NK=>CCC*dll pu`0xhbGߏl6;kzŢrj 6@n af{M ,jdMd1]̧ަDG D$bBCCCe$/_իWk1hZQujZz5VZE#fR*> V_4"jF8xk 6Q3[H_`D'<5K\~шh9K_aTpFDQ5͉T)J͋pFD+S_s^KٳhkkktqͥK`&&pe^#ҥKuZ< 6f """+DDDTW 6lQ] ;gql]=~ g]lDܸn < ߺu9ΆiH=s^<tnZ%""s诿 xpGź3\ioMǟ%0}ŸۆqU*35Kпԁ#X҂?Eצ >6S?[gH6n`6{] Z;d0:"@dxso-vMtl# .-wua}w`iܹ"""dKg+o 6vp8{""""bADDDu`$ \D"b ð+4H5x*BP* r^e1 2IDAT a.T( u]>:|>% `Ahi4M˅Nu9=6E@<_R媕C0R W߂@ 6LM3 j0䲢Z y<e*{P,+N2V˫VId2m+e6+UILF__~?x\/_}2hat]7L]t1M}ni4Mj83W܆Zr˦R+i4ﯸ|ث_=N5We>է3.l-/Sn###|>4̤Ų`tt,1߿ik(n%eB%e%jUخ:Muppв|\$i:ڒ*`0rU=#ٳq~" \D`.dYyx<|e)~鱼.U,bݕe!d2)/}}}v崡iSSS|>yD ATFsJA3D\m`uvvL˸nit{''N/" 255e6nNR{B!T _;@Ԟ!aAa V!Q# x 'j`o{TkN&sQ)"cAݻVTKdHD`0(J-3jOrm$B]]x8ԇh266V|0 ji\.t̕'qhPhJ I6(z~ Em*і\U -C^m뺼?|֥6ᰥVAu٣hI0fQicn?RW6K;VߺY"""Z9^~MEg?Fajm6lQ]1 ""bo"""+f6lQ]W_Z[޻r2rby3NN7HDDD'qmphٲ 6v^8Λ-(^<"""Ιlqoga也\<"""jf2WG{q-K??fO~lvl [6múwϼcǧ1^|c]6 6;Pƻ.6DDD|>n-ln m%PqΞgADDDp~ 6hyZWrը$DRnthq˭YWADs3 seLMMy7%"?FDDDu` """+DDDTW 6lQ]1 P(Jg2D".Ӳ8Dr0 ax/ 40L\Љ;LZ.h~_>_ ¾}>yE|d2~R)c)3W2`Νy.D,FDV3͖|>Հ(t? "jj2ˍajt1622~U*1fjE\&Y떬xit]ٜr d2\.b1Pccc5;K "jj2ȋ21PrY ð,'.@\XL^v Ned2iL"o/}kNkgr 122Rzو irr000`ɊطNeܞ={\& ʌSfa)`AT:.ssQmfNV?Dz5 Ðm= behhq~q,^yr?,~?ᰬ4MǡiZ2e|w,i>m~! "Yw"Sw9A,YzppJN+{5QP>Ƿ1 ZFD*4M\pqy\|y뷯O<\bb6k :z]v^%2 b2_4LNNP :G%Q ʪVAe]]]4\QCB4߿U0x <\r!b;UƠNB]N4Mhfir9y캺iXu !̚fѲQ-C3ө?1Ö 5k055 .d'Oti&ZjL&c\8Q)zPF6=mIJN Q'''K2 5P(@uƞjPnߪ1ת! СC( -t:-15X"(HDn# 9C`&$:::*͈ er궝M6auD,ݭrw\ DrW;[NmgSJ/|3=z?GGG^"zF144BBP(4'c!>Wn[vlYTvgVNU͎2UpB<G,+Ɠ-U=(VtvvgaVYƢ TZJKCQ'b >{ϳ,j٫m|>ڦDd/G͎2VQ`lTI#c13h80C]-z{{꼡P ]e'VȈSVZw,S&" N_~yh9rjawС)F#N[=˩g\r"4q՚hd!D"E٩p8"O$0F$AOO}N$@__l?tAxݾ@ Ax^2 Sۥ S֫\ R4hXh  "/D+|D 6V 4hl@ 4lP 4Vl`DTKJDnmmm{" DڀAi2 \ODTkebK8s6?MMDr5D7\֗f 6S1֙l [h&l?q|Bwކ7kHy{?{/+_WE|~ʈm@/wU--~j+`o hт޵W~[6XrF?Z;h96y hph9h8:5 h9z]lhmph9ٲqwn^}DDDDM7`e!""e """+D"\.WG&it1BRD""Մ};HX^/ ަx@ˈreQ瓙 `n]\ À+YVd+׶A샦iؼy ,WH$,C;&twwWڵ p!߁y@ `Ɏr9)p  jbpXV0 4N85Mˊw%h&Uׂlb6a&Ţ㾉$]K  ľRXip0;wZ+w0! "SOhIfDԌJfZR}j{Gd~)g>118]ꃁQlՐzw/.cccZ}h4*J:Q a_Ӵd2D#O]eBTg85*vߝjFIQ=UG,N^Y D^E֩C5Z}" 8Pl@4U+j W p@RT;;թxDu]5"3D5vKM^$sRDF +LӴ\k-LZ{5Bpj!TdRO@{PM"W6;>ZDDDDڛ͋82ADDD` """+v}%""bfbꪥՙF7~ooX:ptTDDDDѩih[-7_~z?7N'q-l{nC; \<"""jfWMO>ss߂? m;_ßBQsz—S?]Nw~.!W0zx֮–mC7DDD~}}_?]>t/6nX'_plj&ql~}ҢˍׯƝmtbm%^<\8ip)IENDB`pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/practice/000077500000000000000000000000001471400134000253315ustar00rootroot00000000000000pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/practice/Bert_Example.md000066400000000000000000000424631471400134000302330ustar00rootroot00000000000000# 开发新模型--以Bert为例 本文将介绍如何在mindformers中构建一个自己的模型。这里将以Bert为例，构建其训练、推理的相关代码。 ## 模型注册 在mindformers中，我们主要使用注册机制来方便用户能使用AutoConfig、AutoModel等通用接口进行模型的调用。因此，开发一个全新的模型，我们也希望用户能使用库上的相应接口，对模型进行注册。 以Bert模型为例。首先，我们通常将模型相关的文件放置在`mindformers/models`文件夹中。为了新建立一个模型，我们在`models`路径下，新建了一个`bert`文件夹。在该文件夹中，我们可以放置模型文件、配置文件、处理文件、分词文件等与模型相关的文件。bert模型注册的代码结构如下： ```python # mindformers/models/bert/bert.py from mindformers.tools.register import MindFormerRegister from mindformers.models.base_model import BaseModel @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertNetwork(BaseModel): _support_list = MindFormerBook.get_model_support_list()['bert'] def __init__(self, config): super(BertNetwork, self).__init__(config) def construct(self, **inputs): # bert model code pass ``` 在上述代码中，有三点需要详细介绍： * bert模型通过`MindFormerRegister`的`register`函数，将bert模型注册到`MindFormerRegister`的`registry`字典中，方便后续的调用。具体注册方式，可以参考[register.py](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/tools/register/register.py)。 * `BertNetwork`需要继承`BaseModel`，从而继承其自带的方法，包括`from_pretrained`、`save_pretrained`等，可以帮助用户快速从模型名称、路径等方式，生成模型。 * `_support_list`是模型自带的支持列表，其包含了模型所对应的支持列表。用户可以调用`BertNetwork.show_support_list()`方法，获取不同大小的bert模型的支持列表，从而选择自己所需要对应的模型结构。在support_list中，我们将当前的模型命名为"bert_base_uncased"。 具体代码，可以参见[bert.py](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/models/bert/bert.py)。 ### 配置文件注册 对于`BertNetwork`而言，需要传入一个`config`文件进行模型的配置，其中包含`num_layers`、`embedding_size`等配置信息，当然，这是由用户自己决定需要传给模型什么样的信息的。对于这样的配置信息，我们也提供了相应的注册机制，可以方便用户快速地获取到模型的配置。在`bert`文件夹中新建`bert_config.py`，其配置信息的注册代码如下： ```python # mindformers/models/bert/bert_config.py from mindformers.tools.register import MindFormerRegister from mindformers.models.base_config import BaseConfig @MindFormerRegister.register(MindFormerModuleType.CONFIG) class BertConfig(BaseConfig): _support_list = MindFormerBook.get_model_support_list()['bert'] def __init__(self, num_layers: int = 12, **kwargs): super(BertConfig, self).__init__(**kwargs) self.num_layers = num_layers ``` 同样地，在注册类的构造中，我们也需要完成`register`、`_support_list`等相关信息的注册，从而完成后期的调用。由于继承了`BaseConfig`基类，`BertConfig`也继承了其对应的方法，可以方便地进行配置类的生成和保存，详情可以参见[base_config.py](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/models/bert/bert_config.py)。 完成配置文件之后，还需要在[`mindformer_book.py`](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/mindformer_book.py)中进行注册。 首先，我们需要在`_MODEL_SUPPORT_LIST`列表中添加对应的模型名称： ```python _MODEL_SUPPORT_LIST = OrderedDict([ ('bert', [ 'bert_base_uncased', ]), ]) ``` 注册完成之后，模型即可通过`AutoConfig`即可通过模型名称`bert_base_uncased`来获取其yaml配置文件的路径地址。 ### yaml文件配置 为了方便用户管理和修改其配置文件信息，mindformers中提供了读取`yaml`文件进行配置的方法。yaml文件可以放置在： ```text /configs/bert/model_config ``` 其包含的yaml文件的格式如下： ```text # /configs/bert/model_config/bert_base_uncased.yaml model: model_config: type: BertConfig num_labels: 1 ``` 用户只需要将对应的属性填写到`model_config`中，即可完成yaml文件的修改。 具体代码，可以参见[bert_config.py](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/models/bert/bert.py)、[bert_base_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bert/model_config/bert_base_uncased.yaml) 在完成配置文件和模型文件的构造后，我们就可以直接使用如下代码进行`Bert`模型的构造： ```python import os from mindformers.mindformer_book import MindFormerBook from mindformers import BertNetwork, BertConfig # yaml_path yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", "bert", "model_config", "bert_base_uncased.yaml") # 先生成config，再生成模型 config = BertConfig.from_pretrained(yaml_path) model = BertNetwork(config) print(model) ``` ## AutoModel/AutoConfig使用 Mindformers提供了更加泛化的`AutoModel`和`AutoConfig`接口，进行模型及其配置文件的构造。用户在完成上述的模型的构造和配置文件构造后，得益于`MindFormerRegister`, 可以直接使用`AutoModel`和`AutoConfig`调用刚才注册的`BertNetwork`和`BertConfig`进行使用，代码如下： ```python import os from mindformers.mindformer_book import MindFormerBook from mindformers import AutoModel, AutoConfig # yaml_path yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", "bert", "model_config", "bert_base_uncased.yaml") # 方法一：先生成config，再生成模型 config = AutoConfig.from_pretrained(yaml_path) model = AutoModel.from_config(config) print(model) # 方法二：AutoModel 可以直接通过config的yaml文件进行模型的生成 model = AutoModel.from_config(yaml_path) print(model) ``` 在Mindformers中，我们提供了从云上下载配置的功能。代码如下： ```python import os from mindformers.mindformer_book import MindFormerBook from mindformers import AutoModel, AutoConfig model_name = "bert_base_uncased" config = AutoConfig.from_pretrained(model_name) model = AutoModel.from_config(config) ``` ## Trainer构建 构造完成模型和配置文件后，就可以构建代码进行模型的训练、评估、预测等操作。同样地，mindformers提供了`trainer`类，通过`trainer`，调用模型、数据集等。在mindformers中，需要在trainer中构建对应的类，一般我们使用任务名称来建立`trainer`类。 ### 模型tranier构建 根据需要构建的模型任务类型，我们可以在`trainer`文件夹下的新建一个`masked_language_modeling`文件夹，在该文件夹下面，我们可以构建对应的任务的文件，如：`masked_language_modeling_pretrain.py`。与上面的配置和模型相同，我们也提供了`BaseTrainer`作为基类，新构建的模型的类可以继承该基类。具体代码如下 ```python from mindspore.train.model import Model @MindFormerRegister.register(MindFormerModuleType.TRAINER, alias="mlm") class MaskedLanguageModelingTrainer(BaseTrainer): r"""MaskedLanguageModeling Task For Trainer. """ def __init__(self, model_name: str = None): super(MaskedLanguageModelingTrainer, self).__init__(model_name) def train(self, config=None, network=None, dataset=None, wrapper=None, optimizer=None, callbacks=None, **kwargs): # your code model = Model(wrapper) model.train(config.epochs, dataset=dataset, callbacks=callbacks, dataset_sink_mode=config.sink_mode, sink_size=config.sink_mode) def eval(self, **kwargs): pass def predict(self, **kwargs): pass ``` 在`MaskedLanguageModelingTrainer`中，我们注意到`train`的方法中，我们需要将模型训练所需要的参数都作为入参传入到该函数中，以供模型进行训练。在`train`的方法中，我们可以按MindSpore的训练方法进行训练, 用户可以参考MindSpore的[教程](https://www.mindspore.cn/tutorials/zh-CN/r1.9/advanced/model/model.html)。具体代码可以参考[masked_language_modeling_pretrain.py](https://gitee.com/jiahongqian/mindformers/blob/dev/mindformers/trainer/masked_language_modeling/masked_language_modeling_pretrain.py)。 ### 数据集构建 在mindformers中，用户可以在`mindformers/dataset`路径下构建自己的数据集文件，其代码结构如下： ```python from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class MaskLanguageModelDataset(BaseDataset): def __new__(cls, dataset_config = None): # fake dataset generate def generator(): """dataset generator""" data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32) input_mask = np.ones_like(data) token_type_id = np.zeros_like(data) next_sentence_lables = np.array([1]).astype(np.int32) masked_lm_positions = np.array([1, 2]).astype(np.int32) masked_lm_ids = np.array([1, 2]).astype(np.int32) masked_lm_weights = np.ones_like(masked_lm_ids) train_data = (data, input_mask, token_type_id, next_sentence_lables, masked_lm_positions, masked_lm_ids, masked_lm_weights) for _ in range(256): yield train_data # Dataset and operations dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids", "next_sentence_labels", "masked_lm_positions", "masked_lm_ids", "masked_lm_weights"]) dataset = dataset.batch(batch_size=dataset_config.batch_size) return dataset ``` 对于`Bert`，用户可以参考库上的[`mask_language_model_dataset.py`](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/dataset/mask_language_model_dataset.py)。 ### 使用模型trainer进行训练 在完成`trainer`的构建后，用户可以使用如下代码进行模型的训练： ```Python import os from mindformers.mindformer_book import MindFormerBook from mindformers import BertNetwork, BertConfig from mindformers.trainer import MaskedLanguageModelingTrainer from mindformers.trainer.config_args import ConfigArguments, RunnerConfig, WrapperConfig from mindformers.dataset import MaskLanguageModelDataset # yaml_path yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", "bert", "model_config", "bert_base_uncased.yaml") # 先生成config，再生成模型 config = BertConfig.from_pretrained(yaml_path) bert_model = BertNetwork(config) # train config batch_size = 16 train_config = RunnerConfig(epochs=1, batch_size=batch_size, sink_mode=True, sink_size=2) # train dataset dataset = MaskLanguageModelDataset(train_config) # optimizer lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, learning_rate=lr_schedule, params=bert_model.trainable_params()) # wrapper bert_wrapper = TrainOneStepCell(bert_model, optimizer, sens=1024) # callback loss_cb = LossMonitor(per_print_times=2) time_cb = TimeMonitor() callbacks = [loss_cb, time_cb] # trainer bert_trainer = MaskedLanguageModelingTrainer(model_name="bert_base_uncased") # train model bert_trainer.train( config=train_config, network=bert_model, dataset=dataset, wrapper=bert_wrapper, optimizer=optimizer, callbacks=callbacks) ``` 在这里，我们简单地使用MindSpore自带的`Model`进行模型的训练，在这里，我们需要传入数据集、优化器等组件，这些组件由用户手动定义。为了更好地方便用户管理训练所需要的各个组件，我们提供了一整套任务的配置文件读取，可以让用户快速地获取模型配置、模型训练、数据集配置等所需要的所有信息。 ### task config 配置 在建立`trainer`类之后，我们发现，使用模型的`trainer`训练模型，我们需要预先定义好network、dataset、oprimizer等 我们需要一个类来存储模型的训练各项超参数，包括训练的epoch、learning rate、batch size等重要参数。在mindformers中，我们通常使用`yaml`文件进行全局控制。与配置文件类型，一般我们会在`configs`下面的`bert`文件夹构建以下几个文件: ```text └─configs ├─bert ├─model_config ├─bert_base_uncased.yaml # 模型的配置文件 ├─task_config ├─context.yaml # 全局context配置文件 ├─runner.yaml # 运行超参配置文件 ├─bert_dataset.yaml # 数据集配置文件 run_bert_base_uncased.yaml # 总体配置文件 ``` * `bert_base_uncased.yaml`: 模型配置文件，同上述章节所介绍的，提供了模型的配置信息。 * `context.yaml`: 该文件主要提供了全局的参数配置信息，在配置文件中，可以直接使用对应的`context`、`parallel_context`方法中所需要的属性，用户需要了解以下两个MindSpore的函数的相关特性: ```python from mindspore import context context.set_context() context.set_auto_parallel_context() ``` * `runner.yaml`: 运行配置文件，该文件中存放了运行所需要的runner_config、runner_wrapper、optimizer等信息。其文件格式如下： ```text # config runner_config: epochs: 1 batch_size: 16 sink_mode: True per_epoch_size: -1 initial_epoch: 0 # wrapper runner_wrapper: type: TrainOneStepCell sens: 1024 # optimizer optimizer: type: AdamWeightDecay beta1: 0.9 beta2: 0.999 # lr sechdule lr_schedule: type: WarmUpLR learning_rate: 0.0001 end_learning_rate: 0.000001 warmup_steps: 10000 # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True ``` * `bert_dataset.yaml`: 在数据集的配置文件中, 我们也可以指定模型的数据集的相关信息。 ```text train_dataset: &train_dataset batch_size = 16 train_dataset_task: type: MaskLanguageModelDataset dataset_config: *train_dataset ``` * `run_bert_base_uncased.yaml`: 该文件会包含上述构建的yaml文件，其格式如下： ```text base_config: [ '../__base__.yaml', './task_config/context.yaml', './task_config/runner.yaml', './task_config/bert_dataset.yaml', './model_config/bert_tiny_uncased.yaml' ] profile: False use_parallel: False seed: 0 resume_or_finetune_checkpoint: '' run_status: 'train' trainer: type: mlm model_name: 'bert' ``` 这里，使用`base_config`将所有的关键配置全部包含，并且使用`run_status`确定模型的运行状态。在`trainer`中，使用`type`关键字，来获取上节构建的`trainer`，并且通过`model_name`获取模型的名称。 用户可以参考mindformers中已经完成的[Bert配置文件](/home/jenkins/qianjiahong/mindformers/20230113/transformer/configs/bert)进行配置。 ### trainer 配置文件注册 同模型的配置文件注册相同, 这里也需要将`trainer`的yaml文件配置信息注册在`mindformer_book.py`中，其注册的位置在： ```python import os _TRAINER_SUPPORT_TASKS_LIST = OrderedDict([ ("fill_mask", OrderedDict([ ("bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_base_uncased.yaml")),]) ), ]) ``` 这里值得注意的是，`fill_mask`是任务的名称，用户也可以根据任务的特殊性，来命名合适的任务。 ## Trainer接口使用 在完成训练任务的配置文件和注册后，用户就可以使用mindformer提供的`Trainer`接口快速调用其网络进行训练，代码如下： ```python from mindformers.trainer import Trainer bert_trainer = Trainer(task='fill_mask') bert_trainer.train() ``` 可以看到，相比于`使用模型trainer进行训练`中使用的`MaskedLanguageModelingTrainer`而言，使用`Trainer`及yaml文件进行训练，会更加方便和简洁。 在完成上述训练类的定义后，`MindFormers`也提供了一个统一的运行文件[run_mindformer.py](https://gitee.com/mindspore/mindformers/blob/dev/run_mindformer.py)，方便读者训练自己的网络，运行指令如下： ```shell python run_mindformer.py --config /path/run_bert_base_uncased.yaml --run_mode train --dataset_dir /datasetpath/ ``` pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/practice/Develop_With_Api.md000066400000000000000000000272671471400134000310530ustar00rootroot00000000000000# 高阶接口开发样例 MindFormers套件的Trainer高阶接口提供了`train`、`finetune`、`evaluate`、`predict` 4个关键属性函数，帮助用户快速拉起任务的训练、微调、评估、推理流程：[Trainer.train](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L334) [Trainer.finetune](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L419) [Trainer.evaluate](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L516) [Trainer.predict](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L583) 使用`Trainer.train` `Trainer.finetune` `Trainer.evaluate` `Trainer.predict` 拉起任务的训练、微调、评估、推理流程，以下为使用`Trainer` 高阶接口进行全流程开发的使用样例（多卡分布式并行），命名为`task.py`： ```python import argparse from mindformers import Trainer, TrainingArguments from mindformers import init_context, ContextConfig, ParallelContextConfig def context_init(use_parallel=False, optimizer_parallel=False): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, enable_parallel_optimizer=optimizer_parallel, full_batch=True) rank_id, device_num = init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) def main(use_parallel=False, run_mode='train', task='text_generation', model_type='gpt2', pet_method='', train_dataset='./train', eval_dataset='./eval', predict_data='hello!', dp=1, mp=1, pp=1, micro_size=1, op=False): # 环境初始化 context_init(use_parallel, op) # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=1, batch_size=2, learning_rate=0.001, warmup_steps=100, sink_mode=True, sink_size=2) # 定义任务，预先准备好相应数据集 task = Trainer(task=task, model=model_type, pet_method=pet_method, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) task.set_parallel_config(data_parallel=dp, model_parallel=mp, pipeline_stage=pp, optimizer_shard=op, micro_batch_num=micro_size) if run_mode == 'train': task.train() elif run_mode == 'finetune': task.finetune() elif run_mode == 'eval': task.evaluate() elif run_mode == 'predict': result = task.predict(input_data=predict_data) print(result) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--run_mode', default='train', required=True, help='set run mode for model.') parser.add_argument('--use_parallel', default=False, help='open parallel for model.') parser.add_argument('--task', default='text_generation', required=True, help='set task type.') parser.add_argument('--model_type', default='gpt2', required=True, help='set model type.') parser.add_argument('--train_dataset', default=None, help='set train dataset.') parser.add_argument('--eval_dataset', default=None, help='set eval dataset.') parser.add_argument('--pet_method', default='', help="set finetune method, now support type: ['', 'lora']") parser.add_argument('--data_parallel', default=1, type=int, help='set data parallel number. Default: None') parser.add_argument('--model_parallel', default=1, type=int, help='set model parallel number. Default: None') parser.add_argument('--pipeline_parallel', default=1, type=int, help='set pipeline parallel number. Default: None') parser.add_argument('--micro_size', default=1, type=int, help='set micro batch number. Default: None') parser.add_argument('--optimizer_parallel', default=False, type=bool, help='whether use optimizer parallel. Default: None') args = parser.parse_args() main(run_mode=args.run_mode, task=args.task, use_parallel=args.use_parallel, model_type=args.model_type, pet_method=args.pet_method, train_dataset=args.train_dataset, eval_dataset=args.eval_dataset, dp=args.data_parallel, mp=args.model_parallel, pp=args.pipeline_parallel, micro_size=args.micro_size, op=args.optimizer_parallel) ``` - 单卡使用样例： ```shell # 训练 python task.py --task text_generation --model_type gpt2 --train_dataset ./train --run_mode train # 评估 python task.py --task text_generation --model_type gpt2 --eval_dataset ./eval --run_mode eval # 微调，支持 python task.py --task text_generation --model_type gpt2 --train_dataset ./finetune --pet_method lora --run_mode finetune # 推理 python task.py --task text_generation --model_type gpt2 --predict_data 'hello!' --run_mode predict ``` - 多卡分布式使用样例： - 单机多卡标准启动脚本：`run_distribute_single_node.sh` ```bash #!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 4 ] then echo "Usage Help: bash run_distribute_single_node.sh [EXECUTE_ORDER] [RANK_TABLE_PATH] [DEVICE_RANGE] [RANK_SIZE] For Multiple Devices In Single Machine" exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } EXECUTE_ORDER=$1 RANK_TABLE_PATH=$(check_real_path $2) DEVICE_RANGE=$3 DEVICE_RANGE_LEN=${#DEVICE_RANGE} DEVICE_RANGE=${DEVICE_RANGE:1:DEVICE_RANGE_LEN-2} PREFIX=${DEVICE_RANGE%%","*} INDEX=${#PREFIX} START_DEVICE=${DEVICE_RANGE:0:INDEX} END_DEVICE=${DEVICE_RANGE:INDEX+1:DEVICE_RANGE_LEN-INDEX} if [ ! -f $RANK_TABLE_PATH ] then echo "error: RANK_TABLE_FILE=$RANK_TABLE_PATH is not a file" exit 1 fi if [[ ! $START_DEVICE =~ ^[0-9]+$ ]]; then echo "error: start_device=$START_DEVICE is not a number" exit 1 fi if [[ ! $END_DEVICE =~ ^[0-9]+$ ]]; then echo "error: end_device=$END_DEVICE is not a number" exit 1 fi ulimit -u unlimited export RANK_SIZE=$4 export RANK_TABLE_FILE=$RANK_TABLE_PATH shopt -s extglob for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export DEVICE_ID=${i} export RANK_ID=$((i-START_DEVICE)) mkdir -p ./output/log/rank_$RANK_ID echo "start training for rank $RANK_ID, device $DEVICE_ID" $EXECUTE_ORDER &> ./output/log/rank_$RANK_ID/mindformer.log & done shopt -u extglob ``` - 多机多卡标准启动脚本：`run_distribute_multi_node.sh` ```bash #!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 4 ] then echo "Usage Help: bash run_distribute_multi_node.sh [EXECUTE_ORDER] [RANK_TABLE_FILE] [DEVICE_RANGE] [RANK_SIZE]" exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } EXECUTE_ORDER=$1 RANK_TABLE_PATH=$(check_real_path $2) DEVICE_RANGE=$3 DEVICE_RANGE_LEN=${#DEVICE_RANGE} DEVICE_RANGE=${DEVICE_RANGE:1:DEVICE_RANGE_LEN-2} PREFIX=${DEVICE_RANGE%%","*} INDEX=${#PREFIX} START_DEVICE=${DEVICE_RANGE:0:INDEX} END_DEVICE=${DEVICE_RANGE:INDEX+1:DEVICE_RANGE_LEN-INDEX} if [ ! -f $RANK_TABLE_PATH ] then echo "error: RANK_TABLE_FILE=$RANK_TABLE_PATH is not a file" exit 1 fi if [[ ! $START_DEVICE =~ ^[0-9]+$ ]]; then echo "error: start_device=$START_DEVICE is not a number" exit 1 fi if [[ ! $END_DEVICE =~ ^[0-9]+$ ]]; then echo "error: end_device=$END_DEVICE is not a number" exit 1 fi ulimit -u unlimited export RANK_SIZE=$4 export RANK_TABLE_FILE=$RANK_TABLE_PATH shopt -s extglob for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export RANK_ID=${i} export DEVICE_ID=$((i-START_DEVICE)) echo "start training for rank $RANK_ID, device $DEVICE_ID" mkdir -p ./output/log/rank_$RANK_ID $EXECUTE_ORDER &> ./output/log/rank_$RANK_ID/mindformer.log & done shopt -u extglob ``` - 分布式并行执行`task.py`样例：需提前生成`RANK_TABLE_FILE`，同时`task.py`中默认使用**半自动并行模式**。 **注意单机时使用{single}，多机时使用{multi}** ```shell # 分布式训练 bash run_distribute_{single/multi}_node.sh "python task.py --task text_generation --model_type gpt2 --train_dataset ./train --run_mode train --use_parallel True --data_parallel 1 --model_parallel 2 --pipeline_parallel 2 --micro_size 2" hccl_4p_0123_192.168.89.35.json [0,4] 4 # 分布式评估 bash run_distribute_{single/multi}_node.sh "python task.py --task text_generation --model_type gpt2 --eval_dataset ./eval --run_mode eval --use_parallel True --data_parallel 1 --model_parallel 2 --pipeline_parallel 2 --micro_size 2" hccl_4p_0123_192.168.89.35.json [0,4] 4 # 分布式微调 bash run_distribute_{single/multi}_node.sh "python task.py --task text_generation --model_type gpt2 --train_dataset ./train --run_mode finetune --pet_method lora --use_parallel True --data_parallel 1 --model_parallel 2 --pipeline_parallel 2 --micro_size 2" hccl_4p_0123_192.168.89.35.json [0,4] 4 # 分布式推理（不支持流水并行,试用特性） bash run_distribute_{single/multi}_node.sh "python task.py --task text_generation --model_type gpt2 --predict_data 'hello!' ./train --run_mode predict --pet_method lora --use_parallel True --data_parallel 1 --model_parallel 2" hccl_4p_0123_192.168.89.35.json [0,4] 4 ```pengcheng.mind-test/docs/readthedocs/source_zh_cn/docs/practice/Using_Api.md000066400000000000000000000266111471400134000275370ustar00rootroot00000000000000# 高阶接口使用样例 ## AutoClass MindFormers大模型套件提供了AutoClass类，包含AutoConfig、AutoModel、AutoTokenizer、AutoProcessor4个便捷高阶接口，方便用户调用套件中已封装的API接口。上述4类提供了相应领域模型的ModelConfig、Model、Tokenzier、Processor的实例化功能。 ### AutoConfig&&AutoModel 使用AutoConfig和AutoModel自动实例化MindFormers中支持的模型配置或模型架构： ```python from mindformers import AutoConfig, AutoModel gpt_config = AutoConfig.from_pretrained("gpt2") gpt_model = AutoModel.from_pretrained("gpt2") # 自动加载预置权重到网络中 ``` 使用已有模型配置或网络架构实例化相应的模型配置或网络架构实例： ```python from mindformers import GPT2LMHeadModel, GPT2Config gpt_13b_config = GPT2Config.from_pretrained("gpt2_13b") gpt_13b_model = GPT2LMHeadModel.from_pretrained("gpt2_13b") # 自动加载预置权重到网络中 ``` 使用已有模型配置或模型架构进行二次开发： ```python from mindformers import GPT2LMHeadModel, GPT2Config gpt_config = GPT2Config(hidden_size=768, num_layers=12) gpt_model = GPT2LMHeadModel(gpt_config) ``` ### AutoTokenizer 使用AutoTokenizer自动获取文本分词器： ```python from mindformers import AutoTokenizer gpt_tokenizer = AutoTokenizer.from_pretrained("gpt2") gpt_tokenizer("hello!") ``` ```text {'input_ids': [50256, 31373, 0, 50256], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]} ``` 使用已有Tokenizer函数对文本进行分词： ```python from mindformers import GPT2Tokenizer gpt_tokenizer = GPT2Tokenizer.from_pretrained("gpt2") gpt_tokenizer("hello!") ``` ```text {'input_ids': [50256, 31373, 0, 50256], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]} ``` ### AutoProcessor 使用AutoProcessor自动获取模型数据预处理过程： ```python from mindformers import AutoProcessor gpt_processor = AutoProcessor.from_pretrained("gpt2") gpt_processor("hello!") ``` ```text {'text': Tensor(shape=[1, 128], dtype=Int32, value= [[50256, 31373, 0 ... 50256, 50256, 50256]])} ``` 使用已有模型数据预处理过程对相应数据做模型输入前的预处理： ```python from mindformers import GPT2Processor gpt_processor = GPT2Processor.from_pretrained('gpt2') gpt_processor("hello!") ``` ```text {'text': Tensor(shape=[1, 128], dtype=Int32, value= [[50256, 31373, 0 ... 50256, 50256, 50256]])} ``` 使用已有数据预处理过程进行二次开发： ```python from mindformers import GPT2Processor, GPT2Tokenizer # 自定义 tokenizer tok = GPT2Tokenizer.from_pretrained("gpt2") gpt_processor = GPT2Processor(tokenizer=tok, max_length=256, return_tensors='ms') gpt_processor("hello!") ``` ```text {'text': Tensor(shape=[1, 256], dtype=Int32, value= [[50256, 31373, 0 ... 50256, 50256, 50256]])} ``` ## pipeline MindFormers大模型套件面向任务设计pipeline推理接口，旨在让用户可以便捷的体验不同AI领域的大模型在线推理服务，当前已集成10+任务的推理流程； ### pipeline使用样例 使用MindFormers预置任务和模型开发一个推理流： ```python from mindformers import pipeline text_generation = pipeline(task='text_generation', model='gpt2', max_length=50) text_generation("I love Beijing, because", top_k=3) ``` ```text [{'text_generation_text': ['I love Beijing, because of all its beautiful scenery and beautiful people," said the mayor of Beijing, who was visiting the capital for the first time since he took office.\n\n"The Chinese government is not interested in any of this," he added']}] ``` 使用自定义的模型、tokenizer等进行任务推理： ```python from mindformers import pipeline from mindformers import GPT2LMHeadModel, GPT2Config, GPT2Tokenizer tok = GPT2Tokenizer.from_pretrained("gpt2") gpt_model = GPT2LMHeadModel.from_pretrained("gpt2") text_generation = pipeline(task='text_generation', model=gpt_model, tokenizer=tok, max_length=50) text_generation("I love Beijing, because", top_k=3) ``` ```text [{'text_generation_text': ['I love Beijing, because of all its beautiful scenery and beautiful people," said the mayor of Beijing, who was visiting the capital for the first time since he took office.\n\n"The Chinese government is not interested in any of this," he added']}] ``` ## Trainer MindFormers大模型套件面向任务设计Trainer接口，旨在让用户可以快速使用我们预置任务和模型的训练、微调、评估、推理能力，当前已集成10+任务和10+模型的全流程开发能力； ### init_context mindspore相关环境的初始化，MindFormers中提供了init_context标准接口帮助用户完成单卡或多卡并行环境的初始化： [init_context](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/core/context/build_context.py#L53) [ContextConfig](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/config_args.py#L26) [ParallelContextConfig](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/config_args.py#L26) 单卡初始化： ```python from mindformers import init_context, ContextConfig def context_init(): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) rank_id, device_num = init_context(use_parallel=False, context_config=context_config) ``` 多卡数据并行模式初始化： ```python from mindformers import init_context, ContextConfig, ParallelContextConfig def context_init(): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = ParallelContextConfig(parallel_mode='DATA_PARALLEL', gradients_mean=True, enable_parallel_optimizer=False) rank_id, device_num = init_context(use_parallel=True, context_config=context_config, parallel_config=parallel_config) ``` 多卡半自动并行模式初始化： ```python from mindformers import init_context, ContextConfig, ParallelContextConfig def context_init(): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, enable_parallel_optimizer=False, full_batch=True) rank_id, device_num = init_context(use_parallel=True, context_config=context_config, parallel_config=parallel_config) ``` ### TrainingArguments&&Trainer MindFormers套件对用户提供了`TrainingArguments`类，用于自定义大模型训练过程中的各类参数，支持参数详见：[TrainingArguments](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/training_args.py) 同时，MindFormers也提供了`Trainer`高阶接口，用于大模型任务的开发、训练、微调、评估、推理等流程； 使用TrainingArguments自定义大模型训练过程参数： ```python from mindformers import TrainingArguments # 环境初始化，参考上述`init_context`章节实现 context_init() # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=3, batch_size=2, learning_rate=0.001, warmup_steps=1000, sink_mode=True) ``` 使用Trainer接口创建内部预置任务：数据集按照官方教程准备[GPT预训练数据集准备](../model_cards/gpt2.md)，自定义训练参数 ```python from mindformers import Trainer, TrainingArguments # 环境初始化，参考上述`init_context`章节实现 context_init() # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=3, batch_size=2, learning_rate=0.001, warmup_steps=1000, sink_mode=True) text_generation = Trainer(task='text_generation', model='gpt2', args=training_args, train_dataset='./train', eval_dataset='./eval') ``` 使用Trainer接口创建内部预置任务：自定义数据集，模型，训练参数 ```python from mindspore.dataset import GeneratorDataset from mindformers import Trainer, TrainingArguments, AutoModel def generator(): """text dataset generator.""" seq_len = 1025 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(512): yield input_ids # 环境初始化，参考上述`init_context`章节实现 context_init() # 自定义训练超参数 training_args = TrainingArguments(num_train_epochs=3, batch_size=2, learning_rate=0.001, warmup_steps=1000, sink_mode=True) # 自定义模型 pangu_model = AutoModel.from_pretrained("pangualpha_2_6b") # 自定义数据集 dataset = GeneratorDataset(generator, column_names=["input_ids"]) train_dataset = dataset.batch(batch_size=4) eval_dataset = dataset.batch(batch_size=4) # 定义文本生成任务，传入自定义模型、数据集、超参数 text_generation = Trainer(task='text_generation', model=pangu_model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) ``` ### 并行&&重计算配置 MindFormers的Trainer接口提供了并行的配置接口`set_parallel_config`和重计算配置接口`set_recompute_config`，其中`set_parallel_config`接口仅在**半自动并行** 或**全自动并行模式**下生效，同时需要模型本身已支持或已配置[并行策略](https://www.mindspore.cn/tutorials/experts/zh-CN/r2.0/parallel/introduction.html); [set_parallel_config](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L690) [set_recompute_config](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L731) 使用Trainer高阶接口，自定义并行和重计算配置： ```python import numpy as np from mindspore.dataset import GeneratorDataset from mindformers import Trainer, TrainingArguments from mindformers import PanguAlphaHeadModel, PanguAlphaConfig def generator(): """text dataset generator.""" seq_len = 1025 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(512): yield input_ids # 环境初始化，参考上述`init_context`章节实现 context_init() # 自定义训练超参数 training_args = TrainingArguments(num_train_epochs=3, batch_size=2, learning_rate=0.001, warmup_steps=1000, sink_mode=True) # 自定义模型 pangu_config = PanguAlphaConfig(hidden_size=768, ffn_hidden_size=768 * 4, num_layers=12, num_heads=12, checkpoint_name_or_path='') pangu_model = PanguAlphaHeadModel(pangu_config) # 自定义数据集 dataset = GeneratorDataset(generator, column_names=["input_ids"]) train_dataset = dataset.batch(batch_size=4) eval_dataset = dataset.batch(batch_size=4) # 定义文本生成任务，传入自定义模型、数据集、超参数 text_generation = Trainer(task='text_generation', model=pangu_model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) # 设定并行策略，比如2机16卡,设定数据并行4 模型并行2 流水并行2 微批次大小为2 打开优化器并行 text_generation.set_parallel_config(data_parallel=4, model_parallel=2, pipeline_stage=2, micro_batch_num=2, optimizer_shard=True) # 设置重计算配置，打开重计算 text_generation.set_recompute_config(recompute=True) ``` pengcheng.mind-test/docs/readthedocs/source_zh_cn/index.rst000066400000000000000000000033601471400134000244520ustar00rootroot00000000000000目录 ========================== .. toctree:: :glob: :maxdepth: 1 :caption: 开始 Introduction.md Install.md Version_Match.md Quick_Tour.md docs/model_support_list.md .. toctree:: :glob: :maxdepth: 1 :caption: 设计 docs/design/AutoClass_Design.md docs/design/Trainer_Design.md docs/design/Pipeline_Design.md docs/design/Config_Design.md docs/design/Parallel_Design.md docs/design/Pet_Tuners_Design.md .. toctree:: :glob: :maxdepth: 1 :caption: 高级特性 docs/feature_cards/Resume_Training.md docs/feature_cards/Eval_While_Training.md docs/feature_cards/Transform_Ckpt.md docs/feature_cards/Text_Generator.md docs/feature_cards/Pet_Tuners.md docs/feature_cards/Precision_Optimization.md docs/feature_cards/Performance_Tuning.md docs/feature_cards/Chat_Web.md docs/feature_cards/Inference.md .. toctree:: :glob: :maxdepth: 1 :caption: AICC docs/aicc_cards/README.md .. toctree:: :glob: :maxdepth: 2 :caption: 模型 & 任务 docs/model_cards/index docs/task_cards/index .. toctree:: :glob: :maxdepth: 1 :caption: 开发实践 docs/practice/Using_Api.md docs/practice/Develop_With_Api.md docs/practice/Bert_Example.md .. toctree:: :glob: :maxdepth: 1 :caption: API docs/api_python/README.md docs/api_python/mindformers docs/api_python/mindformers.core docs/api_python/mindformers.dataset docs/api_python/mindformers.models docs/api_python/mindformers.modules docs/api_python/mindformers.pipeline docs/api_python/mindformers.trainer docs/api_python/mindformers.wrapper .. toctree:: :glob: :maxdepth: 1 :caption: FAQ docs/FAQ.md docs/transformer仓Python编程规范.md pengcheng.mind-test/docs/task_cards/000077500000000000000000000000001471400134000177575ustar00rootroot00000000000000pengcheng.mind-test/docs/task_cards/contrastive_language_image_pretrain.md000066400000000000000000000044261471400134000275610ustar00rootroot00000000000000# Contrastive Language Image Pretrain ## 任务描述 语言图像对比预训练：对模型进行图文对比学习，增强模型对文本图片的匹配度认识能力，预训练完的模型可用于零样本图像分类等下游任务 [相关论文](https://arxiv.org/abs/2103.00020) Alec Radford, Jong Wook Kim, et al., Learning Transferable Visual Models From Natural Language Supervision, 2021. ## 已支持数据集性能 | model | type | Datasets | Performance | stage | example | |:-----:|:----------------------------------------------------------------------------:|:--------:|:-----------:|:--------:|:-----------------------:| | clip | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | Flickr8k | -- | pretrain | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/contrastive_language_image_pretrain/clip_vit_b_32_pretrain_on_flickr8k.sh) | ### Flickr8k([链接](https://pan.baidu.com/s/1LRlQUL1MRipPL4MLOdExzg)，密码: s4be) - 数据集大小：2.2G，共8000张彩色图像，每张图像都与五个不同的标题配对，这些标题提供了对图片中物体和事件的内容描述 - 训练集：6000张图像 - 验证集：1000张图像 - 测试集：1000张图像 - 数据格式：RGB ```bash 数据集目录格式 └─Flickr8k ├─Flickr8k_Dataset | └─Flickr8k_Dataset └─Flickr8k_text ├─Flickr8k.devImages.txt ├─Flickr8k.testImages.txt ├─Flickr8k.trainImages.txt └─Flickr8k.token.txt ``` ## 快速任务接口 - Trainer接口开启训练： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("contrastive_language_image_pretrain") # INFO - Trainer support model list for contrastive_language_image_pretrain task is: # INFO - ['clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336'] # INFO - ------------------------------------- # 初始化trainer trainer = Trainer(task='contrastive_language_image_pretrain', model='clip_vit_b_32', train_dataset='./Flickr8k' ) trainer.train() ``` pengcheng.mind-test/docs/task_cards/image_classification.md000066400000000000000000000133731471400134000244450ustar00rootroot00000000000000# Image Classification ## 任务描述 图像分类：模型基于图像数据集进行训练后，可以在给定任意图片的情况下，完成对图像的分类，分类结果仅限于数据集中所包含的类别。 [相关论文-vit](https://gitee.com/link?target=https%3A%2F%2Farxiv.org%2Fabs%2F2010.11929): Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. 2021. [相关论文-swin](https://arxiv.org/abs/2103.14030) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo, 2021 ## 已支持数据集性能 | model | type | datasets | Top1-accuracy | stage | example | | :---: | :------------: | :---------: | :-----------: | :----------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | vit | vit_base_p16 | ImageNet-1K | 83.71% | train
finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/vit_base_p16_train_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/vit_base_p16_finetune_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/vit_base_p16_eval_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/vit_base_p16_predict_on_imagenet1k.sh) | | swin | swin_base_p4w7 | ImageNet-1K | 83.44% | train
finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/swin_base_p4w7_train_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/swin_base_p4w7_finetune_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/swin_base_p4w7_eval_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/swin_base_p4w7_predict_on_imagenet1k.sh) | ### [ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```text 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速任务接口 - Trainer接口开启训练/评估/推理： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("image_classification") # INFO - Trainer support model list for image_classification task is: # INFO - ['vit_base_p16', 'swin_base_p4w7'] # INFO - ------------------------------------- # 下面以ViT模型为例，Swin同理 # 初始化trainer vit_trainer = Trainer( task='image_classification', model='vit_base_p16', train_dataset="imageNet-1k/train", eval_dataset="imageNet-1k/val") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 vit_trainer.train(resume_or_finetune_from_checkpoint="mae_vit_base_p16", do_finetune=True) vit_trainer.evaluate(eval_checkpoint=True) predict_result = vit_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式2: 重头开始训练，并使用训练好的权重进行eval和推理 vit_trainer.train() vit_trainer.evaluate(eval_checkpoint=True) predict_result = vit_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式3： 从obs下载训练好的权重并进行eval和推理 vit_trainer.evaluate() predict_result = vit_trainer.predict(input_data=img, top_k=3) print(predict_result) ``` - pipeline接口开启快速推理 ```python from mindformers import pipeline, MindFormerBook from mindformers.tools.image_tools import load_image # 显示pipeline支持的模型列表 MindFormerBook.show_pipeline_support_model_list("image_classification") # INFO - Pipeline support model list for image_classification task is: # INFO - ['vit_base_p16', 'swin_base_p4w7'] # INFO - ------------------------------------- # 下面以ViT模型为例，Swin同理 # pipeline初始化 from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("image_classification", model='vit_base_p16') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img, top_k=3) # 输出 # [[{'score': 0.8846962, 'label': 'daisy'}, {'score': 0.005090589, 'label': 'bee'}, {'score': 0.0031510447, 'label': 'vase'}]] ``` pengcheng.mind-test/docs/task_cards/index.rst000066400000000000000000000053271471400134000216270ustar00rootroot00000000000000.. role:: raw-html-m2r(raw) :format: html 任务 ========================== .. toctree:: :glob: :maxdepth: 1 contrastive_language_image_pretrain image_classification question_answering text_classification text_generation token_classification zero_shot_image_classification 此处给出了MindFormers套件中支持的任务名称和模型名称，用于高阶开发时的索引名 .. list-table:: :header-rows: 1 * - 模型 - 任务（task name） - 模型（model name） * - `BERT <../model_cards/bert.html>`_ - masked_language_modeling\ :raw-html-m2r:`
`\ `text_classification <../task_cards/text_classification.html>`_\ :raw-html-m2r:`
`\ `token_classification <../task_cards/token_classification.html>`_\ :raw-html-m2r:`
`\ `question_answering <../task_cards/question_answering.html>`_ - bert_base_uncased :raw-html-m2r:`
`\ txtcls_bert_base_uncased\ :raw-html-m2r:`
`\ txtcls_bert_base_uncased_mnli :raw-html-m2r:`
`\ tokcls_bert_base_chinese\ :raw-html-m2r:`
`\ tokcls_bert_base_chinese_cluener :raw-html-m2r:`
`\ qa_bert_base_uncased\ :raw-html-m2r:`
`\ qa_bert_base_chinese_uncased * - `T5 <../model_cards/t5.html>`_ - translation - t5_small * - `GPT2 <../model_cards/gpt2.html>`_ - text_generation - gpt2_small :raw-html-m2r:`
`\ gpt2_13b :raw-html-m2r:`
`\ gpt2_52b * - `PanGuAlpha <../model_cards/pangualpha.html>`_ - text_generation - pangualpha_2_6_b\ :raw-html-m2r:`
`\ pangualpha_13b * - `GLM <../model_cards/glm.html>`_ - text_generation - glm_6b\ :raw-html-m2r:`
`\ glm_6b_lora * - `LLama <../model_cards/llama.html>`_ - text_generation - llama_7b :raw-html-m2r:`
`\ llama_13b :raw-html-m2r:`
`\ llama_65b :raw-html-m2r:`
`\ llama_7b_lora * - `Bloom <../model_cards/bloom.html>`_ - text_generation - bloom_560m\ :raw-html-m2r:`
`\ bloom_7.1b :raw-html-m2r:`
`\ bloom_65b\ :raw-html-m2r:`
`\ bloom_176b * - `MAE <../model_cards/mae.html>`_ - masked_image_modeling - mae_vit_base_p16 * - `VIT <../model_cards/vit.html>`_ - `image_classification <../task_cards/image_classification.html>`_ - vit_base_p16 * - `Swin <../model_cards/swin.html>`_ - `image_classification <../task_cards/image_classification.html>`_ - swin_base_p4w7 * - `CLIP <../model_cards/clip.html>`_ - `contrastive_language_image_pretrain <../task_cards/contrastive_language_image_pretrain.html>`_\ :raw-html-m2r:`
`\ `zero_shot_image_classification <../task_cards/zero_shot_image_classification.html>`_ - clip_vit_b_32\ :raw-html-m2r:`
`\ clip_vit_b_16 :raw-html-m2r:`
`\ clip_vit_l_14\ :raw-html-m2r:`
`\ clip_vit_l_14@336 pengcheng.mind-test/docs/task_cards/masked_image_modeling.md000066400000000000000000000054211471400134000245670ustar00rootroot00000000000000# Masked Image Modeling ## 任务描述 掩码图像建模：对图像中的部分图像块进行掩码，用剩下的图像块重建整张图像，从而对被掩码的图像块进行预测。 [相关论文-MAE](https://arxiv.org/abs/2111.06377): Kaiming He and Xinlei Chen and Saining Xie and Yanghao Li and Piotr Dollár and Ross Girshick. 2021. ### [ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```text 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速任务接口 - Trainer接口开启训练/推理： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("masked_image_modeling") # INFO - Trainer support model list for masked_image_modeling task is: # INFO - ['mae_vit_base_p16'] # INFO - ------------------------------------- # 初始化trainer mae_trainer = Trainer( task='masked_image_modeling', model='mae_vit_base_p16', train_dataset="imageNet-1k/train") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1: 重头开始训练，并使用训练好的权重推理 mae_trainer.train() predict_result = mae_trainer.predict(predict_checkpoint=True, input_data=img) print(predict_result) # 方式2： 从obs下载训练好的权重并进行eval和推理 predict_result = mae_trainer.predict(input_data=img) print(predict_result) # 输出 # [{'info': './output/output_image0.jpg', 'data': }] ``` - pipeline接口开启快速推理 ```python from mindformers import pipeline, MindFormerBook from mindformers.tools.image_tools import load_image # 显示pipeline支持的模型列表 MindFormerBook.show_pipeline_support_model_list("masked_image_modeling") # INFO - Pipeline support model list for masked_image_modeling task is: # INFO - ['mae_vit_base_p16'] # INFO - ------------------------------------- # pipeline初始化 from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("masked_image_modeling", model='mae_vit_base_p16') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img) print(pipeline_result) # 输出 # [{'info': './output/output_image0.jpg', 'data': }] ``` pengcheng.mind-test/docs/task_cards/question_answering.md000066400000000000000000000120321471400134000242230ustar00rootroot00000000000000# Question Answering ## 任务描述 **问答任务**：模型在基于问答数据集的微调后，输入为上下文（context）和问题（question），模型根据上下文（context）给出相应的回答。 **相关论文** - Jacob Devlin, Ming-Wei Chang, et al., [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/pdf/1810.04805.pdf), 2019. - Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang, [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://arxiv.org/pdf/1606.05250.pdf), 2016. ## 已支持数据集性能 | model | type | datasets | EM | F1 | stage | example | | :---: | :------------------------: | :--------: | :---: | ----- | :------------------------: | :----------------------------------------------------------: | | q'a | qa_bert_case_uncased_squad | SQuAD v1.1 | 80.74 | 88.33 | finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/question_answering/qa_bert_base_uncased_finetune_on_squad.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/question_answering/qa_bert_base_uncased_eval_on_squad.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/question_answering/qa_bert_base_uncased_predict_on_squad.sh) | ### [SQuAD v1.1](https://rajpurkar.github.io/SQuAD-explorer/) - 下载地址：[SQuAD v1.1训练集](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)，[SQuAD v1.1验证集](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json) - 数据集：该数据集包含 10 万个（问题，原文，答案）三元组，原文来自于 536 篇维基百科文章，而问题和答案的构建主要是通过众包的方式，让标注人员提出最多 5 个基于文章内容的问题并提供正确答案，且答案出现在原文中。 - 数据格式：json文件 ```bash 数据集目录格式 └─squad ├─train-v1.1.json └─dev-v1.1.json ``` ## 快速任务接口 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 在脚本执行目录创建 `squad` 文件夹，然后将数据集放入其中 - 脚本运行测试 ```shell # finetune python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode finetune --load_checkpoint qa_bert_base_uncased # evaluate python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode eval --load_checkpoint qa_bert_base_uncased_squad # predict python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode predict --load_checkpoint qa_bert_base_uncased_squad --predict_data [TEXT] ``` ### 调用API启动 - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer # 初始化trainer trainer = Trainer(task='question_answering', model='qa_bert_base_uncased', train_dataset='./squad/', eval_dataset='./squad/') #方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 trainer.train(resume_or_finetune_from_checkpoint="qa_bert_base_uncased", do_finetune=True) trainer.evaluate(eval_checkpoint=True) # 测试数据，测试数据分为context和question两部分，两者以 “-” 分隔 input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] trainer.predict(predict_checkpoint=True, input_data=input_data) # 方式2： 从obs下载训练好的权重并进行eval和推理 trainer.evaluate() # INFO - QA Metric = {'QA Metric': {'exact_match': 80.74739829706716, 'f1': 88.33552874684968}} # 测试数据，测试数据分为context和question两部分，两者以 “-” 分隔 input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] trainer.predict(input_data=input_data) # INFO - output result is [{'text': 'Berlin', 'score': 0.9941, 'start': 34, 'end': 40}] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import QuestionAnsweringPipeline from mindformers import AutoTokenizer, BertForQuestionAnswering, AutoConfig # 测试数据，测试数据分为context和question两部分，两者以 “-” 分隔 input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] tokenizer = AutoTokenizer.from_pretrained('qa_bert_base_uncased_squad') qa_squad_config = AutoConfig.from_pretrained('qa_bert_base_uncased_squad') # This is a known issue, you need to specify batch size equal to 1 when creating bert model. qa_squad_config.batch_size = 1 model = BertForQuestionAnswering(qa_squad_config) qa_pipeline = QuestionAnsweringPipeline(task='question_answering', model=model, tokenizer=tokenizer) results = qa_pipeline(input_data) print(results) # 输出 # [{'text': 'Berlin', 'score': 0.9941, 'start': 34, 'end': 40}] ``` pengcheng.mind-test/docs/task_cards/text_classification.md000066400000000000000000000141401471400134000243400ustar00rootroot00000000000000# Text Classification ## 任务描述 文本分类：模型在基于文本对的微调后，可以在给定任意文本对与候选标签列表的情况下，完成对文本对关系的分类，文本对的两个文本之间以-分割。 [相关论文](https://arxiv.org/pdf/1810.04805.pdf) Jacob Devlin, Ming-Wei Chang, et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019. ## 已支持数据集性能 | model | type | datasets | Top1-accuracy | stage | example | |:------:|:----------------------------------------------------------:|:--------:|:---------------:|:--------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | bert | txtcls_bert_base_uncased | Mnli | 30.9% | pretrain | -- | | txtcls | txtcls_bert_case_uncased_mnli | Mnli | 84.8% | finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/text_classification/txtcls_bert_base_uncased_finetune_on_mnli.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/text_classification/txtcls_bert_base_uncased_mnli_eval_on_mnli.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/text_classification/txtcls_bert_base_uncased_mnli_predict_on_mnli.sh) | ### [Mnli](https://dl.fbaipublicfiles.com/glue/data/MNLI.zip) - 数据集大小：298M，共431992个样本，3个类别 - 训练集：392702个样本 - 匹配测试集：9796个样本 - 非匹配测试集：9847个样本 - 匹配开发集：9815个样本 - 非匹配开发集：9832个样本 - 数据格式：tsv文件 ```bash 数据集目录格式 └─mnli ├─dev ├─test └─train ``` - 用户可以参考[BERT](https://github.com/google-research/bert)代码仓中的run_classifier.py文件，进行Mnli数据集`TFRecord`格式文件的生成。 ## 快速任务接口 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 ```shell # finetune python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode finetune --load_checkpoint txtcls_bert_base_uncased # evaluate python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode eval --load_checkpoint txtcls_bert_base_uncased_mnli # predict python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode predict --load_checkpoint txtcls_bert_base_uncased_mnli --predict_data [TEXT] ``` ### 调用API启动 - Trainer接口开启训练/评估/推理： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("text_classification") # INFO - Trainer support model list for txt_classification task is: # INFO - ['txtcls_bert_base_uncased'] # INFO - ------------------------------------- # 初始化trainer trainer = Trainer(task='text_classification', model='txtcls_bert_base_uncased', train_dataset='./mnli/train', eval_dataset='./mnli/eval') # 测试数据，该input_data有两个测试案例，即两个文本对，单个文本对的两个文本之间用-分割 input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", "i don't know um do you do a lot of camping-I know exactly."] #方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 trainer.train(resume_or_finetune_from_checkpoint="txtcls_bert_base_uncased", do_finetune=True) trainer.evaluate(eval_checkpoint=True) trainer.predict(predict_checkpoint=True, input_data=input_data, top_k=1) # 方式2： 从obs下载训练好的权重并进行eval和推理 trainer.evaluate() # INFO - Top1 Accuracy=84.8% trainer.predict(input_data=input_data, top_k=1) # INFO - output result is [[{'label': 'neutral', 'score': 0.9714198708534241}], # [{'label': 'contradiction', 'score': 0.9967639446258545}]] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import TextClassificationPipeline from mindformers import AutoTokenizer, BertForMultipleChoice, AutoConfig input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", "i don't know um do you do a lot of camping-I know exactly."] tokenizer = AutoTokenizer.from_pretrained('txtcls_bert_base_uncased_mnli') txtcls_mnli_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased_mnli') # Because batch_size parameter is required when bert model is created, and pipeline # function deals with samples one by one, the batch_size parameter is seted one. txtcls_mnli_config.batch_size = 1 model = BertForMultipleChoice(txtcls_mnli_config) txtcls_pipeline = TextClassificationPipeline(task='text_classification', model=model, tokenizer=tokenizer, max_length=model.config.seq_length, padding="max_length") results = txtcls_pipeline(input_data, top_k=1) print(results) # 输出 # [[{'label': 'neutral', 'score': 0.9714198708534241}], [{'label': 'contradiction', 'score': 0.9967639446258545}]] ``` pengcheng.mind-test/docs/task_cards/text_generation.md000066400000000000000000000004111471400134000234740ustar00rootroot00000000000000# Text Generation ## 任务描述 文本生成：生成新文本的任务，模型基于输入的不完整的文本或解释进行填充，或对输入的源语言翻译成目标语言。 ## [已支持模型](../model_support_list.md#text-generator支持度表) pengcheng.mind-test/docs/task_cards/token_classification.md000066400000000000000000000123561471400134000245030ustar00rootroot00000000000000# Token Classification ## 任务描述 命名实体识别：模型在基于命名实体识别数据集的微调后，可以在给定任意文本与候选标签列表的情况下，完成对文本中命名实体的识别。 [相关论文](https://arxiv.org/abs/2001.04351) Xu, Liang and Dong, Qianqian and Yu, Cong and Tian, Yin and Liu, Weitang and Li, Lu and Zhang, Xuanwei, CLUENER2020: Fine-grained Name Entity Recognition for Chinese, 2020. ## 已支持数据集性能 | model | type | datasets | Entity F1 | stage | example | | :----: | :------------------------------: | :------: | :-------: | :------------------------: | :----------------------------------------------------------: | | tokcls | tokcls_bert_case_chinese_cluener | CLUENER | 0.7905 | finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/token_classification/tokcls_bert_base_chinese_finetune_on_cluener.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/token_classification/tokcls_bert_base_chinese_eval_on_cluener.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/token_classification/tokcls_bert_base_chinese_predict_on_cluener.sh) | ### [CLUENER](https://github.com/CLUEbenchmark/CLUENER2020) - 数据集：训练集大小为10748个样本，验证集大小为1343个样本，10个类别，分别为: 地址（address），书名（book），公司（company），游戏（game），政府（government），电影（movie），姓名（name），组织机构（organization），职位（position），景点（scene）。 - 数据格式：json文件 ```bash 数据集目录格式 └─cluener ├─train.json ├─dev.json ├─test.json ├─cluener_predict.json └─README.md ``` ## 快速任务接口 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 在脚本执行目录创建 `cluener` 文件夹，然后将数据集放入其中 - 脚本运行测试 ```shell # finetune python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode finetune --load_checkpoint tokcls_bert_base_chinese # evaluate python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode eval --load_checkpoint tokcls_bert_base_chinese_cluener # predict python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode predict --load_checkpoint tokcls_bert_base_chinese_cluener --predict_data [TEXT] ``` ### 调用API启动 - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer # 初始化trainer trainer = Trainer(task='token_classification', model='tokcls_bert_base_chinese', train_dataset='./cluener/', eval_dataset='./cluener/') # 测试数据 input_data = ["结果上周六他们主场0：3惨败给了中游球队瓦拉多利德，近7个多月以来西甲首次输球。"] #方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 trainer.train(resume_or_finetune_from_checkpoint="tokcls_bert_base_chinese", do_finetune=True) trainer.evaluate(eval_checkpoint=True) trainer.predict(predict_checkpoint=True, input_data=input_data) # 方式2： 从obs下载训练好的权重并进行eval和推理 trainer.evaluate() # INFO - Entity F1=0.7853 trainer.predict(input_data=input_data) # INFO - output result is [[{'entity_group': 'organization', 'start': 20, 'end': 24, 'score': 0.94914, 'word': '瓦拉多利德'}, # {'entity_group': 'organization', 'start': 33, 'end': 34, 'score': 0.9496, 'word': '西甲'}]] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import TokenClassificationPipeline from mindformers import AutoTokenizer, BertForTokenClassification, AutoConfig from mindformers.dataset.labels import cluener_labels input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。"] id2label = {label_id: label for label_id, label in enumerate(cluener_labels)} tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') tokcls_cluener_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese_cluener') # This is a known issue, you need to specify batch size equal to 1 when creating bert model. tokcls_cluener_config.batch_size = 1 model = BertForTokenClassification(tokcls_cluener_config) tokcls_pipeline = TokenClassificationPipeline(task='token_classification', model=model, id2label=id2label, tokenizer=tokenizer, max_length=model.config.seq_length, padding="max_length") results = tokcls_pipeline(input_data) print(results) # 输出 # [[{'entity_group': 'address', 'start': 6, 'end': 8, 'score': 0.52329, 'word': '日内瓦'}, # {'entity_group': 'name', 'start': 12, 'end': 25, 'score': 0.83922, 'word': 'freresoltramar'}]] ``` pengcheng.mind-test/docs/task_cards/zero_shot_image_classification.md000066400000000000000000000076171471400134000265450ustar00rootroot00000000000000# Zero Shot Image Classification ## 任务描述 零样本图像分类：模型在基于图文对的预训练后，可以在给定任意图片与候选标签列表的情况下，完成对图像的分类，而无需任何微调。 [相关论文](https://arxiv.org/abs/2103.00020) Alec Radford, Jong Wook Kim, et al., Learning Transferable Visual Models From Natural Language Supervision, 2021. ## 已支持数据集性能 | model | type | datasets | Top1-accuracy | stage | example | |:-----:|:------------------------------------------------------------------------:|:--------:|:--------------------------------------------:|:----------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | clip | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | Cifar100 | 57.24%
61.41%
69.67%
68.19% | eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/zero_shot_image_classification/clip_vit_b_32_eval_on_cifar100.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/zero_shot_image_classification/clip_vit_b_32_predict_on_cifar100.sh) | ### [Cifar100](http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz) - 数据集大小：161M，共60000张图片，100个类别 - 训练集：50000张图片 - 测试集：10000张图片 - 数据格式：二进制文件 ```bash 数据集目录格式 └─cifar-100-python ├─meta ├─test └─train ``` ## 快速任务接口 - Trainer接口开启评估/推理： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("zero_shot_image_classification") # INFO - Trainer support model list for zero_shot_image_classification task is: # INFO - ['clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336'] # INFO - ------------------------------------- # 初始化trainer trainer = Trainer(task='zero_shot_image_classification', model='clip_vit_b_32', eval_dataset='cifar-100-python' ) img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") trainer.evaluate() #下载权重进行评估 # INFO - Top1 Accuracy=57.24% trainer.predict(input_data=img) #下载权重进行推理 # INFO - output result is saved at ./results.txt ``` - pipeline接口开启快速推理 ```python from mindformers import pipeline, MindFormerBook from mindformers.tools.image_tools import load_image # 显示pipeline支持的模型列表 MindFormerBook.show_pipeline_support_model_list("zero_shot_image_classification") # INFO - Pipeline support model list for zero_shot_image_classification task is: # INFO - ['clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336'] # INFO - ------------------------------------- # pipeline初始化 classifier = pipeline("zero_shot_image_classification", model="clip_vit_b_32" candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") classifier(img) # 输出 # [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, # {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, # {'score': 2.396818e-06, 'label': 'cat'}]] ``` pengcheng.mind-test/docs/transformer仓Python编程规范.md000066400000000000000000000324441471400134000271550ustar00rootroot00000000000000# 说明 本规范以[PEP8](https://www.python.org/dev/peps/pep-0008/)为基础，参考华为Python通用编码规范、安全编程规范，并结合业界共识整理而成，参与MindSpore社区开发需要首先遵循本规范内容（与PEP8冲突部分），其余遵循PEP8规范。 如果对规则异议，建议提交issue并说明理由，经MindSpore社区运营团队评审接纳后可修改生效。 a ## 适用范围 MindSpore开源社区 ------------------------ ## 1. 代码风格 ### 1.1 命名 **规则 1.1.1 包名，模块名：小写，不使用下划线。** **规则 1.1.2 类名：使用驼峰格式，首字母大写，私有类下划线前缀。** ```python class _Foo: _instance = None pass ``` **规则 1.1.3 函数名、变量名：小写，多个单词下划线分割。** ```python def _func_example(path): pass ``` **建议 1.1.4 除迭代器与计数器除外，禁止使用单字符命名。** ### 1.2 格式 **规则 1.2.1 每行字符数不要超过 120 个。** 如果超过120个字符，请选择合理的方式进行换行。 **规则 1.2.2 使用空格进行缩进，每次缩进4个空格，禁止tab缩进。** **规则 1.2.3 import顺序：标准库、第三方、自定义模块。** **规则 1.2.4 返回语句和条件语句中不使用括号。** **规则 1.2.5 模块级函数和类之间双空行，类成员函数之间一空行，注释与代码间按需添加空行，原则上不超过两空行。** **规则 1.2.6 无效或冗余代码直接删除，不要以注释、TODO等方式保留在代码中，建议提issue记录。** ### 1.3 注释 **规则 1.3.1 文件头注释必须包含版权声明。** 所有python文件，均需包含如下版权声明： ```python # Copyright 2019 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Add notes. """ import xxx ``` > 关于版权说明，应注意： > 2020年新建的文件，应该是`Copyright 2020 Huawei Technologies Co., Ltd` > 2019年创建年份，2020年修改年份，应该是`Copyright 2019-2020 Huawei Technologies Co., Ltd` **规则 1.3.2 对外的类、方法、算子、Cell注释格式。** - `class` 和 `def` 的注释格式相同，采用业界通用的python注释语法，写在声明下方并缩进，所有的 `class` 和 `def` 都需要写注释，模块内部的类和方法可以只写一条简介。 - 注释格式详见[MindSpore注释规范](https://gitee.com/mindspore/community/blob/master/security/comments_specification_zh_cn.md)。 **规则 1.3.3 不允许通过注释屏蔽pylint告警。** ### 1.4 日志 **规则 1.4.1 异常日志文本首字母大写。** **规则 1.4.2 日志文本中变量名必须使用单引号注明。** ## 2. 通用编码 ### 2.1 接口声明 **规则 2.1.1 用户接口在文件的__all__中说明，__all__摆放在import与代码之间。** **规则 2.1.2 当前文件使用的非对外方法命名采用下划线前缀，内部跨模块使用的方法无需下划线前缀，用户接口在__all__中声明。** ### 2.2 数据校验 **规则 2.2.1 对所有外部数据进行合法性检查，包括但不限于：函数入参、外部输入命名行、文件格式，文件大小、环境变量、用户数据等。** **建议 2.2.2 必须对文件路径进行规范化后再使用。** 当文件路径来自外部数据时，需要先将文件路径规范化，如果没有作规范化处理，攻击者就有机会通过恶意构造文件路径进行文件的越权访问： 例如，攻击者可以构造“../../../etc/passwd”的方式进行任意文件访问。 在linux下，使用realpath函数，在windows下，使用PathCanonicalize函数进行文件路径的规范化。 **规则 2.2.3 禁止调用OS命令解析器执行命令或运行程序。** 使用未经校验的不可信输入作为系统命令的参数或命令的一部分，可能导致命令注入漏洞。对于命令注入漏洞，命令将会以与Python应用程序相同的特权级别执行，它向攻击者提供了类似系统shell的功能。在Python中，os.system 或 os.popen 经常被用来调用一个新的进程，如果被执行的命令来自于外部输入，则可能会产生命令和参数注入。 执行命令的时候，请注意以下几点： 1. 命令执行的字符串不要去拼接输入的参数，如果必须拼接时，要对输入参数进行白名单过滤。 2. 对传入的参数要做类型校验，例如：整数数据，可以对数据进行整数强制转换。 3. 保证格式化字符串的正确性，例如：int类型参数的拼接，对于参数要用%d，不能用%s。 【错误代码示例1】 攻击者可以通过找到环境变量APPHOME对应的值，并且在相应目录下放置常量INITCMD对应的攻击程序，达到执行的效果： ```python home = os.getenv('APPHOME') cmd = os.path.join(home, INITCMD) os.system(cmd) ``` 【错误代码示例2】 没有校验属性 backuptype 的值，这个是用户输入的，攻击者可能进行攻击， 例如：用户输入的是：" && del c:\\dbms\\*.* "： ```python # 值来自用户配置 btype = req.field('backuptype') cmd = "cmd.exe /K \"c:\\util\\rmanDB.bat " + btype + "&&c:\\util\\cleanup.bat\"" os.system(cmd) ``` 【错误代码示例3】 没有校验属性 backuptype 的值，这个是用户输入的，攻击者可能进行攻击，例如：用户输入的是：" && del c:\\dbms\\*.* "： ```python import os import sys try: print(os.system("ls " + sys.argv[1])) except Exception as ex: print('exception:', ex) ``` 攻击者可以通过以下命令来利用这个漏洞程序： ```python python test.py ". && echo bad" ``` 实际将会执行两个命令： ```python ls . echo bad ``` 【正确代码示例】 避免使用 os.system，可以使用标准的 API 替代运行系统命令来完成任务： ```python import os import sys try: print(os.listdir(sys.argv[1])) except Exception as ex: print(ex) ``` ### 2.3 异常行为 **规则 2.3.1 异常必须被妥当处理，禁止抑制或者忽略已检查异常。** 每一个except 块都应该确保程序只会在继续有效的情况下才会继续运行下去。except 块必须要么从异常情况中恢复，要么重新抛出适合当前catch块上下文的另一个异常以允许最邻近的外层try-except 语句块来进行恢复工作。 【正确代码示例】 正确的做法是，避免使用 os.system，可以使用标准的 API 替代运行系统命令来完成任务： ```python validFlag = False while not validFlag: try: # If requested file does not exist, throws FileNotFoundError # If requested file exists, sets validFlag to true validFlag = True except FileNotFoundError: import traceback traceback.print_exc() ``` 【例外情况】： 1. 在资源释放失败不会影响程序后续行为的情况下，释放资源时发生的异常可以被抑制。释放资源的例子包括关闭文件、网络套接字、线程等等。这些资源通常是在except或者fianlly块中被释放，并且在后续的程序运行中都不会再被使用。因此，除非资源被耗尽，否则不会有其他途径使得这些异常会影响程序后续的行为。在充分处理了资源耗尽问题的情况下，只需对异常进行净化和记录日志（以备日后改进）就足够了；在这种情况下没必要做其他额外的错误处理。 2. 如果在特定的抽象层次上不可能从异常情况中恢复过来，则在那个层级的代码就不用处理这个异常，而是应该抛出一个合适的异常，让更高层次的代码去捕获处理，并尝试恢复。对于这种情况，最通常的实现方法是省略掉catch语句块，允许异常被广播出去。 **规则 2.3.2 使用try…except…结构对代码作保护时，需要在异常后使用finally…结构保证操作对象的释放。** 使用try…except…结构对代码作保护时，如果代码执行出现了异常，为了能够可靠地关闭操作对象，需要使用finally…结构确保释放操作对象。 【正确代码示例】 ```python handle = open(r"/tmp/sample_data.txt") # May raise IOError try: data = handle.read() # May raise UnicodeDecodeError except UnicodeDecodeError as decode_error: print(decode_error) finally: handle.close() # Always run after try: ``` **规则 2.3.3 不要使用“except:”语句来捕获所有异常。** 在异常这方面, Python非常宽容,“except:”语句真的会捕获包括Python语法错误在内的任何错误。使用“except:”很容易隐藏真正的bug，我们在使用try…except…结构对代码作保护时，应该明确期望处理的异常。Exception类是大多数运行时异常的基类，一般也应当避免在except语句中使用。通常，try只应当包含必须要在当前位置处理异常的语句，except只捕获必须处理的异常。比如对于打开文件的代码，try应当只包含open语句，except只捕获FileNotFoundError异常。对于其他预料外的异常，则让上层函数捕获，或者透传到程序外部来充分暴露问题。 【错误代码示例】 如下代码可能抛出两种异常，使用“except:”语句进行统一处理时，如果是open执行异常，将在“except:”语句之后handle无效的情况下调用close，报错handle未定义。 ```python try: handle = open(r"/tmp/sample_data.txt") # May raise IOError data = handle.read() # May raise UnicodeDecodeError except: handle.close() ``` 【正确代码示例】 ```python try: handle = open(r"/tmp/sample_data.txt") # May raise IOError try: data = handle.read() # May raise UnicodeDecodeError except UnicodeDecodeError as decode_error: print(decode_error) finally: handle.close() except(FileNotFoundError, IOError) as file_open_except: print(file_open_except) ``` **规则 2.3.4 不在except分支里面的raise都必须带异常。** raise关键字单独使用只能出现在try-except语句中，重新抛出except抓住的异常。 【错误代码示例】 ```python a = 1 if a == 1: raise ``` 【正确代码示例1】raise一个Exception或自定义的Exception ```python a = 1 if a == 1: raise Exception ``` 【正确代码示例2】在try-except语句中使用 ```python try: f = open('myfile.txt') s = f.readline() i = int(s.strip()) except IOError as e: print("I/O error({0}): {1}".format(e.errno, e.strerror)) except ValueError: print("Could not convert data to an integer.") except Exception: print("Unexpected error:", sys.exc_info()[0]) raise ``` ### 2.4 序列化和反序列化 **规则 2.4.1 pickle存在安全性问题，禁止使用pickle.load、cPickle.load和shelve模块加载不可信数据。** **规则 2.4.2 使用安全随机数。** Python产生随机数的功能在random模块中实现，实现了各种分布的伪随机数生成器。产生的随机数可以是均匀分布，高斯分布，对数正态分布，负指数分布以及alpha，beta分布，但是这些随机数都是伪随机数，不能应用于安全加密目的的应用中。 请使用/dev/random生成安全随机数，或者使用在python 3.6版本官方引入的secrets模块生成安全随机数。 【错误代码示例】 ```python import random # 伪随机数 func = random.SystemRandom() print(func.random()) print(func.randint(0, 10)) ``` 【正确代码示例】 ```python import platform # 长度请参见密码算法规范，不同场景要求长度不一样 randLength = 16 if platform.system() == 'Linux': with open("/dev/random", 'rb') as file: sr = file.read(randLength) print(sr) ``` **规则 2.4.3 assert语句通常只在测试代码中使用，禁止在Release版本中包含assert功能。** assert只应在研发过程中内部测试时使用，出现了AssertionError异常说明存在软件设计或者编码上的错误，应当修改软件予以解决。在对外发布的生产版本中禁止包含assert功能。 pengcheng.mind-test/flash_attention/000077500000000000000000000000001471400134000200735ustar00rootroot00000000000000pengcheng.mind-test/flash_attention/__init__.py000066400000000000000000000001101471400134000221740ustar00rootroot00000000000000# Copyright (C) 2021.Huawei Technologies Co., Ltd. All rights reserved. pengcheng.mind-test/flash_attention/attention.py000066400000000000000000000466051471400134000224650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """the base class of flash attention""" from abc import ABCMeta from abc import abstractmethod from collections import defaultdict from functools import partial import te.platform as tbe_platform from tbe import tik from tbe.common.platform import get_soc_spec from constants import FP16 from constants import FP32 from constants import GM from constants import HARDWARE_MAX_STRIDE_LMT from constants import MASK_FILL_VALUE from constants import UB from constants import TILING_BLOCK_SIZE from tik_ops_utils import TikOpsUtils from tiling_strategy.sparse_tiling import SparseTiling from tiling_strategy.strategy import TilingPara from tiling_strategy.strategy import TilingStrategy class FlashAttention(metaclass=ABCMeta): """The base class of FlashAttention""" def __init__(self, q, k, v, attn_mask, dropout_mask, alibi_mask, kernel_name, tiling_stgy_cls, prev_block_num=65536, next_block_num=65536, disable_debug=True): """ Init parameter shape :param q: with shape: (B, h, N, d) :param k: with shape: (B, h, N, d) :param v: with shape: (B, h, N, d) :param attn_mask: with shape: (1, N, N) or (B, N, N) :param dropout_mask: with shape: (B, h, N, N) :param alibi_mask: with shape: (B, h, 1, N) :param kernel_name: :param tiling_stgy_cls: :param prev_block_num: :param next_block_num: :param disable_debug: """ self.tik_instance = tik.Tik(disable_debug=disable_debug) self.core_num = get_soc_spec(tbe_platform.CORE_NUM) self.M = tbe_platform.get_soc_spec(tbe_platform.L1_SIZE) self.kernel_name = kernel_name self.cont_data_mv_1_bust = partial(self.tik_instance.data_move, sid=0, nburst=1, src_stride=0, dst_stride=0) self.tik_ops_utils = TikOpsUtils(self.tik_instance) self.parse_input_shape(alibi_mask, attn_mask, dropout_mask, k, q, v) # NZ _, _, N1, M1, M0, N0 = self.q_shape self.M1 = M1 self.N1 = N1 self.M0 = M0 self.N0 = N0 self.d = N1 * N0 # ND batch_size, h, Nq, actual_d = self.q_ori_shape self.head_num = h self.B, self.Nq = batch_size * h, Nq self.N = self.k_ori_shape[2] self.actual_d = actual_d self.Nq_N0 = self.Nq * self.N0 self.N_N0 = self.N * self.N0 self.l_shape = [batch_size, h, self.Nq] self.m_shape = [batch_size, h, self.Nq] self.O_shape = self.q_shape self.prev_block_num = prev_block_num self.next_block_num = next_block_num if tiling_stgy_cls is None: self.tiling_stgy = SparseTiling(self.Nq, self.N, self.d) else: self.tiling_stgy: TilingStrategy = tiling_stgy_cls(self.Nq, self.N, self.d) self.Br = None self.last_Br = None self.Bc = None self.last_Bc = None self.Tr = None self.Tc = None self.Q_gm = None self.K_gm = None self.V_gm = None self.dim_mask_gm = None self.att_mask_gm = None self.drop_mask_gm = None self.alibi_mask_gm = None @staticmethod def get_l_gm_offset(batch_start, batch_idx, h, block_h, block_idx): """get l gm offset""" gm_offset = (batch_start + batch_idx) * h + block_idx * block_h return gm_offset @staticmethod def get_alibi_gm_offset(batch_start, batch_idx, w, block_w, block_idx): """get alibi gm offset""" gm_offset = (batch_start + batch_idx) * w + block_idx * block_w return gm_offset @staticmethod def get_drop_mask_gm_offset(batch_start, batch_idx, h, w, block_h, block_h_idx, block_w, block_w_idx): """get drop mask gm offset""" gm_offset = (batch_start + batch_idx) * h * w + block_h_idx * (w * block_h) + block_w_idx * block_w return gm_offset @abstractmethod def define_custom_inputs(self): """define custom inputs""" raise NotImplementedError @abstractmethod def define_outputs(self): """define outputs""" raise NotImplementedError @abstractmethod def collect_inputs(self): """collect inputs""" raise NotImplementedError @abstractmethod def collect_outputs(self): """collect outputs""" raise NotImplementedError @abstractmethod def compute_one_core(self, batch_start_s, batch_num_s, core_idx_to_tr_info, core_idx): """compute one core""" raise NotImplementedError @abstractmethod def prepare_global_ones(self): """prepare global ones""" raise NotImplementedError def get_gm_offset(self, batch_start, batch_idx, h, w, block_h, block_idx): """get gm offset""" gm_offset = (batch_start + batch_idx) * h * w + block_idx * block_h * self.N0 return gm_offset def get_cur_tr_block_num(self, tr_idx): """get cur tr block_num""" cur_prev_block_num = min(tr_idx, self.prev_block_num) cur_next_block_num = min(self.next_block_num, self.Tc - tr_idx - 1) block_num = cur_prev_block_num + 1 + cur_next_block_num return block_num def get_total_block_num(self): """get total block num""" block_num = 0 for b_idx in range(self.B): for tr_idx in range(self.Tr): block_num += self.get_cur_tr_block_num(tr_idx) return block_num def update_core_task_map(self, core_b_map, core_b_tr_map, core_idx, b_start, b_end, tr_start, tr_end): """update core task map""" core_b_map[core_idx][0] = min(core_b_map[core_idx][0], b_start) if tr_end == 0: # 跨head，但跨过的head不会被当前的core处理 core_b_map[core_idx][1] = max(core_b_map[core_idx][1], b_end - 1) else: core_b_map[core_idx][1] = max(core_b_map[core_idx][1], b_end) for b_idx in range(b_start, b_end + 1): if b_idx == b_end and tr_end == 0: # 跨head，但跨过的head不会被当前的core处理 break elif b_idx == b_start and b_idx == b_end: # 没跨head core_b_tr_map[core_idx][b_idx] = (tr_start, tr_end) elif b_idx == b_start: # 跨head，第一个head core_b_tr_map[core_idx][b_idx] = (tr_start, self.Tr) elif b_idx == b_end: # 跨head，最后一个head core_b_tr_map[core_idx][b_idx] = (0, tr_end) else: # 跨head，中间的head core_b_tr_map[core_idx][b_idx] = (0, self.Tr) def convert_py_dict_to_tik_tensor(self, core_b_map, core_b_tr_map): """convert py dict to tik tensor""" # python dict -> tik tensor # [batch_start, batch_idx_end] -> [batch_start, batch_num] # [tr_start, tr_idx_end] -> [tr_start, tr_idx_end) core_idx_to_batch_info = self.tik_instance.Tensor( "int32", (self.core_num, 2), name="core_idx_to_batch_info", scope=UB ) core_idx_to_tr_info = self.tik_instance.Tensor( "int32", (self.core_num, self.B, 2), name="core_idx_to_tr_info", scope=UB ) for core_idx in core_b_map.keys(): batch_start, batch_end = core_b_map[core_idx] core_idx_to_batch_info[core_idx, 0] = batch_start core_idx_to_batch_info[core_idx, 1] = batch_end - batch_start + 1 for batch_idx in core_b_tr_map[core_idx].keys(): tr_start, tr_end = core_b_tr_map[core_idx][batch_idx] core_idx_to_tr_info[core_idx, batch_idx, 0] = tr_start core_idx_to_tr_info[core_idx, batch_idx, 1] = tr_end return core_idx_to_batch_info, core_idx_to_tr_info def get_core_task_info(self): """ Get batch start and batch number of each NPU core. :return: Tensor([[core_1_batch_start, core_1_batch_num],...,[core_m_batch_start, core_m_batch_num]]), Tensor([[[core_1_batch_1_Tr_start, core_1_batch_1_Tr_end],...[core_1_batch_n_Tr_start, core_1_batch_n_Tr_end]],...,[[core_m_batch_1_Tr_start, core_m_batch_1_Tr_end],...[core_m_batch_n_Tr_start, core_m_batch_n_Tr_end]] """ if self.core_num > self.B * self.Tr: self.core_num = self.B * self.Tr total_blk_num = self.get_total_block_num() b_start = 0 tr_start = 0 remain_blk_num = total_blk_num core_b_map = defaultdict(lambda: [100000, -1]) core_b_tr_map = defaultdict(lambda: defaultdict(list)) for core_idx in range(self.core_num): cur_core_blk_num = 0 cur_each_core_blk_num = remain_blk_num // (self.core_num - core_idx) cur_core_finished = False b_end = b_start tr_end = tr_start while b_end < self.B: while tr_end < self.Tr: cur_tr_blk_num = self.get_cur_tr_block_num(tr_end) if abs(cur_core_blk_num - cur_each_core_blk_num) <= \ (cur_core_blk_num + cur_tr_blk_num - cur_each_core_blk_num): self.update_core_task_map(core_b_map, core_b_tr_map, core_idx, b_start, b_end, tr_start, tr_end) remain_blk_num -= cur_core_blk_num cur_core_finished = True break else: cur_core_blk_num += cur_tr_blk_num tr_end += 1 if tr_end == self.Tr: tr_end = 0 b_end += 1 if cur_core_finished: b_start = b_end tr_start = tr_end break core_idx_to_batch_info, core_idx_to_tr_info = self.convert_py_dict_to_tik_tensor(core_b_map, core_b_tr_map) return core_idx_to_batch_info, core_idx_to_tr_info def get_attn_mask_gm_offset(self, batch_start, batch_idx, h, w, block_h, block_h_idx, block_w, block_w_idx): """get attn mask gm offset""" if self.att_mask_shape[0] == 1: gm_offset = block_w_idx * (h * block_w) + block_h_idx * block_h * self.N0 else: gm_offset = ((batch_start + batch_idx) // self.head_num) * h * w \ + block_w_idx * (h * block_w) + block_h_idx * block_h * self.N0 return gm_offset def is_only_causal_mask(self, attn_mask_ori_shape): """check attn mask is only causal mask""" if len(attn_mask_ori_shape) == 2: return attn_mask_ori_shape[0] == TILING_BLOCK_SIZE and attn_mask_ori_shape[1] == TILING_BLOCK_SIZE if len(attn_mask_ori_shape) == 3: return attn_mask_ori_shape[0] == 1 and attn_mask_ori_shape[1] == TILING_BLOCK_SIZE and \ attn_mask_ori_shape[2] == TILING_BLOCK_SIZE def parse_input_shape(self, alibi_mask, attn_mask, dropout_mask, k, q, v): """parser input shape""" self.has_attn_mask = False self.has_drop_mask = False self.has_alibi_mask = False # NZ self.q_shape = q["shape"] self.k_shape = k["shape"] self.v_shape = v["shape"] # ND self.q_ori_shape = q["ori_shape"] self.k_ori_shape = k["ori_shape"] if attn_mask is not None: self.has_attn_mask = True self.att_mask_shape = attn_mask["shape"] attn_mask_ori_shape = attn_mask["ori_shape"] self.only_causal = self.is_only_causal_mask(attn_mask_ori_shape) if dropout_mask is not None: self.has_drop_mask = True self.drop_mask_shape = dropout_mask["shape"] if alibi_mask is not None: self.has_alibi_mask = True self.alibi_mask_shape = alibi_mask["shape"] def define_inputs_outputs(self): """define inputs outputs""" self.define_common_inputs() self.define_custom_inputs() self.define_outputs() def init(self): """init parameters""" tiling_para: TilingPara = self.tiling_stgy.tiling() self.Br = tiling_para.Br self.last_Br = tiling_para.last_Br self.Bc = tiling_para.Bc self.last_Bc = tiling_para.last_Bc self.Tr = tiling_para.Tr self.Tc = tiling_para.Tc self.define_inputs_outputs() def define_common_inputs(self): """define common input gm tensors""" self.Q_gm = self.tik_instance.Tensor(FP16, self.q_shape, name="Q_gm", scope=GM) self.K_gm = self.tik_instance.Tensor(FP16, self.k_shape, name="K_gm", scope=GM) self.V_gm = self.tik_instance.Tensor(FP16, self.v_shape, name="V_gm", scope=GM) if self.has_attn_mask: self.att_mask_gm = self.tik_instance.Tensor(FP16, self.att_mask_shape, name="att_mask_gm", scope=GM) if self.has_drop_mask: self.drop_mask_gm = self.tik_instance.Tensor(FP16, self.drop_mask_shape, name="drop_mask_gm", scope=GM) if self.has_alibi_mask: self.alibi_mask_gm = self.tik_instance.Tensor(FP16, self.alibi_mask_shape, name="alibi_mask_gm", scope=GM) def do_alibi_mask(self, Sij_ub, alibi_mask_gm_offset, m_aligned, n_aligned): """load alibi mask from gm to ub, then add Sij""" with self.tik_instance.new_stmt_scope(disable_sync=False): alibi_mask_ub = self.tik_instance.Tensor(FP16, (1, n_aligned), scope=UB, name="alibi_mask_ub") self.tik_instance.data_move(alibi_mask_ub, self.alibi_mask_gm[alibi_mask_gm_offset], 0, 1, n_aligned // 16, 0, 0) alibi_mask_ub_broadcast = self.tik_ops_utils.broadcast_row(alibi_mask_ub, (m_aligned, n_aligned)) self.tik_instance.h_add(Sij_ub, Sij_ub, alibi_mask_ub_broadcast) def do_att_mask(self, Sij_ub_zN, attn_mask_gm_offset, q_blk_height, kv_blk_height, q_blk_h_aligned, kv_blk_h_aligned, only_causal=False): """load attn mask from gm to ub, then mul it by MASK_FILL_VALUE and add Sij""" with self.tik_instance.new_stmt_scope(disable_sync=False): att_mask_ub_zN = self.tik_instance.Tensor(FP16, (kv_blk_h_aligned // self.N0, q_blk_h_aligned, self.N0), scope=UB, name="att_mask_ub_zN") if only_causal: self.cont_data_mv_1_bust(dst=att_mask_ub_zN, src=self.att_mask_gm, burst=q_blk_h_aligned * kv_blk_h_aligned // 16) else: self.tik_instance.data_move(att_mask_ub_zN, self.att_mask_gm[attn_mask_gm_offset], 0, kv_blk_height // self.N0, q_blk_height * self.N0 // 16, (self.Nq - q_blk_height) * self.N0 // 16, 0) self.tik_instance.h_mul(att_mask_ub_zN, att_mask_ub_zN, MASK_FILL_VALUE) self.tik_instance.h_add(Sij_ub_zN, Sij_ub_zN, att_mask_ub_zN) def do_dropout_mask(self, Pij_ub, dropout_mask_gm_offset, kv_blk_h_aligned, kv_blk_height, q_blk_h_aligned, q_blk_height, precision_type=FP16, workspace=None): """load drop mask from gm to ub, then mul it by Pij""" with self.tik_instance.new_stmt_scope(disable_sync=False): dropout_mask_ub = self.tik_instance.Tensor(FP16, (q_blk_h_aligned, kv_blk_h_aligned), scope=UB, name="drop_mask_ub") self.tik_instance.data_move(dropout_mask_ub, self.drop_mask_gm[dropout_mask_gm_offset], 0, q_blk_height, kv_blk_height // 16, (self.N - kv_blk_height) // 16, 0) dropout_mask_ub = dropout_mask_ub.reshape((kv_blk_h_aligned // self.N0, q_blk_h_aligned, self.N0)) if precision_type == FP32: dropout_mask_ub_fp32 = self.tik_instance.Tensor(FP32, (kv_blk_h_aligned // self.N0, q_blk_h_aligned, self.N0), scope=UB, name="dropout_mask_ub_fp32") self.tik_instance.h_cast(dropout_mask_ub_fp32, dropout_mask_ub, "none") if workspace is None: self.tik_instance.h_mul(Pij_ub, Pij_ub, dropout_mask_ub_fp32) else: self.tik_instance.h_mul(workspace, Pij_ub, dropout_mask_ub_fp32) else: if workspace is None: self.tik_instance.h_mul(Pij_ub, Pij_ub, dropout_mask_ub) else: self.tik_instance.h_mul(workspace, Pij_ub, dropout_mask_ub) def compute_process(self): """The compute process of FlashAttention""" self.init() self.prepare_global_ones() core_idx_to_batch_info, core_idx_to_tr_info = self.get_core_task_info() with self.tik_instance.for_range(begint=0, endt=self.core_num, name="core_index", block_num=self.core_num) as core_idx: batch_start_s = self.tik_instance.Scalar("int32", name="batch_start_s") batch_num_s = self.tik_instance.Scalar("int32", name="batch_num_s") batch_start_s.set_as(core_idx_to_batch_info[core_idx, 0]) batch_num_s.set_as(core_idx_to_batch_info[core_idx, 1]) self.compute_one_core(batch_start_s, batch_num_s, core_idx_to_tr_info, core_idx) self.tik_instance.BuildCCE( kernel_name=self.kernel_name, inputs=self.collect_inputs(), outputs=self.collect_outputs(), config={"dump_cce_code": False, "save_temp_cce_file": True, "enable_const_fold": True}, enable_l2=True ) def unroll_data_move(self, dst, dst_offset, src, src_offset, nburst, burst, src_stride, dst_stride, dst_offset_inc, src_offset_inc): if dst_stride <= HARDWARE_MAX_STRIDE_LMT and src_stride <= HARDWARE_MAX_STRIDE_LMT: self.tik_instance.data_move( dst=dst[dst_offset], src=src[src_offset], sid=0, nburst=nburst, burst=burst, dst_stride=dst_stride, src_stride=src_stride ) else: for _ in range(nburst): self.cont_data_mv_1_bust(dst=dst[dst_offset], src=src[src_offset], burst=burst) src_offset += src_offset_inc dst_offset += dst_offset_inc pengcheng.mind-test/flash_attention/constants.py000066400000000000000000000022771471400134000224710ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """define constants""" from tbe import tik HARDWARE_MAX_STRIDE_LMT = 65535 # < 64K TILING_BLOCK_SIZE = 128 BLOCK_NUM = 16 FP16 = "float16" INT8 = "int8" UINT8 = "uint8" INT32 = "int32" FP32 = "float32" REPEAT_SZ = 128 BLK_STRIDE = 1 REPEAT_STRIDE = 8 TRANS_CUBE_TGT = 8 FP16_MIN_VAL = -65504.0 MASK_FILL_VALUE = -10000.0 GM = tik.scope_gm L1 = tik.scope_cbuf L1OUT = tik.scope_cbuf_out UB = tik.scope_ubuf L0A = tik.scope_ca L0B = tik.scope_cb L0C = tik.scope_cc DTYPE_SIZE = { "int8": 1, "float16": 2, "int16": 2, "float32": 4, } pengcheng.mind-test/flash_attention/flash_attention_bwd.py000066400000000000000000000646031471400134000244740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The backward tik ops of flash attention""" from tbe import tik from attention import FlashAttention from constants import FP16 from constants import FP32 from constants import GM from constants import L1 from constants import UB from tiling_strategy.strategy import TilingStrategy class FlashAttentionBwd(FlashAttention): """The implementation of FlashAttention backward This function contains the flash attention backward implementation used in flash attention (see paper) `FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning ` """ def __init__(self, q, k, v, O, dO, l, attn_mask, dropout_mask, alibi_mask, prev_block_num, next_block_num, kernel_name, tiling_stgy: TilingStrategy, disable_debug): super().__init__(q, k, v, attn_mask, dropout_mask, alibi_mask, kernel_name, tiling_stgy, prev_block_num, next_block_num, disable_debug) self.dO_shape = dO["shape"] # [B, Nq, d] self.dV_shape = self.v_shape self.dQ_shape = self.q_shape self.dK_shape = self.k_shape self.dQ_gm = None self.dK_gm = None self.dV_gm = None self.O_gm = None self.dO_gm = None self.l_gm = None def define_outputs(self): """define output gm tensors""" self.dQ_gm = self.tik_instance.Tensor(FP32, self.dQ_shape, name="dQ_gm", scope=GM, is_atomic_add=True) self.dK_gm = self.tik_instance.Tensor(FP32, self.dK_shape, name="dK_gm", scope=GM, is_atomic_add=True) self.dV_gm = self.tik_instance.Tensor(FP32, self.dV_shape, name="dV_gm", scope=GM, is_atomic_add=True) def define_custom_inputs(self): """define input gm tensors""" self.O_gm = self.tik_instance.Tensor(FP16, self.O_shape, name="O_gm", scope=GM) self.dO_gm = self.tik_instance.Tensor(FP16, self.dO_shape, name="dO_gm", scope=GM) self.l_gm = self.tik_instance.Tensor(FP32, self.l_shape, name="l_gm", scope=GM) def collect_inputs(self): """collect all input gm tensors into input_gm_list, the input list should keep order with the para order in Primitive and init """ input_gm_list = [ self.Q_gm, self.K_gm, self.V_gm, self.O_gm, self.dO_gm, self.l_gm ] if self.has_attn_mask: input_gm_list.append(self.att_mask_gm) if self.has_drop_mask: input_gm_list.append(self.drop_mask_gm) if self.has_alibi_mask: input_gm_list.append(self.alibi_mask_gm) return input_gm_list def prepare_global_ones(self): """Prepare global ones tensor in L1 for cube impl row_sum""" self.ones_l1 = self.tik_instance.Tensor(FP16, (self.d, 16), name="ones_l1", scope=L1) with self.tik_instance.new_stmt_scope(disable_sync=False): ones_ub = self.tik_instance.Tensor(FP16, (self.d, 16), name="ones_ub", scope=UB) self.tik_instance.h_duplicate(ones_ub, 1.0) self.cont_data_mv_1_bust(dst=self.ones_l1, src=ones_ub, burst=self.d) def compute_Pij(self, Qi_l1_K1MK0_ed, KjT_l1_K1NK0_ed, m, k, n, l_gm_offset, attn_mask_gm_offset, dropout_mask_gm_offset, alibi_mask_gm_offset, q_blk_idx, kv_blk_idx): """Refer to Algorithm 4 line11-14 in FlashAttention implement Pij computation""" m_aligned = self.tik_ops_utils.up_align_to_K0(m) n_aligned = self.tik_ops_utils.up_align_to_K0(n) # Sij <- Qi * KjT Sij_ub = self.tik_ops_utils.matmul_compute(Qi_l1_K1MK0_ed, KjT_l1_K1NK0_ed, m, k, n, N1MN0_to_MN=False) with self.tik_instance.new_stmt_scope(disable_sync=False): # TODO NZ 适配 alibi if self.has_attn_mask: if self.only_causal: with self.tik_instance.if_scope(q_blk_idx == kv_blk_idx): self.do_att_mask(Sij_ub, attn_mask_gm_offset, m, n, m_aligned, n_aligned, self.only_causal) else: self.do_att_mask(Sij_ub, attn_mask_gm_offset, m, n, m_aligned, n_aligned) # move li (ith block of l_gm) and mi (ith block of m_gm) from gm to ub li_ub = self.tik_instance.Tensor(FP32, (m_aligned,), name="li_ub", scope=UB) self.tik_ops_utils.move_vector_from_gm_to_ub(li_ub, self.l_gm, l_gm_offset, m) # broadcast broadcast_li_ub = self.tik_instance.Tensor(FP32, (1, m_aligned, self.N0), scope=UB, name="broadcast_li_ub") src_scalar = self.tik_instance.Scalar(FP32, name="src_scalar") for idx in range(0, m): src_scalar.set_as(li_ub[idx]) self.tik_instance.h_duplicate(broadcast_li_ub[0, idx, :], src_scalar) # fp16 -> fp32 Sij_ub_fp32 = self.tik_instance.Tensor(FP32, (n_aligned // self.N0, m_aligned, self.N0), name="Sij_ub_fp32", scope=UB) self.tik_instance.h_cast(Sij_ub_fp32, Sij_ub, "none") # Sij - Li for idx in range(n_aligned // self.N0): self.tik_instance.h_sub(Sij_ub_fp32[idx, :, :], Sij_ub_fp32[idx, :, :], broadcast_li_ub) # exp self.tik_instance.h_exp(Sij_ub_fp32, Sij_ub_fp32) # fp32 -> fp16 self.tik_instance.h_cast(Sij_ub, Sij_ub_fp32, "none") # dropout_mask if self.has_drop_mask: Pij_drop_ed_ub = self.tik_instance.Tensor(FP16, (n_aligned // self.N0, m_aligned, self.N0), name="Pij_drop_ed_ub", scope=UB) self.do_dropout_mask(Sij_ub, dropout_mask_gm_offset, n_aligned, n, m_aligned, m, workspace=Pij_drop_ed_ub) else: Pij_drop_ed_ub = Sij_ub return Sij_ub, Pij_drop_ed_ub def compute_Di(self, Di_ub, dOi_ub, qo_gm_offset, q_blk_height): """Refer to Algorithm 4 line19 in FlashAttention implement Di computation""" q_blk_height_aligned = self.tik_ops_utils.up_align_to_K0(q_blk_height) with self.tik_instance.new_stmt_scope(disable_sync=False): Oi_ub = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_aligned, self.N0), scope=UB, name="Oi_ub") self.unroll_data_move(dst=Oi_ub, dst_offset=0, src=self.O_gm, src_offset=qo_gm_offset, nburst=self.N1, burst=q_blk_height * self.N0 // 16, src_stride=(self.Nq - q_blk_height) * self.N0 // 16, dst_stride=0, dst_offset_inc=q_blk_height * self.N0, src_offset_inc=self.Nq * self.N0) self.tik_instance.h_mul(Oi_ub, dOi_ub, Oi_ub) dOi_Oi_l1_K1MK0 = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_aligned, self.N0), name="dOi_Oi_l1_K1MK0", scope=L1) self.cont_data_mv_1_bust(dst=dOi_Oi_l1_K1MK0, src=Oi_ub, burst=q_blk_height_aligned * self.d // 16) self.tik_ops_utils.row_sum_cube_impl(dOi_Oi_l1_K1MK0, self.ones_l1, Di_ub, q_blk_height, self.actual_d, precision_type=FP16) def compute_dSij(self, Pij_ub, dOi_l1_K1MK0_ed, VjT_K1NK0_ed, Di_ub, kv_blk_height, q_blk_height, dropout_mask_gm_offset): """Refer to Algorithm 4 line20 in FlashAttention implement dSij computation""" with self.tik_instance.new_stmt_scope(disable_sync=False): dPij_ub = self.tik_ops_utils.matmul_compute(dOi_l1_K1MK0_ed, VjT_K1NK0_ed, q_blk_height, self.actual_d, kv_blk_height, N1MN0_to_MN=False) q_blk_height_aligned = self.tik_ops_utils.up_align_to_K0(q_blk_height) kv_blk_height_aligned = self.tik_ops_utils.up_align_to_K0(kv_blk_height) # dropout_mask if self.has_drop_mask: self.do_dropout_mask(dPij_ub, dropout_mask_gm_offset, kv_blk_height_aligned, kv_blk_height, q_blk_height_aligned, q_blk_height) # dPij - Di with self.tik_instance.new_stmt_scope(disable_sync=False): broadcast_Di_ub = self.tik_ops_utils.broadcast_vec_from_M_to_MN0(Di_ub) broadcast_Di_ub = broadcast_Di_ub.reshape((1, q_blk_height_aligned, self.N0)) n1 = kv_blk_height_aligned // self.N0 for idx in range(n1): self.tik_instance.h_sub(dPij_ub[idx, :, :], dPij_ub[idx, :, :], broadcast_Di_ub) self.tik_instance.h_mul(Pij_ub, Pij_ub, dPij_ub) return Pij_ub def update_dVj(self, Pij_l1_zN, dOi_l1_zN, kv_gm_offset, kv_blk_height, q_blk_height): """Refer to Algorithm 4 line16 in FlashAttention implement dVj update""" with self.tik_instance.new_stmt_scope(disable_sync=False): PijT_Oi_ub = self.tik_ops_utils.mmad_compute_transpose(Pij_l1_zN, dOi_l1_zN, kv_blk_height, q_blk_height, self.actual_d, N1MN0_to_MN=False, a_transpose=True, precision_type=FP32) self.tik_instance.set_atomic_add(1) # 实测 2 (fp16) 不支持 self.unroll_data_move(dst=self.dV_gm, dst_offset=kv_gm_offset, src=PijT_Oi_ub, src_offset=0, nburst=self.N1, burst=kv_blk_height * self.N0 // 8, src_stride=0, dst_stride=(self.Nq - kv_blk_height) * self.N0 // 8, dst_offset_inc=self.Nq * self.N0, src_offset_inc=kv_blk_height * self.N0) self.tik_instance.set_atomic_add(0) def update_dQi(self, dSij_l1_zN, Kj_l1_zN, qo_gm_offset, q_blk_height, kv_blk_height): """Refer to Algorithm 4 line21 in FlashAttention implement dQi update""" with self.tik_instance.new_stmt_scope(disable_sync=False): dSij_Kj_ub = self.tik_ops_utils.mmad_compute(dSij_l1_zN, Kj_l1_zN, q_blk_height, kv_blk_height, self.actual_d, N1MN0_to_MN=False, precision_type=FP32) self.tik_instance.set_atomic_add(1) self.unroll_data_move(dst=self.dQ_gm, dst_offset=qo_gm_offset, src=dSij_Kj_ub, src_offset=0, nburst=self.d // self.N0, burst=q_blk_height * self.N0 // 8, src_stride=0, dst_stride=(self.Nq - q_blk_height) * self.N0 // 8, dst_offset_inc=self.Nq * self.N0, src_offset_inc=q_blk_height * self.N0) self.tik_instance.set_atomic_add(0) def update_dKj(self, dSij_l1_zN, Qi_l1_zN, kv_gm_offset, kv_blk_height, q_blk_height): """Refer to Algorithm 4 line22 in FlashAttention implement dKi update""" with self.tik_instance.new_stmt_scope(disable_sync=False): dSijT_Qi_ub = self.tik_ops_utils.mmad_compute_transpose(dSij_l1_zN, Qi_l1_zN, kv_blk_height, q_blk_height, self.actual_d, N1MN0_to_MN=False, a_transpose=True, precision_type=FP32) self.tik_instance.set_atomic_add(1) self.unroll_data_move(dst=self.dK_gm, dst_offset=kv_gm_offset, src=dSijT_Qi_ub, src_offset=0, nburst=self.d // self.N0, burst=kv_blk_height * self.N0 // 8, src_stride=0, dst_stride=(self.Nq - kv_blk_height) * self.N0 // 8, dst_offset_inc=self.Nq * self.N0, src_offset_inc=kv_blk_height * self.N0) self.tik_instance.set_atomic_add(0) def compute_in_each_kv_block(self, batch_start, batch_idx, kv_blk_idx, kv_blk_height, core_idx_to_tr_info, core_idx): """The backward computation in each outer loop""" kv_blk_height_aligned = self.tik_ops_utils.up_align_to_K0(kv_blk_height) kv_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.N, self.d, self.Bc, kv_blk_idx) # load KjT Kj_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, kv_blk_height_aligned, self.N0), name="Kj_l1_zN", scope=L1) # for Qi*KjT (算法11行) self.unroll_data_move(dst=Kj_l1_zN, dst_offset=0, src=self.K_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=kv_blk_height_aligned * self.N0 // 16, src_stride=(self.N - kv_blk_height_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=kv_blk_height_aligned * self.N0, src_offset_inc=self.N * self.N0) # load VjT Vj_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, kv_blk_height_aligned, self.N0), name="Vj_l1_zN", scope=L1) self.unroll_data_move(dst=Vj_l1_zN, dst_offset=0, src=self.V_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=kv_blk_height_aligned * self.N0 // 16, src_stride=(self.N - kv_blk_height_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=kv_blk_height_aligned * self.N0, src_offset_inc=self.N * self.N0) tr_start_s = self.tik_instance.Scalar("int32", name="tr_start_s") tr_end_s = self.tik_instance.Scalar("int32", name="tr_end_s") tr_start_s.set_as(core_idx_to_tr_info[core_idx, batch_start + batch_idx, 0]) tr_end_s.set_as(core_idx_to_tr_info[core_idx, batch_start + batch_idx, 1]) with self.tik_instance.for_range(tr_start_s, tr_end_s, name="q_blk_idx") as q_blk_idx: with self.tik_instance.if_scope(tik.all(kv_blk_idx - self.next_block_num <= q_blk_idx, q_blk_idx <= kv_blk_idx + self.prev_block_num)): with self.tik_instance.if_scope(q_blk_idx != self.Tr - 1): self.compute_in_each_q_block(Kj_l1_zN, Vj_l1_zN, batch_idx, batch_start, kv_gm_offset, kv_blk_height, self.Br, kv_blk_idx, q_blk_idx) with self.tik_instance.else_scope(): self.compute_in_each_q_block(Kj_l1_zN, Vj_l1_zN, batch_idx, batch_start, kv_gm_offset, kv_blk_height, self.last_Br, kv_blk_idx, q_blk_idx) def compute_in_each_q_block(self, Kj_l1_zN, Vj_l1_zN, batch_idx, batch_start, kv_gm_offset, kv_blk_height, q_blk_height, kv_blk_idx, q_blk_idx): """The backward computation in each inner loop""" kv_blk_height_alig = self.tik_ops_utils.up_align_to_K0(kv_blk_height) q_blk_height_alig = self.tik_ops_utils.up_align_to_K0(q_blk_height) # load Qi并做分形 (L1: 256K) qo_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.Nq, self.d, self.Br, q_blk_idx) Qi_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_alig, self.N0), name="Qi_l1_zN", scope=L1) # for Qi*KjT (算法11行) self.unroll_data_move(dst=Qi_l1_zN, dst_offset=0, src=self.Q_gm, src_offset=qo_gm_offset, nburst=self.N1, burst=q_blk_height_alig * self.N0 // 16, src_stride=(self.Nq - q_blk_height_alig) * self.N0 // 16, dst_stride=0, dst_offset_inc=q_blk_height_alig * self.N0, src_offset_inc=self.Nq * self.N0) l_gm_offset = self.get_l_gm_offset(batch_start, batch_idx, self.Nq, self.Br, q_blk_idx) attn_mask_gm_offset, dropout_mask_gm_offset, alibi_mask_gm_offset = None, None, None if self.has_attn_mask: if self.only_causal: attn_mask_gm_offset = 0 else: attn_mask_gm_offset = self.get_attn_mask_gm_offset(batch_start, batch_idx, self.Nq, self.N, self.Br, q_blk_idx, self.Bc, kv_blk_idx) if self.has_drop_mask: dropout_mask_gm_offset = self.get_drop_mask_gm_offset(batch_start, batch_idx, self.Nq, self.N, self.Br, q_blk_idx, self.Bc, kv_blk_idx) if self.has_alibi_mask: alibi_mask_gm_offset = self.get_alibi_gm_offset(batch_start, batch_idx, self.N, self.Bc, kv_blk_idx) # 算法11~15行 Pij_ub, Pij_drop_ed_ub = self.compute_Pij(Qi_l1_zN, Kj_l1_zN, q_blk_height, self.actual_d, kv_blk_height, l_gm_offset, attn_mask_gm_offset, dropout_mask_gm_offset, alibi_mask_gm_offset, q_blk_idx, kv_blk_idx) dOi_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_alig, self.N0), name="dOi_l1_zN", scope=L1) # for dOi * VjT (算法17行) okk self.unroll_data_move( dst=dOi_l1_zN, dst_offset=0, src=self.dO_gm, src_offset=qo_gm_offset, nburst=self.N1, burst=q_blk_height_alig * self.N0 // 16, src_stride=(self.Nq - q_blk_height_alig) * self.N0 // 16, dst_stride=0, dst_offset_inc=q_blk_height_alig * self.N0, src_offset_inc=self.Nq * self.N0 ) # load dOi并做分形，提前计算Di，否则dOi_ub无法释放，或者后面计算Di时需要再次load dOi (L1: 320K) Di_ub = self.tik_instance.Tensor(FP16, (q_blk_height_alig,), name="Di_ub", scope=UB) # 为了释放dOi_ub，提前计算Di with self.tik_instance.new_stmt_scope(disable_sync=False): dOi_ub = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_alig, self.N0), name="dOi_ub", scope=UB) self.cont_data_mv_1_bust(dst=dOi_ub, src=dOi_l1_zN, burst=q_blk_height_alig * self.d // 16) self.compute_Di(Di_ub, dOi_ub, qo_gm_offset, q_blk_height) # 分形，为PijT*dOi(算法16行)做准备 Pij_l1_zN = self.tik_instance.Tensor(FP16, (q_blk_height_alig, kv_blk_height_alig), name="Pij_l1_zN", scope=L1) self.cont_data_mv_1_bust(dst=Pij_l1_zN, src=Pij_drop_ed_ub, burst=q_blk_height_alig * kv_blk_height_alig // 16) # 算法24行：write dKj dVj (放入内循环=>增大tiling的blocksize) self.update_dVj(Pij_l1_zN, dOi_l1_zN, kv_gm_offset, kv_blk_height, q_blk_height) # (L1: 512K) dSij_l1_Nz = self.tik_instance.Tensor(FP16, (kv_blk_height_alig // self.N0, q_blk_height_alig, self.N0), name="dSij_l1_Nz", scope=L1) # for dSij*Kj (算法21行) with self.tik_instance.new_stmt_scope(disable_sync=False): # 为了释放dSij_ub dSij_ub = self.compute_dSij(Pij_ub, dOi_l1_zN, Vj_l1_zN, Di_ub, kv_blk_height, q_blk_height, dropout_mask_gm_offset) # for dSij*Kj (算法21行) self.cont_data_mv_1_bust(dst=dSij_l1_Nz, src=dSij_ub, burst=kv_blk_height_alig * q_blk_height_alig // 16) # 算法21行 self.update_dQi(dSij_l1_Nz, Kj_l1_zN, qo_gm_offset, q_blk_height, kv_blk_height) # 算法22行 self.update_dKj(dSij_l1_Nz, Qi_l1_zN, kv_gm_offset, kv_blk_height, q_blk_height) def compute_one_core(self, batch_start_sc, batch_num_sc, core_idx_to_tr_info, core_idx): """The computation of FlashAttention backward on each core""" with self.tik_instance.for_range(0, batch_num_sc, name="batch_index") as batch_idx: with self.tik_instance.for_range(0, self.Tc, name="kv_blk_idx") as kv_blk_idx: with self.tik_instance.if_scope(kv_blk_idx != self.Tc - 1): self.compute_in_each_kv_block(batch_start_sc, batch_idx, kv_blk_idx, self.Bc, core_idx_to_tr_info, core_idx) with self.tik_instance.else_scope(): self.compute_in_each_kv_block(batch_start_sc, batch_idx, kv_blk_idx, self.last_Bc, core_idx_to_tr_info, core_idx) def collect_outputs(self): """collect all output gm tensors into output_gm_list, the output list should keep order with the para order in Primitive and init """ output_gm_list = [self.dQ_gm, self.dK_gm, self.dV_gm] return output_gm_list def flash_attention_grad(query, key, value, output, dO, rowsum, attn_mask, dropout_mask, alibi_mask, dq, dk, dv, prev_block_num=65536, next_block_num=65536, tiling_stgy_name='sparse', kernel_name="flash_attention_grad", disable_debug=True): """ algorithm: flash_attention_backward Parameters ---------- query : dict. shape and dtype of input, only support float16 key : dict. shape and dtype of input, only support float16 value: dict. shape and dtype of input, only support float16 output: dict. shape and dtype of input, only support float16 dO: dict. shape and dtype of input, only support float16 rowsum: dict. shape and dtype of input, only support float32 attn_mask: dict. shape and dtype of input, only support float16 dropout_mask: dict. shape and dtype of input, only support float16 alibi_mask: dict. shape and dtype of input, only support float16 dq: dict. shape and dtype of output, only support float16 dk: dict. shape and dtype of output, only support float16 dv: dict. shape and dtype of output, only support float16 prev_block_num: int. an attribute used to define sparse attention next_block_num: int. an attribute used to define sparse attention tiling_stgy_name: str. an attribute used to choose the tiling strategy kernel_name: str. cce kernel name, default value is real_div disable_debug: bool. whether disable debug Returns ------- tik_instance """ fa_grad = FlashAttentionBwd(q=query, k=key, v=value, O=output, dO=dO, l=rowsum, attn_mask=attn_mask, dropout_mask=dropout_mask, alibi_mask=alibi_mask, prev_block_num=prev_block_num, next_block_num=next_block_num, kernel_name=kernel_name, tiling_stgy=TilingStrategy.from_strategy_name(tiling_stgy_name), disable_debug=disable_debug) fa_grad.compute_process() return fa_grad.tik_instance pengcheng.mind-test/flash_attention/flash_attention_fwd.py000066400000000000000000000706751471400134000245060ustar00rootroot00000000000000from tbe import tik from attention import FlashAttention from constants import FP16 from constants import FP32 from constants import GM from constants import L1 from constants import UB from tiling_strategy.strategy import TilingStrategy class FlashAttentionFwd(FlashAttention): """The implementation of flash attention forward This function contains the flash attention forward implementation used in flash attention (see paper) `FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning ` """ def __init__(self, q, k, v, attn_mask, dropout_mask, alibi_mask, kernel_name, tiling_stgy: TilingStrategy, prev_block_num=65536, next_block_num=65536, disable_debug=True): super().__init__(q, k, v, attn_mask, dropout_mask, alibi_mask, kernel_name, tiling_stgy, prev_block_num, next_block_num, disable_debug) def define_custom_inputs(self): pass def define_outputs(self): """define output gm tensors""" self.O_gm = self.tik_instance.Tensor(FP16, self.O_shape, name="O_gm", scope=GM, is_atomic_add=True) self.l_gm = self.tik_instance.Tensor(FP32, self.l_shape, name="l_gm", scope=GM, is_atomic_add=True) def prepare_global_ones(self): """Prepare global ones tensor in L1 for cube impl row_sum""" Bc_aligned = (self.Bc + 15) // 16 * 16 last_Bc_aligned = (self.last_Bc + 15) // 16 * 16 self.ones_l1 = self.tik_instance.Tensor(FP16, (Bc_aligned, 16), name="ones_l1", scope=L1) self.last_ones_l1 = self.tik_instance.Tensor(FP16, (last_Bc_aligned, 16), name="last_ones_l1", scope=L1) with self.tik_instance.new_stmt_scope(disable_sync=False): ones_ub = self.tik_instance.Tensor(FP16, (Bc_aligned, 16), name="ones_ub", scope=UB) self.tik_instance.h_duplicate(ones_ub, 1.0) self.cont_data_mv_1_bust(dst=self.ones_l1, src=ones_ub, burst=Bc_aligned) last_ones_ub = self.tik_instance.Tensor(FP16, (last_Bc_aligned, 16), name="last_ones_ub", scope=UB) self.tik_instance.h_duplicate(ones_ub, 1.0) self.cont_data_mv_1_bust(dst=self.last_ones_l1, src=last_ones_ub, burst=last_Bc_aligned) def rowmax_compute(self, Sij_ub_zN, mi_ub, kv_blk_idx): """Calculate row max for softmax of flashattention2 :param Sij_ub_zN: with shape N1 x M x N0 :param mi_ub: :param kv_blk_idx: :return: """ n1, m, n0 = Sij_ub_zN.shape # update_scale_ub: for update li and Oi(exp(mi_old - mi_new)) update_scale_ub = self.tik_instance.Tensor(FP32, (m,), scope=UB, name="update_scale_ub") with self.tik_instance.if_scope(kv_blk_idx != 0): self.tik_instance.h_cast(update_scale_ub, mi_ub, "none") with self.tik_instance.new_stmt_scope(disable_sync=False): # TODO: 二分 mn0_block_max = self.tik_instance.Tensor(FP16, (1, m, n0), name="mn0_block_max", scope=UB) self.cont_data_mv_1_bust(dst=mn0_block_max, src=Sij_ub_zN, burst=m) # 用第一个(m, n0)始化，少一次duplicate和max for idx in range(1, n1): self.tik_instance.h_max(mn0_block_max, mn0_block_max, Sij_ub_zN[idx, :, :]) # reduce_max mn0_block_max = mn0_block_max.reshape((m, n0)) mij_ub = self.tik_instance.Tensor(FP16, (m,), scope=UB, name="mij_ub") self.tik_instance.h_reduce_max(mij_ub, mn0_block_max, 1) # update mi with self.tik_instance.if_scope(kv_blk_idx == 0): self.cont_data_mv_1_bust(dst=mi_ub, src=mij_ub, burst=m // 16) with self.tik_instance.else_scope(): self.tik_instance.h_max(mi_ub, mi_ub, mij_ub) with self.tik_instance.if_scope(kv_blk_idx != 0): # exp(mi_old - mi_new) mi_ub_fp32 = self.tik_instance.Tensor(FP32, (m,), scope=UB, name="mi_ub_fp32") self.tik_instance.h_cast(mi_ub_fp32, mi_ub, "none") self.tik_instance.h_sub(update_scale_ub, update_scale_ub, mi_ub_fp32) self.tik_instance.h_exp(update_scale_ub, update_scale_ub) return mi_ub, update_scale_ub def rowsum_compute(self, Pij_l1_zN, li_ub, update_scale_ub, kv_blk_idx): """Calculate row sum for softmax of flashattention2 :param Pij_l1_zN: with shape N1 x M x N0 :param li_ub: :param update_scale_ub: :param kv_blk_idx: :return: """ n1, m, n0 = Pij_l1_zN.shape # cube impl rowsum of Pij lij_ub = self.tik_instance.Tensor(FP32, (m,), scope=UB, name="lij_ub") if n1 * n0 == self.Bc: lij_ub = self.tik_ops_utils.row_sum_cube_impl(Pij_l1_zN, self.ones_l1, lij_ub, m, self.Bc, precision_type=FP32) else: lij_ub = self.tik_ops_utils.row_sum_cube_impl(Pij_l1_zN, self.last_ones_l1, lij_ub, m, self.last_Bc, precision_type=FP32) # update li with self.tik_instance.if_scope(kv_blk_idx == 0): self.cont_data_mv_1_bust(dst=li_ub, src=lij_ub, burst=m // 8) with self.tik_instance.else_scope(): self.tik_instance.h_mul(li_ub, li_ub, update_scale_ub) self.tik_instance.h_add(li_ub, li_ub, lij_ub) return li_ub def softmax_compute(self, Sij_ub_zN, li_ub, mi_ub, kv_blk_idx, drop_mask_gm_offset): """Calculate softmax :param Sij_ub_zN: with shape N1 x M x N0 :param li_ub: :param mi_ub: :param kv_blk_idx: :param drop_mask_gm_offset: :return: """ # rowmax mi_ub, update_scale_ub = self.rowmax_compute(Sij_ub_zN, mi_ub, kv_blk_idx) # Sij - mi n1, m, n0 = Sij_ub_zN.shape with self.tik_instance.new_stmt_scope(disable_sync=False): broadcast_mij_ub = self.tik_ops_utils.broadcast_vec_from_M_to_MN0(mi_ub) broadcast_mij_ub = broadcast_mij_ub.reshape((1, m, n0)) for idx in range(n1): self.tik_instance.h_sub(Sij_ub_zN[idx, :, :], Sij_ub_zN[idx, :, :], broadcast_mij_ub) # exp with self.tik_instance.new_stmt_scope(disable_sync=False): Sij_ub_zN_fp32 = self.tik_instance.Tensor(FP32, (n1, m, n0), scope=UB, name="Sij_ub_zN_fp32") self.tik_instance.h_cast(Sij_ub_zN_fp32, Sij_ub_zN, "none") self.tik_instance.h_exp(Sij_ub_zN_fp32, Sij_ub_zN_fp32) self.tik_instance.h_cast(Sij_ub_zN, Sij_ub_zN_fp32, "none") # rowsum Pij_l1_zN = self.tik_instance.Tensor(FP16, (n1, m, n0), scope=L1, name="Pij_l1_zN") self.cont_data_mv_1_bust(dst=Pij_l1_zN, src=Sij_ub_zN, burst=n1 * m * n0 // 16) li_ub = self.rowsum_compute(Pij_l1_zN, li_ub, update_scale_ub, kv_blk_idx) # dropout if self.has_drop_mask: self.do_dropout_mask(Sij_ub_zN, drop_mask_gm_offset, n1 * n0, n1 * n0, m, m) self.cont_data_mv_1_bust(dst=Pij_l1_zN, src=Sij_ub_zN, burst=n1 * m * n0 // 16) return Pij_l1_zN, li_ub, mi_ub, update_scale_ub def output_update(self, Oi_l1_zN, Oij_ub_zN, broadcast_scale_ub, offset, n1_begin, n1_size, m): """Update output of flashattention2 :param Oi_l1_zN: with shape N1 x M x N0 :param Oij_ub_zN: :param broadcast_scale_ub: :param offset: :param n1_begin: :param n1_size: :param m: :return: """ n1_end = n1_begin + n1_size Oi_ub_zN = self.tik_instance.Tensor(FP32, (n1_size, m, self.N0), scope=UB, name="Oi_ub_zN") # mte1: L1 -> UB self.cont_data_mv_1_bust(dst=Oi_ub_zN, src=Oi_l1_zN[offset], burst=n1_size * m * self.N0 // 8) # vector: mul and add for i in range(n1_size): self.tik_instance.h_mul(Oi_ub_zN[i, :, :], Oi_ub_zN[i, :, :], broadcast_scale_ub) self.tik_instance.h_add(Oi_ub_zN, Oi_ub_zN, Oij_ub_zN[n1_begin:n1_end, :, :]) # mte3: UB -> L1 self.cont_data_mv_1_bust(dst=Oi_l1_zN[offset], src=Oi_ub_zN, burst=n1_size * m * self.N0 // 8) def output_compute(self, Oi_l1_zN, Pij_l1_zN, Vj_l1_zN, update_scale_ub, q_blk_h, kv_blk_idx, kv_blk_h): """Calculate output of flashattention2 :param Oi_l1_zN: with shape N1 x M x N0 :param Pij_l1_zN: :param Vj_l1_zN: :param update_scale_ub: :param q_blk_h: :param kv_blk_idx: :param kv_blk_h: :return: """ m = Pij_l1_zN.shape[1] with self.tik_instance.new_stmt_scope(disable_sync=False): Oij_ub_zN = self.tik_ops_utils.mmad_compute(Pij_l1_zN, Vj_l1_zN, q_blk_h, kv_blk_h, self.actual_d, N1MN0_to_MN=False, precision_type=FP32) with self.tik_instance.if_scope(kv_blk_idx == 0): self.cont_data_mv_1_bust(dst=Oi_l1_zN, src=Oij_ub_zN, burst=m * self.d // 8) with self.tik_instance.else_scope(): broadcast_scale_ub = self.tik_instance.Tensor(FP32, (1, m, self.N0), scope=UB, name="broadcast_scale_ub") src_scalar = self.tik_instance.Scalar(FP32, name="src_scalar") for idx in range(0, m): src_scalar.set_as(update_scale_ub[idx]) self.tik_instance.h_duplicate(broadcast_scale_ub[0, idx, :], src_scalar) single_loop_size = 8 # prevent ub overflow loop_times, tail_num = divmod(self.N1, single_loop_size) for idx in range(0, loop_times): # double buffer with self.tik_instance.for_range(0, 2, thread_num=2) as t_idx: single_thread_size = single_loop_size // 2 offset = idx * single_loop_size * m * self.N0 + t_idx * single_thread_size * m * self.N0 n1_begin = idx * single_loop_size + t_idx * single_thread_size self.output_update(Oi_l1_zN, Oij_ub_zN, broadcast_scale_ub, offset, n1_begin, single_thread_size, m) if tail_num > 0: offset = loop_times * single_loop_size * m * self.N0 thread_num = min(tail_num, 2) # thread1 n1_begin = loop_times * single_loop_size n1_size1 = tail_num // thread_num self.output_update(Oi_l1_zN, Oij_ub_zN, broadcast_scale_ub, offset, n1_begin, n1_size1, m) # thread2 n1_size2 = tail_num - n1_size1 if n1_size2 > 0: offset = offset + (tail_num // thread_num) * m * self.N0 n1_begin = loop_times * single_loop_size + tail_num // thread_num self.output_update(Oi_l1_zN, Oij_ub_zN, broadcast_scale_ub, offset, n1_begin, n1_size2, m) return Oi_l1_zN def output_norm(self, Oi_l1_zN, broadcast_li_rec_ub, qo_gm_offset, o_l1_offset, n1_size, m): """Normalize output of flashattention2 :param Oi_l1_zN: with shape N1 x M x N0 :param broadcast_li_rec_ub: :param qo_gm_offset: :param o_l1_offset: :param n1_size: :param m: :return: """ Oi_ub_zN = self.tik_instance.Tensor(FP32, (n1_size, m, self.N0), scope=UB, name="Oi_ub_zN") # mte1: L1 -> UB self.cont_data_mv_1_bust(dst=Oi_ub_zN, src=Oi_l1_zN[o_l1_offset], burst=n1_size * m * self.N0 // 8) # vector: mul and cast for i in range(n1_size): self.tik_instance.h_mul(Oi_ub_zN[i, :, :], Oi_ub_zN[i, :, :], broadcast_li_rec_ub) Oi_ub_zN_fp16 = self.tik_instance.Tensor(FP16, (n1_size, m, self.N0), scope=UB, name="Oi_ub_zN_fp16") self.tik_instance.h_cast(Oi_ub_zN_fp16, Oi_ub_zN, "none") # mte3: ub -> out self.unroll_data_move(dst=self.O_gm, dst_offset=qo_gm_offset, src=Oi_ub_zN_fp16, src_offset=0, nburst=n1_size, burst=m * self.N0 // 16, src_stride=0, dst_stride=(self.Nq - m) * self.N0 // 16, dst_offset_inc=self.Nq * self.N0, src_offset_inc=m * self.N0 ) def output_norm_and_write(self, Oi_l1_zN, li_ub, mi_ub, qo_gm_offset, l_gm_offset, q_blk_h): """Normalize output and write it back to gm :param Oi_l1_zN: with shape N1 x M x N0 :param li_ub: :param mi_ub: :param qo_gm_offset: :param l_gm_offset: :param q_blk_h: :return: """ m = Oi_l1_zN.shape[1] li_rec_ub = self.tik_ops_utils.calc_vec_rec(li_ub, m) with self.tik_instance.new_stmt_scope(disable_sync=False): broadcast_li_rec_ub = self.tik_instance.Tensor(FP32, (1, m, self.N0), scope=UB, name="broadcast_li_rec_ub") src_scalar = self.tik_instance.Scalar(FP32, name="src_scalar") for idx in range(0, m): src_scalar.set_as(li_rec_ub[idx]) self.tik_instance.h_duplicate(broadcast_li_rec_ub[0, idx, :], src_scalar) single_loop_size = 12 # prevent ub overflow loop_times, tail_num = divmod(self.N1, single_loop_size) for idx in range(0, loop_times): # double buffer with self.tik_instance.for_range(0, 2, thread_num=2) as t_idx: single_thread_size = single_loop_size // 2 cur_qo_gm_offset = qo_gm_offset + \ (idx * single_loop_size + t_idx * single_thread_size) * self.Nq * self.N0 o_l1_offset = (idx * single_loop_size + t_idx * single_thread_size) * m * self.N0 self.output_norm(Oi_l1_zN, broadcast_li_rec_ub, cur_qo_gm_offset, o_l1_offset, single_thread_size, m) if tail_num > 0: thread_num = min(tail_num, 2) # thread1 cur_qo_gm_offset1 = qo_gm_offset + loop_times * single_loop_size * self.Nq * self.N0 o_l1_offset1 = loop_times * single_loop_size * m * self.N0 n1_size1 = tail_num // thread_num self.output_norm(Oi_l1_zN, broadcast_li_rec_ub, cur_qo_gm_offset1, o_l1_offset1, n1_size1, m) # thread2 n1_size2 = tail_num - n1_size1 if n1_size2 > 0: cur_qo_gm_offset2 = cur_qo_gm_offset1 + n1_size1 * self.Nq * self.N0 o_l1_offset2 = o_l1_offset1 + n1_size1 * m * self.N0 self.output_norm(Oi_l1_zN, broadcast_li_rec_ub, cur_qo_gm_offset2, o_l1_offset2, n1_size2, m) # compute li mi_ub_fp32 = self.tik_instance.Tensor(FP32, (m,), scope=UB, name="mi_ub_fp32") self.tik_instance.h_cast(mi_ub_fp32, mi_ub, "none") self.tik_instance.h_ln(li_ub, li_ub) self.tik_instance.h_add(li_ub, li_ub, mi_ub_fp32) self.tik_ops_utils.move_vector_from_ub_to_gm(self.l_gm, li_ub, l_gm_offset, q_blk_h) def compute_in_each_kv_block(self, Qi_l1_zN, Oi_l1_zN, li_ub, mi_ub, batch_start, batch_idx, q_blk_idx, kv_blk_idx, q_blk_h, kv_blk_h, workspace=None): """The forward computation in each inner loop""" q_blk_h_aligned = self.tik_ops_utils.up_align_to_K0(q_blk_h) kv_blk_h_aligned = self.tik_ops_utils.up_align_to_K0(kv_blk_h) kv_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.N, self.d, self.Bc, kv_blk_idx) # load Vj (kv_blk_idx_th block of V_gm), then reorder for Pij*Vj Vj_l1_zN = self.tik_instance.Tensor(FP16, (kv_blk_h_aligned, self.d), name="Vj_l1_zN", scope=L1) self.unroll_data_move(dst=Vj_l1_zN, dst_offset=0, src=self.V_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=kv_blk_h_aligned * self.N0 // 16, src_stride=(self.N - kv_blk_h_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=kv_blk_h_aligned * self.N0, src_offset_inc=self.N * self.N0 ) if workspace is None: # load Kj (kv_blk_idx_th block of K_gm) KjT_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, kv_blk_h_aligned, self.N0), name="KjT_l1_K1MK0_ed", scope=L1) self.unroll_data_move(dst=KjT_l1_zN, dst_offset=0, src=self.K_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=kv_blk_h_aligned * self.N0 // 16, src_stride=(self.N - kv_blk_h_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=kv_blk_h_aligned * self.N0, src_offset_inc=self.N * self.N0 ) # Qi*Kj.T Sij_ub_zN = self.tik_ops_utils.matmul_compute(Qi_l1_zN, KjT_l1_zN, q_blk_h, self.actual_d, kv_blk_h, N1MN0_to_MN=False) else: Sij_ub_zN = workspace # Sij <- Sij + attn_mask if self.has_attn_mask: if self.only_causal: with self.tik_instance.if_scope(q_blk_idx == kv_blk_idx): # main diagonal self.do_att_mask(Sij_ub_zN, 0, q_blk_h, kv_blk_h, q_blk_h_aligned, kv_blk_h_aligned, self.only_causal) else: attn_mask_gm_offset = self.get_attn_mask_gm_offset(batch_start, batch_idx, self.Nq, self.N, self.Br, q_blk_idx, self.Bc, kv_blk_idx) self.do_att_mask(Sij_ub_zN, attn_mask_gm_offset, q_blk_h, kv_blk_h, q_blk_h_aligned, kv_blk_h_aligned) # Pij = softmax(Sij), Pij <- dropout(Pij) drop_mask_gm_offset = None if self.has_drop_mask: drop_mask_gm_offset = self.get_drop_mask_gm_offset(batch_start, batch_idx, self.Nq, self.N, self.Br, q_blk_idx, self.Bc, kv_blk_idx) Pij_l1_zN, li_ub, mi_ub, update_scale_ub = self.softmax_compute(Sij_ub_zN, li_ub, mi_ub, kv_blk_idx, drop_mask_gm_offset) # do Sij = matmul(Qi, Kj.T) for next inner loop with self.tik_instance.if_scope(kv_blk_idx < self.Tc - 2): kv_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.N, self.d, self.Bc, kv_blk_idx + 1) KjT_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, self.Bc, self.N0), name="KjT_l1_zN", scope=L1) self.unroll_data_move(dst=KjT_l1_zN, dst_offset=0, src=self.K_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=self.Bc * self.N0 // 16, src_stride=(self.N - self.Bc) * self.N0 // 16, dst_stride=0, dst_offset_inc=self.Bc * self.N0, src_offset_inc=self.N * self.N0 ) Sij_ub_zN = self.tik_ops_utils.matmul_compute(Qi_l1_zN, KjT_l1_zN, q_blk_h, self.actual_d, self.Bc, N1MN0_to_MN=False, workspace=Sij_ub_zN) Oi_l1_zN = self.output_compute(Oi_l1_zN, Pij_l1_zN, Vj_l1_zN, update_scale_ub, q_blk_h, kv_blk_idx, kv_blk_h) return Oi_l1_zN, li_ub, mi_ub def compute_in_each_q_block(self, batch_start, batch_idx, q_blk_idx, q_blk_h): """The forward computation in each outer loop""" q_blk_h_aligned = self.tik_ops_utils.up_align_to_K0(q_blk_h) # initialize Oi_l1, li_ub, mi_ub Oi_l1_zN = self.tik_instance.Tensor(FP32, (self.d // self.N0, q_blk_h_aligned, self.N0), scope=L1, name="Oi_l1_zN") li_ub = self.tik_instance.Tensor(FP32, (q_blk_h_aligned,), scope=UB, name="li_ub") mi_ub = self.tik_instance.Tensor(FP16, (q_blk_h_aligned,), scope=UB, name="mi_ub") # load Qi (q_blk_idx_th block of Q_gm) qo_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.Nq, self.d, self.Br, q_blk_idx) Qi_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_h_aligned, self.N0), scope=L1, name="Qi_l1_zN") self.unroll_data_move(dst=Qi_l1_zN, dst_offset=0, src=self.Q_gm, src_offset=qo_gm_offset, nburst=self.N1, burst=q_blk_h_aligned * self.N0 // 16, src_stride=(self.Nq - q_blk_h_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=q_blk_h_aligned * self.N0, src_offset_inc=self.Nq * self.N0 ) with self.tik_instance.new_stmt_scope(disable_sync=False): # load K0 (0_th block of K_gm) kv_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.N, self.d, self.Bc, 0) # zN等价于转置后的K1NK0 KjT_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, self.Bc, self.N0), name="KjT_l1_zN", scope=L1) self.unroll_data_move(dst=KjT_l1_zN, dst_offset=0, src=self.K_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=self.Bc * self.N0 // 16, src_stride=(self.N - self.Bc) * self.N0 // 16, dst_stride=0, dst_offset_inc=self.Bc * self.N0, src_offset_inc=self.N * self.N0 ) # do Sij = matmul(Qi, Kj.T) for first inner loop Sij_ub_zN = self.tik_instance.Tensor(FP16, (self.Bc // 16, q_blk_h_aligned, 16), scope=UB, name="Sij_ub_zN") Sij_ub_zN = self.tik_ops_utils.matmul_compute(Qi_l1_zN, KjT_l1_zN, q_blk_h, self.actual_d, self.Bc, N1MN0_to_MN=False, workspace=Sij_ub_zN) with self.tik_instance.for_range(0, self.Tc - 1, name="kv_blk_idx") as kv_blk_idx: with self.tik_instance.if_scope(tik.all(kv_blk_idx - self.next_block_num <= q_blk_idx, q_blk_idx <= kv_blk_idx + self.prev_block_num)): Oi_l1_zN, li_ub, mi_ub = self.compute_in_each_kv_block(Qi_l1_zN, Oi_l1_zN, li_ub, mi_ub, batch_start, batch_idx, q_blk_idx, kv_blk_idx, q_blk_h, self.Bc, workspace=Sij_ub_zN) # last kv blk with self.tik_instance.if_scope(tik.all(self.Tc - 1 - self.next_block_num <= q_blk_idx, q_blk_idx <= self.Tc - 1 + self.prev_block_num)): Oi_l1_zN, li_ub, mi_ub = self.compute_in_each_kv_block(Qi_l1_zN, Oi_l1_zN, li_ub, mi_ub, batch_start, batch_idx, q_blk_idx, self.Tc - 1, q_blk_h, self.last_Bc) # normalize output and write it to gm l_gm_offset = self.get_l_gm_offset(batch_start, batch_idx, self.Nq, self.Br, q_blk_idx) self.output_norm_and_write(Oi_l1_zN, li_ub, mi_ub, qo_gm_offset, l_gm_offset, q_blk_h) def compute_one_core(self, batch_start_sc, batch_num_sc, core_idx_to_tr_info, core_idx): """The computation of FlashAttention forward on each core""" with self.tik_instance.for_range(0, batch_num_sc, name="batch_index") as batch_idx: tr_start_s = self.tik_instance.Scalar("int32", init_value=core_idx_to_tr_info[ core_idx, batch_start_sc + batch_idx, 0], name="tr_start_s") tr_end_s = self.tik_instance.Scalar("int32", init_value=core_idx_to_tr_info[core_idx, batch_start_sc + batch_idx, 1], name="tr_end_s") with self.tik_instance.for_range(tr_start_s, tr_end_s - 1, name="q_blk_idx") as q_blk_idx: self.compute_in_each_q_block(batch_start_sc, batch_idx, q_blk_idx, self.Br) with self.tik_instance.if_scope(tr_end_s - 1 != self.Tr - 1): self.compute_in_each_q_block(batch_start_sc, batch_idx, tr_end_s - 1, self.Br) with self.tik_instance.else_scope(): self.compute_in_each_q_block(batch_start_sc, batch_idx, self.Tr - 1, self.last_Br) def collect_inputs(self): """collect all input gm tensors into input_gm_list, the input list should keep order with the para order in Primitive and init """ input_gm_list = [self.Q_gm, self.K_gm, self.V_gm] if self.has_attn_mask: input_gm_list.append(self.att_mask_gm) if self.has_drop_mask: input_gm_list.append(self.drop_mask_gm) if self.has_alibi_mask: input_gm_list.append(self.alibi_mask_gm) return input_gm_list def collect_outputs(self): """collect all output gm tensors into output_gm_list, the output list should keep order with the para order in Primitive and init """ output_gm_list = [self.O_gm, self.l_gm] return output_gm_list def flash_attention(query, key, value, attn_mask, dropout_mask, alibi_mask, output, rowsum, prev_block_num=65536, next_block_num=65536, tiling_stgy_name='sparse', kernel_name="flash_attention", disable_debug=True): """ algorithm: flash_attention_backward Parameters ---------- query : dict. shape and dtype of input, only support float16 key : dict. shape and dtype of input, only support float16 value: dict. shape and dtype of input, only support float16 attn_mask: dict. shape and dtype of input, only support float16 dropout_mask: dict. shape and dtype of input, only support float16 alibi_mask: dict. shape and dtype of input, only support float16 output: dict. shape and dtype of output, only support float16 rowsum: dict. shape and dtype of output, only support float16 prev_block_num: int. an attribute used to define sparse attention next_block_num: int. an attribute used to define sparse attention tiling_stgy_name: str. an attribute used to choose the tiling strategy kernel_name: str. cce kernel name, default value is real_div disable_debug: bool. whether disable debug Returns ------- tik_instance """ fa = FlashAttentionFwd(q=query, k=key, v=value, attn_mask=attn_mask, dropout_mask=dropout_mask, alibi_mask=alibi_mask, kernel_name=kernel_name, tiling_stgy=TilingStrategy.from_strategy_name(tiling_stgy_name), prev_block_num=prev_block_num, next_block_num=next_block_num, disable_debug=disable_debug) fa.compute_process() return fa.tik_instance pengcheng.mind-test/flash_attention/flash_attention_impl.py000066400000000000000000000152441471400134000246560ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The impl of flash attention""" from __future__ import absolute_import import sys import os import mindspore.ops as ops import mindspore.common.dtype as mstype from mindspore.ops import Custom from mindspore.ops import DataType from mindspore.ops import TBERegOp from mindspore.ops.composite.multitype_ops.zeros_like_impl import zeros_like sys.path.append(os.path.dirname(__file__)) from flash_attention_bwd import flash_attention_grad from flash_attention_fwd import flash_attention KERNEL_NAME = "flash_attention" cus_flash_atten_op_info = TBERegOp("FlashAttentionPrimitive") \ .fusion_type("OPAQUE") \ .partial_flag(True) \ .async_flag(False) \ .binfile_name("flash_attention.so") \ .compute_cost(10) \ .kernel_name(KERNEL_NAME) \ .attr("prev_block_num", "required", "int", "all", "65536") \ .attr("next_block_num", "required", "int", "all", "65536") \ .attr("tiling_stgy_name", "required", "str", "all", "sparse") \ .input(0, "query", False, "required", "all") \ .input(1, "key", False, "required", "all") \ .input(2, "value", False, "required", "all") \ .input(3, "attn_mask", False, "optional", "all") \ .input(4, "dropout_mask", False, "optional", "all") \ .input(5, "alibi_mask", False, "optional", "all") \ .output(0, "output", False, "required", "all") \ .output(1, "rowsum", False, "required", "all") \ .dtype_format(DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_Default, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F32_Default,) \ .get_op_info() GRAD_KERNEL_NAME = "flash_attention_grad" cus_flash_atten_grad_op_info = TBERegOp("FlashAttentionGradPrimitive") \ .fusion_type("OPAQUE") \ .partial_flag(True) \ .async_flag(False) \ .binfile_name("flash_attention_grad.so") \ .compute_cost(10) \ .kernel_name(GRAD_KERNEL_NAME) \ .attr("prev_block_num", "required", "int", "all", "65536") \ .attr("next_block_num", "required", "int", "all", "65536") \ .attr("tiling_stgy_name", "required", "str", "all", "sparse") \ .input(0, "query", False, "required", "all") \ .input(1, "key", False, "required", "all") \ .input(2, "value", False, "required", "all") \ .input(3, "output", False, "required", "all") \ .input(4, "do", False, "required", "all") \ .input(5, "rowsum", False, "required", "all") \ .input(7, "attn_mask", False, "optional", "all") \ .input(8, "dropout_mask", False, "optional", "all") \ .input(9, "alibi_mask", False, "optional", "all") \ .output(0, "dq", False, "required", "all") \ .output(1, "dk", False, "required", "all") \ .output(2, "dv", False, "required", "all") \ .dtype_format(DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F32_Default, DataType.F16_FracNZ, DataType.F16_Default, DataType.F16_FracNZ, DataType.F32_FracNZ, DataType.F32_FracNZ, DataType.F32_FracNZ) \ .get_op_info() def get_flash_attention_grad(prev_block_num=65536, next_block_num=65536, tiling_stgy_name='sparse'): """get flash attention grad""" def infer_shape(q_shape, k_shape, v_shape, o_shape, do_shape, l_shape, att_mask_shape, dropout_mask_shape, alibi_mask_shape): return q_shape, k_shape, v_shape def infer_dtype(q_dtype, k_dtype, v_dtype, o_dytpe, do_dtype, l_dtype, attn_mask_dtype, dropout_mask_dtype, alibi_mask_type): return mstype.float32, mstype.float32, mstype.float32 fa_grad = Custom(flash_attention_grad, out_shape=infer_shape, out_dtype=infer_dtype, func_type="tbe", reg_info=cus_flash_atten_grad_op_info) fa_grad.add_prim_attr("prev_block_num", prev_block_num) fa_grad.add_prim_attr("next_block_num", next_block_num) fa_grad.add_prim_attr("tiling_stgy_name", tiling_stgy_name) fa_grad.init_prim_io_names( inputs=["query", "key", "value", "output", "do", "rowsum", "attn_mask", "dropout_mask", "alibi_mask"], outputs=["dq", "dk", "dv"] ) def bprop(query, key, value, attn_mask, dropout_mask, alibi_mask, outs, douts): output, rowsum, = outs dout, _, = douts dq, dk, dv = fa_grad(query, key, value, output, dout, rowsum, attn_mask, dropout_mask, alibi_mask) dq = ops.cast(dq, mstype.float16) dk = ops.cast(dk, mstype.float16) dv = ops.cast(dv, mstype.float16) return dq, dk, dv, zeros_like(attn_mask), zeros_like(dropout_mask), zeros_like(alibi_mask) return bprop def get_flash_attention(prev_block_num=65536, next_block_num=65536, tiling_stgy_name='sparse'): """get_flash_attention""" def infer_shape(q_shape, k_shape, v_shape, attn_mask_shape=None, dropout_mask_shape=None, alibi_mask_shape=None): """infer shape""" batch, head_num, seq_len, _ = q_shape l_shape = (batch, head_num, seq_len) return q_shape, l_shape def infer_dtype(q_dtype, k_dtype, v_dtype, attn_mask_dtype=None, dropout_mask_dtype=None, alibi_mask_type=None): """infer type""" return q_dtype, mstype.float32 fa_grad = get_flash_attention_grad(prev_block_num, next_block_num, tiling_stgy_name) fa_forward = Custom(flash_attention, out_shape=infer_shape, out_dtype=infer_dtype, func_type="tbe", bprop=fa_grad, reg_info=cus_flash_atten_op_info) fa_forward.add_prim_attr("prev_block_num", prev_block_num) fa_forward.add_prim_attr("next_block_num", next_block_num) fa_forward.add_prim_attr("tiling_stgy_name", tiling_stgy_name) fa_forward.init_prim_io_names( inputs=["query", "key", "value", "attn_mask", "dropout_mask", "alibi_mask"], outputs=["output", "rowsum"] ) return fa_forward pengcheng.mind-test/flash_attention/nn/000077500000000000000000000000001471400134000205065ustar00rootroot00000000000000pengcheng.mind-test/flash_attention/nn/layer/000077500000000000000000000000001471400134000216225ustar00rootroot00000000000000pengcheng.mind-test/flash_attention/nn/layer/flash_attention.py000066400000000000000000000212711471400134000253610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ A FlashAttention Layer. """ import math import mindspore.common.dtype as mstype from mindspore import ops from mindspore.common.tensor import Tensor from mindspore.nn.cell import Cell __all__ = ['FlashAttention'] HEAD_DIM_MAX_LIMIT = 256 from acctransformer.flash_attention.ops.flash_attention.flash_attention_impl import get_flash_attention class FlashAttention(Cell): """Flash Attention Layer. This function contains the flash attention primitives used in FlashAttention (see paper) `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ` Specifically, it includes the following: 1. An interface for calling flashattention operation. 2. Two configuration parameters for enabling local block sparse of flashattention. Args: head_dim(int): The hidden size of input. dropout_rate(float): The dropout rate of the attention score. Default 0.0. prev_block_num(int): A integer to define the number of blocks to look ahead for local block sparse attention. Default 65536. next_block_num(int): A integer to define the number of blocks to look behind for local block sparse attention. Default 65536. tiling_stgy_name(str): A str to define tiling strategy of flash attention. dp(int): data parallel. Default 1. mp(int): model parallel. Default 1. have_attention_mask_batch(bool): indicates whether attention_mask contains the batch dimension. Default True alibi(bool): This parameter indicates whether the flashattention supports the Alibi. Default: False Inputs: - **query** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **key** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **value** (Tensor) - Tensor value (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp16` [batch_size, seq_length, seq_length]): A matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, head_num, seq_length, head_dim] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from acctransformer.flash_attention.nn.layer.flash_attention import FlashAttention >>> from mindspore import Tensor >>> model = FlashAttention(head_dim=128, ... dropout_rate=0.1, ... prev_block_num=7, ... next_block_num=0 ... ) >>> query = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> key = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> value = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> attention_mask = Tensor(np.triu(np.ones((1, 128, 128)), k=1), mstype.float16) >>> output = model(query, key, value, attention_mask) >>> print(output.shape) (2, 16, 4096, 128) """ def __init__(self, head_dim, dropout_rate=0.0, prev_block_num=65536, next_block_num=65536, tiling_stgy_name="sparse", dp=1, mp=1, have_attention_mask_batch=True, alibi=False ): super(FlashAttention, self).__init__() if head_dim > HEAD_DIM_MAX_LIMIT: raise ValueError("head_dim too large, reduce head_dim and try again.") if alibi: raise ValueError("alibi not supported in the current version.") self.flash_attention = get_flash_attention( prev_block_num=prev_block_num, next_block_num=next_block_num, tiling_stgy_name=tiling_stgy_name ) self.flash_attention.add_prim_attr("primitive_target", "Ascend") self.scale_factor = Tensor([1. / math.sqrt(math.sqrt(head_dim))], dtype=mstype.float16) self.scale_mul = ops.Mul().shard(((dp, mp, 1, 1), (1,))) self.dropout_rate = dropout_rate self.enable_dropout = self.dropout_rate > 0.0 self.have_attention_mask_batch = have_attention_mask_batch self.alibi = alibi if self.enable_dropout: self.keep_prob = Tensor(1 - self.dropout_rate, dtype=mstype.float16) self.fill_v2 = ops.FillV2().shard(((dp, mp, 1, 1), ())) self.tensor_one = Tensor(1.0, mstype.float16) self.drop_gen_mask = ops.DropoutGenMask() self.do_dropout = ops.DropoutDoMask().shard(((dp, mp, 1, 1),)) self.depend = ops.Depend() def shard(self, in_strategy=None, out_strategy=None): """Distributed configuration of FlashAttention :param in_strategy: Describe the split strategy of operator input. Default: None. :param out_strategy: Describe the split strategy of operator output, it is only for certain operators, such as MatMul. Default: None. :return: """ if in_strategy is None: # default: dp=1, mp=1, construct inputs only contain q, k, v in_strategy = ( (1, 1, 1, 1), (1, 1, 1, 1), (1, 1, 1, 1), ) self.flash_attention.shard(in_strategy) dp = in_strategy[0][0] mp = in_strategy[0][1] self.flash_attention.add_prim_attr("dev_matrix_shape", [dp, mp, 1, 1]) inputs_tensor_map = [ [3, 2, 1, 0], [3, 2, 1, 0], [3, 2, 1, 0], ] if self.have_attention_mask_batch: inputs_tensor_map.append([3, 1, 0]) else: inputs_tensor_map.append([-1, 1, 0]) input_empty_args_num = 2 # dropout_mask if self.enable_dropout: input_empty_args_num -= 1 inputs_tensor_map.append([3, 2, 1, 0]) if self.alibi: input_empty_args_num -= 1 inputs_tensor_map.append([3, 2, 1, 0]) self.flash_attention.add_prim_attr("inputs_tensor_map", inputs_tensor_map) self.flash_attention.add_prim_attr("outputs_tensor_map", [ [3, 2, 1, 0], # O [3, 2, 1], # L ]) self.flash_attention.add_prim_attr("as_loss_divisor", 0) self.flash_attention.add_prim_attr("empty_mirror_ops", input_empty_args_num) def construct(self, q, k, v, attn_mask=None, alibi_mask=None): """FlashAttention forward :param q: [bsz, head_num, seq_len, head_dim] :param k: [bsz, head_num, seq_len, head_dim] :param v: [bsz, head_num, seq_len, head_dim] :param attn_mask: [1 or bsz, seq_len, seq_len], if not None :param alibi_mask: [bsz, head_num, 1, seq_len], if not None :return: o [bsz, head_num, seq_len, head_dim] """ q = self.scale_mul(q, self.scale_factor) k = self.scale_mul(k, self.scale_factor) bsz, head_num, seq_len, head_dim = q.shape _, k_head_num, _, _ = k.shape _, v_head_num, _, _ = v.shape if head_num != k_head_num or head_num != v_head_num: raise ValueError( "the head_num of query, key and value must be the same, " "If different head_num are used, users need to change themselves to be same by tile.") if self.enable_dropout: drop_mask_bits = self.drop_gen_mask((bsz, head_num, seq_len, seq_len), self.keep_prob) tensor_shape = Tensor((bsz, head_num, seq_len, seq_len), mstype.int32) ones = self.fill_v2(tensor_shape, self.tensor_one) ones = self.depend(ones, q) drop_mask = self.do_dropout(ones, drop_mask_bits, self.keep_prob) else: drop_mask = None o, _ = self.flash_attention(q, k, v, attn_mask, drop_mask, alibi_mask) return o pengcheng.mind-test/flash_attention/tik_ops_utils.py000066400000000000000000000633401471400134000233430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """the common about tik ops""" from functools import partial from tbe import tik from constants import DTYPE_SIZE from constants import FP16 from constants import FP32 from constants import L0A from constants import L0B from constants import L0C from constants import UB class TikOpsUtils: """Utils function class about tik ops""" def __init__(self, tik_instance): self.tik_instance = tik_instance self.dtype = "float16" self.cont_data_mv_1_bust = partial(self.tik_instance.data_move, sid=0, nburst=1, src_stride=0, dst_stride=0) def MK_TO_K1MK0(self, mk_input_tensor, workspace_tensor=None): """change data shape from (M, K) to (K1, M, K0), K1 = K // K0, the effect is equant to: new_tensor = np.stack(np.hsplit(mk_input_tensor, K1), axis=0) :param mk_input_tensor: input tensor in GM with shape: (M, K) :param workspace_tensor: workspace tensor with shape: (K1, M, K0) tensor will be changed, otherwise the new data will be copied to the workspace tensor, and input tensor will stay unchanged. :return: Tensor with shape (K1,M, K0) """ dtype = mk_input_tensor.dtype m, k = mk_input_tensor.shape K0 = 16 K1 = k // K0 M = self.up_align_to_K0(m) if workspace_tensor is not None: with self.tik_instance.for_range(0, K1) as i: self.tik_instance.data_move( workspace_tensor[i * M * K0:], mk_input_tensor[i * K0:], 0, M, K0 * DTYPE_SIZE[dtype] // 32, (K1 - 1) * K0 * DTYPE_SIZE[dtype] // 32, 0, ) return workspace_tensor.reshape((K1, M, K0)) with self.tik_instance.new_stmt_scope(disable_sync=False): tmp_ub = self.tik_instance.Tensor(dtype, (K1, M, K0), name="tmp_ub", scope=UB) # data_move(m,k) --> (k1,m,K0) with self.tik_instance.for_range(0, K1) as i: self.tik_instance.data_move( tmp_ub[i * M * K0:], mk_input_tensor[i * K0:], 0, M, K0 * DTYPE_SIZE[dtype] // 32, (K1 - 1) * K0 * DTYPE_SIZE[dtype] // 32, 0, ) self.cont_data_mv_1_bust( dst=mk_input_tensor, src=tmp_ub, burst=K1 * M * K0 * DTYPE_SIZE[dtype] // 32) return mk_input_tensor.reshape((K1, M, K0)) def transpose_matrix(self, src_ub, dst_ub, N, nk0=False): """ transpose matrix, default support shape: (16, n) -> (n, 16) if nk0 is true, support shape: (n, 16) -> (16, n) """ K0 = 16 rep_times = N // K0 if nk0: src_list = [src_ub[16 * i] for i in range(16)] dst_list = [dst_ub[N * i] for i in range(16)] else: src_list = [src_ub[N * i] for i in range(16)] dst_list = [dst_ub[16 * i] for i in range(16)] dst_rep_stride = K0 src_rep_stride = 1 if rep_times == 1: dst_rep_stride = 0 src_rep_stride = 0 if nk0: src_rep_stride, dst_rep_stride = dst_rep_stride, src_rep_stride self.tik_instance.vec_trans_scatter( False, False, dst_list, src_list, rep_times, dst_rep_stride, src_rep_stride ) return dst_ub def KN_TO_K1NK0(self, kn_input_tensor, workspace_tensor=None): """change data shape from (K,N) to (K1, N, K0), K1 = K // K0, the effect is equvilent to: new_tensor = np.reshape(kn_input_tensor, newshape=(K1, K0, N)).swapaxes(1, 2) :param kn_input_tensor: input tensor with shape: (K, N) :param workspace_tensor: workspace tensor with shape: (K1, N, K0) tensor will be changed, otherwise the new data will be copied to the workspace tensor, and input tensor will stay unchanged. :return: Tensor with shape: (K1, N, K0) """ dtype = kn_input_tensor.dtype k, n = kn_input_tensor.shape K0 = 16 K1 = k // K0 N = n with self.tik_instance.for_range(0, K1) as index: k1nk0_ub = self.tik_instance.Tensor(dtype, (N, K0), UB, "k1nk0_ub") src_ub = self.tik_instance.Tensor(dtype, (K0, N), UB, "src_ub") burst_len = K0 * N * DTYPE_SIZE[dtype] // 32 self.cont_data_mv_1_bust(dst=src_ub, src=kn_input_tensor[index * K0 * N], burst=burst_len) k1nk0_ub = self.transpose_matrix(src_ub, k1nk0_ub, N) if workspace_tensor is None: self.cont_data_mv_1_bust(dst=kn_input_tensor[index * K0 * N], src=k1nk0_ub, burst=burst_len) else: self.cont_data_mv_1_bust(dst=workspace_tensor[index * K0 * N], src=k1nk0_ub, burst=burst_len) if workspace_tensor is None: return kn_input_tensor.reshape((K1, N, K0)) return workspace_tensor.reshape((K1, N, K0)) def N1MN0_TO_MN(self, N1MN0_input): """change data shape from (N1, M, N0) to (M, N), N0=16, N = N1 * K0, the effect is equant to: N1MN0_input = np.concatenate(list(map(np.squeeze, np.split(N1MN0_input, N1))), axis=1) :param N1MN0_input: input tensor with shape (N, M, N0) in GM or L1. :return: """ dtype = N1MN0_input.dtype N1, M, N0 = N1MN0_input.shape with self.tik_instance.new_stmt_scope(disable_sync=False): tmp_ub = self.tik_instance.Tensor(dtype, (M, N1 * N0), name="tmp_ub", scope=UB) # data_move (n1,m,n0) --> (m,n) with self.tik_instance.for_range(0, N1) as i: self.tik_instance.data_move( tmp_ub[i * N0:], N1MN0_input[i * M * N0:], 0, M, N0 * DTYPE_SIZE[dtype] // 32, 0, (N1 - 1) * N0 * DTYPE_SIZE[dtype] // 32, ) # data_move out self.cont_data_mv_1_bust(dst=N1MN0_input, src=tmp_ub, burst=M * N1 * N0 * DTYPE_SIZE[ dtype] // 32) return N1MN0_input.reshape((M, N1 * N0)) def broadcast_vec_from_M_to_MN0(self, vec_ub): """broadcast a vec from (M,) to (M, 16)""" M = vec_ub.shape[0] dst_ub = self.tik_instance.Tensor(FP16, (M, 16), name="dst_ub", scope=UB) with self.tik_instance.new_stmt_scope(disable_sync=False): broadcast_l0c = self.tik_instance.Tensor(FP16, (16, M), name="broadcast_l0c", scope=L0C) self.tik_instance.broadcast_ub_to_l0c(broadcast_l0c, vec_ub, 1, M // 16, 1, 1) broadcast_ub = self.tik_instance.Tensor(FP16, (16, M), name="broadcast_ub", scope=UB) self.cont_data_mv_1_bust(dst=broadcast_ub, src=broadcast_l0c, burst=M * 16 * 2 // 512) # (M // N0, 16, N0) -> (M, 16), N0 = 16 self.tik_instance.vec_trans(dst_ub, broadcast_ub, M // 16, 1, 1) return dst_ub def broadcast(self, vec_ub, shape): """ broadcast a vector to a matrix :param vec_ub: a tensor in UB with shape of (M,), and dtype is float16 :param shape: the target shape, a tuple with value (M, N), M and N are integer multiples of 16 :return: a tensor in UB with shape of (M, N) """ M, N = shape dst_ub = self.tik_instance.Tensor(FP16, shape, name="dst_ub", scope=UB) with self.tik_instance.new_stmt_scope(disable_sync=False): M_16_ub = self.broadcast_vec_from_M_to_MN0(vec_ub) # (M, 16) -> (M, 32) -> (M, 64) -> ... -> (M, N) self.tik_instance.data_move(dst_ub, M_16_ub, 0, M, 1, 0, N // 16 - 1) times = self.tik_instance.Scalar("int32", name="times", init_value=1) with self.tik_instance.for_range(begint=0, endt=N) as idx: offset = times * 16 with self.tik_instance.if_scope(offset * 2 <= N): burst = offset // 16 src_stride = N // 16 - burst dst_stride = N // 16 - burst self.tik_instance.data_move(dst_ub[offset], dst_ub, 0, M, burst, src_stride, dst_stride) with self.tik_instance.else_scope(): burst = (N - offset) // 16 src_stride = N // 16 - burst dst_stride = N // 16 - burst with self.tik_instance.if_scope(burst > 0): self.tik_instance.data_move(dst_ub[offset], dst_ub, 0, M, burst, src_stride, dst_stride) self.tik_instance.tik_break() times.set_as(times * 2) return dst_ub def broadcast_row(self, vec_ub, shape): """broadcast row""" M, N = shape dst_ub = self.tik_instance.Tensor(FP16, shape, name="dst_ub", scope=UB) self.tik_instance.data_move(dst_ub, vec_ub, 0, 1, N // 16, 0, 0) times = self.tik_instance.Scalar("int32", name="times", init_value=1) # (1, N) -> (2, M) -> (4, N) -> ... -> (M, N) with self.tik_instance.for_range(begint=0, endt=M): with self.tik_instance.if_scope(times * 2 <= M): burst = times * N // 16 offset = times * N self.tik_instance.data_move(dst_ub[offset], dst_ub, 0, 1, burst, 0, 0) with self.tik_instance.else_scope(): burst = (M - times) * N // 16 offset = times * N with self.tik_instance.if_scope(burst > 0): self.tik_instance.data_move(dst_ub[offset], dst_ub, 0, 1, burst, 0, 0) self.tik_instance.tik_break() times.set_as(times * 2) return dst_ub def get_K0(self, dtype=None): """get K0""" if dtype is None: dtype = self.dtype return 32 // DTYPE_SIZE[dtype] def up_align_to_K0(self, n, dtype=None): """byte alignment by dtype""" if dtype is None: dtype = self.dtype K0 = 32 // DTYPE_SIZE[dtype] return (n + K0 - 1) // K0 * K0 def calc_vec_rec(self, vec_ub, vec_len): """cal the reciprocal of a vector""" dtype = vec_ub.dtype vec_len_aligned = self.up_align_to_K0(vec_len) vec_rec_ub = self.tik_instance.Tensor(dtype, (vec_len_aligned,), scope=UB, name="li_new_rec_ub") mask_len = 256 // DTYPE_SIZE[dtype] block_len = 32 // DTYPE_SIZE[dtype] work_size = 8 // DTYPE_SIZE[dtype] with self.tik_instance.new_stmt_scope(disable_sync=False): repeat_times = vec_len // mask_len if repeat_times > 0: dst_rep_stride = 8 src_rep_stride = 8 src_extent_size = (repeat_times - 1) * src_rep_stride * block_len + mask_len wk_size_unit = ((src_extent_size + block_len - 1) // block_len) * block_len wk_size = work_size * wk_size_unit # 定义work_tensor work_tensor_ub = self.tik_instance.Tensor( "float32", (wk_size,), name="work_tensor_ub", scope=UB ) # 如果work_tensor有索引，需要写成 work_tensor[index:] self.tik_instance.vec_rec_high_preci( mask_len, vec_rec_ub[0:], vec_ub[0:], work_tensor_ub[0:], repeat_times, dst_rep_stride, src_rep_stride, ) mask_len = vec_len - repeat_times * mask_len if mask_len > 0: wk_size = work_size * ((mask_len + block_len - 1) // block_len) * block_len work_tensor_ub2 = self.tik_instance.Tensor( "float32", (wk_size,), name="work_tensor_ub2", scope=UB ) self.tik_instance.vec_rec_high_preci( mask_len, vec_rec_ub[repeat_times * mask_len:], vec_ub[repeat_times * mask_len:], work_tensor_ub2[0:], 1, 0, 0, ) return vec_rec_ub def row_sum_cube_impl(self, matrix_l1_K1MK0_ed, right_all_one_matrix_l1, rowsum_ub, m, k, precision_type): """用cube实现矩阵行和：右乘一个shape=(n,1)全一矩阵 :param matrix_l1_K1MK0_ed: input tensor with shape (K1, M, K0) :param right_all_one_matrix_l1: input tensor with shape (K, 16) :param rowsum_ub: output tensor stores the row sum of input tensor :param m: actual tensor height :param k: actual tensor width :return: row sum of the output tensor """ K1, M, K0 = matrix_l1_K1MK0_ed.shape # 调用matmul实现rowsum，结果shape=(m, 16)，取每行的第一个数 with self.tik_instance.new_stmt_scope(disable_sync=False): row_sum_ub_N1MN0 = self.matmul_compute(matrix_l1_K1MK0_ed, right_all_one_matrix_l1, m, k, 16, N1MN0_to_MN=False, precision_type=precision_type) row_sum_ub_MN_ed = row_sum_ub_N1MN0.reshape((M, 16)) if precision_type == FP32: for idx in range(0, m): cur_row_sum = self.tik_instance.Scalar(FP32, init_value=row_sum_ub_MN_ed[idx, 0]) rowsum_ub[idx].set_as(cur_row_sum) else: # row_sum_ub_MN_ed 先转置，然后取一行, 替换原来按行操作: lij_ub[i].set_as(row_sum_ub_MN_ed[i, 0]) row_sum_ub_trans = self.tik_instance.Tensor(FP16, (16, M), name="row_sum_ub_trans", scope=UB) row_sum_ub_trans = self.transpose_matrix(row_sum_ub_MN_ed, row_sum_ub_trans, M, True) self.cont_data_mv_1_bust(dst=rowsum_ub, src=row_sum_ub_trans, burst=M // 16) return rowsum_ub def matmul_compute(self, A_l1, B_l1, m, k, n, N1MN0_to_MN=True, precision_type=FP16, workspace=None): """calculate matrix multiplication A_l1 * B_l1, and move the result to C_ub, then rearrange C_ub :param A_l1: input tensor in L1 with shape of (K1, M, K0) :param B_l1: input tensor in L1 with shape of (K1, N, K0) :param m: the actual number of rows of A_l1 :param k: the actual number of cols of A_l1 :param n: the actual number of cols of B_l1 :param N1MN0_to_MN: Whether reorder the result tensor. :return: C_ub with tensor with shape of (M, N) if N1MN0_to_MN else (N1, M, N0) """ M = self.up_align_to_K0(m) N = self.up_align_to_K0(n) # 根据精度类型，决定matmul返回的ub精度类型，如果fp16, tensor_move做随路精度转换 if workspace is None: C_ub = self.tik_instance.Tensor(precision_type, (N // 16, M, 16), name="C_ub", scope=UB) else: C_ub = workspace with self.tik_instance.new_stmt_scope(disable_sync=False): # matmul C_l0c = self.tik_instance.Tensor( FP32, (N // 16, M, 16), scope=L0C, name="C_l0c" ) # n1mn0 (n0=16) self.tik_instance.matmul(C_l0c, A_l1, B_l1, m, k, n) # L0C -> ub, fp32 -> fp16 (tensor_mov可做随路转换) self.tik_instance.tensor_mov(C_ub, C_l0c, "m", 1, M * N * DTYPE_SIZE[FP32] // 1024, 0, 0) if N1MN0_to_MN: return self.N1MN0_TO_MN(C_ub) else: return C_ub def mmad_compute_transpose(self, A_l1, B_l1, m, k, n, a_transpose=False, b_transpose=False, N1MN0_to_MN=False, precision_type=FP16): """calculate matrix multiplication A_l1 * B_l1, and move the result to C_ub, then rearrange C_ub :param A_l1: input tensor in L1 zN format :param B_l1: input tensor in L1 zN format :param m: the actual number of rows of A_l1 :param k: the actual number of cols of A_l1 :param n: the actual number of cols of B_l1 :param a_transpose: A Whether to transpose :param b_transpose: B Whether to transpose :param N1MN0_to_MN: Whether reorder the result tensor. :return: C_ub with tensor with shape of (M, N) if N1MN0_to_MN else (N1, M, N0) """ M = self.up_align_to_K0(m) N = self.up_align_to_K0(n) K = self.up_align_to_K0(k) # todo: if m == 1: # m=1时, mmad 退化成gemv, l0A 排布: m=1 Nd, m=2 zZ; # 根据精度类型，决定matmul返回的ub精度类型，如果fp16, tensor_move做随路精度转换 C_ub = self.tik_instance.Tensor(precision_type, (N // 16, M, 16), name="C_ub", scope=UB) with self.tik_instance.new_stmt_scope(disable_sync=False): # matmul C_l0c = self.tik_instance.Tensor( FP32, (N // 16, M, 16), scope=L0C, name="C_l0c" ) # n1mn0 (n0=16) A_l0a = self.tik_instance.Tensor(FP16, (M, K), scope=L0A, name="A_l0a") # Nz -> zZ if a_transpose: self.tik_instance.load2dv1(dst=A_l0a, src=A_l1, index=0, repeat_times=M * K * DTYPE_SIZE[FP16] // 512, src_stride=1, sid=0, if_transpose=True) else: for i in range(M // 16): # todo: 使用少的循环次数 self.tik_instance.load2dv1(dst=A_l0a[i * K * 16], src=A_l1[i * 256], index=0, repeat_times=16 * K * DTYPE_SIZE[FP16] // 512, src_stride=M//16, sid=0, if_transpose=False) B_l0b = self.tik_instance.Tensor(FP16, (K, N), scope=L0B, name="B_l0b") if b_transpose: self.tik_instance.load2dv1(dst=B_l0b, src=B_l1, index=0, repeat_times=N * K * DTYPE_SIZE[FP16] // 512, src_stride=1, sid=0, if_transpose=False) else: for i in range(K // 16): self.tik_instance.load2dv1(dst=B_l0b[i * N * 16], src=B_l1[i * 256], index=0, repeat_times=16 * N * DTYPE_SIZE[FP16] // 512, src_stride=K//16, sid=0, if_transpose=True) self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, k, n, is_bias=0) # L0C -> ub, fp32 -> fp16 (tensor_mov可做随路转换) self.tik_instance.tensor_mov(C_ub, C_l0c, "m", 1, M * N * DTYPE_SIZE[FP32] // 1024, 0, 0) if N1MN0_to_MN: return self.N1MN0_TO_MN(C_ub) else: return C_ub def mmad_compute(self, A_l1, B_l1, m, k, n, N1MN0_to_MN=False, precision_type=FP16): """calculate matrix multiplication A_l1 * B_l1, and move the result to C_ub, then rearrange C_ub :param A_l1: input tensor in L1 zN format :param B_l1: input tensor in L1 zN format :param m: the actual number of rows of A_l1 :param k: the actual number of cols of A_l1 :param n: the actual number of cols of B_l1 :param N1MN0_to_MN: Whether reorder the result tensor. :return: C_ub with tensor with shape of (M, N) if N1MN0_to_MN else (N1, M, N0) """ M = self.up_align_to_K0(m) N = self.up_align_to_K0(n) K = self.up_align_to_K0(k) k_tail = 16 if k % 16 == 0 else k % 16 # 根据精度类型，决定matmul返回的ub精度类型，如果fp16, tensor_move做随路精度转换 C_ub = self.tik_instance.Tensor(precision_type, (N // 16, M, 16), name="C_ub", scope=UB) with self.tik_instance.new_stmt_scope(disable_sync=False): # matmul C_l0c = self.tik_instance.Tensor(FP32, (N // 16, M, 16), scope=L0C, name="C_l0c") # n1mn0 (n0=16) if K > 16: with self.tik_instance.for_range(0, K // 16 // 2) as i: with self.tik_instance.for_range(0, 2, thread_num=2) as t_idx: A_l0a = self.tik_instance.Tensor(FP16, (M, 16), scope=L0A, name="A_l0a") B_l0b = self.tik_instance.Tensor(FP16, (16, N), scope=L0B, name="B_l0b") self.tik_instance.load2dv1(dst=A_l0a, src=A_l1[(i * 2 + t_idx) * M * 16], index=0, repeat_times=16 * M * DTYPE_SIZE[FP16] // 512, src_stride=1, sid=0, if_transpose=False) self.tik_instance.load2dv1(dst=B_l0b, src=B_l1[(i * 2 + t_idx) * 256], index=0, repeat_times=16 * N * DTYPE_SIZE[FP16] // 512, src_stride=K//16, sid=0, if_transpose=True) with self.tik_instance.if_scope(tik.all(i == 0, t_idx == 0)): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, 16, n, is_bias=0) with self.tik_instance.elif_scope(tik.all(K // 16 % 2 == 0, i == K // 16 // 2 - 1, t_idx == 1)): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, k_tail, n, is_bias=1) with self.tik_instance.else_scope(): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, 16, n, is_bias=1) if K // 16 % 2 > 0: # 尾快处理 A_l0a = self.tik_instance.Tensor(FP16, (M, 16), scope=L0A, name="A_l0a") B_l0b = self.tik_instance.Tensor(FP16, (16, N), scope=L0B, name="B_l0b") self.tik_instance.load2dv1(dst=A_l0a, src=A_l1[(K // 16 - 1) * M * 16], index=0, repeat_times=16 * M * DTYPE_SIZE[FP16] // 512, src_stride=1, sid=0, if_transpose=False) self.tik_instance.load2dv1(dst=B_l0b, src=B_l1[(K // 16 - 1) * 256], index=0, repeat_times=16 * N * DTYPE_SIZE[FP16] // 512, src_stride=K // 16, sid=0, if_transpose=True) with self.tik_instance.if_scope(K > 16): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, k_tail, n, is_bias=1) with self.tik_instance.else_scope(): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, k, n, is_bias=0) # L0C -> ub, fp32 -> fp16 (tensor_mov可做随路转换) self.tik_instance.tensor_mov(C_ub, C_l0c, "m", 1, M * N * DTYPE_SIZE[FP32] // 1024, 0, 0) if N1MN0_to_MN: return self.N1MN0_TO_MN(C_ub) else: return C_ub def move_vector_from_gm_to_ub(self, dst_tensor, src_tensor, gm_offset, vec_len): """load the vector from gm to ub :param dst_tensor: :param src_tensor: :param gm_offset: :return: """ dtype = src_tensor.dtype a_burst_num = 32 // DTYPE_SIZE[dtype] full_tik_blk_num, tail_num = divmod(vec_len, a_burst_num) with self.tik_instance.if_scope(full_tik_blk_num > 0): self.cont_data_mv_1_bust(dst=dst_tensor, src=src_tensor[gm_offset], burst=full_tik_blk_num) # 地址回退处理尾部数据 with self.tik_instance.if_scope(tail_num > 0): offset = vec_len - a_burst_num last_blk_ub = self.tik_instance.Tensor(dtype, (a_burst_num,), name="last_blk_ub", scope=UB) self.cont_data_mv_1_bust(dst=last_blk_ub, src=src_tensor[gm_offset + offset], burst=1) with self.tik_instance.for_range(0, a_burst_num) as idx: # offset非32bytes对齐，无法用datamove dst_tensor[offset + idx].set_as(last_blk_ub[idx]) def move_vector_from_ub_to_gm(self, dst_tensor, src_tensor, gm_offset, block_h): """write the vector back to gm :param dst_tensor: :param src_tensor: :param gm_offset: :param block_h: :return: """ dtype = src_tensor.dtype a_burst_num = 32 // DTYPE_SIZE[dtype] full_tik_blk_num = block_h // a_burst_num with self.tik_instance.if_scope(full_tik_blk_num > 0): self.cont_data_mv_1_bust(dst=dst_tensor[gm_offset], src=src_tensor, burst=full_tik_blk_num) tail_num = block_h % a_burst_num with self.tik_instance.if_scope(tail_num > 0): offset = block_h - a_burst_num tmp_ub = self.tik_instance.Tensor(dtype, (a_burst_num,), name="tmp_ub", scope=UB) with self.tik_instance.for_range(0, a_burst_num) as idx: tmp_ub[idx].set_as(src_tensor[offset + idx]) self.cont_data_mv_1_bust(dst=dst_tensor[gm_offset + offset], src=tmp_ub, burst=1) def scale_compute_vector(self, Sij_ub, dim): """scale compute vector""" scale_value = dim ** -0.5 scale = self.tik_instance.Scalar(dtype=FP16) scale.set_as(scale_value) self.tik_instance.h_mul(Sij_ub, Sij_ub, scale) return Sij_ub pengcheng.mind-test/flash_attention/tiling_strategy/000077500000000000000000000000001471400134000233035ustar00rootroot00000000000000pengcheng.mind-test/flash_attention/tiling_strategy/__init__.py000066400000000000000000000001101471400134000254040ustar00rootroot00000000000000# Copyright (C) 2021.Huawei Technologies Co., Ltd. All rights reserved. pengcheng.mind-test/flash_attention/tiling_strategy/sparse_tiling.py000066400000000000000000000027121471400134000265220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ from tiling_strategy.strategy import TilingPara from tiling_strategy.strategy import TilingStrategy class SparseTiling(TilingStrategy): """A tiling strategy implementation for sparse shape""" @classmethod def strategy_name(cls): return "sparse" def tiling(self) -> TilingPara: self.Br = min(128, self.Nq) self.Bc = min(128, self.N) self.Tr = self.Nq // self.Br self.Tc = self.N // self.Bc if self.Nq % self.Br != 0: self.last_Br = self.Nq - self.Tr * self.Br self.Tr += 1 else: self.last_Br = self.Br if self.N % self.Bc != 0: self.last_Bc = self.N - self.Tc * self.Bc self.Tc += 1 else: self.last_Bc = self.Bc return self.gen_tiling_para()pengcheng.mind-test/flash_attention/tiling_strategy/strategy.py000066400000000000000000000041411471400134000255170ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The base class of tiling strategy""" from abc import ABCMeta from abc import abstractmethod from collections import namedtuple TilingPara = namedtuple("TilingPara", "Br last_Br Bc last_Bc Tr Tc") class TilingStrategy(metaclass=ABCMeta): """Tiling strategy interface. All implementations should be defined in this module, otherwise, the UT will fail. """ _strategies = {} def __init__(self, Nq, N, head_dim) -> None: super().__init__() self.Nq = Nq self.N = N self.Br = None self.last_Br = None self.Bc = None self.last_Bc = None self.Tr = None self.Tc = None self.d = head_dim def __init_subclass__(cls, **kwargs): TilingStrategy._strategies[cls.strategy_name()] = cls @classmethod @abstractmethod def strategy_name(cls): """strategy name""" raise NotImplementedError @classmethod def from_strategy_name(cls, stgy_name: str): """from strategy name""" stgy_clz = TilingStrategy._strategies.get(stgy_name) if stgy_clz is None: raise Exception(f"Strategy:{stgy_name} not supported") return stgy_clz @abstractmethod def tiling(self) -> TilingPara: """tiling""" raise NotImplementedError def gen_tiling_para(self) -> TilingPara: """gen tiling para""" return TilingPara(self.Br, self.last_Br, self.Bc, self.last_Bc, self.Tr, self.Tc) pengcheng.mind-test/flash_attention/tiling_strategy/wukong_tiling.py000066400000000000000000000045751471400134000265500ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """wukong tiling""" from tiling_strategy.strategy import TilingPara from tiling_strategy.strategy import TilingStrategy class WukongTiling(TilingStrategy): """A tiling strategy implementation for wukonghuahua model shape""" @classmethod def strategy_name(cls): return "wukong" def tiling(self) -> TilingPara: """ 反向的空间分布待详细分析 N = (4096, 1024, 256, 64) 或 77 Nq = (4096, 1024, 256, 64) d = dv = (40, 80, 160， 160) """ if self.N <= 77: # [77, 64] # cross-attention or self-attention of (64, 64, 160) self.Bc = self.N self.Tc = self.N // self.Bc if self.d <= 80: # [40, 80] # 内存瓶颈为在ub中对P*V结果[Br, dv]进行cast # ub: 512 * 80 * 6 // 1024 = 240KB self.Br = min(self.Nq, 64) self.Tr = self.Nq // self.Br else: # ub: dv = 160， 256 * 160 * 6 // 1024 = 240KB self.Br = min(self.Nq, 64) self.Tr = self.Nq // self.Br else: # self-attention if self.N == 256: self.Bc = 64 self.Tc = 1 # 内存瓶颈为在ub中对Q*K的结果[Br, Bc]进行cast # ub: 128 * 256 * 6 // 1024 = 192KB self.Br = 64 self.Tr = self.Nq // self.Br else: self.Bc = 64 self.Tc = self.N // self.Bc # ub: 64 * 512 * 6 // 1024 = 192KB self.Br = 64 self.Tr = self.Nq // self.Br self.last_Br = self.Br self.last_Bc = self.Bc return self.gen_tiling_para() pengcheng.mind-test/mindformers/000077500000000000000000000000001471400134000172365ustar00rootroot00000000000000pengcheng.mind-test/mindformers/__init__.py000066400000000000000000000025731471400134000213560ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """mindformers init""" from mindformers import core, auto_class, dataset, \ models, modules, wrapper, tools from mindformers.pipeline import * from mindformers.trainer import * from mindformers.core import * from mindformers.dataset import * from mindformers.models import * from mindformers.modules import * from mindformers.wrapper import * from mindformers.tools import * from mindformers.auto_class import * from mindformers.generation import * #from mindformers.pet import * from .mindformer_book import MindFormerBook __all__ = [] __all__.extend(dataset.__all__) __all__.extend(models.__all__) __all__.extend(core.__all__) __all__.extend(tools.__all__) __all__.extend(auto_class.__all__) __all__.extend(generation.__all__) pengcheng.mind-test/mindformers/__pycache__/000077500000000000000000000000001471400134000214465ustar00rootroot00000000000000pengcheng.mind-test/mindformers/__pycache__/__init__.cpython-39.pyc000066400000000000000000000014751471400134000256450ustar00rootroot00000000000000a >cf{@sdZddlmZmZmZmZmZmZmZddl Tddl Tddl Tddl Tddl TddlTddlTddlTddlTddlTddlmZgZeejeejeejeejeejeejdS)zmindformers init)core auto_classdatasetmodelsmoduleswrappertools)*)MindFormerBookN)__doc__ mindformersrrrrrrrZmindformers.pipelinemindformers.trainermindformers.coreZmindformers.datasetZmindformers.modelsZmindformers.modulesZmindformers.wrappermindformers.toolsZmindformers.auto_classZmindformers.generationmindformer_bookr __all__extendZ generationrr?/home/ma-user/work/mindformers-baichuan/mindformers/__init__.pys&$      pengcheng.mind-test/mindformers/__pycache__/auto_class.cpython-39.pyc000066400000000000000000000557441471400134000262530ustar00rootroot00000000000000a >cf@sdZddlZddlZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZgd ZGd d d ZGdddZGdddZGdddZdS)u AutoConfig、AutoModel N)MindFormerBook print_dict)build_processor) BaseConfig) build_model)build_model_config)logger)MindFormerConfig) AutoConfig AutoModel AutoProcessor AutoTokenizerc@sXeZdZdZeZdZdZddZ e ddZ e dd Z e d d Z e d d ZdS)r a AutoConfig class, helps instantiates a config by yaml model name or path. If using a model name, the config yaml will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoConfig >>> >>> # 1) instantiates a config by yaml model name >>> config_a = AutoConfig.from_pretrained('clip_vit_b_32') >>> # 2) instantiates a config by yaml model path >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> config_b = AutoConfig.from_pretrained(config_path) rrcCs tddS)NzkAutoConfig is designed to be instantiated using the `AutoConfig.from_pretrained(yaml_name_or_path)` method.EnvironmentErrorselfrA/home/ma-user/work/mindformers-baichuan/mindformers/auto_class.py__init__8szAutoConfig.__init__c Cs|dr|d|j}|d|j|jvr8dS|d|j}|j|}t|tsh||vrhdS|}t|d|jks|d|j|vrt d|d|d||d|j}|||vrt d|d|d|d||dS %Check whether it is a valid yaml name mindspore/_TF'' is not supported by '', please select from startswithsplit _model_name _model_type _support_listkeys isinstancedictlen ValueErrorclsyaml_name_or_pathlocal_model_typelocal_model_listlocal_model_nameslocal_model_namerrrinvalid_yaml_name>s*   zAutoConfig.invalid_yaml_namec Ks|dd}|dur|}t|ts6tdt|dtj|rr|dsZt |dt |}t d|n| |rt |d|jdn|}|d r|d |j}tjt|d |j}ntjt|d |j}tj|s tj|d d tj||d}dd}tj|s||} tj| rrtj| rrt| |t d|ntd| t |}t|jj} tt | |jj!j| S)a From pretrain method, which instantiates a config by yaml model name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. pretrained_model_name_or_pathN+yaml_name_or_path should be a str, but got ..yamlz) should be a .yaml file for model config.z.the content in %s is used for config building.g is not a supported model type or a valid path to model config. supported model could be selected from rrrTexist_okcSs0d}tD]}||vr||}q,q|SNrget_trainer_support_task_listvaluesget model_namedefault_yaml_file model_dictrrrget_default_yaml_files  z9AutoConfig.from_pretrained..get_default_yaml_file"default yaml config in %s is used.0default yaml file path must be correct, but get )"popr%str TypeErrortypeospathexistsendswithr(r r infor0r#rr r!joinr#get_xihe_checkpoint_download_folderr"&get_default_checkpoint_download_foldermakedirsrealpathshutilcopyFileNotFoundErrorrmodel model_configset_model_config_to_nameidarch) r*r+kwargsr1 config_args yaml_namecheckpoint_path yaml_filerBr@configrrrfrom_pretrainedYsR            zAutoConfig.from_pretrainedcCstd|jt|jdSzshow support list methodzsupport list of %s is:Nr rM__name__rr#r*rrrshow_support_listszAutoConfig.show_support_listcCs|jSzget support list methodr#rerrrget_support_listszAutoConfig.get_support_listN)rd __module__ __qualname____doc__rget_config_support_listr#r"r!r classmethodr0rarfrirrrrr #s  M r c@s|eZdZdZeZdZdZddZ e ddZ e dd Z e d d Z e d d Ze ddZe ddZe ddZdS)r a AutoModel class helps instantiates a model by yaml model name, path or config. If using a model name, the config yaml and checkpoint file will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoModel >>> >>> # 1) input model name, load model and weights >>> model_a = AutoModel.from_pretrained('clip_vit_b_32') >>> # 2) input model directory, load model and weights >>> from mindformers.mindformer_book import MindFormerBook >>> checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip') >>> model_b = AutoModel.from_pretrained(checkpoint_dir) >>> # 3) input yaml path, load model without weights >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> model_c = AutoModel.from_config(config_path) >>> # 4) input config, load model without weights >>> config = AutoConfig.from_pretrained('clip_vit_b_32') >>> model_d = AutoModel.from_config(config) rrcCs tddS)NzAutoModel is designed to be instantiated using the `AutoModel.from_pretrained(pretrained_model_name_or_dir)` method or `AutoModel.from_config(config)` method.rrrrrrszAutoModel.__init__c Cs|dr|d|j}|d|j|jvr8dS|d|j}|j|}t|tsh||vrhdS|}t|d|jks|d|j|vrt d|d|d||d|j}|||vrt d|d|d|d||dS) z&Check whether it is a valid model namerrrTFrrrr)r*pretrained_model_name_or_dirr,r-r.r/rrrinvalid_model_names2    zAutoModel.invalid_model_namecKs|durtd|dd}t|tr<||}||}n(tj|r\| dr\t |}ntd|srd|j j _ t|j }td|S)a' From config method, which instantiates a Model by config. Args: config (str, BaseConfig): A model config inherited from BaseConfig, or a path to .yaml file for model config. Returns: A model, which inherited from BaseModel. Nz8a model cannot be built from config with config is None.download_checkpointTr4zUconfig should be inherited from BaseConfig, or a path to .yaml file for model config.model built successfully!)r(rEr%r_inverse_parse_config _wrap_configrIrJrKrLr rVrWcheckpoint_name_or_pathrr rM)r*r`r[rqZinversed_configr\rVrrr from_configs         zAutoModel.from_configcCsRt|ts|S|jj}|d|i|D] \}}||}|||iq,|S)a Inverse parse config method, which builds yaml file content for model config. Args: config (BaseConfig): a model config inherited from BaseConfig. Returns: A model config, which follows the yaml content. rH)r%r __class__rdupdateitemsrs)r*r` class_namekeyvalnew_valrrrrss  zAutoModel._inverse_parse_configcCsH|dd}|dur(tt|d}t|d}t||d}t|dS)a Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A model config, which has the same content as a yaml file. r?N)rH)rWrZ)rV)rErget_model_config_to_namer=rYr)r*r`r?rZrVrrrrt(s   zAutoModel._wrap_configcKsj|dd}|dd}|dur$|}t|ts@tdt|tj|}tj|}|rp|st |dn | |rt |d|j d|r2d d t |D}d d t |D}|r|st d |tj|||j} tj|||j} td | | t| } | jjd| it| j} n"|} |drr|d|j} tjt| d|j}ntjt|d|j}tj|stj|ddtj|| d} dd}tj| s || }tj|rtj|rt || td| nt d|t| } | jjd|i|sJd| jj_!t| j} | j"|_"td| S)ai From pretrain method, which instantiates a Model by pretrained model name or path. Args: pretrained_model_name_or_dir (str): A supported model name or a directory to model checkpoint (including .yaml file for config and .ckpt file for weights), the supported model name could be selected from AutoModel.show_support_list(). If pretrained_model_name_or_dir is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "pretrained_model_name_or_dir", if "pretrained_model_name_or_path" is set, "pretrained_model_name_or_dir" is useless. Returns: A model, which inherited from BaseModel. r1NrqTz6pretrained_model_name_or_dir should be a str, but got z is not a directory.r5r3cSsg|]}|dr|qSr4rL.0filerrr bs z-AutoModel.from_pretrained..cSsg|]}|dr|qS)z.ckptrrrrrrds zIthere is no yaml file for model config or ckpt file for model weights in z;config in %s and weights in %s are used for model building.rurrrr6r4cSs0d}tD]}||vr||}q,q|Sr8r:r>rrrrBs  z8AutoModel.from_pretrained..get_default_yaml_filerCrDrr)#rEr%rFrGrHrIrJrKisdirr(rpr#listdirrUrNr"r rMr rVrWrxrrr r!rrOrPrQrRrSrTru default_checkpoint_download_path)r*ror[r1rqis_existis_dir yaml_list ckpt_listr_ ckpt_filer\rVZpretrained_checkpoint_namer^rBr@rrrra;s~            zAutoModel.from_pretrainedcCstd|jt|jdSrbrcrerrrrfszAutoModel.show_support_listcCs|jSrgrhrerrrriszAutoModel.get_support_listN)rdrjrkrlrget_model_support_listr#r"r!rrnrprvrsrtrarfrirrrrr s&     i r c@sXeZdZdZeZdZdZddZ e ddZ e dd Z e d d Z e d d ZdS)r a AutoProcessor helps instantiates a processor by yaml model name or path. If using a model name, the config yaml will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoProcessor >>> >>> # 1) instantiates a processor by yaml model name >>> pro_a = AutoProcessor.from_pretrained('clip_vit_b_32') >>> # 2) instantiates a processor by yaml model path >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> pro_b = AutoProcessor.from_pretrained(config_path) rrcCs tddS)NzqAutoProcessor is designed to be instantiated using the `AutoProcessor.from_pretrained(yaml_name_or_path)` method.rrrrrrszAutoProcessor.__init__c Cs|dr|d|j}|d|j}|j|}t|tsJ||vrJdS|}t|d|jksz|d|j|vrt d|d|d||d|j}|||vrt d|d|d|d||dS)rrrrFrrr) rr r!r"r#r%r&r$r'r(r)rrrr0s&   zAutoProcessor.invalid_yaml_namecKsV|dd}|dur|}t|ts4tdt|tj|}|drf| d|j  d|j n| d|j }|s||j vrt|d|jd|j d |rtd |tj|rd d t|D}tj|||j }t|}nt|}n"|}||sn|drN| d|j }tjt| d|j } ntjt| d|j } nt|d |jd|j d tj| stj| ddtj| |d} dd} tj| s| |} tj| rtj| rt| | td| ntd| t| }|} tj| s:d} t|j | d}td|S)a From pretrain method, which instantiated a processor by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A processor which inherited from BaseProcessor. r1Nr2rrrz, does not exist, and it is not supported by z. please select from r3z1config in %s is used for auto processor building.cSsg|]}|dr|qSrrrrrrr z1AutoProcessor.from_pretrained..z+ does not exist, or it is not supported by Tr6r4cSs0d}tD]}||vr||}q,q|Sr8r:r>rrrrB(s  z.get_default_yaml_filerCrD)lib_pathzprocessor built successfully!)!rEr%rFrGrHrIrJrKrr r!r"r#r$r(rdr rMrrrNr r0rrOrPrQrRrSrTrUr processor)r*r+r[r1rr?rr]r\r^r_rBr@rrrrrrasv              zAutoProcessor.from_pretrainedcCstd|jt|jdSrbrcrerrrrf@szAutoProcessor.show_support_listcCs|jSrgrhrerrrriFszAutoProcessor.get_support_listN)rdrjrkrlrget_processor_support_listr#r"r!rrnr0rarfrirrrrr s  Z r c@sheZdZdZeZdZdZe ddZ e ddZ e dd Z e d d Z e d d Ze ddZdS)ra Load the tokenizer according to the `yaml_name_or_path`. It supports the following situations 1. `yaml_name_or_path` is the model name. 2. `yaml_name_or_path` is the path to the downloaded files. Examples: >>> from mindformers.auto_class import AutoTokenizer >>> >>> # 1) instantiates a tokenizer by the model name >>> tokenizer_a = AutoTokenizer.from_pretrained("clip_vit_b_32") >>> # 2) instantiates a tokenizer by the path to the downloaded files. >>> from mindformers.models.clip.clip_tokenizer import CLIPTokenizer >>> clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") >>> clip_tokenizer.save_pretrained(path_saved) >>> restore_tokenizer = AutoTokenizer.from_pretrained(path_saved) rrc Cs|dr|d|j}|d|j|jvr8dS|d|j}|j|}t|tsh||vrhdSt|tsvdS|}t|d|jks|d|j|vrt d|d|d||d|j}|||vrt d|d|d|d||dSrrr)rrrr0as.   zAutoTokenizer.invalid_yaml_namec Cstj|}tj|}tj|}|s~|s:t|d|sLt|dddt|D}|shdStj|||j}n|}t d|t |}d}|rd|vrd|dvrd |ddvr|dd d d}t d |||S) z Try to find the yaml from the given path Args: yaml_name_or_path (str): the directory of the config yaml Returns: The class name of the tokenizer in the config yaml. z3 does not exist, Please pass a valid the directory.z3 is not a directory. You should pass the directory.cSsg|]}|dr|qSrrrrrrrrz;AutoTokenizer._get_class_name_from_yaml..Nz;Config in the yaml file %s are used for tokenizer building.r tokenizerrHz&Load the tokenizer name %s from the %s) rIrJrKrisfiler(rrNr"r rMr rE) r*r+rris_filerr_r`rzrrr_get_class_name_from_yamls,    z'AutoTokenizer._get_class_name_from_yamlcCstj|d}tj|s*td|dt|d}t|}Wdn1sT0Y|dd}|st d|d| |S) z try to get the tokenizer type from tokenizer_config.json Args: yaml_name_or_path (str): the directory of tokenizer_config.json Returns: The class name of the tokenizer in tokenizer_config.json ztokenizer_config.jsonz:The file `tokenizer_config.json` should exits in the path z, but not found.rNtokenizer_classz1There should be the key word`tokenizer_class` in z', but not found. The optional keys are ) rIrJrNrKrUopenjsonloadrEr(r$)r*r+tokenizer_config_pathfpZ config_kwargsrzrrr*_get_class_name_from_tokenizer_config_files    (  z8AutoTokenizer._get_class_name_from_tokenizer_config_filec Ks|dd}|dur|}ddlm}t|ts@tdt|tj |rh| |}|sd| |}n | |sj|}| dr|d|j}tjt|d|j}ntjt|d|j}tj|stj|d d tj||d }d d } tj|s^| |} tj| rPtj| rPt| |td|ntd| | |}ntd|d|jd|d|jd|d} | |} td| j j| S)a From pretrain method, which instantiates a tokenizer by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/clip_vit_b_32" or "clip_vit_b_32". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A tokenizer which inherited from PretrainedTokenizer. r1Nr)MindFormerRegisterr2rrrTr6r4cSs0d}tD]}||vr||}q,q|Sr8r:r>rrrrBs  z.get_default_yaml_filerCrDzTokenizer type `z` does not exist. Use `zJ.show_support_list()` to check the supported tokenizer. Or make sure the `z` is a directory.r) module_typerzz %s Tokenizer built successfully!)!rEr9rr%rFrGrHrIrJrrrr0rr r!rNrrOr"rPrKrQrRrSrTr rMrUrdget_clsrarw) r*r+r[r1rrzr]r^r_rBr@ dynamic_classZinstanced_classrrrrasT               zAutoTokenizer.from_pretrainedcCstd|jt|jdSrbrcrerrrrfszAutoTokenizer.show_support_listcCs|jSrgrhrerrrri szAutoTokenizer.get_support_listN)rdrjrkrlrget_tokenizer_support_listr#r"r!rnr0rrrarfrirrrrrLs   #  K r)rlrIrrSmindformer_bookrrZmodels.build_processorrZmodels.base_configrZmodels.build_modelrZmodels.build_configrtoolsr Ztools.register.configr __all__r r r rrrrrs&      pengcheng.mind-test/mindformers/__pycache__/mindformer_book.cpython-39.pyc000066400000000000000000001034551471400134000272630ustar00rootroot00000000000000a >cf0@sNdZddlZddlZddlmZddlmZddZddZGd d d Z dS) z| MindFormerBook class, which contains the lists of models, pipelines, tasks, and default settings in MindFormer repository. N) OrderedDict)loggercCsbt|trL|D]6\}}t|tr:td|t|qtd||qntt|ddS)z Print dict function for show support method of MindFormerBook or other BaseClasses. Args: input_dict (dict): the dict to be printed. z%s:z %s : %sz is unspoorted by print_dictN) isinstancedictitemsrinfo print_dict TypeErrortype) input_dictkeyvalrF/home/ma-user/work/mindformers-baichuan/mindformers/mindformer_book.pyrs    rcCs<t|ttfr&td|tdntt|ddS)z Print path or list function for show support method of MindFormerBook or other BaseClasses. Args: input_path_or_list (str, list): the path or list to be printed. z %sz%-------------------------------------z% is unsupported by print_path_or_listN)rstrlistrrr r )input_path_or_listrrrprint_path_or_list-s  rc5@seZdZdZejejejeZ e dej ddZ e dej ddZ ededej e d fgfd ed ej e d fdej e d fgfd edej e dfdej e dfdej e dfdej e dfdej e dfgfdedej e dfdej e dfdej e dfgfdedej e dfdej e dfdej e dfd ej e d!fd"ej e dfd#ej e d$fd%ej e d&fd'ej e d(fdej e dfg fd)edej e d*fgfd+edej e d,fd"ej e d,fd#ej e d-fd%ej e d.fd'ej e d/fd0ej e d1fdej e d,fgfd2ed3ej e d4fd5ej e d6fdej e d7fgfd8ed9ej e d:fd;ej e d<fdej e d:fgfd=ed>ej e d?fd@ej e dAfdBej e dAfdCej e dDfdej e d?fgfdEedFej e dGfdHej e dIfdej e dGfgfdJedKej e dLfdMej e dLfdNej e dLfdej e dLfgfdOedPej e dQfdRej e dSfdTej e dUfdVej e dWfdXej e dYfdZej e d[fd\ej e d]fd^ej e d_fd`ej e dafdbej e dcfddej e defdfej e dgfdhej e difdjej e dkfdlej e dmfdnej e dofdpej e dqfdrej e dsfdtej e dufdvej e dwfdxej e dyfdzej e d{fd|ej e d}fd~ej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dQfg fdedej e dfdej e dfdej e dfdej e dfgfgZed+ed#ej e d-fd"ej e d,fdej e d,fd%ej e d.fd'ej e d/fdej e d,fd0ej e d1fgfd2ed3ej e d4fd5ej e d6fdej e d6fgfd ed ej e d fdej e d fgfd edej e dfdej e dfdej e dfdej e dfdej e dfgfd8ed9ej e d:fdej e d:fgfdedej e dfdej e dfdej e dfgfd=ed>ej e d?fd@ej e dAfdBej e dAfdCej e dDfdej e d?fgfdEedFej e dGfdHej e dIfdej e dGfgfdJedKej e dLfdej e dLfgfdOedPej e dQfdRej e dSfdXej e dYfdZej e d[fdTej e dUfd\ej e d]fd^ej e d_fd`ej e dafdbej e dcfddej e defdfej e dgfdhej e difdjej e dkfdlej e dmfdnej e dofdpej e dqfdrej e dsfdtej e dufdvej e dwfdxej e dyfdzej e d{fd|ej e d}fd~ej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dQfgfd)edej e d*fgfdedej e dfdej e dfdej e dfdej e dfgfg Zedgdfdgdfdedd3d5gfgfdd gfdgdfdddgfdddgfdedgdfgfdedgdfdPdCgfgfdeddKdMgfgfdd9gfddgfdPgdfdgdfdgdfddjdlgfdgdfdgdfddvdxgfdgdfgZedgdfdeddd0gfdgdfgfdedd3d5gfgfdd gfdgdfdddgfdddgfdedgdfgfdedgdfdPdCgfgfdeddKdMgfgfdd9gfddgfddgfdPgdfdgdfdgdfddjdlgfdgdfdgdfddvdxgfdddgfdgdfgZedgdfdgdfdedd3d5gfgfdd gfdgdfdddgfdddgfdedgdfgfdedgdfdPdCgfgfdd9gfdPgdfddgfddjdlgfddgfdgdfdgdfgZedgdfddd0gfdddgfdedgdfgfdedgdfdPdCgfgfdeddKdMgfgfdd9gfdPdPgfdgdfdgdfddjdlgfddngfddvgfdgdfdddgfdddgfgZegZeddgfd"dgfd#dgfd%dgfd'dgfd dgfddgfddgfd0dgfddgfddgfd3dgfd dgfd5dgfddgfddgfddgfd9dgfddgfddgfdFdgfdHdgfddgfd>dgfd@dgfdBdgfdKdgfdMdgfdPdgfdRdgfdXdgfdZdgfdTdgfdjdgfdldgfdndgfdpdgfdrdgfdtdgfdvdgfdxdgfd\dgfd^dgfdbdgfdddgfdhdgfdzdgfd|dgfddgfddgfddgfg3Zeddgfd#dgfd%dgfd'dgfddgfd0dgfd dgfddgfddgfddgfd9dgfddgfddgfdFdgfdHdgfd>dgfd@dgfdKdgfdMdgfdPdgfgZedgZeddgfd"dgfd#dgfd%dgfd'dgfd9dgfddgfddgfdFdgfdHdgfdӐd gfd>d gfd@d gfdBd gfdKd gfdMd gfdPd d gfdXd d gfdRd d gfdZd d gfdTd d gfdjdgfdldgfdndgfdvdgfd\dgfdhdgfdbdgfdzdgfd|dgfd~dgfddgfg ZegdZed`ddZedaddZeddZeddZeddZ ed d!Z!ed"d#Z"ed$d%Z#ed&d'Z$ed(d)Z%ed*d+Z&ed,d-Z'ed.d/Z(ed0d1Z)ed2d3Z*ed4d5Z+ed6d7Z,ed8d9Z-ed:d;Z.ed<d=Z/ed>d?Z0ed@dAZ1edBdCZ2edDdEZ3edFdGZ4edHdIZ5edJdKZ6edLdMZ7edNdOZ8edPdQZ9edRdSZ:edTdUZ;edVdWZed\d]Z?ed^d_Z@dS(bMindFormerBookaR MindFormerBook class, which contains the lists of models, pipelines, tasks, and default settings in MindFormer repository When adding a new Model or assemble in this project, the following constants list and dict need adding. Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> >>> # 1) Fill the following constant list and dict in this class >>> # 2) Overwrite the support_list when define a new Model or Pipeline. >>> @MindFormerRegister.register(MindFormerModuleType.MODELS) ... class ViTForImageClassification(BaseModel): ... _support_list = MindFormerBook.get_model_support_list()['vit'] >>> # 3) Then you can use auto class and from pretrain to init an instance. >>> vit_model = AutoModel.from_pretrained('vit_base_p16') ZCHECKPOINT_DOWNLOAD_FOLDER.Zcheckpoint_downloadZCHECKPOINT_SAVE_FOLDERZcheckpoint_savegeneralcommonz%configs/general/run_general_task.yamlZmasked_image_modelingmae_vit_base_p16z/configs/mae/run_mae_vit_base_p16_224_800ep.yamlZimage_classification vit_base_p16z+configs/vit/run_vit_base_p16_224_100ep.yamlZswin_base_p4w7z.configs/swin/run_swin_base_p4w7_224_100ep.yamlmindspore/vit_base_p16zmindspore/swin_base_p4w7 fill_maskZbert_base_uncasedz'configs/bert/run_bert_base_uncased.yamlZbert_tiny_uncasedz'configs/bert/run_bert_tiny_uncased.yamlZ#contrastive_language_image_pretrain clip_vit_b_32z5configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yamlblip2_stage1_vit_gz:configs/blip2/run_blip2_stage1_vit_g_qformer_pretrain.yamlblip2_stage2_vit_g_baichuan_7bz5configs/blip2/run_blip2_stage2_vit_g_baichuan_7b.yamlblip2_stage2_vit_g_llama_7bz2configs/blip2/run_blip2_stage2_vit_g_llama_7b.yamlmindspore/clip_vit_b_32 clip_vit_b_16z5configs/clip/run_clip_vit_b_16_pretrain_flickr8k.yaml clip_vit_l_14z5configs/clip/run_clip_vit_l_14_pretrain_flickr8k.yamlclip_vit_l_14@336z9configs/clip/run_clip_vit_l_14@336_pretrain_flickr8k.yamlZimage_to_text_retrievalz=configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yamlzero_shot_image_classificationzKconfigs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yamlzKconfigs/clip/run_clip_vit_b_16_zero_shot_image_classification_cifar100.yamlzKconfigs/clip/run_clip_vit_l_14_zero_shot_image_classification_cifar100.yamlzOconfigs/clip/run_clip_vit_l_14@336_zero_shot_image_classification_cifar100.yamlblip2_stage1_classificationzQconfigs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yamlZimage_to_text_generation"itt_blip2_stage2_vit_g_baichuan_7bzNconfigs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yamlitt_blip2_stage2_vit_g_llama_7bzKconfigs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yamlzJconfigs/clip/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml translationZt5_smallz%configs/t5/run_t5_small_on_wmt16.yamlZt5_tinyz$configs/t5/run_t5_tiny_on_wmt16.yamlZtext_classificationtxtcls_bert_base_uncasedz0configs/txtcls/run_txtcls_bert_base_uncased.yamltxtcls_bert_base_uncased_mnliz5configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml'mindspore/txtcls_bert_base_uncased_mnliZ gpt2_txtclsz!configs/gpt2/run_gpt2_txtcls.yamlZtoken_classificationtokcls_bert_base_chinesez0configs/tokcls/run_tokcls_bert_base_chinese.yaml tokcls_bert_base_chinese_cluenerz8configs/tokcls/run_tokcls_bert_base_chinese_cluener.yamlZquestion_answeringZqa_bert_base_uncasedz(configs/qa/run_qa_bert_base_uncased.yamlZqa_bert_base_uncased_squadzmindspore/qa_bert_base_uncasedZtext_generationgpt2zconfigs/gpt2/run_gpt2.yaml gpt2_lorazconfigs/gpt2/run_gpt2_lora.yamlgpt2_13bzconfigs/gpt2/run_gpt2_13b.yamlZgpt2_52bzconfigs/gpt2/run_gpt2_52b.yamlgpt2_xlzconfigs/gpt2/run_gpt2_xl.yaml gpt2_xl_loraz"configs/gpt2/run_gpt2_xl_lora.yamlllama_7bzconfigs/llama/run_llama_7b.yaml llama_13bz configs/llama/run_llama_13b.yaml llama_65bz configs/llama/run_llama_65b.yaml llama2_7bz!configs/llama2/run_llama2_7b.yaml llama2_13bz"configs/llama2/run_llama2_13b.yaml llama2_70bz"configs/llama2/run_llama2_70b.yaml llama_7b_loraz$configs/llama/run_llama_7b_lora.yamlZpangualpha_2_6bz+configs/pangualpha/run_pangualpha_2_6b.yamlZpangualpha_13bz*configs/pangualpha/run_pangualpha_13b.yamlglm_6bz$configs/glm/run_glm_6b_finetune.yaml glm_6b_chatz!configs/glm/run_glm_6b_infer.yaml glm_6b_loraz configs/glm/run_glm_6b_lora.yamlglm_6b_lora_chatz&configs/glm/run_glm_6b_lora_infer.yamlZglm2_6bzconfigs/glm2/run_glm2_6b.yamlZ glm2_6b_loraz"configs/glm2/run_glm2_6b_lora.yaml bloom_560mz!configs/bloom/run_bloom_560m.yaml bloom_7.1bz!configs/bloom/run_bloom_7.1b.yaml bloom_65bz configs/bloom/run_bloom_65b.yaml bloom_176bz!configs/bloom/run_bloom_176b.yaml baichuan_7bz&research/baichuan/run_baichuan_7b.yaml baichuan2_7bz(research/baichuan2/run_baichuan2_7b.yaml baichuan2_13bz)research/baichuan2/run_baichuan2_13b.yamlziya_13bzresearch/ziya/run_ziya_13b.yaml internlm_7bz&research/internlm/run_internlm_7b.yamlinternlm_7b_loraz+research/internlm/run_internlm_7b_lora.yamlZsegment_anything sam_vit_bzconfigs/sam/run_sam_vit-b.yaml sam_vit_lzconfigs/sam/run_sam_vit-l.yaml sam_vit_hzconfigs/sam/run_sam_vit-h.yamlblip2_stage1_vig_gclip)rr!r"r#r Zblip2)rKr%rrr&r'ZittmaeZvit)rr vit_g_p16Zswinberttokcls)r,r-*mindspore/tokcls_bert_base_chinese_cluenertxtcls)r)r*r+qat5ZfilipZfilip_vit_l_14)r.r/r1r2r0llama) r3r4r5r6r7r8r9rBrCrDrErFrGZllama2)r6r7r8Z pangualphaZbloom)r>r?r@rAglm)r:r;r<r=Zglm2Zsam)rHrIrJz1-stgz2-stg)blip2_stage1_pretrainedrr)rrrrNZmt5Zt5_pegasus_base)r3r4r5r6r7r8r9Zinternlm)rr%rr) r3r4r5r6r7r8r9rBrEZ baichuan2zjhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/clip_vit_b_32.ckptz[https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/clip_vit_b_32/file/clip_vit_b_32.ckptz`https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_b_16.ckptz`https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_l_14.ckptzfhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_l_14%40336.ckptzlhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/mae/mae_vit_base_p16.ckptzhhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/vit/vit_base_p16.ckptrNz]https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/vit_g_p16.ckptzdhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/blip2_pretrained.ckptrWzYhttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/vit_base_p16/file/vit_base_p16.ckptzkhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/swin/swin_base_p4w7.ckptz]https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/swin_base_p4w7/file/swin_base_p4w7.ckptzjhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/t5/mindspore_model.ckptzxhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_tiny/bert_tiny_uncased.ckptzxhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_base/bert_base_uncased.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese_cluener.ckptrQzhttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/tokcls_bert_base_chineses_cluener/file/tokcls_bert_base_chinese_cluener.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased_mnli.ckptz{https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/txtcls_bert_base_uncased_mnli/file/txtcls_bert_base_uncased_mnli.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased_squad.ckptzahttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2.ckptzfhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_lora.ckptzdhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_xl.ckptzihttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_xl_lora.ckptzehttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_13b.ckptzrhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/pangualpha/pangualpha_2_6b.ckptzqhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/pangualpha/pangualpha_13b.ckptzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm/glm_6b.ckptzghttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm/glm_6b_lora.ckptzdhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm2/glm2_6b.ckptzihttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm2/glm2_6b_lora.ckptzkhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/llama/open_llama_7b.ckptzlhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/llama/open_llama_13b.ckptz^https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/llama2/llama2_7b.ckptzdhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/llama2/llama2-13b-fp16.ckptzphttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/llama/open_llama_7b_lora.ckptz^https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/bloom/bloom_560m.ckptz^https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/bloom/bloom_7.1b.ckptzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/sam/sam_vit_b_01ec64.ckptzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/sam/sam_vit_l_0b3195.ckptzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/sam/sam_vit_h_4b8939.ckptzjhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/clip_vit_b_32.yamlz`https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_b_16.yamlz`https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_l_14.yamlzfhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_l_14%40336.yamlzfhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/blip2_stage1_vit_g.yamlzohttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/blip2_stage1_classification.yamlzlhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/mae/mae_vit_base_p16.yamlzhhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/vit/vit_base_p16.yamlz]https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/vit_g_p16.yamlzkhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/swin/swin_base_p4w7.yamlzjhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/t5/mindspore_model.yamlzxhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_tiny/bert_tiny_uncased.yamlzxhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_base/bert_base_uncased.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese_cluener.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased_mnli.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased_squad.yamlzahttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2.yaml)r$zehttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.pngzthttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gzzehttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/clip_vit_b_32/file/bpe_simple_vocab_16e6.txt.gzzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/t5/spiece.modelzahttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/vocab.txtzshttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_base_chinese/vocab.txtzfhttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/tokcls_bert_base_chineses_cluener/file/vocab.txtzshttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_base_english/vocab.txtzbhttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/txtcls_bert_base_uncased_mnli/file/vocab.txtzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/vocab.jsonzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/merges.txtzihttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/pangualpha/vocab.modelzehttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm/ice_text.modelzghttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm2/tokenizer.modelzhhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/llama/tokenizer.modelz_https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/llama2/tokenizer.modelz]https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/bloom/tokenizer.json) )ZChatGLMTokenizer GLMProcessor)ZChatGLM2TokenizerrX)Z CLIPTokenizerZ CLIPProcessor) BertTokenizer BertProcessor)Z T5TokenizerZ T5Processor)ZLlamaTokenizerLlamaProcessor)Z GPT2TokenizerZ GPT2Processor)ZPanguAlphaTokenizerZPanguAlphaProcessor)ZBloomTokenizerZBloomProcessor)ZInternLMTokenizerr[)ZBaichuan2Tokenizerr[NcCst|j}|d|D]:\}}|dg}|D]\}}||q<|||<q|r||vr||}tdt |t |qt dntdt |dS)show_trainer_support_model_listrrz+Trainer support model list for %s task is: unsupported taskz-Trainer support model list of MindFormer is: N) copydeepcopy_TRAINER_SUPPORT_TASKS_LISTpoprappendkeysrrrrKeyErrorrclstaskZall_listr r temp_list model_name_rrrr\s          z.MindFormerBook.show_trainer_support_model_listcCst|j}|D]:\}}|dg}|D]\}}||q2|||<q|r||vr||}tdt |t |qt dntdt |dS) show_pipeline_support_model_listrz,Pipeline support model list for %s task is: r]z.Pipeline support model list of MindFormer is: N) r^r__PIPELINE_SUPPORT_TASK_LISTrrarbrcrrrrrdrrerrrrks        z/MindFormerBook.show_pipeline_support_model_listcCstdt|jdS)z)show_tokenizer_name_to_processor function._TRAINER_SUPPORT_TASKS_LIST of MindFormer is: N)rrr_TOKENIZER_NAME_TO_PROCESSORrfrrr show_tokenizer_name_to_processors z/MindFormerBook.show_tokenizer_name_to_processorcCs|jS)z(get_tokenizer_name_to_processor function)rnrorrrget_tokenizer_name_to_processor sz.MindFormerBook.get_tokenizer_name_to_processorcCstdt|jdS)z'show_trainer_support_task_list functionrmN)rrrr`rorrrshow_trainer_support_task_list%s z-MindFormerBook.show_trainer_support_task_listcCstdt|jdS)z.show_pipeline_support_input_data_list functionz,_PIPELINE_INPUT_DATA_LIST of MindFormer is: N)rrr_PIPELINE_INPUT_DATA_LISTrorrr%show_pipeline_support_input_data_list+s z4MindFormerBook.show_pipeline_support_input_data_listcCs|jS)z-get_pipeline_support_input_data_list function)rsrorrr$get_pipeline_support_input_data_list1sz3MindFormerBook.get_pipeline_support_input_data_listcCs|jS)z&get_trainer_support_task_list function)r`rorrrget_trainer_support_task_list6sz,MindFormerBook.get_trainer_support_task_listcCstdt|jdS)z(show_pipeline_support_task_list functionz._PIPELINE_SUPPORT_TASK_LIST of MindFormer is: N)rrrrlrorrrshow_pipeline_support_task_list;s z.MindFormerBook.show_pipeline_support_task_listcCs|jS)z'get_pipeline_support_task_list function)rlrorrrget_pipeline_support_task_listAsz-MindFormerBook.get_pipeline_support_task_listcCst|jdS)"show_model_config_to_name functionN)r_MODEL_CONFIG_TO_NAMErorrrshow_model_config_to_nameFsz(MindFormerBook.show_model_config_to_namecCs|jS)ry)rzrorrrget_model_config_to_nameKsz'MindFormerBook.get_model_config_to_namecCs|j||idS)z set_model_config_to_name function Args: model_config (str): the name of model config model_name (str): the name of model N)rzupdate)rf model_configrirrrset_model_config_to_namePsz'MindFormerBook.set_model_config_to_namecCstdt|jdS)z#show_model_config_url_list functionz)_MODEL_CONFIG_URL_LIST of MindFormer is: N)rrr_MODEL_CONFIG_URL_LISTrorrrshow_model_config_url_list\s z)MindFormerBook.show_model_config_url_listcCs|jS)z"get_model_config_url_list function)rrorrrget_model_config_url_listbsz(MindFormerBook.get_model_config_url_listcCstdt|jdS)zshow_project_path functionz _PROJECT_PATH of MindFormer is: N)rrr _PROJECT_PATHrorrrshow_project_pathgs z MindFormerBook.show_project_pathcCs|jS)zget_project_path function)rrorrrget_project_pathmszMindFormerBook.get_project_pathcCstj|jdS)z0get xihe's mindspore checkpoint download folder. mindspore)ospathjoin#_DEFAULT_CHECKPOINT_DOWNLOAD_FOLDERrorrr#get_xihe_checkpoint_download_folderrsz2MindFormerBook.get_xihe_checkpoint_download_foldercCstdt|jdS)z0show_default_checkpoint_download_folder functionz6_DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER of MindFormer is: N)rrrrrorrr'show_default_checkpoint_download_folderws z6MindFormerBook.show_default_checkpoint_download_foldercCs|jS)z/get_default_checkpoint_download_folder function)rrorrr&get_default_checkpoint_download_folder}sz5MindFormerBook.get_default_checkpoint_download_foldercCs$tj|st|d||_dS)z set_default_checkpoint_download_folder function Args: download_folder (str): the path of default checkpoint download folder  should be a directory.N)rrisdirr r)rfZdownload_folderrrr&set_default_checkpoint_download_folders z5MindFormerBook.set_default_checkpoint_download_foldercCs|jS)z+get_default_checkpoint_save_folder function)_DEFAULT_CHECKPOINT_SAVE_FOLDERrorrr"get_default_checkpoint_save_foldersz1MindFormerBook.get_default_checkpoint_save_foldercCs$tj|st|d||_dS)z set_default_checkpoint_save_folder function Args: save_folder (str): the path of default checkpoint save folder rN)rrrr r)rfZ save_folderrrr"set_default_checkpoint_save_folders z1MindFormerBook.set_default_checkpoint_save_foldercCstdt|jdS)z,show_default_checkpoint_save_folder functionz2_DEFAULT_CHECKPOINT_SAVE_FOLDER of MindFormer is: N)rrrrrorrr#show_default_checkpoint_save_folders z2MindFormerBook.show_default_checkpoint_save_foldercCstdt|jdS)z!show_config_support_list functionz&CONFIG_SUPPORT_LIST of MindFormer is: N)rrr_CONFIG_SUPPORT_LISTrorrrshow_config_support_lists z'MindFormerBook.show_config_support_listcCs|jS)z get_config_support_list function)rrorrrget_config_support_listsz&MindFormerBook.get_config_support_listcCstdt|jdS)z show_model_support_list functionz%MODEL_SUPPORT_LIST of MindFormer is: N)rrr_MODEL_SUPPORT_LISTrorrrshow_model_support_lists z&MindFormerBook.show_model_support_listcCs|jS)zget_model_support_list function)rrorrrget_model_support_listsz%MindFormerBook.get_model_support_listcCstdt|jdS)z$show_processor_support_list functionz)PROCESSOR_SUPPORT_LIST of MindFormer is: N)rrr_PROCESSOR_SUPPORT_LISTrorrrshow_processor_support_lists z*MindFormerBook.show_processor_support_listcCs|jS)z#get_processor_support_list function)rrorrrget_processor_support_listsz)MindFormerBook.get_processor_support_listcCs|jS)z#get_tokenizer_support_list function)_TOKENIZER_SUPPORT_LISTrorrrget_tokenizer_support_listsz)MindFormerBook.get_tokenizer_support_listcCst|jdS)z$show_tokenizer_support_list functionN)rrrorrrshow_tokenizer_support_listsz*MindFormerBook.show_tokenizer_support_listcCs|jS)z'get_tokenizer_url_support_list function)_TOKENIZER_SUPPORT_URL_LISTrorrrget_tokenizer_url_support_listsz-MindFormerBook.get_tokenizer_url_support_listcCst|jdS)z(show_tokenizer_url_support_list functionN)rrrorrrshow_tokenizer_url_support_listsz.MindFormerBook.show_tokenizer_url_support_listcCs2g}|jD]}||j|qt|S)get_model_name_support_list)r`rcextendgetset)rfZsupport_model_name task_namerrrrsz*MindFormerBook.get_model_name_support_listcCstdt|jdS)z!show_model_ckpt_url_list functionz&MODEL_CKPT_URL_LIST of MindFormer is: N)rrr_MODEL_CKPT_URL_LISTrorrrshow_model_ckpt_url_lists z'MindFormerBook.show_model_ckpt_url_listcCs|jS)z get_model_ckpt_url_list function)rrorrrget_model_ckpt_url_listsz&MindFormerBook.get_model_ckpt_url_list)N)N)A__name__ __module__ __qualname____doc__rrdirnameabspath__file__rgetenvrrrrr`rlrrrrrzrrrsrrn classmethodr\rkrprqrrrtrurvrwrxr{r|rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr;s    BD  @$            k            k         M     OMV    r) rrr^ collectionsrmindformers.toolsrrrrrrrrs  pengcheng.mind-test/mindformers/__pycache__/version_control.cpython-39.pyc000066400000000000000000000056731471400134000273370ustar00rootroot00000000000000a >cf@sdZddlZddlZddlmZddlmmZddl mm Z ddl m Z ddlmZddZd d Zd d Zd dZdddZddZddZdS)zMindSpore Version ControlN)nn) is_version_ge)loggercCs dd}|S)zCell reuse decorator.cSsFtdddksttjds |Std|jjddl m }||S)NZMS_DEV_CELL_REUSE0z2.1.0zEnable cell use mode at %s.r)cell_attr_register) osgetenvrms __version__rinfo __class____name__Zmindspore._extendsr)funcrrF/home/ma-user/work/mindformers-baichuan/mindformers/version_control.py decorators  z!get_cell_reuse..decoratorr)rrrrget_cell_reusesrcCs.ttjdrtj|d}ntjd|d}|S)N1.11.0)pr) keep_prob)rr r rDropout)Z dropout_probdropoutrrr get_dropout"s rcCs"ttjdrt}nt}|SNr)rr r PTrilr)trilrrrget_tril)s  rcCs(d dd}d dd}ttjdr$|S|S) zreturn ops.normNFcSstj|||||dS)N)orddimkeepdimdtype)FnormArr r!r"rrr tensor_norm14szget_norm..tensor_norm1cSs6|durtdt|ts$tdtj||||dS)Nz>The 'dtype' is not available when mindspore version < '1.11.0'zAThe type of 'ord' should be int when mindspore version < '1.11.0')r keep_dims)rwarning isinstanceint TypeErrorr#r$r%rrr tensor_norm29s   zget_norm..tensor_norm2r)NNFN)r(NFNrr r )r'r.rrrget_norm0s    r0cKslttjdr&|j|f|||d|S|j||||||dd|dd|dd|dd|d dd S) Nr) input_columnsoutput_columnsnum_parallel_workerspython_multiprocessingF max_rowsizecache callbacksoffload) r1r2 column_orderr3r4r5r7r8r9)rr r mapget)dataset operationsr1r2r3kwargsrrrget_dataset_mapDs&      r@cCsttjdrtStjSr)rr r rIdentityr#identityrrrr get_identityVs rCcCs ttjdS)Nz2.2.0r/rrrrfix_optim_global_step_sig[srD)NNN)__doc__r mindsporer rZmindspore.ops.operationsopsr>rZmindspore.ops.functional functionalr#Z tools.utilsrZ tools.loggerrrrrr0r@rCrDrrrrs     pengcheng.mind-test/mindformers/auto_class.py000066400000000000000000001124161471400134000217520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ AutoConfig、AutoModel """ import os import json import shutil from .mindformer_book import MindFormerBook, print_dict from .models.build_processor import build_processor from .models.base_config import BaseConfig from .models.build_model import build_model from .models.build_config import build_model_config from .tools import logger from .tools.register.config import MindFormerConfig __all__ = ['AutoConfig', 'AutoModel', 'AutoProcessor', 'AutoTokenizer'] class AutoConfig: """ AutoConfig class, helps instantiates a config by yaml model name or path. If using a model name, the config yaml will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoConfig >>> >>> # 1) instantiates a config by yaml model name >>> config_a = AutoConfig.from_pretrained('clip_vit_b_32') >>> # 2) instantiates a config by yaml model path >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> config_b = AutoConfig.from_pretrained(config_path) """ _support_list = MindFormerBook.get_config_support_list() _model_type = 0 _model_name = 1 def __init__(self): raise EnvironmentError( "AutoConfig is designed to be instantiated " "using the `AutoConfig.from_pretrained(yaml_name_or_path)` method." ) @classmethod def invalid_yaml_name(cls, yaml_name_or_path): """Check whether it is a valid yaml name""" if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name_or_path = yaml_name_or_path.split('/')[cls._model_name] if not yaml_name_or_path.split('_')[cls._model_type] in cls._support_list.keys(): return True local_model_type = yaml_name_or_path.split('_')[cls._model_type] local_model_list = cls._support_list[local_model_type] if not isinstance(local_model_list, dict) and yaml_name_or_path in local_model_list: return False local_model_names = local_model_list.keys() if len(yaml_name_or_path.split('_')) <= cls._model_name or \ not yaml_name_or_path.split('_')[cls._model_name] in local_model_names: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}\', ' f'please select from {local_model_list}') local_model_name = yaml_name_or_path.split('_')[cls._model_name] if not yaml_name_or_path in local_model_list[local_model_name]: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}_{local_model_name}\', ' f'please select from {local_model_list[local_model_name]}') return False @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a config by yaml model name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}.") if os.path.exists(yaml_name_or_path): if not yaml_name_or_path.endswith(".yaml"): raise ValueError(f"{yaml_name_or_path} should be a .yaml file for model" " config.") config_args = MindFormerConfig(yaml_name_or_path) logger.info("the content in %s is used for" " config building.", yaml_name_or_path) elif cls.invalid_yaml_name(yaml_name_or_path): raise ValueError(f"{yaml_name_or_path} is not a supported" f" model type or a valid path to model config." f" supported model could be selected from {cls._support_list}.") else: yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) config = build_model_config(config_args.model.model_config) MindFormerBook.set_model_config_to_name(id(config), config_args.model.arch.type) return config @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list class AutoModel: """ AutoModel class helps instantiates a model by yaml model name, path or config. If using a model name, the config yaml and checkpoint file will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoModel >>> >>> # 1) input model name, load model and weights >>> model_a = AutoModel.from_pretrained('clip_vit_b_32') >>> # 2) input model directory, load model and weights >>> from mindformers.mindformer_book import MindFormerBook >>> checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip') >>> model_b = AutoModel.from_pretrained(checkpoint_dir) >>> # 3) input yaml path, load model without weights >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> model_c = AutoModel.from_config(config_path) >>> # 4) input config, load model without weights >>> config = AutoConfig.from_pretrained('clip_vit_b_32') >>> model_d = AutoModel.from_config(config) """ _support_list = MindFormerBook.get_model_support_list() _model_type = 0 _model_name = 1 def __init__(self): raise EnvironmentError( "AutoModel is designed to be instantiated " "using the `AutoModel.from_pretrained(pretrained_model_name_or_dir)` method " "or `AutoModel.from_config(config)` method." ) @classmethod def invalid_model_name(cls, pretrained_model_name_or_dir): """Check whether it is a valid model name""" if pretrained_model_name_or_dir.startswith('mindspore'): # Adaptation the name of model at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" pretrained_model_name_or_dir = pretrained_model_name_or_dir.split('/')[cls._model_name] if not pretrained_model_name_or_dir.split('_')[cls._model_type] in cls._support_list.keys(): return True local_model_type = pretrained_model_name_or_dir.split('_')[cls._model_type] local_model_list = cls._support_list[local_model_type] if not isinstance(local_model_list, dict) and pretrained_model_name_or_dir in local_model_list: return False local_model_names = local_model_list.keys() if len(pretrained_model_name_or_dir.split('_')) <= cls._model_name or \ not pretrained_model_name_or_dir.split('_')[cls._model_name] in local_model_names: raise ValueError(f'\'{pretrained_model_name_or_dir}\' is not supported by \'{local_model_type}\', ' f'please select from {local_model_list}') local_model_name = pretrained_model_name_or_dir.split('_')[cls._model_name] if not pretrained_model_name_or_dir in local_model_list[local_model_name]: raise ValueError(f'\'{pretrained_model_name_or_dir}\' is not supported by ' f'\'{local_model_type}_{local_model_name}\', please select from ' f'{local_model_list[local_model_name]}') return False @classmethod def from_config(cls, config, **kwargs): """ From config method, which instantiates a Model by config. Args: config (str, BaseConfig): A model config inherited from BaseConfig, or a path to .yaml file for model config. Returns: A model, which inherited from BaseModel. """ if config is None: raise ValueError("a model cannot be built from config with config is None.") download_checkpoint = kwargs.pop("download_checkpoint", True) if isinstance(config, BaseConfig): inversed_config = cls._inverse_parse_config(config) config_args = cls._wrap_config(inversed_config) elif os.path.exists(config) and config.endswith(".yaml"): config_args = MindFormerConfig(config) else: raise ValueError("config should be inherited from BaseConfig," " or a path to .yaml file for model config.") if not download_checkpoint: config_args.model.model_config.checkpoint_name_or_path = None model = build_model(config_args.model) logger.info("model built successfully!") return model @classmethod def _inverse_parse_config(cls, config): """ Inverse parse config method, which builds yaml file content for model config. Args: config (BaseConfig): a model config inherited from BaseConfig. Returns: A model config, which follows the yaml content. """ if not isinstance(config, BaseConfig): return config class_name = config.__class__.__name__ config.update({"type": class_name}) for key, val in config.items(): new_val = cls._inverse_parse_config(val) config.update({key: new_val}) return config @classmethod def _wrap_config(cls, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A model config, which has the same content as a yaml file. """ model_name = config.pop("model_name", None) if model_name is None: model_name = MindFormerBook.get_model_config_to_name().get(id(config), None) arch = BaseConfig(type=model_name) model = BaseConfig(model_config=config, arch=arch) return BaseConfig(model=model) @classmethod def from_pretrained(cls, pretrained_model_name_or_dir, **kwargs): """ From pretrain method, which instantiates a Model by pretrained model name or path. Args: pretrained_model_name_or_dir (str): A supported model name or a directory to model checkpoint (including .yaml file for config and .ckpt file for weights), the supported model name could be selected from AutoModel.show_support_list(). If pretrained_model_name_or_dir is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "pretrained_model_name_or_dir", if "pretrained_model_name_or_path" is set, "pretrained_model_name_or_dir" is useless. Returns: A model, which inherited from BaseModel. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) download_checkpoint = kwargs.pop("download_checkpoint", True) if pretrained_model_name_or_path is not None: pretrained_model_name_or_dir = pretrained_model_name_or_path if not isinstance(pretrained_model_name_or_dir, str): raise TypeError(f"pretrained_model_name_or_dir should be a str," f" but got {type(pretrained_model_name_or_dir)}") is_exist = os.path.exists(pretrained_model_name_or_dir) is_dir = os.path.isdir(pretrained_model_name_or_dir) if is_exist: if not is_dir: raise ValueError(f"{pretrained_model_name_or_dir} is not a directory.") else: if cls.invalid_model_name(pretrained_model_name_or_dir): raise ValueError(f"{pretrained_model_name_or_dir} is not a supported model" f" type or a valid path to model config. supported model" f" could be selected from {cls._support_list}.") if is_dir: yaml_list = [file for file in os.listdir(pretrained_model_name_or_dir) if file.endswith(".yaml")] ckpt_list = [file for file in os.listdir(pretrained_model_name_or_dir) if file.endswith(".ckpt")] if not yaml_list or not ckpt_list: raise FileNotFoundError(f"there is no yaml file for model config or ckpt file" f" for model weights in {pretrained_model_name_or_dir}") yaml_file = os.path.join(pretrained_model_name_or_dir, yaml_list[cls._model_type]) ckpt_file = os.path.join(pretrained_model_name_or_dir, ckpt_list[cls._model_type]) logger.info("config in %s and weights in %s are used for model" " building.", yaml_file, ckpt_file) config_args = MindFormerConfig(yaml_file) config_args.model.model_config.update({"checkpoint_name_or_path": ckpt_file}) model = build_model(config_args.model) else: pretrained_checkpoint_name = pretrained_model_name_or_dir if pretrained_model_name_or_dir.startswith('mindspore'): # Adaptation the name of model at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" pretrained_checkpoint_name = pretrained_model_name_or_dir.split('/')[cls._model_name] checkpoint_path = os.path.join( MindFormerBook.get_xihe_checkpoint_download_folder(), pretrained_checkpoint_name.split('_')[cls._model_type]) else: # Default the name of model, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join( MindFormerBook.get_default_checkpoint_download_folder(), pretrained_model_name_or_dir.split("_")[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, pretrained_checkpoint_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(pretrained_checkpoint_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) config_args.model.model_config.update( {"checkpoint_name_or_path": pretrained_model_name_or_dir}) if not download_checkpoint: config_args.model.model_config.checkpoint_name_or_path = None model = build_model(config_args.model) cls.default_checkpoint_download_path = model.default_checkpoint_download_path logger.info("model built successfully!") return model @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list class AutoProcessor: """ AutoProcessor helps instantiates a processor by yaml model name or path. If using a model name, the config yaml will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoProcessor >>> >>> # 1) instantiates a processor by yaml model name >>> pro_a = AutoProcessor.from_pretrained('clip_vit_b_32') >>> # 2) instantiates a processor by yaml model path >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> pro_b = AutoProcessor.from_pretrained(config_path) """ _support_list = MindFormerBook.get_processor_support_list() _model_type = 0 _model_name = 1 def __init__(self): raise EnvironmentError( "AutoProcessor is designed to be instantiated " "using the `AutoProcessor.from_pretrained(yaml_name_or_path)` method." ) @classmethod def invalid_yaml_name(cls, yaml_name_or_path): """Check whether it is a valid yaml name""" if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name_or_path = yaml_name_or_path.split('/')[cls._model_name] local_model_type = yaml_name_or_path.split('_')[cls._model_type] local_model_list = cls._support_list[local_model_type] if not isinstance(local_model_list, dict) and yaml_name_or_path in local_model_list: return False local_model_names = local_model_list.keys() if len(yaml_name_or_path.split('_')) <= cls._model_name or \ not yaml_name_or_path.split('_')[cls._model_name] in local_model_names: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}\', ' f'please select from {local_model_list}') local_model_name = yaml_name_or_path.split('_')[cls._model_name] if not yaml_name_or_path in local_model_list[local_model_name]: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}_{local_model_name}\', ' f'please select from {local_model_list[local_model_name]}') return False @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiated a processor by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A processor which inherited from BaseProcessor. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}") is_exist = os.path.exists(yaml_name_or_path) model_name = yaml_name_or_path.split('/')[cls._model_name].split("_")[cls._model_type] \ if yaml_name_or_path.startswith('mindspore') else yaml_name_or_path.split("_")[cls._model_type] if not is_exist and model_name not in cls._support_list.keys(): raise ValueError(f'{yaml_name_or_path} does not exist,' f' and it is not supported by {cls.__name__}. ' f'please select from {cls._support_list}.') if is_exist: logger.info("config in %s is used for auto processor" " building.", yaml_name_or_path) if os.path.isdir(yaml_name_or_path): yaml_list = [file for file in os.listdir(yaml_name_or_path) if file.endswith(".yaml")] yaml_name = os.path.join(yaml_name_or_path, yaml_list[cls._model_type]) config_args = MindFormerConfig(yaml_name) else: config_args = MindFormerConfig(yaml_name_or_path) else: yaml_name = yaml_name_or_path if not cls.invalid_yaml_name(yaml_name_or_path): if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) else: raise ValueError(f'{yaml_name_or_path} does not exist,' f' or it is not supported by {cls.__name__}.' f' please select from {cls._support_list}.') if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) lib_path = yaml_name_or_path if not os.path.isdir(lib_path): lib_path = None processor = build_processor(config_args.processor, lib_path=lib_path) logger.info("processor built successfully!") return processor @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list class AutoTokenizer: """ Load the tokenizer according to the `yaml_name_or_path`. It supports the following situations 1. `yaml_name_or_path` is the model name. 2. `yaml_name_or_path` is the path to the downloaded files. Examples: >>> from mindformers.auto_class import AutoTokenizer >>> >>> # 1) instantiates a tokenizer by the model name >>> tokenizer_a = AutoTokenizer.from_pretrained("clip_vit_b_32") >>> # 2) instantiates a tokenizer by the path to the downloaded files. >>> from mindformers.models.clip.clip_tokenizer import CLIPTokenizer >>> clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") >>> clip_tokenizer.save_pretrained(path_saved) >>> restore_tokenizer = AutoTokenizer.from_pretrained(path_saved) """ _support_list = MindFormerBook.get_tokenizer_support_list() _model_type = 0 _model_name = 1 @classmethod def invalid_yaml_name(cls, yaml_name_or_path): """Check whether it is a valid yaml name""" if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name_or_path = yaml_name_or_path.split('/')[cls._model_name] if not yaml_name_or_path.split('_')[cls._model_type] in cls._support_list.keys(): return True local_model_type = yaml_name_or_path.split('_')[cls._model_type] local_model_list = cls._support_list[local_model_type] if not isinstance(local_model_list, dict) and yaml_name_or_path in local_model_list: return False if not isinstance(local_model_list, dict): return True local_model_names = local_model_list.keys() if len(yaml_name_or_path.split('_')) <= cls._model_name or \ not yaml_name_or_path.split('_')[cls._model_name] in local_model_names: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}\', ' f'please select from {local_model_list}') local_model_name = yaml_name_or_path.split('_')[cls._model_name] if not yaml_name_or_path in local_model_list[local_model_name]: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}_{local_model_name}\', ' f'please select from {local_model_list[local_model_name]}') return False @classmethod def _get_class_name_from_yaml(cls, yaml_name_or_path): """ Try to find the yaml from the given path Args: yaml_name_or_path (str): the directory of the config yaml Returns: The class name of the tokenizer in the config yaml. """ is_exist = os.path.exists(yaml_name_or_path) is_dir = os.path.isdir(yaml_name_or_path) is_file = os.path.isfile(yaml_name_or_path) if not is_file: if not is_exist: raise ValueError(f"{yaml_name_or_path} does not exist, Please pass a valid the directory.") if not is_dir: raise ValueError(f"{yaml_name_or_path} is not a directory. You should pass the directory.") # If passed a directory, load the file from the yaml files yaml_list = [file for file in os.listdir(yaml_name_or_path) if file.endswith(".yaml")] if not yaml_list: return None yaml_file = os.path.join(yaml_name_or_path, yaml_list[cls._model_type]) else: yaml_file = yaml_name_or_path logger.info("Config in the yaml file %s are used for tokenizer building.", yaml_file) config = MindFormerConfig(yaml_file) class_name = None if config and 'processor' in config and 'tokenizer' in config['processor'] \ and 'type' in config['processor']['tokenizer']: class_name = config['processor']['tokenizer'].pop('type', None) logger.info("Load the tokenizer name %s from the %s", class_name, yaml_name_or_path) return class_name @classmethod def _get_class_name_from_tokenizer_config_file(cls, yaml_name_or_path): """ try to get the tokenizer type from tokenizer_config.json Args: yaml_name_or_path (str): the directory of tokenizer_config.json Returns: The class name of the tokenizer in tokenizer_config.json """ tokenizer_config_path = os.path.join(yaml_name_or_path, 'tokenizer_config.json') if not os.path.exists(tokenizer_config_path): raise FileNotFoundError(f"The file `tokenizer_config.json` should exits in the " f"path {tokenizer_config_path}, but not found.") with open(tokenizer_config_path, 'r') as fp: config_kwargs = json.load(fp) class_name = config_kwargs.pop('tokenizer_class', None) if not class_name: raise ValueError(f"There should be the key word`tokenizer_class` in {tokenizer_config_path}, but " f"not found. The optional keys are {config_kwargs.keys()}") return class_name @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a tokenizer by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/clip_vit_b_32" or "clip_vit_b_32". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A tokenizer which inherited from PretrainedTokenizer. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path from . import MindFormerRegister if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}") # Try to load from the remote if os.path.isdir(yaml_name_or_path): class_name = cls._get_class_name_from_yaml(yaml_name_or_path) if not class_name: class_name = cls._get_class_name_from_tokenizer_config_file(yaml_name_or_path) elif not cls.invalid_yaml_name(yaml_name_or_path): # Should download the files from the remote storage yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') class_name = cls._get_class_name_from_yaml(yaml_file) else: raise FileNotFoundError(f"Tokenizer type `{yaml_name_or_path}` does not exist. " f"Use `{cls.__name__}.show_support_list()` to check the supported tokenizer. " f"Or make sure the `{yaml_name_or_path}` is a directory.") dynamic_class = MindFormerRegister.get_cls(module_type='tokenizer', class_name=class_name) instanced_class = dynamic_class.from_pretrained(yaml_name_or_path) logger.info("%s Tokenizer built successfully!", instanced_class.__class__.__name__) return instanced_class @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list pengcheng.mind-test/mindformers/core/000077500000000000000000000000001471400134000201665ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/__init__.py000066400000000000000000000024211471400134000222760ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Core.""" from .lr import build_lr from .loss import build_loss from .optim import build_optim from .metric import build_metric from .callback import build_callback from .lr import * from .loss import * from .optim import * from .metric import * from .callback import * from .context import * from .clip_grad import ClipGradNorm from .parallel_config import build_parallel_config __all__ = ['build_parallel_config', 'ClipGradNorm'] __all__.extend(lr.__all__) __all__.extend(loss.__all__) __all__.extend(optim.__all__) __all__.extend(metric.__all__) __all__.extend(callback.__all__) __all__.extend(context.__all__) pengcheng.mind-test/mindformers/core/__pycache__/000077500000000000000000000000001471400134000223765ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/__pycache__/__init__.cpython-39.pyc000066400000000000000000000012741471400134000265720ustar00rootroot00000000000000a ?cf@sdZddlmZddlmZddlmZddlmZddl m Z ddlTddlTddlTddlTddl Tddl Tddl m Z dd lmZd d gZeejeejeejeejee jee jd S) zMindFormers Core.)build_lr) build_loss) build_optim) build_metric)build_callback)*) ClipGradNorm)build_parallel_configr rN)__doc__lrrlossroptimrmetricrcallbackrcontext clip_gradrparallel_configr __all__extendrrD/home/ma-user/work/mindformers-baichuan/mindformers/core/__init__.pys(            pengcheng.mind-test/mindformers/core/__pycache__/clip_grad.cpython-39.pyc000066400000000000000000000056311471400134000267600ustar00rootroot00000000000000a Hlf@sdZddlZddlmZddlmZddlmZ ddlm Z ddlm Z ddlmZdd lmZzdd lmZdd lmZWn*eyddlmZddlmZYn0e d d Ze dZe dZedddZ edddddZ!GdddeZ"dS)z,Operations for clipping grad by global norm.N)ops)Cell) composite) functional) operations)Tensor)dtype) Validator)Rel grad_scaleTget_square_sumapply_global_normrcCs.tdt|d}tt|tjd}|S)NFr)P ReduceSumFsquare expand_dimscastmstypefloat32)xnormrrE/home/ma-user/work/mindformers-baichuan/mindformers/core/clip_grad.py_get_square_sum)srcCsTt|}||d}tj|tdtjttdtjd}||}t ||}|S)Ngư>?)clip_value_maxclip_value_min) rrr clip_by_valuerrrnplogr) clip_norm global_normrx_dtype clip_coefZclip_coef_clampedrrr_apply_global_norm0s   r&cs*eZdZdZdfdd ZddZZS) ClipGradNormae Clips tensor values by the ratio of the sum of their norms. Args: max_norm (Union(float, int)): The clipping ratio. Default: 1.0 use_norm (Union(float, None)): The global norm. Default: None Inputs: - **x** (Union(tuple[Tensor], list[Tensor])) - Input data to clip. Outputs: Tensor, a clipped Tensor. rNcsjtt||dur,td|jd|td|dtj|jt |gt j |_ t |_t|_dS)NzFor 'zF', input 'use_norm' only supports None currently, but got 'use_norm': r"r)superr'__init__ ValueErrorcls_name validator check_numberr GTrrrr"CHyperMap hyper_mapr GreaterEqual greater_equal)selfmax_normuse_norm __class__rrr)Js  zClipGradNorm.__init__cCs^|t|}tt|}|}|||j}t|||j}|tt |j||}||fS)z clip grad.) r1r rsqrtaddnr3r"selectpartialr )r4r square_sumr#Zorigin_global_normcondclip_xrrr constructUs zClipGradNorm.construct)rN)__name__ __module__ __qualname____doc__r)r@ __classcell__rrr7rr';s r')#rDnumpyr mindsporermindspore.nn.cellr mindspore.opsrr/rrrrmindspore.common.tensorrmindspore.commonrrZmindspore._checkparamr r,r ImportError _checkparam ExpandDims add_prim_attrrMultitypeFuncGraphr r registerrr&r'rrrrs,               pengcheng.mind-test/mindformers/core/__pycache__/parallel_config.cpython-39.pyc000066400000000000000000000023751471400134000301570ustar00rootroot00000000000000a ?cf @sLdZddlmZmZddlmZmZddlmZeZ ee dZ ddZ dS) zParallel Config Init.)default_moe_config MoEConfig)TransformerOpParallelConfigTransformerRecomputeConfig)logger) recomputecCs|jr4t|jts:td|jtfi|j|_nt|_|jrnt|jtsttd|jtfi|j|_nt|_|j rt|j t std|j |j j dkrtd|j j d|j _ t fd|ji|j |_ nt |_ dS) zBuild context config.z initial moe_config from dict: %sz&initial recompute_config from dict: %sz%initial parallel_config from dict: %sz=pipeline_stage = %s > 1, vocab_emd_dp will be reset to False.FrN) moe_config isinstancerrinforZrecompute_configrdefault_recompute_configparallel_configrpipeline_stage vocab_emb_dpdefault_parallel_config)configrK/home/ma-user/work/mindformers-baichuan/mindformers/core/parallel_config.pybuild_parallel_configs,      rN) __doc__#mindformers.modules.transformer.moerrmindformers.modules.transformerrrmindformers.tools.loggerrr rrrrrrs   pengcheng.mind-test/mindformers/core/callback/000077500000000000000000000000001471400134000217225ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/callback/.ipynb_checkpoints/000077500000000000000000000000001471400134000255135ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/callback/.ipynb_checkpoints/build_callback-checkpoint.py000066400000000000000000000063551471400134000331360ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Callback API.""" import inspect from mindspore.train import callback from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_callback( config: [dict, list] = None, default_args: dict = None, module_type: str = 'callback', class_name: str = None, **kwargs): r"""Build callback For MindFormer. Instantiate the callback from MindFormerRegister's registry. Args: config (dict, list): The task callback's config. Default: None. default_args (dict): The default argument of callback API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'callback'. class_name (str): The class name of callback API. Default: None. Return: The function instance of callback API. Examples: >>> from mindformers import build_callback >>> callback_config = [{'type': 'MFLossMonitor'}, {'type': 'SummaryMonitor'}] >>> # 1) use config dict to build callback >>> callback_from_config = build_callback(callback_config) >>> # 2) use class name to build callback >>> callback_class_name = build_callback(class_name='MFLossMonitor') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) cfg_callback = config if not isinstance(cfg_callback, list): return MindFormerRegister.get_instance_from_cfg( cfg_callback, MindFormerModuleType.CALLBACK, default_args=default_args) callbacks = [] for callback_type in cfg_callback: if callback_type.get('type') == 'MFLossMonitor': callback_op = MindFormerRegister.get_instance_from_cfg( callback_type, MindFormerModuleType.CALLBACK, default_args) else: callback_op = MindFormerRegister.get_instance_from_cfg( callback_type, MindFormerModuleType.CALLBACK) callbacks.append(callback_op) return callbacks return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_cb(): """ register MindSpore builtin LR class. """ for module_name in dir(callback): if module_name.startswith('__'): continue monitor = getattr(callback, module_name) if inspect.isclass(monitor): MindFormerRegister.register_cls( monitor, MindFormerModuleType.CALLBACK) register_ms_cb() pengcheng.mind-test/mindformers/core/callback/.ipynb_checkpoints/callback-checkpoint.py000066400000000000000000000756561471400134000317710ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormer Self-Define Callback.""" import json import os import time import datetime from collections import OrderedDict from copy import deepcopy from typing import Callable, Optional, Union import numpy as np import mindspore as ms from mindspore import Callback, Profiler, ModelCheckpoint, CheckpointConfig, context, save_checkpoint, Tensor from mindspore.train.callback import SummaryCollector from mindspore.nn.learning_rate_schedule import LearningRateSchedule from mindspore.train.callback._callback import set_cur_net from mindspore.train.serialization import _get_merged_param_data from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.cloud_adapter.cloud_adapter import Local2ObsMonitor from mindformers.tools.logger import logger from mindformers.tools.utils import get_output_root_path, get_output_subpath, get_remote_save_url, check_in_modelarts __all__ = ['ObsMonitor', 'MFLossMonitor', 'CheckpointMointor', 'SummaryMonitor', 'ProfileMonitor', 'EvalCallBack'] _cur_dir = os.getcwd() SAVE_DIR = _cur_dir @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class ObsMonitor: """Obs Monitor For AICC and Local""" def __new__(cls, src_dir: str = None, target_dir: str = None, step_upload_frequence: int = 100, epoch_upload_frequence: int = -1, keep_last: bool = True): if src_dir is None: src_dir = get_output_root_path() if target_dir is None: target_dir = get_remote_save_url() return Local2ObsMonitor(src_dir, target_dir, step_upload_frequence, epoch_upload_frequence, keep_last) def _get_loss_output(output): """Get output of task for MFLossMonitor.""" overflow = False scaling_sens = False loss = output learning_rate = None grad_norm = None if isinstance(output, (tuple, list)): if len(output) == 3: loss, overflow, scaling_sens = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() elif len(output) == 4: loss, overflow, scaling_sens, learning_rate = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() elif len(output) == 5: loss, overflow, scaling_sens, learning_rate, grad_norm = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() else: if isinstance(output[0], ms.Tensor) and isinstance(output[0].asnumpy(), np.ndarray): loss = output[0] if isinstance(loss, ms.Tensor) and isinstance(loss.asnumpy(), np.ndarray): loss = np.mean(loss.asnumpy()) # Boundary check. if isinstance(loss, float) and (np.isnan(loss) or np.isinf(loss)): invalid_loss_info = "NaN" if np.isnan(loss) else "Inf" raise ValueError(f"The current value of loss is {invalid_loss_info}, terminate training.") return loss, overflow, scaling_sens, learning_rate, grad_norm @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class MFLossMonitor(Callback): """ Loss Monitor for classification. Args: learning_rate (Union[float, LearningRateSchedule], optional): The learning rate schedule. Default: None. per_print_times (int): Every how many steps to print the log information. Default: 1. micro_batch_num (int): MicroBatch size for Pipeline Parallel. Default: 1. micro_batch_interleave_num (int): split num of batch size. Default: 1. origin_epochs (int): Training epoches. Default: None. dataset_size (int): Training dataset size. Default: None. Examples: >>> from mindformers.core.callback import MFLossMonitor >>> lr = [0.01, 0.008, 0.006, 0.005, 0.002] >>> monitor = MFLossMonitor(per_print_times=10) """ def __init__(self, learning_rate: Optional[Union[float, LearningRateSchedule]] = None, per_print_times: int = 1, micro_batch_num: int = 1, micro_batch_interleave_num: int = 1, origin_epochs: int = None, dataset_size: int = None, initial_epoch: int = 0, global_batch_size: int = 0, device_num: int = 0): super(MFLossMonitor, self).__init__() self.per_print_times = per_print_times self.learning_rate = deepcopy(learning_rate) self.last_print_time = 0 self.mirco_size = micro_batch_num self.print_warning_flag = True self.loss_list = [] self.step_time = time.time() self.epoch_time = time.time() self.run_context = None self.steps_per_epoch = dataset_size self.micro_batch_interleave_num = micro_batch_interleave_num self.origin_epochs = origin_epochs self.initial_epoch = initial_epoch self.global_batch_size = global_batch_size self.device_num = device_num def epoch_begin(self, run_context): """ Record time at the beginning of epoch. Args: run_context (RunContext): Context of the process running. """ self.loss_list = [] self.epoch_time = time.time() self.run_context = run_context def epoch_end(self, run_context): """ Print training info at the end of epoch. Args: run_context (RunContext): Context of the process running. """ def step_begin(self, run_context): """ Record time at the beginning of step. Args: run_context (RunContext): Context of the process running. """ self.step_time = time.time() self.run_context = run_context def step_end(self, run_context): """ Print training info at the end of step. Args: run_context (RunContext): Context of the process running. """ parallel_mode = ms.get_auto_parallel_context("parallel_mode") full_batch = ms.get_auto_parallel_context("full_batch") auto_parallel = parallel_mode in ['semi_auto_parallel', 'auto_parallel'] if auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) cb_params = run_context.original_args() step_seconds = (time.time() - self.step_time) * 1000 net_outputs = cb_params.net_outputs loss, overflow, scaling_sens, learning_rate, grad_norm = _get_loss_output(net_outputs) if learning_rate is not None: self.learning_rate = learning_rate loss = self._fix_loss_for_parallel(loss) self.loss_list.append(loss) if not overflow: overflow = "False" if not scaling_sens: scaling_sens = "unavailable" if cb_params.dataset_sink_mode: origin_epochs = self.origin_epochs per_step_seconds = step_seconds / cb_params.batch_num steps_per_epoch = self.steps_per_epoch cur_epoch_num = (cb_params.cur_step_num - 1) // steps_per_epoch \ + self.initial_epoch * cb_params.batch_num // steps_per_epoch + 1 cur_step_num = (cb_params.cur_step_num - 1) % steps_per_epoch + 1 else: origin_epochs = self.origin_epochs per_step_seconds = step_seconds steps_per_epoch = cb_params.batch_num cur_epoch_num = cb_params.cur_epoch_num cur_step_num = (cb_params.cur_step_num - 1) % cb_params.batch_num + 1 # compute time remaining step_remain = (origin_epochs - cur_epoch_num + 1) * steps_per_epoch - cur_step_num time_remain = step_remain * per_step_seconds / 1000 # compute throughput throughput = self.global_batch_size / self.device_num / (per_step_seconds / 1000) # compute percent percent = ((cur_epoch_num - 1) * steps_per_epoch + cur_step_num) / origin_epochs / steps_per_epoch * 100 if (cb_params.cur_step_num - self.last_print_time) >= self.per_print_times: self.last_print_time = cb_params.cur_step_num self.print_output_info(cb_params, cur_epoch_num, origin_epochs, throughput, cur_step_num, steps_per_epoch, loss, per_step_seconds, overflow, scaling_sens, time_remain, percent, grad_norm) if check_in_modelarts() and int(os.getenv("RANK_ID", "0")) == int(os.getenv("RANK_SIZE", "1")) - 1: self.dump_info_to_modelarts(ma_step_num=cur_step_num, ma_loss=loss) if auto_parallel: ms.context.set_auto_parallel_context(parallel_mode=parallel_mode, full_batch=full_batch) def _fix_loss_for_parallel(self, loss): """Fix loss value in pipeline or double parallel mode.""" pipeline_stages = ms.context.get_auto_parallel_context("pipeline_stages") if pipeline_stages > 1 and self.print_warning_flag: logger.warning("pipeline stages: %s > 1, the loss on the last card is valid.", pipeline_stages) if self.micro_batch_interleave_num > 1 and self.print_warning_flag: logger.warning("micro_batch_interleave_num: %s > 1, multiple copies in parallel is open.") if pipeline_stages > 1: loss = loss / (self.mirco_size * self.micro_batch_interleave_num) elif self.micro_batch_interleave_num > 1: loss = loss / self.micro_batch_interleave_num return loss def print_output_info(self, cb_params, cur_epoch_num, origin_epochs, throughput, cur_step_num, steps_per_epoch, loss, per_step_seconds, overflow, scaling_sens, time_remain, percent, grad_norm): """print output information.""" if self.learning_rate is not None: if isinstance(self.learning_rate, (float, Tensor)): current_lr = str(self.learning_rate) elif isinstance(self.learning_rate, LearningRateSchedule): if ms.context.get_context('device_target') == 'CPU': if self.print_warning_flag: logger.warning( "device target not support CPU when generating the learning rate value, " "please use: mindspore.context.set_context(device_target='Ascend')") self.print_warning_flag = False current_lr = None else: if cb_params.optimizer is not None: global_step = cb_params.optimizer.global_step else: global_step = cb_params.network.optimizer.global_step # temporary set_train to avoid error on 910B origin_phase = cb_params.train_network.phase cb_params.train_network.set_train(False) current_lr = self.learning_rate(global_step) cb_params.train_network.set_train(origin_phase) current_lr = np.array2string(current_lr.asnumpy()) else: if self.print_warning_flag: logger.warning( "The current learning rate cannot be calculated in real time." "Only the type of LearningRateSchedule is supported in the callback of MFLossMonitor," "but the input learning rate function type is %s", type(self.learning_rate) ) self.print_warning_flag = False current_lr = None else: if self.print_warning_flag: logger.warning( "MFLossMonitor callback is not set learning rate arguments." "To display the learning rate, you must input the arguments, " "which can be LearningRateSchedule or a fixed float" ) self.print_warning_flag = False current_lr = None if current_lr is not None: if cb_params.dataset_sink_mode: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, " "per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, int(per_step_seconds), current_lr, overflow, scaling_sens, grad_norm) else: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], " "per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, np.mean(self.loss_list), int(per_step_seconds), current_lr, overflow, scaling_sens, grad_norm) show_str = ('|%%-%ds|' % 50) % (int(50 * percent / 100) * "▮") logger.info(" %4.1f%% %s %.2f samples/s/p %s }", percent, show_str, throughput, datetime.timedelta(seconds=int(time_remain))) else: if cb_params.dataset_sink_mode: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, " "per_step_time: %dms, overflow cond: %s, loss_scale: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, int(per_step_seconds), overflow, scaling_sens) else: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], " "per_step_time: %dms, overflow cond: %s, loss_scale: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, np.mean(self.loss_list), int(per_step_seconds), overflow, scaling_sens) show_str = ('|%%-%ds|' % 50) % (int(50 * percent / 100) * "▮") logger.info(" %4.1f%% %s %.2f samples/s/p %s }", percent, show_str, throughput, datetime.timedelta(seconds=int(time_remain))) def dump_info_to_modelarts(self, ma_step_num, ma_loss): """dump modelarts info to display evaluation result page""" ma_loss = float(ma_loss) obj = None modelarts_dir = os.path.join(get_output_root_path(), "modelarts") if not os.path.exists(modelarts_dir): os.mkdir(modelarts_dir) if not os.path.exists(os.path.join(modelarts_dir, "model_analysis_results.json")): obj = { "en-us": { "common": {}, "precision_performance": { "pr": { "title": "loss", "description": "loss of model", "value": {"current_loss": 0}, "line_chart": { "pr_line_chart": { "name": "loss line chart of model", "x_axis_name": "step", "y_axis_name": "loss", "curve": {"loss": []}}}}}, "feature_sensitivity": {}, "computational_performance": {}, "abstract_feature": {}, "adversary": {} }, "zh-cn": { "common": {}, "precision_performance": { "pr": { "title": "loss", "description": "模型损失", "value": {"当前loss": 0}, "line_chart": { "pr_line_chart": { "name": "loss line chart of model", "x_axis_name": "step", "y_axis_name": "loss", "curve": {"loss": []}}}}}, "feature_sensitivity": {}, "computational_performance": {}, "abstract_feature": {}, "adversary": {} } } else: with open(os.path.join(modelarts_dir, "model_analysis_results.json"), "r") as fp: obj = json.load(fp) if obj is not None: en_precision_performance = obj["en-us"]["precision_performance"] en_precision_performance["pr"]["value"]["loss_value"] = ma_loss en_loss_list = en_precision_performance["pr"]["line_chart"]["pr_line_chart"]["curve"]["loss"] en_loss_list.append([ma_step_num, ma_loss]) zh_precision_performance = obj["zh-cn"]["precision_performance"] zh_precision_performance["pr"]["value"]["当前loss"] = ma_loss zh_loss_list = zh_precision_performance["pr"]["line_chart"]["pr_line_chart"]["curve"]["loss"] zh_loss_list.append([ma_step_num, ma_loss]) with open(os.path.join(modelarts_dir, "model_analysis_results.json"), "w") as fp: json.dump(obj, fp) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class SummaryMonitor: """Summary Monitor For AICC and Local""" def __new__(cls, summary_dir=None, collect_freq=10, collect_specified_data=None, keep_default_action=True, custom_lineage_data=None, collect_tensor_freq=None, max_file_size=None, export_options=None): if summary_dir is None: rank_id = int(os.getenv("RANK_ID", '0')) summary_dir = get_output_subpath('summary', rank_id) kwargs = { "summary_dir": summary_dir, "collect_freq": collect_freq, "collect_specified_data": collect_specified_data, "keep_default_action": keep_default_action, "custom_lineage_data": custom_lineage_data, "collect_tensor_freq": collect_tensor_freq, "max_file_size": max_file_size, "export_options": export_options } return SummaryCollector(**kwargs) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class CheckpointMointor(ModelCheckpoint): """Checkpoint Monitor For Save LossScale""" def __init__(self, prefix='CKP', directory=None, config=None, save_checkpoint_steps=1, save_checkpoint_seconds=0, keep_checkpoint_max=5, keep_checkpoint_per_n_minutes=0, integrated_save=True, save_network_params=True, save_trainable_params=False, async_save=False, saved_network=None, append_info=None, enc_key=None, enc_mode='AES-GCM', exception_save=False): self.config = config self.save_network_params = save_network_params self.save_trainable_params = save_trainable_params self.rank_id = int(os.getenv("RANK_ID", '0')) prefix = prefix + "_rank_{}".format(self.rank_id) if append_info is None: append_info = [{ "epoch_num": 0, "step_num": 0, "global_step": 0, "loss_scale": 1 }] ckpt_directory = os.path.join(directory, f"checkpoint/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint', self.rank_id) self.network_directory = os.path.join(directory, f"checkpoint_network/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint_network', self.rank_id) self.trainable_directory = os.path.join(directory, f"checkpoint_trainable/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint_trainable', self.rank_id) if context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: logger.info("Integrated_save is changed to False when using auto_parallel.") integrated_save = False config_ck = CheckpointConfig(save_checkpoint_steps=save_checkpoint_steps, save_checkpoint_seconds=save_checkpoint_seconds, keep_checkpoint_max=keep_checkpoint_max, keep_checkpoint_per_n_minutes=keep_checkpoint_per_n_minutes, integrated_save=integrated_save, async_save=async_save, saved_network=saved_network, append_info=append_info, enc_key=enc_key, enc_mode=enc_mode, exception_save=exception_save) super(CheckpointMointor, self).__init__(prefix, ckpt_directory, config=config_ck) def _save_ckpt(self, cb_params, force_to_save=False): """Save checkpoint files.""" # pylint: disable=E0203 if cb_params.cur_step_num == self._last_triggered_step: return # if param is cache enable, flush data from cache to host before save_ckpt if self._need_flush_from_cache: self._flush_from_cache(cb_params) save_ckpt = self._check_save_ckpt(cb_params, force_to_save) step_num_in_epoch = int((cb_params.cur_step_num - 1) % cb_params.batch_num + 1) if save_ckpt: logger.info('......Saving ckpt......') cur_ckpoint_file = self._prefix + "-" + str(cb_params.cur_epoch_num) + "_" \ + str(step_num_in_epoch) + ".ckpt" # update checkpoint file list. self._manager.update_ckpoint_filelist(self._directory, self._prefix) # keep checkpoint files number equal max number. if self._config.keep_checkpoint_max and 0 < self._config.keep_checkpoint_max <= self._manager.ckpoint_num: self._manager.remove_oldest_ckpoint_file() elif self._config.keep_checkpoint_per_n_minutes and self._config.keep_checkpoint_per_n_minutes > 0: # pylint: disable=E0203 self._cur_time_for_keep = time.time() if (self._cur_time_for_keep - self._last_time_for_keep) \ < self._config.keep_checkpoint_per_n_minutes * 60: self._manager.keep_one_ckpoint_per_minutes(self._config.keep_checkpoint_per_n_minutes, self._cur_time_for_keep) # generate the new checkpoint file and rename it. global SAVE_DIR SAVE_DIR = self._directory cur_file = os.path.join(self._directory, cur_ckpoint_file) self._last_time_for_keep = time.time() self._last_triggered_step = cb_params.cur_step_num if context.get_context("enable_ge") and os.getenv("MS_ENABLE_REF_MODE", "0") == "0": set_cur_net(cb_params.train_network) cb_params.train_network.exec_checkpoint_graph() if "epoch_num" in self._append_dict: self._append_dict["epoch_num"] = self._append_epoch_num + cb_params.cur_epoch_num if "step_num" in self._append_dict: self._append_dict["step_num"] = self._append_step_num + cb_params.cur_epoch_num * cb_params.batch_num if cb_params.optimizer is not None: self._append_dict["global_step"] = cb_params.optimizer.global_step else: self._append_dict["global_step"] = cb_params.network.optimizer.global_step if "loss_scale" in self._append_dict: outputs = cb_params.net_outputs if isinstance(outputs, (tuple, list)) and len(outputs) >= 3: self._append_dict["loss_scale"] = outputs[2] network = self._config.saved_network if self._config.saved_network is not None else cb_params.train_network save_checkpoint(network, cur_file, self._config.integrated_save, self._config.async_save, self._append_dict, self._config.enc_key, self._config.enc_mode) def save_only_network_params(): save_obj = cb_params.network if save_obj.optimizer is not None: save_obj = save_obj.network if self.save_network_params: cb_cur_ckpoint_file = self._prefix + "-" + "network" + ".ckpt" cb_cur_file = os.path.join(self.network_directory, cb_cur_ckpoint_file) os.makedirs(self.network_directory, exist_ok=True) save_checkpoint(save_obj, cb_cur_file, self._config.integrated_save, self._config.async_save, {}, self._config.enc_key, self._config.enc_mode) if self.save_trainable_params: save_obj.init_parameters_data() param_dict = OrderedDict() for param in save_obj.trainable_params(): param_dict[param.name] = param param_list = [] for (key, value) in param_dict.items(): each_param = {"name": key} param_data = Tensor(value.data.asnumpy()) # in automatic model parallel scenario, some parameters were split to all the devices, # which should be combined before saving if key in save_obj.parameter_layout_dict: param_data = _get_merged_param_data(save_obj, key, param_data, self._config.integrated_save) each_param["data"] = param_data param_list.append(each_param) save_obj = param_list cb_cur_ckpoint_file = self._prefix + "-" + "trainable_params" + ".ckpt" cb_cur_file = os.path.join(self.trainable_directory, cb_cur_ckpoint_file) os.makedirs(self.trainable_directory, exist_ok=True) save_checkpoint(save_obj, cb_cur_file, self._config.integrated_save, self._config.async_save, {}, self._config.enc_key, self._config.enc_mode) save_only_network_params() self._latest_ckpt_file_name = cur_file @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class ProfileMonitor(Callback): """ Profile analysis in training. """ def __init__(self, start_step=1, stop_step=10, output_path=None, start_profile=True, profile_communication=False, profile_memory=True, **kwargs): super(ProfileMonitor, self).__init__() self.start_step = start_step self.stop_step = stop_step self.start_profile = start_profile self.profile_communication = profile_communication if profile_communication and not start_profile: raise ValueError("When profile_communication is True, start_profile must also be True") if output_path is None: rank_id = int(os.getenv("RANK_ID", '0')) output_path = get_output_subpath('profile', rank_id) if ms.get_context("device_target") == "GPU" and profile_memory: logger.warning("The parameter profile_memory is not supported on GPU currently, so is changed to False. ") profile_memory = False self.profiler = Profiler( start_profile=start_profile, output_path=output_path, profile_communication=profile_communication, profile_memory=profile_memory, **kwargs) self.run_context = None self.output_path = output_path def step_begin(self, run_context): """ Start profile at the begin of step. Args: run_context (RunContext): Context of the train running. """ cb_params = run_context.original_args() step_num = cb_params.cur_step_num if step_num == self.start_step and not self.start_profile: self.profiler.start() def step_end(self, run_context): """ Stop profile at the end of step. Args: run_context (RunContext): Context of the train running. """ cb_params = run_context.original_args() step_num = cb_params.cur_step_num if step_num == self.stop_step: self.profiler.stop() self.profiler.analyse() logger.info("End of Profiling, please view the profile data under %s and analyze it using mindinsight." "MindInsight order as follow: " "mindinsight start --summary-base-dir %s", self.output_path, self.output_path) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class EvalCallBack(Callback): """Evaluate Callback used in training progress. Args: eval_func (Callable): the function to calculate eval result, task specific. step_interval (int): determine the num of step intervals between each eval. Default -1, means only eval on epoch end, do not eval between steps. Note that it will not take effects when running in data sink mode. epoch_interval (int): determine the num of epoch intervals between each eval. Default 1, means eval on every epoch end. """ def __init__(self, eval_func: Callable, step_interval: int = 100, epoch_interval: int = -1): self.eval_func = eval_func self.step_interval = step_interval self.epoch_interval = epoch_interval def epoch_end(self, run_context): # if not use epoch end if self.epoch_interval <= 0: return callback_params = run_context.original_args() cur_epoch_num = callback_params.cur_epoch_num if cur_epoch_num % self.epoch_interval == 0: self._execute_eval() def step_end(self, run_context): # if not use step end if self.step_interval <= 0: return callback_params = run_context.original_args() cur_step_num = callback_params.cur_step_num if cur_step_num % self.step_interval == 0: self._execute_eval() def _execute_eval(self): start_time = time.time() output = self.eval_func() eval_time = time.time() - start_time logger.info("Eval result: %s, eval time is %f s.", output, eval_time) pengcheng.mind-test/mindformers/core/callback/__init__.py000066400000000000000000000014511471400134000240340ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Callback.""" from .build_callback import build_callback from .callback import * __all__ = [] __all__.extend(callback.__all__) pengcheng.mind-test/mindformers/core/callback/__pycache__/000077500000000000000000000000001471400134000241325ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/callback/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004611471400134000303230ustar00rootroot00000000000000a ?cf)@s,dZddlmZddlTgZeejdS)zMindFormers Callback.)build_callback)*N)__doc__rcallback__all__extendrrM/home/ma-user/work/mindformers-baichuan/mindformers/core/callback/__init__.pys pengcheng.mind-test/mindformers/core/callback/__pycache__/build_callback.cpython-39.pyc000066400000000000000000000043261471400134000315030ustar00rootroot00000000000000a ?cf @sXdZddlZddlmZddlmZmZmZd ee gee e dddZ d d Z e dS) zBuild Callback API.N)callback)MindFormerRegisterMindFormerModuleTypeMindFormerConfigr)config default_args module_type class_namec Ks|dur|durdS|durt|tr>t|ts>tfi|}|}t|ts^tj|tj|dSg}|D]<}|ddkrt|tj|}nt|tj}| |qf|Stj ||fi|S)atBuild callback For MindFormer. Instantiate the callback from MindFormerRegister's registry. Args: config (dict, list): The task callback's config. Default: None. default_args (dict): The default argument of callback API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'callback'. class_name (str): The class name of callback API. Default: None. Return: The function instance of callback API. Examples: >>> from mindformers import build_callback >>> callback_config = [{'type': 'MFLossMonitor'}, {'type': 'SummaryMonitor'}] >>> # 1) use config dict to build callback >>> callback_from_config = build_callback(callback_config) >>> # 2) use class name to build callback >>> callback_class_name = build_callback(class_name='MFLossMonitor') N)rtypeZ MFLossMonitor) isinstancedictrlistrget_instance_from_cfgrCALLBACKgetappend get_instance) rrrr kwargsZ cfg_callback callbacks callback_typeZ callback_oprS/home/ma-user/work/mindformers-baichuan/mindformers/core/callback/build_callback.pybuild_callbacks,  rcCs@ttD]2}|drqtt|}t|rt|tj qdS)z& register MindSpore builtin LR class. __N) dirr startswithgetattrinspectisclassr register_clsrr) module_namemonitorrrrregister_ms_cbDs    r")NNrN) __doc__rZmindspore.trainrmindformers.tools.registerrrrr r strrr"rrrrs  - pengcheng.mind-test/mindformers/core/callback/__pycache__/callback.cpython-39.pyc000066400000000000000000000456101471400134000303250ustar00rootroot00000000000000a kf{@sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl ZddlZddlmZmZmZmZmZmZmZddlmZddlmZdd lmZdd lmZdd l m!Z!m"Z"dd l#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+gdZ,e-Z.e.a/e!0e"j1GdddZ2ddZ3e!0e"j1GdddeZ4e!0e"j1GdddZ5e!0e"j1GdddeZ6e!0e"j1GdddeZ7e!0e"j1GdddeZ8dS)z MindFormer Self-Define Callback.N) OrderedDict)deepcopy)CallableOptionalUnion)CallbackProfilerModelCheckpointCheckpointConfigcontextsave_checkpointTensor)SummaryCollector)LearningRateSchedule) set_cur_net)_get_merged_param_data)MindFormerRegisterMindFormerModuleType)Local2ObsMonitor)logger)get_output_root_pathget_output_subpathget_remote_save_urlcheck_in_modelarts) ObsMonitor MFLossMonitorCheckpointMointorSummaryMonitorProfileMonitor EvalCallBackc@s(eZdZdZd eeeeedddZdS) rzObs Monitor For AICC and LocalNdT)src_dir target_dirstep_upload_frequenceepoch_upload_frequence keep_lastcCs,|durt}|durt}t|||||SN)rrr)clsr"r#r$r%r&r)M/home/ma-user/work/mindformers-baichuan/mindformers/core/callback/callback.py__new__/s zObsMonitor.__new__)NNr r!T)__name__ __module__ __qualname____doc__strintboolr+r)r)r)r*r+srcCs\d}d}|}d}d}t|ttfrt|dkrN|\}}}t|tjr|}nt|dkr||\}}}}t|tjr|}n\t|dkr|\}}}}}t|tjr|}n,t|dtjrt|dtjr|d}t|tjrt|tjrt |}t|t rNt |s*t |rNt |r:dnd}t d |d |||||fS) z%Get output of task for MFLossMonitor.FNrNaNInfzThe current value of loss is z, terminate training.) isinstancetuplelistlenmsr asnumpynpndarraymeanfloatisnanisinf ValueError)outputoverflow scaling_sensloss learning_rate grad_normZinvalid_loss_infor)r)r*_get_loss_output<s4           $ $rKc s|eZdZdZdeeeefeeeeeeeed fdd Z dd Z d d Z d d Z ddZ ddZddZddZZS)ra Loss Monitor for classification. Args: learning_rate (Union[float, LearningRateSchedule], optional): The learning rate schedule. Default: None. per_print_times (int): Every how many steps to print the log information. Default: 1. micro_batch_num (int): MicroBatch size for Pipeline Parallel. Default: 1. micro_batch_interleave_num (int): split num of batch size. Default: 1. origin_epochs (int): Training epoches. Default: None. dataset_size (int): Training dataset size. Default: None. Examples: >>> from mindformers.core.callback import MFLossMonitor >>> lr = [0.01, 0.008, 0.006, 0.005, 0.002] >>> monitor = MFLossMonitor(per_print_times=10) Nr) rIper_print_timesmicro_batch_nummicro_batch_interleave_num origin_epochs dataset_size initial_epochglobal_batch_size device_numc sxtt|||_t||_d|_||_d|_g|_ t |_ t |_ d|_ ||_||_||_||_||_| |_dS)NrT)superr__init__rMrrIlast_print_time mirco_sizeprint_warning_flag loss_listtime step_time epoch_time run_contextsteps_per_epochrOrPrRrSrT) selfrIrMrNrOrPrQrRrSrT __class__r)r*rVqs    zMFLossMonitor.__init__cCsg|_t|_||_dS)z Record time at the beginning of epoch. Args: run_context (RunContext): Context of the process running. N)rZr[r]r^r`r^r)r)r* epoch_begins zMFLossMonitor.epoch_begincCsdS)z Print training info at the end of epoch. Args: run_context (RunContext): Context of the process running. Nr)rcr)r)r* epoch_endszMFLossMonitor.epoch_endcCst|_||_dS)z Record time at the beginning of step. Args: run_context (RunContext): Context of the process running. N)r[r\r^rcr)r)r* step_begins zMFLossMonitor.step_begincCstd}td}|dv}|r0tjjddd|}t|jd}|j}t|\}} } } } | durp| |_ | |}|j || sd } | sd } |j r|j} ||j}|j}|jd ||j|j|d }|jd |d }n*|j} |}|j}|j}|jd |jd }| |d ||}||d}|j|j|d}|d ||| |d }|j|j|jkr|j|_|||| |||||| | ||| trttd dttddd kr|j||d|rtjj||ddS)z Print training info at the end of step. Args: run_context (RunContext): Context of the process running. parallel_mode full_batch)semi_auto_parallel auto_parallel data_parallelF)rgrhiNFalse unavailablerLr RANK_ID0 RANK_SIZE1) ma_step_numma_loss)r<get_auto_parallel_contextr set_auto_parallel_context original_argsr[r\ net_outputsrKrI_fix_loss_for_parallelrZappenddataset_sink_moderP batch_numr_ cur_step_numrR cur_epoch_numrSrTrWrMprint_output_inforr1osgetenvdump_info_to_modelarts)r`r^rgrhrj cb_params step_secondsrwrHrFrGrIrJrPper_step_secondsr_r}r|Z step_remain time_remain throughputpercentr)r)r*step_ends\         .zMFLossMonitor.step_endcCsrtjd}|dkr&|jr&td||jdkr@|jr@td|dkrZ||j|j}n|jdkrn||j}|S)z3Fix loss value in pipeline or double parallel mode.pipeline_stagesrLz 1, the loss on the last card is valid.zHmicro_batch_interleave_num: %s > 1, multiple copies in parallel is open.)r<r rtrYrwarningrOrX)r`rHrr)r)r*rxs    z$MFLossMonitor._fix_loss_for_parallelcCs |jdurt|jttfr&t|j}qt|jtrtjddkr^|j rXt dd|_ d}q|j durr|j j }n |jj j }|jj}|jd||}|j|t|}q|j rt dt|jd|_ d}n|j rt dd|_ d}|dur|jr*t d|||||t||| | | n,t d |||||t|jt||| | | d d td | d d }t d| ||tjt| dn|jrt d|||||t|| | n(t d|||||t|jt|| | d d td | d d }t d| ||tjt| ddS)zprint output information.N device_targetCPUzdevice target not support CPU when generating the learning rate value, please use: mindspore.context.set_context(device_target='Ascend')FzThe current learning rate cannot be calculated in real time.Only the type of LearningRateSchedule is supported in the callback of MFLossMonitor,but the input learning rate function type is %szMFLossMonitor callback is not set learning rate arguments.To display the learning rate, you must input the arguments, which can be LearningRateSchedule or a fixed floatz}{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %sz{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %sz|%%-%ds|2r u▮z# %4.1f%% %s %.2f samples/s/p %s })secondszf{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, per_step_time: %dms, overflow cond: %s, loss_scale: %szm{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], per_step_time: %dms, overflow cond: %s, loss_scale: %s)rIr8rAr r0rr<r get_contextrYrr optimizer global_stepnetwork train_networkphase set_trainr> array2stringr=typerzinfor1r@rZdatetime timedelta)r`rr}rPrr|r_rHrrFrGrrrJZ current_lrr origin_phaseZshow_strr)r)r*r~sv                zMFLossMonitor.print_output_infoc Cst|}d}tjtd}tj|s2t|tjtj|dsidddddid d d ddgid id iiiiididddddid d d ddgid id iiiiidd}n>ttj|dd}t |}Wdn1s0Y|dur|dd}||ddd<|ddd dd}| ||g|dd}||ddd<|ddd dd} | ||gttj|dd}t ||Wdn1s0YdS)z5dump modelarts info to display evaluation result pageNZ modelartszmodel_analysis_results.jsonprrHz loss of modelZ current_lossrZ pr_line_chartzloss line chart of modelstep)nameZ x_axis_nameZ y_axis_namecurve)title descriptionvalue line_chart)commonprecision_performanceZfeature_sensitivityZcomputational_performanceZabstract_featureZ adversaryu 模型损失u 当前loss)en-uszh-cnrrrr loss_valuerrrw) rArpathjoinrexistsmkdiropenjsonloadrydump) r`rrrsobjZ modelarts_dirfpZen_precision_performanceZ en_loss_listZzh_precision_performanceZ zh_loss_listr)r)r*rEsl      #(   z$MFLossMonitor.dump_info_to_modelarts) NrLrLrLNNrrr)r,r-r.r/rrrArr1rVrdrerfrrxr~r __classcell__r)r)rar*r_s8  ?Lrc@seZdZdZdddZdS)rz"Summary Monitor For AICC and LocalN Tc CsF|dur"ttdd} td| }||||||||d} tfi| S)Nrnrosummary) summary_dir collect_freqcollect_specified_datakeep_default_actioncustom_lineage_datacollect_tensor_freq max_file_sizeexport_options)r1rrrr) r(rrrrrrrrrank_idkwargsr)r)r*r+s   zSummaryMonitor.__new__)NrNTNNNN)r,r-r.r/r+r)r)r)r*rsrcs,eZdZdZdfd d Zdd d ZZS)rz%Checkpoint Monitor For Save LossScaleCKPNrLrr5TFAES-GCMc s||_| |_| |_ttdd|_|d|j}| durLdddddg} |rftj |d|jn t d|j}|rtj |d |jn t d |j|_ |rtj |d |jn t d |j|_ t d dvrtdd}t|||||| | | |||d }tt|j|||ddS)Nrnroz_rank_{}rrL) epoch_numstep_numr loss_scalezcheckpoint/rank_ checkpointzcheckpoint_network/rank_Zcheckpoint_networkzcheckpoint_trainable/rank_Zcheckpoint_trainablerg)rirjhybrid_parallelz=Integrated_save is changed to False when using auto_parallel.F) save_checkpoint_stepssave_checkpoint_secondskeep_checkpoint_maxkeep_checkpoint_per_n_minutesintegrated_save async_save saved_network append_infoenc_keyenc_modeexception_save)config)rsave_network_paramssave_trainable_paramsr1rrrformatrrrnetwork_directorytrainable_directoryr rtrrr rUrrV)r`prefix directoryrrrrrrrrrrrrrrZckpt_directoryZ config_ckrar)r*rVsR     zCheckpointMointor.__init__c stjjkrdSjr |}tjdjd}|rptdj dt j dt |d}j jj jjrdjjkrj jkrnn j nPjjrjjdkrt_jjjjdkrj jjjjatjj|}t_j_td rhtd d d krhtj j !d j"vrj#j j"d <d j"vrj$j jj"d <j%durƈj%j&j"d<nj'j%j&j"d<dj"vrj(}t)|t*t+frt,|dkr|dj"d<jj-dur*jj-nj }t.||jj/jj0j"jj1jj2fdd} | |_3dS)zSave checkpoint files.NrLz......Saving ckpt......-_.ckptr< enable_geZMS_ENABLE_REF_MODErorrrrr3c sXj}|jdur|j}jrrjddd}tjj|}tjjddt ||j j j j ij j j jjrT|t}|D]}|||j<qg}|D]L\}}d|i}t|j} ||jvrt||| j j } | |d<||q|}jddd}tjj|}tjjddt ||j j j j ij j j jdS) NrrrT)exist_okrdatatrainable_params)rrr_prefixrrrrmakedirsr _configrrrrrinit_parameters_datarrritemsr rr=parameter_layout_dictrryr) save_objZcb_cur_ckpoint_fileZ cb_cur_file param_dictparam param_listkeyr each_param param_datarr`r)r*save_only_network_paramss<      z>CheckpointMointor._save_ckpt..save_only_network_params)4r|_last_triggered_step_need_flush_from_cache_flush_from_cache_check_save_ckptr1r{rrrr0r}_managerupdate_ckpoint_filelist _directoryrr ckpoint_numremove_oldest_ckpoint_filerr[_cur_time_for_keep_last_time_for_keepkeep_one_ckpoint_per_minutesSAVE_DIRrrrr rrrrexec_checkpoint_graph _append_dict_append_epoch_num_append_step_numrrrrwr8r9r:r;rr rrrr_latest_ckpt_file_name) r`r force_to_save save_ckptstep_num_in_epochcur_ckpoint_filecur_fileoutputsrrr)rr* _save_ckptsb    (            $zCheckpointMointor._save_ckpt)rNNrLrr5rTTFFNNNrF)F)r,r-r.r/rVr rr)r)rar*rs&5rcs2eZdZdZd fdd Zd d Zd d ZZS)rz' Profile analysis in training. rLrNTFc  stt|||_||_||_||_|r6|s6td|durXtt dd}t d|}t ddkrx|rxtdd}tf||||d ||_d|_||_dS) NzCWhen profile_communication is True, start_profile must also be TruernroprofilerGPUzXThe parameter profile_memory is not supported on GPU currently, so is changed to False. F) start_profile output_pathprofile_communicationprofile_memory)rUrrV start_step stop_steprrrDr1rrrr<rrrrprofilerr^r) r`rrrrrrrrrar)r*rV@s*  zProfileMonitor.__init__cCs,|}|j}||jkr(|js(|jdS)z Start profile at the begin of step. Args: run_context (RunContext): Context of the train running. N)rvr|rrrstartr`r^rrr)r)r*rf[szProfileMonitor.step_begincCsB|}|j}||jkr>|j|jtd|j|jdS)z Stop profile at the end of step. Args: run_context (RunContext): Context of the train running. zEnd of Profiling, please view the profile data under %s and analyze it using mindinsight.MindInsight order as follow: mindinsight start --summary-base-dir %sN) rvr|rrstopanalyserrrrr)r)r*rgs   zProfileMonitor.step_end)rLrNTFT)r,r-r.r/rVrfrrr)r)rar*r:s rc@s<eZdZdZdeeedddZddZd d Zd d Z d S)raEvaluate Callback used in training progress. Args: eval_func (Callable): the function to calculate eval result, task specific. step_interval (int): determine the num of step intervals between each eval. Default -1, means only eval on epoch end, do not eval between steps. Note that it will not take effects when running in data sink mode. epoch_interval (int): determine the num of epoch intervals between each eval. Default 1, means eval on every epoch end. r r! eval_func step_intervalepoch_intervalcCs||_||_||_dSr'r)r`rrrr)r)r*rVszEvalCallBack.__init__cCs6|jdkrdS|}|j}||jdkr2|dSNr)rrvr} _execute_eval)r`r^callback_paramsr}r)r)r*res  zEvalCallBack.epoch_endcCs6|jdkrdS|}|j}||jdkr2|dSr)rrvr|r)r`r^r r|r)r)r*rs  zEvalCallBack.step_endcCs.t}|}t|}td||dS)Nz#Eval result: %s, eval time is %f s.)r[rrr)r` start_timerEZ eval_timer)r)r*rs zEvalCallBack._execute_evalN)r r!) r,r-r.r/rr1rVrerrr)r)r)r*rxs    r)9r/rrr[r collectionsrcopyrtypingrrrnumpyr> mindsporer<rrr r r r r mindspore.train.callbackr#mindspore.nn.learning_rate_scheduler"mindspore.train.callback._callbackrmindspore.train.serializationrmindformers.tools.registerrrZ-mindformers.tools.cloud_adapter.cloud_adapterrmindformers.tools.loggerrmindformers.tools.utilsrrrr__all__getcwd_cur_dirrregisterCALLBACKrrKrrrrrr)r)r)r*sJ  $       # %   = pengcheng.mind-test/mindformers/core/callback/build_callback.py000066400000000000000000000063551471400134000252200ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Callback API.""" import inspect from mindspore.train import callback from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_callback( config: [dict, list] = None, default_args: dict = None, module_type: str = 'callback', class_name: str = None, **kwargs): r"""Build callback For MindFormer. Instantiate the callback from MindFormerRegister's registry. Args: config (dict, list): The task callback's config. Default: None. default_args (dict): The default argument of callback API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'callback'. class_name (str): The class name of callback API. Default: None. Return: The function instance of callback API. Examples: >>> from mindformers import build_callback >>> callback_config = [{'type': 'MFLossMonitor'}, {'type': 'SummaryMonitor'}] >>> # 1) use config dict to build callback >>> callback_from_config = build_callback(callback_config) >>> # 2) use class name to build callback >>> callback_class_name = build_callback(class_name='MFLossMonitor') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) cfg_callback = config if not isinstance(cfg_callback, list): return MindFormerRegister.get_instance_from_cfg( cfg_callback, MindFormerModuleType.CALLBACK, default_args=default_args) callbacks = [] for callback_type in cfg_callback: if callback_type.get('type') == 'MFLossMonitor': callback_op = MindFormerRegister.get_instance_from_cfg( callback_type, MindFormerModuleType.CALLBACK, default_args) else: callback_op = MindFormerRegister.get_instance_from_cfg( callback_type, MindFormerModuleType.CALLBACK) callbacks.append(callback_op) return callbacks return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_cb(): """ register MindSpore builtin LR class. """ for module_name in dir(callback): if module_name.startswith('__'): continue monitor = getattr(callback, module_name) if inspect.isclass(monitor): MindFormerRegister.register_cls( monitor, MindFormerModuleType.CALLBACK) register_ms_cb() pengcheng.mind-test/mindformers/core/callback/callback.py000066400000000000000000000760741471400134000240460ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormer Self-Define Callback.""" import json import os import time import datetime from collections import OrderedDict from copy import deepcopy from typing import Callable, Optional, Union import numpy as np import mindspore as ms from mindspore import Callback, Profiler, ModelCheckpoint, CheckpointConfig, context, save_checkpoint, Tensor from mindspore.train.callback import SummaryCollector from mindspore.nn.learning_rate_schedule import LearningRateSchedule from mindspore.train.callback._callback import set_cur_net from mindspore.train.serialization import _get_merged_param_data from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.cloud_adapter.cloud_adapter import Local2ObsMonitor from mindformers.tools.logger import logger from mindformers.tools.utils import get_output_root_path, get_output_subpath, get_remote_save_url, check_in_modelarts __all__ = ['ObsMonitor', 'MFLossMonitor', 'CheckpointMointor', 'SummaryMonitor', 'ProfileMonitor', 'EvalCallBack'] _cur_dir = os.getcwd() SAVE_DIR = _cur_dir @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class ObsMonitor: """Obs Monitor For AICC and Local""" def __new__(cls, src_dir: str = None, target_dir: str = None, step_upload_frequence: int = 100, epoch_upload_frequence: int = -1, keep_last: bool = True): if src_dir is None: src_dir = get_output_root_path() if target_dir is None: target_dir = get_remote_save_url() return Local2ObsMonitor(src_dir, target_dir, step_upload_frequence, epoch_upload_frequence, keep_last) def _get_loss_output(output): """Get output of task for MFLossMonitor.""" overflow = False scaling_sens = False loss = output learning_rate = None grad_norm = None if isinstance(output, (tuple, list)): if len(output) == 3: loss, overflow, scaling_sens = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() elif len(output) == 4: loss, overflow, scaling_sens, learning_rate = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() elif len(output) == 5: loss, overflow, scaling_sens, learning_rate, grad_norm = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() else: if isinstance(output[0], ms.Tensor) and isinstance(output[0].asnumpy(), np.ndarray): loss = output[0] if isinstance(loss, ms.Tensor) and isinstance(loss.asnumpy(), np.ndarray): loss = np.mean(loss.asnumpy()) # Boundary check. if isinstance(loss, float) and (np.isnan(loss) or np.isinf(loss)): invalid_loss_info = "NaN" if np.isnan(loss) else "Inf" raise ValueError(f"The current value of loss is {invalid_loss_info}, terminate training.") return loss, overflow, scaling_sens, learning_rate, grad_norm @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class MFLossMonitor(Callback): """ Loss Monitor for classification. Args: learning_rate (Union[float, LearningRateSchedule], optional): The learning rate schedule. Default: None. per_print_times (int): Every how many steps to print the log information. Default: 1. micro_batch_num (int): MicroBatch size for Pipeline Parallel. Default: 1. micro_batch_interleave_num (int): split num of batch size. Default: 1. origin_epochs (int): Training epoches. Default: None. dataset_size (int): Training dataset size. Default: None. Examples: >>> from mindformers.core.callback import MFLossMonitor >>> lr = [0.01, 0.008, 0.006, 0.005, 0.002] >>> monitor = MFLossMonitor(per_print_times=10) """ def __init__(self, learning_rate: Optional[Union[float, LearningRateSchedule]] = None, per_print_times: int = 1, micro_batch_num: int = 1, micro_batch_interleave_num: int = 1, origin_epochs: int = None, dataset_size: int = None, initial_epoch: int = 0, global_batch_size: int = 0, device_num: int = 0): super(MFLossMonitor, self).__init__() self.per_print_times = per_print_times self.learning_rate = deepcopy(learning_rate) self.last_print_time = 0 self.mirco_size = micro_batch_num self.print_warning_flag = True self.loss_list = [] self.step_time = time.time() self.epoch_time = time.time() self.run_context = None self.steps_per_epoch = dataset_size self.micro_batch_interleave_num = micro_batch_interleave_num self.origin_epochs = origin_epochs self.initial_epoch = initial_epoch self.global_batch_size = global_batch_size self.device_num = device_num def epoch_begin(self, run_context): """ Record time at the beginning of epoch. Args: run_context (RunContext): Context of the process running. """ self.loss_list = [] self.epoch_time = time.time() self.run_context = run_context def epoch_end(self, run_context): """ Print training info at the end of epoch. Args: run_context (RunContext): Context of the process running. """ def step_begin(self, run_context): """ Record time at the beginning of step. Args: run_context (RunContext): Context of the process running. """ self.step_time = time.time() self.run_context = run_context def step_end(self, run_context): """ Print training info at the end of step. Args: run_context (RunContext): Context of the process running. """ parallel_mode = ms.get_auto_parallel_context("parallel_mode") full_batch = ms.get_auto_parallel_context("full_batch") auto_parallel = parallel_mode in ['semi_auto_parallel', 'auto_parallel'] if auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) cb_params = run_context.original_args() step_seconds = (time.time() - self.step_time) * 1000 net_outputs = cb_params.net_outputs loss, overflow, scaling_sens, learning_rate, grad_norm = _get_loss_output(net_outputs) if learning_rate is not None: self.learning_rate = learning_rate loss = self._fix_loss_for_parallel(loss) self.loss_list.append(loss) if not overflow: overflow = "False" if not scaling_sens: scaling_sens = "unavailable" if cb_params.dataset_sink_mode: origin_epochs = self.origin_epochs per_step_seconds = step_seconds / cb_params.batch_num steps_per_epoch = self.steps_per_epoch cur_epoch_num = (cb_params.cur_step_num - 1) // steps_per_epoch \ + self.initial_epoch * cb_params.batch_num // steps_per_epoch + 1 cur_step_num = (cb_params.cur_step_num - 1) % steps_per_epoch + 1 else: origin_epochs = self.origin_epochs per_step_seconds = step_seconds steps_per_epoch = cb_params.batch_num cur_epoch_num = cb_params.cur_epoch_num cur_step_num = (cb_params.cur_step_num - 1) % cb_params.batch_num + 1 # compute time remaining # step_remain = (origin_epochs - cur_epoch_num + 1) * steps_per_epoch - cur_step_num # time_remain = step_remain * per_step_seconds / 1000 time_remain = (steps_per_epoch - cur_step_num) * per_step_seconds / 1000 # compute throughput throughput = self.global_batch_size / self.device_num / (per_step_seconds / 1000) # compute percent # percent = ((cur_epoch_num - 1) * steps_per_epoch + cur_step_num) / origin_epochs / steps_per_epoch * 100 percent = cur_step_num / steps_per_epoch * 100 if (cb_params.cur_step_num - self.last_print_time) >= self.per_print_times: self.last_print_time = cb_params.cur_step_num self.print_output_info(cb_params, cur_epoch_num, origin_epochs, throughput, cur_step_num, steps_per_epoch, loss, per_step_seconds, overflow, scaling_sens, time_remain, percent, grad_norm) if check_in_modelarts() and int(os.getenv("RANK_ID", "0")) == int(os.getenv("RANK_SIZE", "1")) - 1: self.dump_info_to_modelarts(ma_step_num=cur_step_num, ma_loss=loss) if auto_parallel: ms.context.set_auto_parallel_context(parallel_mode=parallel_mode, full_batch=full_batch) def _fix_loss_for_parallel(self, loss): """Fix loss value in pipeline or double parallel mode.""" pipeline_stages = ms.context.get_auto_parallel_context("pipeline_stages") if pipeline_stages > 1 and self.print_warning_flag: logger.warning("pipeline stages: %s > 1, the loss on the last card is valid.", pipeline_stages) if self.micro_batch_interleave_num > 1 and self.print_warning_flag: logger.warning("micro_batch_interleave_num: %s > 1, multiple copies in parallel is open.") if pipeline_stages > 1: loss = loss / (self.mirco_size * self.micro_batch_interleave_num) elif self.micro_batch_interleave_num > 1: loss = loss / self.micro_batch_interleave_num return loss def print_output_info(self, cb_params, cur_epoch_num, origin_epochs, throughput, cur_step_num, steps_per_epoch, loss, per_step_seconds, overflow, scaling_sens, time_remain, percent, grad_norm): """print output information.""" if self.learning_rate is not None: if isinstance(self.learning_rate, (float, Tensor)): current_lr = str(self.learning_rate) elif isinstance(self.learning_rate, LearningRateSchedule): if ms.context.get_context('device_target') == 'CPU': if self.print_warning_flag: logger.warning( "device target not support CPU when generating the learning rate value, " "please use: mindspore.context.set_context(device_target='Ascend')") self.print_warning_flag = False current_lr = None else: if cb_params.optimizer is not None: global_step = cb_params.optimizer.global_step else: global_step = cb_params.network.optimizer.global_step # temporary set_train to avoid error on 910B origin_phase = cb_params.train_network.phase cb_params.train_network.set_train(False) current_lr = self.learning_rate(global_step) cb_params.train_network.set_train(origin_phase) current_lr = np.array2string(current_lr.asnumpy()) else: if self.print_warning_flag: logger.warning( "The current learning rate cannot be calculated in real time." "Only the type of LearningRateSchedule is supported in the callback of MFLossMonitor," "but the input learning rate function type is %s", type(self.learning_rate) ) self.print_warning_flag = False current_lr = None else: if self.print_warning_flag: logger.warning( "MFLossMonitor callback is not set learning rate arguments." "To display the learning rate, you must input the arguments, " "which can be LearningRateSchedule or a fixed float" ) self.print_warning_flag = False current_lr = None if current_lr is not None: if cb_params.dataset_sink_mode: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, " "per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, int(per_step_seconds), current_lr, overflow, scaling_sens, grad_norm) else: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], " "per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, np.mean(self.loss_list), int(per_step_seconds), current_lr, overflow, scaling_sens, grad_norm) show_str = ('|%%-%ds|' % 50) % (int(50 * percent / 100) * "▮") logger.info(" %4.1f%% %s %.2f samples/s/p %s }", percent, show_str, throughput, datetime.timedelta(seconds=int(time_remain))) else: if cb_params.dataset_sink_mode: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, " "per_step_time: %dms, overflow cond: %s, loss_scale: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, int(per_step_seconds), overflow, scaling_sens) else: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], " "per_step_time: %dms, overflow cond: %s, loss_scale: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, np.mean(self.loss_list), int(per_step_seconds), overflow, scaling_sens) show_str = ('|%%-%ds|' % 50) % (int(50 * percent / 100) * "▮") logger.info(" %4.1f%% %s %.2f samples/s/p %s }", percent, show_str, throughput, datetime.timedelta(seconds=int(time_remain))) def dump_info_to_modelarts(self, ma_step_num, ma_loss): """dump modelarts info to display evaluation result page""" ma_loss = float(ma_loss) obj = None modelarts_dir = os.path.join(get_output_root_path(), "modelarts") if not os.path.exists(modelarts_dir): os.mkdir(modelarts_dir) if not os.path.exists(os.path.join(modelarts_dir, "model_analysis_results.json")): obj = { "en-us": { "common": {}, "precision_performance": { "pr": { "title": "loss", "description": "loss of model", "value": {"current_loss": 0}, "line_chart": { "pr_line_chart": { "name": "loss line chart of model", "x_axis_name": "step", "y_axis_name": "loss", "curve": {"loss": []}}}}}, "feature_sensitivity": {}, "computational_performance": {}, "abstract_feature": {}, "adversary": {} }, "zh-cn": { "common": {}, "precision_performance": { "pr": { "title": "loss", "description": "模型损失", "value": {"当前loss": 0}, "line_chart": { "pr_line_chart": { "name": "loss line chart of model", "x_axis_name": "step", "y_axis_name": "loss", "curve": {"loss": []}}}}}, "feature_sensitivity": {}, "computational_performance": {}, "abstract_feature": {}, "adversary": {} } } else: with open(os.path.join(modelarts_dir, "model_analysis_results.json"), "r") as fp: obj = json.load(fp) if obj is not None: en_precision_performance = obj["en-us"]["precision_performance"] en_precision_performance["pr"]["value"]["loss_value"] = ma_loss en_loss_list = en_precision_performance["pr"]["line_chart"]["pr_line_chart"]["curve"]["loss"] en_loss_list.append([ma_step_num, ma_loss]) zh_precision_performance = obj["zh-cn"]["precision_performance"] zh_precision_performance["pr"]["value"]["当前loss"] = ma_loss zh_loss_list = zh_precision_performance["pr"]["line_chart"]["pr_line_chart"]["curve"]["loss"] zh_loss_list.append([ma_step_num, ma_loss]) with open(os.path.join(modelarts_dir, "model_analysis_results.json"), "w") as fp: json.dump(obj, fp) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class SummaryMonitor: """Summary Monitor For AICC and Local""" def __new__(cls, summary_dir=None, collect_freq=10, collect_specified_data=None, keep_default_action=True, custom_lineage_data=None, collect_tensor_freq=None, max_file_size=None, export_options=None): if summary_dir is None: rank_id = int(os.getenv("RANK_ID", '0')) summary_dir = get_output_subpath('summary', rank_id) kwargs = { "summary_dir": summary_dir, "collect_freq": collect_freq, "collect_specified_data": collect_specified_data, "keep_default_action": keep_default_action, "custom_lineage_data": custom_lineage_data, "collect_tensor_freq": collect_tensor_freq, "max_file_size": max_file_size, "export_options": export_options } return SummaryCollector(**kwargs) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class CheckpointMointor(ModelCheckpoint): """Checkpoint Monitor For Save LossScale""" def __init__(self, prefix='CKP', directory=None, config=None, save_checkpoint_steps=1, save_checkpoint_seconds=0, keep_checkpoint_max=5, keep_checkpoint_per_n_minutes=0, integrated_save=True, save_network_params=True, save_trainable_params=False, async_save=False, saved_network=None, append_info=None, enc_key=None, enc_mode='AES-GCM', exception_save=False): self.config = config self.save_network_params = save_network_params self.save_trainable_params = save_trainable_params self.rank_id = int(os.getenv("RANK_ID", '0')) prefix = prefix + "_rank_{}".format(self.rank_id) if append_info is None: append_info = [{ "epoch_num": 0, "step_num": 0, "global_step": 0, "loss_scale": 1 }] ckpt_directory = os.path.join(directory, f"checkpoint/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint', self.rank_id) self.network_directory = os.path.join(directory, f"checkpoint_network/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint_network', self.rank_id) self.trainable_directory = os.path.join(directory, f"checkpoint_trainable/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint_trainable', self.rank_id) if context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: logger.info("Integrated_save is changed to False when using auto_parallel.") integrated_save = False config_ck = CheckpointConfig(save_checkpoint_steps=save_checkpoint_steps, save_checkpoint_seconds=save_checkpoint_seconds, keep_checkpoint_max=keep_checkpoint_max, keep_checkpoint_per_n_minutes=keep_checkpoint_per_n_minutes, integrated_save=integrated_save, async_save=async_save, saved_network=saved_network, append_info=append_info, enc_key=enc_key, enc_mode=enc_mode, exception_save=exception_save) super(CheckpointMointor, self).__init__(prefix, ckpt_directory, config=config_ck) def _save_ckpt(self, cb_params, force_to_save=False): """Save checkpoint files.""" # pylint: disable=E0203 if cb_params.cur_step_num == self._last_triggered_step: return # if param is cache enable, flush data from cache to host before save_ckpt if self._need_flush_from_cache: self._flush_from_cache(cb_params) save_ckpt = self._check_save_ckpt(cb_params, force_to_save) step_num_in_epoch = int((cb_params.cur_step_num - 1) % cb_params.batch_num + 1) if save_ckpt: logger.info('......Saving ckpt......') cur_ckpoint_file = self._prefix + "-" + str(cb_params.cur_epoch_num) + "_" \ + str(step_num_in_epoch) + ".ckpt" # update checkpoint file list. self._manager.update_ckpoint_filelist(self._directory, self._prefix) # keep checkpoint files number equal max number. if self._config.keep_checkpoint_max and 0 < self._config.keep_checkpoint_max <= self._manager.ckpoint_num: self._manager.remove_oldest_ckpoint_file() elif self._config.keep_checkpoint_per_n_minutes and self._config.keep_checkpoint_per_n_minutes > 0: # pylint: disable=E0203 self._cur_time_for_keep = time.time() if (self._cur_time_for_keep - self._last_time_for_keep) \ < self._config.keep_checkpoint_per_n_minutes * 60: self._manager.keep_one_ckpoint_per_minutes(self._config.keep_checkpoint_per_n_minutes, self._cur_time_for_keep) # generate the new checkpoint file and rename it. global SAVE_DIR SAVE_DIR = self._directory cur_file = os.path.join(self._directory, cur_ckpoint_file) self._last_time_for_keep = time.time() self._last_triggered_step = cb_params.cur_step_num if context.get_context("enable_ge") and os.getenv("MS_ENABLE_REF_MODE", "0") == "0": set_cur_net(cb_params.train_network) cb_params.train_network.exec_checkpoint_graph() if "epoch_num" in self._append_dict: self._append_dict["epoch_num"] = self._append_epoch_num + cb_params.cur_epoch_num if "step_num" in self._append_dict: self._append_dict["step_num"] = self._append_step_num + cb_params.cur_epoch_num * cb_params.batch_num if cb_params.optimizer is not None: self._append_dict["global_step"] = cb_params.optimizer.global_step else: self._append_dict["global_step"] = cb_params.network.optimizer.global_step if "loss_scale" in self._append_dict: outputs = cb_params.net_outputs if isinstance(outputs, (tuple, list)) and len(outputs) >= 3: self._append_dict["loss_scale"] = outputs[2] network = self._config.saved_network if self._config.saved_network is not None else cb_params.train_network save_checkpoint(network, cur_file, self._config.integrated_save, self._config.async_save, self._append_dict, self._config.enc_key, self._config.enc_mode) def save_only_network_params(): save_obj = cb_params.network if save_obj.optimizer is not None: save_obj = save_obj.network if self.save_network_params: cb_cur_ckpoint_file = self._prefix + "-" + "network" + ".ckpt" cb_cur_file = os.path.join(self.network_directory, cb_cur_ckpoint_file) os.makedirs(self.network_directory, exist_ok=True) save_checkpoint(save_obj, cb_cur_file, self._config.integrated_save, self._config.async_save, {}, self._config.enc_key, self._config.enc_mode) if self.save_trainable_params: save_obj.init_parameters_data() param_dict = OrderedDict() for param in save_obj.trainable_params(): param_dict[param.name] = param param_list = [] for (key, value) in param_dict.items(): each_param = {"name": key} param_data = Tensor(value.data.asnumpy()) # in automatic model parallel scenario, some parameters were split to all the devices, # which should be combined before saving if key in save_obj.parameter_layout_dict: param_data = _get_merged_param_data(save_obj, key, param_data, self._config.integrated_save) each_param["data"] = param_data param_list.append(each_param) save_obj = param_list cb_cur_ckpoint_file = self._prefix + "-" + "trainable_params" + ".ckpt" cb_cur_file = os.path.join(self.trainable_directory, cb_cur_ckpoint_file) os.makedirs(self.trainable_directory, exist_ok=True) save_checkpoint(save_obj, cb_cur_file, self._config.integrated_save, self._config.async_save, {}, self._config.enc_key, self._config.enc_mode) save_only_network_params() self._latest_ckpt_file_name = cur_file @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class ProfileMonitor(Callback): """ Profile analysis in training. """ def __init__(self, start_step=1, stop_step=10, output_path=None, start_profile=True, profile_communication=False, profile_memory=True, **kwargs): super(ProfileMonitor, self).__init__() self.start_step = start_step self.stop_step = stop_step self.start_profile = start_profile self.profile_communication = profile_communication if profile_communication and not start_profile: raise ValueError("When profile_communication is True, start_profile must also be True") if output_path is None: rank_id = int(os.getenv("RANK_ID", '0')) output_path = get_output_subpath('profile', rank_id) if ms.get_context("device_target") == "GPU" and profile_memory: logger.warning("The parameter profile_memory is not supported on GPU currently, so is changed to False. ") profile_memory = False self.profiler = Profiler( start_profile=start_profile, output_path=output_path, profile_communication=profile_communication, profile_memory=profile_memory, **kwargs) self.run_context = None self.output_path = output_path def step_begin(self, run_context): """ Start profile at the begin of step. Args: run_context (RunContext): Context of the train running. """ cb_params = run_context.original_args() step_num = cb_params.cur_step_num if step_num == self.start_step and not self.start_profile: self.profiler.start() def step_end(self, run_context): """ Stop profile at the end of step. Args: run_context (RunContext): Context of the train running. """ cb_params = run_context.original_args() step_num = cb_params.cur_step_num if step_num == self.stop_step: self.profiler.stop() self.profiler.analyse() logger.info("End of Profiling, please view the profile data under %s and analyze it using mindinsight." "MindInsight order as follow: " "mindinsight start --summary-base-dir %s", self.output_path, self.output_path) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class EvalCallBack(Callback): """Evaluate Callback used in training progress. Args: eval_func (Callable): the function to calculate eval result, task specific. step_interval (int): determine the num of step intervals between each eval. Default -1, means only eval on epoch end, do not eval between steps. Note that it will not take effects when running in data sink mode. epoch_interval (int): determine the num of epoch intervals between each eval. Default 1, means eval on every epoch end. """ def __init__(self, eval_func: Callable, step_interval: int = 100, epoch_interval: int = -1): self.eval_func = eval_func self.step_interval = step_interval self.epoch_interval = epoch_interval def epoch_end(self, run_context): # if not use epoch end if self.epoch_interval <= 0: return callback_params = run_context.original_args() cur_epoch_num = callback_params.cur_epoch_num if cur_epoch_num % self.epoch_interval == 0: self._execute_eval() def step_end(self, run_context): # if not use step end if self.step_interval <= 0: return callback_params = run_context.original_args() cur_step_num = callback_params.cur_step_num if cur_step_num % self.step_interval == 0: self._execute_eval() def _execute_eval(self): start_time = time.time() output = self.eval_func() eval_time = time.time() - start_time logger.info("Eval result: %s, eval time is %f s.", output, eval_time) pengcheng.mind-test/mindformers/core/clip_grad.py000066400000000000000000000072451471400134000224740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Operations for clipping grad by global norm.""" import numpy as np from mindspore import ops from mindspore.nn.cell import Cell from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.common.tensor import Tensor from mindspore.common import dtype as mstype # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator as validator from mindspore._checkparam import Rel except ImportError: import mindspore._checkparam as validator import mindspore._checkparam as Rel # The attribute grad_scale is needed for enabling the parallel mode # If this is removed, c.clip_by_global_norm will have precision error in semi/auto parallel mode. expand_dims = P.ExpandDims().add_prim_attr("grad_scale", True) get_square_sum = C.MultitypeFuncGraph("get_square_sum") apply_global_norm = C.MultitypeFuncGraph("apply_global_norm") @get_square_sum.register("Tensor") def _get_square_sum(x): norm = P.ReduceSum(False)(F.square(x), ()) norm = expand_dims(F.cast(norm, mstype.float32), 0) return norm @apply_global_norm.register("Tensor", "Tensor", "Tensor") def _apply_global_norm(clip_norm, global_norm, x): x_dtype = F.dtype(x) clip_coef = clip_norm / (global_norm + 1e-6) clip_coef_clamped = ops.clip_by_value(clip_coef, clip_value_max=Tensor(1.0, mstype.float32), clip_value_min=Tensor(np.log(0.0), mstype.float32)) x = x * clip_coef_clamped x = F.cast(x, x_dtype) return x class ClipGradNorm(Cell): r""" Clips tensor values by the ratio of the sum of their norms. Args: max_norm (Union(float, int)): The clipping ratio. Default: 1.0 use_norm (Union(float, None)): The global norm. Default: None Inputs: - **x** (Union(tuple[Tensor], list[Tensor])) - Input data to clip. Outputs: Tensor, a clipped Tensor. """ def __init__(self, max_norm=1.0, use_norm=None): super(ClipGradNorm, self).__init__() # Add interface. This parameter is not used at present if use_norm is not None: raise ValueError(f"For '{self.cls_name}', input 'use_norm' only supports None currently, " f"but got 'use_norm': {use_norm}") validator.check_number("clip_norm", max_norm, 0.0, Rel.GT, self.cls_name) self.clip_norm = Tensor([max_norm], mstype.float32) self.hyper_map = C.HyperMap() self.greater_equal = P.GreaterEqual() def construct(self, x): """clip grad.""" square_sum = self.hyper_map(get_square_sum, x) global_norm = F.sqrt(F.addn(square_sum)) origin_global_norm = global_norm cond = self.greater_equal(global_norm, self.clip_norm) global_norm = F.select(cond, global_norm, self.clip_norm) clip_x = self.hyper_map(F.partial(apply_global_norm, self.clip_norm, global_norm), x) return clip_x, origin_global_norm pengcheng.mind-test/mindformers/core/context/000077500000000000000000000000001471400134000216525ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/context/.ipynb_checkpoints/000077500000000000000000000000001471400134000254435ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/context/.ipynb_checkpoints/build_context-checkpoint.py000066400000000000000000000162321471400134000330110ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build context.""" import os from mindspore import context from mindspore.communication.management import init, get_group_size, get_rank from mindspore.parallel import set_algo_parameters from mindspore.parallel._cost_model_context import _set_multi_subgraphs from mindformers.core.callback import ProfileMonitor from mindformers.trainer.config_args import ContextConfig, ParallelContextConfig from mindformers.tools.register import MindFormerConfig from mindformers.tools import PARALLEL_MODE, MODE, get_output_subpath,\ get_output_root_path, check_in_modelarts from mindformers.tools.logger import logger CONTEXT_CONFIG = { 'mode': 'GRAPH_MODE', 'device_target': 'Ascend', 'device_id': 0, 'save_graphs': False} PARALLEL_CONFIG = {'parallel_mode': 'DATA_PARALLEL', 'gradients_mean': True} def build_context(config): """Build context.""" if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.use_parallel and config.parallel_config.pipeline_stage > 1: config.parallel.pipeline_stages = config.parallel_config.pipeline_stage local_rank, device_num = init_context(use_parallel=config.use_parallel, context_config=config.context, parallel_config=config.parallel) set_algo_parameters(elementwise_op_strategy_follow=True, fully_use_devices=True) _set_multi_subgraphs() config.device_num = device_num config.local_rank = local_rank if config.parallel.get("strategy_ckpt_load_file"): context.set_auto_parallel_context(strategy_ckpt_load_file=config.parallel.strategy_ckpt_load_file) def init_context(use_parallel=True, context_config=None, parallel_config=None): """Context initialization for MindSpore.""" if isinstance(context_config, ContextConfig): context_config = context_config.__dict__ if isinstance(parallel_config, ParallelContextConfig): parallel_config = parallel_config.__dict__ if context_config is None: context_config = CONTEXT_CONFIG if parallel_config is None: parallel_config = PARALLEL_CONFIG _set_check_context_config(context_config) _set_check_parallel_config(parallel_config) device_num = 1 rank_id = 0 context_config['mode'] = MODE.get(context_config.get('mode')) context.set_context(max_device_memory=context_config.get('max_device_memory'), mode=context_config.get('mode')) del context_config['mode'] del context_config['max_device_memory'] if use_parallel: init() device_id = int(os.getenv('DEVICE_ID', '0')) # 0 ~ 7 rank_id = get_rank() # local_rank device_num = get_group_size() # world_size context_config['device_id'] = device_id parallel_config['parallel_mode'] = PARALLEL_MODE.get(parallel_config.get('parallel_mode')) parallel_config.setdefault('device_num', device_num) context.set_context(**context_config) context.reset_auto_parallel_context() context.set_auto_parallel_context(**parallel_config) else: context.set_context(**context_config) return rank_id, device_num def build_profile_cb(config): """build profile callback from config.""" start_profile = config.init_start_profile profile_communication = config.profile_communication if config.device_num > 1: logger.info("Device number is %s > 1, so profile_communication and start_profile will be set True ") start_profile = True profile_communication = True profile_cb = ProfileMonitor( start_step=config.profile_start_step, stop_step=config.profile_stop_step, start_profile=start_profile, profile_communication=profile_communication, profile_memory=config.profile_memory) logger.warning( "In profiler mode, data sink mode will be turned off. " "Please reduce the data sample size with 'num_samples' in MindSpore data format according to " "https://www.mindspore.cn/mindinsight/docs/zh-CN/master/performance_profiling_ascend.html.") logger.warning("In profiler mode, auto-tune will be turned off.") config.runner_config.sink_mode = False config.auto_tune = False return profile_cb def _set_check_context_config(config): """Set context config.""" mode = config.get('mode') if mode is None: config.setdefault('mode', 0) if mode not in MODE.keys(): raise IndexError('Running mode should be in {}, but get {}'.format(MODE.keys(), mode)) device = config.get('device_id') if device is None: config.setdefault('device_id', 0) max_device_memory = config.get('max_device_memory') if max_device_memory is None: config.setdefault('max_device_memory', '30GB') save_graph = config.get('save_graphs') if save_graph: save_graphs_path = config.get('save_graphs_path') if not check_in_modelarts() else None if save_graphs_path is None: save_graphs_path = get_output_subpath("debug/graphs_info", append_rank=False) config['save_graphs_path'] = save_graphs_path enable_dump = config.get('enable_dump') if enable_dump: save_dump_path = config.get('save_dump_path') if not check_in_modelarts() else None if save_dump_path is None: save_dump_path = get_output_subpath("debug/dump_info", append_rank=False) config['save_dump_path'] = save_dump_path def _set_check_parallel_config(config): """Set parallel config.""" parallel_mode = config.get('parallel_mode') if parallel_mode is None: config.setdefault('parallel_mode', 0) if PARALLEL_MODE.get(config.get('parallel_mode')) not in \ [context.ParallelMode.SEMI_AUTO_PARALLEL, context.ParallelMode.AUTO_PARALLEL] and config.get('full_batch'): logger.info("full_batch will be forced to False when the parallel mode is stand_alone or data_parallel") config.setdefault('full_batch', False) if parallel_mode not in PARALLEL_MODE.keys(): raise IndexError( 'Running parallel mode should be in {}, but get {}'.format(PARALLEL_MODE.keys(), parallel_mode)) strategy_ckpt_save_file = config.get('strategy_ckpt_save_file', "ckpt_strategy.ckpt") rank_id = int(os.getenv("RANK_ID", "0")) os.makedirs(os.path.join(get_output_root_path(), "strategy"), exist_ok=True) config["strategy_ckpt_save_file"] = os.path.join(get_output_root_path(), "strategy", strategy_ckpt_save_file.replace(".ckpt", f"_rank_{rank_id}.ckpt")) pengcheng.mind-test/mindformers/core/context/__init__.py000066400000000000000000000015011471400134000237600ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers context.""" from .build_context import build_context, build_profile_cb, init_context __all__ = ['build_context', 'build_profile_cb', 'init_context'] pengcheng.mind-test/mindformers/core/context/__pycache__/000077500000000000000000000000001471400134000240625ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/context/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004651471400134000302570ustar00rootroot00000000000000a ?cfA@s$dZddlmZmZmZgdZdS)zMindFormers context.) build_contextbuild_profile_cb init_contextN)__doc__rrr__all__rrL/home/ma-user/work/mindformers-baichuan/mindformers/core/context/__init__.pyspengcheng.mind-test/mindformers/core/context/__pycache__/build_context.cpython-39.pyc000066400000000000000000000114411471400134000313570ustar00rootroot00000000000000a ?cf@sdZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZmZmZmZdd lmZd d dddZdddZddZdddZddZddZ ddZ!dS)Build context.N)context)initget_group_sizeget_rank)set_algo_parameters)_set_multi_subgraphs)ProfileMonitor) ContextConfigParallelContextConfig)MindFormerConfig) PARALLEL_MODEMODEget_output_subpathget_output_root_pathcheck_in_modelarts)logger GRAPH_MODEAscendF)mode device_target device_id save_graphs DATA_PARALLELT) parallel_modegradients_meancCst|tr"t|ts"tfi|}|jr@|jjdkr@|jj|j_t|j|j |jd\}}t dddt ||_ ||_ |jdrt j|jjddS)r) use_parallelcontext_configparallel_configT)elementwise_op_strategy_followfully_use_devicesstrategy_ckpt_load_file)r"N) isinstancedictr rrpipeline_stageparallelpipeline_stages init_contextrrr device_num local_rankgetset_auto_parallel_contextr")configr*r)r.Q/home/ma-user/work/mindformers-baichuan/mindformers/core/context/build_context.py build_context$s    r0cCst|tr|j}t|tr |j}|dur,t}|dur8t}t|t|d}d}t | d|d<t j | d| dd|d=|d=|rt t tdd}t}t}||d <t | d |d <|d |t j fi|t t jfi|nt j fi|||fS) z%Context initialization for MindSpore.Nrrrmax_device_memory)r1r DEVICE_ID0rrr))r#r __dict__r CONTEXT_CONFIGPARALLEL_CONFIG_set_check_context_config_set_check_parallel_configrr+r set_contextrintosgetenvrrr setdefaultreset_auto_parallel_contextr,)rrrr)rank_idrr.r.r/r(6s>    r(cCsf|j}|j}|jdkr(tdd}d}t|j|j|||jd}t dt dd|j _ d|_ |S)z#build profile callback from config.rzUDevice number is %s > 1, so profile_communication and start_profile will be set True T) start_step stop_step start_profileprofile_communicationprofile_memoryzIn profiler mode, data sink mode will be turned off. Please reduce the data sample size with 'num_samples' in MindSpore data format according to https://www.mindspore.cn/mindinsight/docs/zh-CN/master/performance_profiling_ascend.html.z/In profiler mode, auto-tune will be turned off.F) Zinit_start_profilerCr)rinfor Zprofile_start_stepZprofile_stop_steprDwarning runner_config sink_mode auto_tune)r-rBrC profile_cbr.r.r/build_profile_cb^s(   rKcCs|d}|dur|dd|tvr>tdt||d}|dur\|dd|d}|durz|dd|d}|rts|d nd}|durtd d d }||d <|d }|rts|dnd}|durtdd d }||d<dS)zSet context config.rNrz(Running mode should be in {}, but get {}rr1Z30GBrsave_graphs_pathzdebug/graphs_infoF) append_rank enable_dumpsave_dump_pathzdebug/dump_info)r+r=rkeys IndexErrorformatrr)r-rdevicer1Z save_graphrLrNrOr.r.r/r7ws.           r7c Cs|d}|dur|ddt|dtjjtjjfvr^|dr^td|dd|t vr~t d t ||dd }t t d d }t jt jtd d dt jtd |dd|d|d<dS)zSet parallel config.rNr full_batchzYfull_batch will be forced to False when the parallel mode is stand_alone or data_parallelFz1Running parallel mode should be in {}, but get {}strategy_ckpt_save_filezckpt_strategy.ckptRANK_IDr3strategyT)exist_okz.ckpt_rank_)r+r=r r ParallelModeSEMI_AUTO_PARALLEL AUTO_PARALLELrrErPrQrRr:r;r<makedirspathjoinrreplace)r-rrUr?r.r.r/r8s(       r8)TNN)"__doc__r; mindsporer"mindspore.communication.managementrrrZmindspore.parallelrZ&mindspore.parallel._cost_model_contextrZmindformers.core.callbackr Zmindformers.trainer.config_argsr r mindformers.tools.registerr mindformers.toolsr rrrrmindformers.tools.loggerrr5r6r0r(rKr7r8r.r.r.r/s$        (pengcheng.mind-test/mindformers/core/context/build_context.py000066400000000000000000000162321471400134000250730ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build context.""" import os from mindspore import context from mindspore.communication.management import init, get_group_size, get_rank from mindspore.parallel import set_algo_parameters from mindspore.parallel._cost_model_context import _set_multi_subgraphs from mindformers.core.callback import ProfileMonitor from mindformers.trainer.config_args import ContextConfig, ParallelContextConfig from mindformers.tools.register import MindFormerConfig from mindformers.tools import PARALLEL_MODE, MODE, get_output_subpath,\ get_output_root_path, check_in_modelarts from mindformers.tools.logger import logger CONTEXT_CONFIG = { 'mode': 'GRAPH_MODE', 'device_target': 'Ascend', 'device_id': 0, 'save_graphs': False} PARALLEL_CONFIG = {'parallel_mode': 'DATA_PARALLEL', 'gradients_mean': True} def build_context(config): """Build context.""" if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.use_parallel and config.parallel_config.pipeline_stage > 1: config.parallel.pipeline_stages = config.parallel_config.pipeline_stage local_rank, device_num = init_context(use_parallel=config.use_parallel, context_config=config.context, parallel_config=config.parallel) set_algo_parameters(elementwise_op_strategy_follow=True, fully_use_devices=True) _set_multi_subgraphs() config.device_num = device_num config.local_rank = local_rank if config.parallel.get("strategy_ckpt_load_file"): context.set_auto_parallel_context(strategy_ckpt_load_file=config.parallel.strategy_ckpt_load_file) def init_context(use_parallel=True, context_config=None, parallel_config=None): """Context initialization for MindSpore.""" if isinstance(context_config, ContextConfig): context_config = context_config.__dict__ if isinstance(parallel_config, ParallelContextConfig): parallel_config = parallel_config.__dict__ if context_config is None: context_config = CONTEXT_CONFIG if parallel_config is None: parallel_config = PARALLEL_CONFIG _set_check_context_config(context_config) _set_check_parallel_config(parallel_config) device_num = 1 rank_id = 0 context_config['mode'] = MODE.get(context_config.get('mode')) context.set_context(max_device_memory=context_config.get('max_device_memory'), mode=context_config.get('mode')) del context_config['mode'] del context_config['max_device_memory'] if use_parallel: init() device_id = int(os.getenv('DEVICE_ID', '0')) # 0 ~ 7 rank_id = get_rank() # local_rank device_num = get_group_size() # world_size context_config['device_id'] = device_id parallel_config['parallel_mode'] = PARALLEL_MODE.get(parallel_config.get('parallel_mode')) parallel_config.setdefault('device_num', device_num) context.set_context(**context_config) context.reset_auto_parallel_context() context.set_auto_parallel_context(**parallel_config) else: context.set_context(**context_config) return rank_id, device_num def build_profile_cb(config): """build profile callback from config.""" start_profile = config.init_start_profile profile_communication = config.profile_communication if config.device_num > 1: logger.info("Device number is %s > 1, so profile_communication and start_profile will be set True ") start_profile = True profile_communication = True profile_cb = ProfileMonitor( start_step=config.profile_start_step, stop_step=config.profile_stop_step, start_profile=start_profile, profile_communication=profile_communication, profile_memory=config.profile_memory) logger.warning( "In profiler mode, data sink mode will be turned off. " "Please reduce the data sample size with 'num_samples' in MindSpore data format according to " "https://www.mindspore.cn/mindinsight/docs/zh-CN/master/performance_profiling_ascend.html.") logger.warning("In profiler mode, auto-tune will be turned off.") config.runner_config.sink_mode = False config.auto_tune = False return profile_cb def _set_check_context_config(config): """Set context config.""" mode = config.get('mode') if mode is None: config.setdefault('mode', 0) if mode not in MODE.keys(): raise IndexError('Running mode should be in {}, but get {}'.format(MODE.keys(), mode)) device = config.get('device_id') if device is None: config.setdefault('device_id', 0) max_device_memory = config.get('max_device_memory') if max_device_memory is None: config.setdefault('max_device_memory', '30GB') save_graph = config.get('save_graphs') if save_graph: save_graphs_path = config.get('save_graphs_path') if not check_in_modelarts() else None if save_graphs_path is None: save_graphs_path = get_output_subpath("debug/graphs_info", append_rank=False) config['save_graphs_path'] = save_graphs_path enable_dump = config.get('enable_dump') if enable_dump: save_dump_path = config.get('save_dump_path') if not check_in_modelarts() else None if save_dump_path is None: save_dump_path = get_output_subpath("debug/dump_info", append_rank=False) config['save_dump_path'] = save_dump_path def _set_check_parallel_config(config): """Set parallel config.""" parallel_mode = config.get('parallel_mode') if parallel_mode is None: config.setdefault('parallel_mode', 0) if PARALLEL_MODE.get(config.get('parallel_mode')) not in \ [context.ParallelMode.SEMI_AUTO_PARALLEL, context.ParallelMode.AUTO_PARALLEL] and config.get('full_batch'): logger.info("full_batch will be forced to False when the parallel mode is stand_alone or data_parallel") config.setdefault('full_batch', False) if parallel_mode not in PARALLEL_MODE.keys(): raise IndexError( 'Running parallel mode should be in {}, but get {}'.format(PARALLEL_MODE.keys(), parallel_mode)) strategy_ckpt_save_file = config.get('strategy_ckpt_save_file', "ckpt_strategy.ckpt") rank_id = int(os.getenv("RANK_ID", "0")) os.makedirs(os.path.join(get_output_root_path(), "strategy"), exist_ok=True) config["strategy_ckpt_save_file"] = os.path.join(get_output_root_path(), "strategy", strategy_ckpt_save_file.replace(".ckpt", f"_rank_{rank_id}.ckpt")) pengcheng.mind-test/mindformers/core/loss/000077500000000000000000000000001471400134000211465ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/loss/.ipynb_checkpoints/000077500000000000000000000000001471400134000247375ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/loss/.ipynb_checkpoints/build_loss-checkpoint.py000066400000000000000000000047751471400134000316120ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Loss API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_loss( config: dict = None, default_args: dict = None, module_type: str = 'loss', class_name: str = None, **kwargs): r"""Build loss For MindFormer. Instantiate the loss from MindFormerRegister's registry. Args: config (dict): The task loss's config. Default: None. default_args (dict): The default argument of loss API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'loss'. class_name (str): The class name of loss API. Default: None. Return: The function instance of loss API. Examples: >>> from mindformers import build_loss >>> loss_config = {'type': 'L1Loss'} >>> # 1) use config dict to build loss >>> loss_from_config = build_loss(loss_config) >>> # 2) use class name to build loss >>> loss_class_name = build_loss(class_name='L1Loss') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.LOSS, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_loss(): """ register MindSpore builtin loss class. """ for module_name in dir(nn.loss): if module_name.startswith('__'): continue loss = getattr(nn.loss, module_name) if inspect.isclass(loss): MindFormerRegister.register_cls( loss, MindFormerModuleType.LOSS) register_ms_loss() pengcheng.mind-test/mindformers/core/loss/.ipynb_checkpoints/loss-checkpoint.py000066400000000000000000000412271471400134000304240ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormer Self-Define Loss.""" from mindspore import nn, Tensor from mindspore import ops as P from mindspore.ops import functional as F from mindspore.common import dtype as mstype from mindspore.nn.loss.loss import LossBase from mindspore.context import ParallelMode from mindspore.parallel import set_algo_parameters from mindspore import log as logger from mindspore.parallel._utils import _get_device_num, _get_pipeline_stages, _get_parallel_mode, _is_sharding_propagation from mindformers.tools.logger import _LogActionOnce from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.modules.transformer.op_parallel_config import default_dpmp_config __all__ = ['SoftTargetCrossEntropy', 'MSELoss', 'L1Loss', 'CrossEntropyLoss', 'CompareLoss'] @MindFormerRegister.register(MindFormerModuleType.LOSS) class SoftTargetCrossEntropy(LossBase): """SoftTargetCrossEntropy for MixUp Augment.""" def __init__(self, parallel_config=default_dpmp_config): super(SoftTargetCrossEntropy, self).__init__() dp = parallel_config.data_parallel self.mean_ops = P.ReduceMean(keep_dims=False).shard(((1,),)) self.sum_ops = P.ReduceSum(keep_dims=False).shard(((dp, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.mul1d = P.Mul().shard(((dp, 1), ())) self.log_softmax = P.LogSoftmax().shard(((dp, 1),)) def construct(self, logit, label): logit = P.Cast()(logit, mstype.float32) label = P.Cast()(label, mstype.float32) logit_softmax = self.log_softmax(logit) neg_target = self.mul1d(label, -1) soft_target = self.mul(neg_target, logit_softmax) loss = self.sum_ops(soft_target, -1) return self.mean_ops(loss) @MindFormerRegister.register(MindFormerModuleType.LOSS) class MSELoss(nn.Cell): """MSELoss for parallel.""" def __init__(self, norm_pixel_loss=True, parallel_config=default_dpmp_config): super(MSELoss, self).__init__() dp = parallel_config.data_parallel self.add_loss = P.Add().shard(((dp, 1, 1), ())) self.sub = P.Sub().shard(((dp, 1, 1), (dp, 1, 1))) self.divide = P.RealDiv().shard(((dp, 1, 1), (dp, 1, 1))) self.pow = P.Pow().shard(((dp, 1, 1), ())) self.divide1 = P.RealDiv().shard(((), ())) self.divide2 = P.RealDiv().shard(((dp, 1, 1), ())) self.square = P.Square().shard(((dp, 1, 1),)) self.cast = P.Cast() self.mean1 = P.ReduceMean(keep_dims=True).shard(((dp, 1, 1),)) self.mean2 = P.ReduceMean().shard(((dp, 1, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.sum = P.ReduceSum().shard(((dp, 1,),)) self.sum2 = P.ReduceSum(keep_dims=True).shard(((dp, 1, 1),)) self.norm_pixel_loss = norm_pixel_loss def construct(self, pred, target, mask): """mse loss construct.""" pred = self.cast(pred, mstype.float32) target = self.cast(target, mstype.float32) mask = self.cast(mask, mstype.float32) if self.norm_pixel_loss: mean = self.mean1(target, -1) var = self.variance(target) var = self.add_loss(var, 1e-6) std = self.pow(var, 0.5) sub = self.sub(target, mean) target = self.divide(sub, std) res = self.sub(pred, target) recon_loss = self.square(res) recon_loss = self.mean2(recon_loss, -1) loss_mask = self.mul(recon_loss, mask) loss_sum = self.sum(loss_mask) mask_sum = self.sum(mask) loss = self.divide1(loss_sum, mask_sum) return loss def variance(self, x): """get variance.""" axis = (x.ndim - 1,) x_mean = self.mean1(x, axis) x_sub = self.sub(x, x_mean) x_pow = self.pow(x_sub, 2) x_sum = self.sum2(x_pow, axis) x_var = self.divide2(x_sum, x.shape[-1]) return x_var @MindFormerRegister.register(MindFormerModuleType.LOSS) class L1Loss(LossBase): """L1Loss for parallel.""" def __init__(self, reduction='mean', parallel_config=default_dpmp_config): super(L1Loss, self).__init__() dp = parallel_config.data_parallel self.abs = P.Abs().shard(((dp, 1, 1, 1),)) self.sub = P.Sub().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.mul = P.Mul().shard(((), (dp, 1, 1, 1))) self.reduce_mean = P.ReduceMean().shard(((dp, 1, 1, 1),)) self.reduce_sum = P.ReduceSum().shard(((dp, 1, 1, 1),)) self.cast = P.Cast() self.average = True self.reduce = True if reduction == 'sum': self.average = False if reduction == 'none': self.reduce = False def get_loss(self, x, weights=1.0): """get loss.""" input_dtype = x.dtype x = self.cast(x, mstype.float32) weights = self.cast(weights, mstype.float32) x = self.mul(weights, x) if self.reduce and self.average: x = self.reduce_mean(x, self.get_axis(x)) if self.reduce and not self.average: x = self.reduce_sum(x, self.get_axis(x)) x = self.cast(x, input_dtype) return x def construct(self, logits, labels): """L1Loss construct.""" x_sub = self.sub(logits, labels) x = self.abs(x_sub) return self.get_loss(x) class _Softmax(nn.Cell): """ Calculate the softmax results with given logits. Note: The bprop of the cell is rewritten, just returns the accepted dout as returns. This cell should be used together with _NLLoss, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. Outputs: Tensor. The corresponding softmax results. """ def __init__(self, parallel_config=default_dpmp_config): super(_Softmax, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel # on/off value for onehot, for smooth labeling, modify the off_value self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.sum = P.ReduceSum().shard(((dp, mp),)) self.max = P.ArgMaxWithValue(axis=-1, keep_dims=True).shard( ((dp, mp),)) self.sub = P.Sub().shard(((dp, mp), (dp, 1))) self.exp = P.Exp().shard(((dp, mp),)) self.div = P.RealDiv().shard(((dp, mp), (dp, 1))) self.onehot = P.OneHot().shard(((dp, mp), (), ())) def construct(self, logits, label): """Forward process """ # LogSoftmax for logits over last dimension logits = F.cast(logits, mstype.float32) _, logit_max = self.max(logits) logit_sub = self.sub(logits, logit_max) logit_exp = self.exp(logit_sub) exp_sum = self.sum(logit_exp, -1) exp_sum = P.Reshape()(exp_sum, (F.shape(exp_sum)[0], 1)) softmax_result = self.div(logit_exp, exp_sum) one_hot_label = self.onehot(label, F.shape(logits)[-1], self.on_value, self.off_value) return softmax_result, one_hot_label def bprop(self, logits, label, _, dout): """just return the loss of the dout. Note this should be used together with _NLLLoss""" d_logits = F.cast(dout[0], F.dtype(logits)) return d_logits, F.zeros_like(label) class _NLLLoss(nn.Cell): """ Calculate the NLLLoss results with given softmax results and the label. Note: The bprop of the cell is rewritten. This cell should be used together with _Softmax, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **loss** (Tensor) - Tensor of shape (N, C). Data type is float32. Outputs: Tensor. The corresponding loss results. """ def __init__(self, parallel_config=default_dpmp_config, eps_const=1e-24): super(_NLLLoss, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.repeat_loss = 1 self.eps_const = Tensor(eps_const, mstype.float32) # In auto parallel, there will be a virtual div in the back propagation begins. As we use custom bprop function # we need to eliminate this virtual div by adding a factor "mp". if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL, ParallelMode.SEMI_AUTO_PARALLEL): self.repeat_loss = mp if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.sum = P.ReduceSum() self.mul = P.Mul() self.neg = P.Neg() self.log = P.Log() self.add = P.Add().shard(((dp, mp), ())) else: self.sum = P.ReduceSum().shard(((dp, mp),)) self.mul = P.Mul().shard(((dp, mp), (dp, mp))) self.neg = P.Neg().shard(((dp, mp),)) self.log = P.Log().shard(((dp, mp),)) self.add = P.Add().shard(((dp, mp), ())) def construct(self, softmax_result, one_hot_label): log_softmax_result = self.log(self.add(softmax_result, self.eps_const)) loss = self.mul(log_softmax_result, one_hot_label) loss_unsum = self.neg(loss) loss_reduce = self.sum(loss_unsum, -1) return loss_reduce def bprop(self, softmax_result, one_hot_label, _, dout): """A simplified function. Note this should be used together with _Softmax""" logits = softmax_result - one_hot_label logits = logits * P.ExpandDims()(dout, -1) * self.repeat_loss return logits, F.zeros_like(one_hot_label) @MindFormerRegister.register(MindFormerModuleType.LOSS) class CrossEntropyLoss(nn.Cell): """ Calculate the cross entropy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. - **labels** (Tensor) - Tensor of shape (N, ). The ground truth label of the sample. - **input_mask** (Tensor) - Tensor of shape (N, ). input_mask indicates whether there are padded inputs and for padded inputs it will not be counted into loss. Outputs: Tensor. The corresponding cross entropy loss. Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> from mindformers.core import CrossEntropyLoss >>> loss = CrossEntropyLoss() >>> >>> logits = Tensor(np.array([[3, 5, 6, 9, 12, 33, 42, 12, 32, 72]]), mstype.float32) >>> labels_np = np.array([1]).astype(np.int32) >>> input_mask = Tensor(np.ones(1).astype(np.float32)) >>> labels = Tensor(labels_np) >>> output = loss(logits, labels, input_mask) >>> print(output.shape) (1,) """ @_LogActionOnce(m_logger=logger, key='CrossEntropyLoss', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) def __init__(self, parallel_config=default_dpmp_config, eps_const=1e-24): super(CrossEntropyLoss, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.enable_force_redistribute = False if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL, ParallelMode.SEMI_AUTO_PARALLEL): self.enable_force_redistribute = True self.add = P.Add().shard(((dp, mp), ())).add_prim_attr("keep_alive", True) self.add_label = P.Add().shard(((dp,), ())).add_prim_attr("keep_alive", True) self._check_and_modify_sharding_context(dp) self.sum2 = P.ReduceSum().shard(((1,),)) self.mul2 = P.Mul().shard(((1,), (1,))) self.add2 = P.Add() self.div2 = P.RealDiv() self.relu = P.ReLU().shard(((1,),)) self._softmax = _Softmax(parallel_config) self._nllloss = _NLLLoss(parallel_config, eps_const) @staticmethod def _check_and_modify_sharding_context(dp): device_num = _get_device_num() stages = _get_pipeline_stages() if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and dp * stages != device_num: set_algo_parameters(fully_use_devices=False) def construct(self, logits, label, input_mask): """Forward process""" # The add is used for forcing the redistribution before stepping in sub graphs, when semi/auto parallel enabled. if self.enable_force_redistribute: logits = self.add(logits, 0) label = self.add_label(label, 0) softmax, one_hot_label = self._softmax(logits, label) loss_reduce = self._nllloss(softmax, one_hot_label) # Using input_mask to mask the loss input_mask = P.Reshape()(input_mask, (-1,)) numerator = self.sum2(self.mul2(loss_reduce, input_mask)) denominator = self.add2( self.sum2(input_mask), P.Cast()(F.tuple_to_array((1e-5,)), mstype.float32)) loss = self.div2(numerator, denominator) return loss @MindFormerRegister.register(MindFormerModuleType.LOSS) class CompareLoss(nn.Cell): """ Calculate the compare loss for reward model. Args: config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **rewards** (Tensor) - Tensor of shape (B, S, 1). Data type must be float16 or float32. The output logits of the backbone. - **loss_mask** (Tensor) - Tensor of shape (B, S, 1). The loss mask of the rewards. - **end_ind** (Tensor) - Tensor of shape (B, ). end index of all tensors. Outputs: Tensor. The corresponding loss. Examples: """ def __init__(self, config): super(CompareLoss, self).__init__() dp = config.data_parallel mp = 1 self.gatherd = P.GatherD() self.log = P.Log() self.reduce_sum = P.ReduceSum(keep_dims=False) self.slice = P.StridedSlice().shard(((1, 1),)) self.slice_ind = P.StridedSlice().shard(((1,),)) self.mul = P.Mul().shard(((dp, mp), (dp, mp))) self.sub = P.Sub().shard(((dp, mp), (dp, mp))) def construct(self, rewards, loss_mask, end_ind): """Forward process""" bs = rewards.shape[0] // 2 # a sample has two bs responses seq_len = rewards.shape[-1] chosen_rewards = self.slice(rewards, (0, 0), (bs, seq_len), (1, 1)) rejected_rewards = self.slice(rewards, (bs, 0), (2*bs, seq_len), (1, 1)) end_ind_chosen = self.slice_ind(end_ind, (0,), (bs,), (1,)) end_ind_reject = self.slice_ind(end_ind, (bs,), (2*bs,), (1,)) temp = P.Concat()((end_ind_chosen, end_ind_reject)) temp = temp.reshape((2, -1)) temp = P.Cast()(temp, mstype.float16) end_ind_final, _ = P.max(temp, axis=0) temp = P.Cast()(temp, mstype.float16) end_ind_final = end_ind_final.reshape((-1, 1)) end_ind_final = P.Cast()(end_ind_final, mstype.int32) loss_mask_final = loss_mask c_truncated_reward = self.mul(chosen_rewards, loss_mask_final) r_truncated_reward = self.mul(rejected_rewards, loss_mask_final) chosen_end_scores = self.gatherd(chosen_rewards, 1, end_ind_final-1) reject_end_scores = self.gatherd(rejected_rewards, 1, end_ind_final-1) compare_len = self.reduce_sum(P.cast(loss_mask_final, mstype.float32), -1) temp_loss = -self.log(P.sigmoid(self.sub(c_truncated_reward, r_truncated_reward))) loss = self.reduce_sum(self.mul(temp_loss, loss_mask_final), -1)/compare_len loss = loss.mean() return loss, chosen_end_scores, reject_end_scores pengcheng.mind-test/mindformers/core/loss/__init__.py000066400000000000000000000014251471400134000232610ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Loss.""" from .build_loss import build_loss from .loss import * __all__ = [] __all__.extend(loss.__all__) pengcheng.mind-test/mindformers/core/loss/__pycache__/000077500000000000000000000000001471400134000233565ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/loss/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004411471400134000275450ustar00rootroot00000000000000a ?cf@s,dZddlmZddlTgZeejdS)zMindFormers Loss.) build_loss)*N)__doc__rloss__all__extendrrI/home/ma-user/work/mindformers-baichuan/mindformers/core/loss/__init__.pys pengcheng.mind-test/mindformers/core/loss/__pycache__/build_loss.cpython-39.pyc000066400000000000000000000035661471400134000301600ustar00rootroot00000000000000a ?cf @sTdZddlZddlmZddlmZmZmZd eee e dddZ d d Z e dS) zBuild Loss API.N)nn)MindFormerRegisterMindFormerModuleTypeMindFormerConfigloss)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)aBuild loss For MindFormer. Instantiate the loss from MindFormerRegister's registry. Args: config (dict): The task loss's config. Default: None. default_args (dict): The default argument of loss API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'loss'. class_name (str): The class name of loss API. Default: None. Return: The function instance of loss API. Examples: >>> from mindformers import build_loss >>> loss_config = {'type': 'L1Loss'} >>> # 1) use config dict to build loss >>> loss_from_config = build_loss(loss_config) >>> # 2) use class name to build loss >>> loss_class_name = build_loss(class_name='L1Loss') N)r) isinstancedictrrget_instance_from_cfgrLOSS get_instance)rrr r kwargsrK/home/ma-user/work/mindformers-baichuan/mindformers/core/loss/build_loss.py build_losssrcCsDttjD]4}|drq ttj|}t|r t|t j q dS)z( register MindSpore builtin loss class. __N) dirrr startswithgetattrinspectisclassr register_clsrr) module_namerrrrregister_ms_loss8s   r)NNrN) __doc__r mindsporermindformers.tools.registerrrrr strrrrrrrs  ! pengcheng.mind-test/mindformers/core/loss/__pycache__/loss.cpython-39.pyc000066400000000000000000000342411471400134000267730ustar00rootroot00000000000000a ?cfB@sjdZddlmZmZddlmZddlmZddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZmZmZmZdd lmZdd lmZmZdd lmZgdZ e!ej"Gddde Z#e!ej"Gdddej$Z%e!ej"Gddde Z&Gdddej$Z'Gdddej$Z(e!ej"Gdddej$Z)e!ej"Gdddej$Z*dS)zMindFormer Self-Define Loss.)nnTensor)ops) functional)dtype)LossBase) ParallelMode)set_algo_parameters)log)_get_device_num_get_pipeline_stages_get_parallel_mode_is_sharding_propagation)_LogActionOnce)MindFormerRegisterMindFormerModuleType)default_dpmp_config)SoftTargetCrossEntropyMSELossL1LossCrossEntropyLoss CompareLosscs,eZdZdZeffdd ZddZZS)rz)SoftTargetCrossEntropy for MixUp Augment.cstt||j}tjddd|_tjdd|dff|_ t |df|dff|_ t |dfdf|_ t |dff|_dS)NF keep_dimsr)superr__init__ data_parallelP ReduceMeanshardmean_ops ReduceSumsum_opsMulmulmul1d LogSoftmax log_softmax)selfparallel_configdp __class__rE/home/ma-user/work/mindformers-baichuan/mindformers/core/loss/loss.pyr(szSoftTargetCrossEntropy.__init__cCsXt|tj}t|tj}||}||d}|||}||d}||SN) r!Castmstypefloat32r+r)r(r&r$)r,logitlabelZ logit_softmaxZ neg_targetZ soft_targetlossrrr1 construct1s    z SoftTargetCrossEntropy.construct)__name__ __module__ __qualname____doc__rrr: __classcell__rrr/r1r$s rcs6eZdZdZdeffdd ZddZddZZS) rzMSELoss for parallel.Tcs`tt||j}t|ddfdf|_t|ddf|ddff|_ t |ddf|ddff|_ t |ddfdf|_ t d|_t |ddfdf|_t|ddff|_t|_tjdd|ddff|_t|ddff|_t|df|dff|_t|dff|_tjdd|ddff|_||_dS)Nrr)rrTr)rrrr r!Addr#add_lossSubsubRealDivdividePowpowdivide1divide2Squaresquarer4castr"mean1mean2r'r(r%sumsum2norm_pixel_loss)r,rQr-r.r/rr1r>s    zMSELoss.__init__cCs||tj}||tj}||tj}|jrv||d}||}||d}||d}|||}| ||}|||}| |} | | d} | | |} | | } | |} || | } | S)zmse loss construct.r3gư>g?)rLr5r6rQrMvariancerArGrCrErKrNr(rOrH)r,predtargetmaskmeanvarstdrCresZ recon_loss loss_maskZloss_sumZmask_sumr9rrr1r:Ps$             zMSELoss.constructcCsR|jdf}|||}|||}||d}|||}|||jd}|S)z get variance.rr3)ndimrMrCrGrPrIshape)r,xaxisx_meanx_subx_powx_sumx_varrrr1rRes     zMSELoss.variance) r;r<r=r>rrr:rRr?rrr/r1r;srcs8eZdZdZdeffdd Zd ddZdd ZZS) rzL1Loss for parallel.rVcstt||j}t|dddff|_t|dddf|dddff|_ t d|dddff|_ t |dddff|_ t|dddff|_t|_d|_d|_|dkrd|_|dkrd|_dS)NrrTrOFnone)rrrr r!Absr#absrBrCr'r(r" reduce_meanr% reduce_sumr4rLaveragereduce)r, reductionr-r.r/rr1rss$ zL1Loss.__init__?cCsz|j}||tj}||tj}|||}|jrL|jrL||||}|jrj|jsj| |||}|||}|S)z get loss.) rrLr5r6r(rkrjrhget_axisri)r,r^weights input_dtyperrr1get_losss    zL1Loss.get_losscCs |||}||}||S)zL1Loss construct.)rCrgrq)r,logitslabelsrar^rrr1r:s  zL1Loss.construct)rm) r;r<r=r>rrrqr:r?rrr/r1rps rcs4eZdZdZeffdd ZddZddZZS)_Softmaxa Calculate the softmax results with given logits. Note: The bprop of the cell is rewritten, just returns the accepted dout as returns. This cell should be used together with _NLLoss, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. Outputs: Tensor. The corresponding softmax results. cstt||j}|j}tdtj|_tdtj|_ t ||ff|_ t jddd ||ff|_t  ||f|dff|_t  ||ff|_t  ||f|dff|_t  ||fddf|_dS)Nrmgr3T)r_rrr)rrtrr model_parallelrr5r6on_value off_valuer!r%r#rOArgMaxWithValuemaxrBrCExpexprDdivOneHotonehot)r,r-r.mpr/rr1rsz_Softmax.__init__c Cst|tj}||\}}|||}||}||d}t |t |ddf}| ||}| |t |d|j |j} || fS)zForward process r3rr)FrLr5r6ryrCr{rOr!Reshaper]r|r~rvrw) r,rrr8_Z logit_maxZ logit_subZ logit_expZexp_sumsoftmax_result one_hot_labelrrr1r:s    z_Softmax.constructcCs$t|dt|}|t|fS)zQjust return the loss of the dout. Note this should be used together with _NLLLossr)rrLr zeros_like)r,rrr8rdoutZd_logitsrrr1bpropsz_Softmax.bprop r;r<r=r>rrr:rr?rrr/r1rtsrtcs6eZdZdZedffdd ZddZddZZS) _NLLLossa7 Calculate the NLLLoss results with given softmax results and the label. Note: The bprop of the cell is rewritten. This cell should be used together with _Softmax, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **loss** (Tensor) - Tensor of shape (N, C). Data type is float32. Outputs: Tensor. The corresponding loss results. 秎W:cstt||j}|j}d|_t|tj|_ t t j t j fvrF||_t t j fvrtrt|_t|_t|_t|_t||fdf|_nvt||ff|_t||f||ff|_t||ff|_t||ff|_t||fdf|_dS)Nrr)rrrr ru repeat_lossrr5r6 eps_constr r AUTO_PARALLELSEMI_AUTO_PARALLELrr!r%rOr'r(NegnegLogr r@r#addr,r-rr.rr/rr1rs$    z_NLLLoss.__init__cCs:||||j}|||}||}||d}|Sr2)r rrr(rrO)r,rrZlog_softmax_resultr9Z loss_unsum loss_reducerrr1r:s    z_NLLLoss.constructcCs.||}|t|d|j}|t|fS)zFA simplified function. Note this should be used together with _Softmaxr3)r! ExpandDimsrrr)r,rrrrrrrrr1rsz_NLLLoss.bproprrrr/r1rsrcsReZdZdZeedeejfvde dffdd Z e ddZ dd Z ZS) ra Calculate the cross entropy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. - **labels** (Tensor) - Tensor of shape (N, ). The ground truth label of the sample. - **input_mask** (Tensor) - Tensor of shape (N, ). input_mask indicates whether there are padded inputs and for padded inputs it will not be counted into loss. Outputs: Tensor. The corresponding cross entropy loss. Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> from mindformers.core import CrossEntropyLoss >>> loss = CrossEntropyLoss() >>> >>> logits = Tensor(np.array([[3, 5, 6, 9, 12, 33, 42, 12, 32, 72]]), mstype.float32) >>> labels_np = np.array([1]).astype(np.int32) >>> input_mask = Tensor(np.ones(1).astype(np.float32)) >>> labels = Tensor(labels_np) >>> output = loss(logits, labels, input_mask) >>> print(output.shape) (1,) )m_loggerkey no_warningrcstt||j}|j}d|_ttjtj fvrd|_t ||fdf dd|_t |fdf dd|_||t  d|_t  d|_t |_t |_t  d|_t||_t|||_dS)NFTr keep_aliver)rr)rrrr ruenable_force_redistributer rrrr!r@r# add_prim_attrr add_label"_check_and_modify_sharding_contextr%rPr'mul2add2rDdiv2ReLUrelurt_softmaxr_nlllossrr/rr1r-s      zCrossEntropyLoss.__init__cCs4t}t}ttjfvr0|||kr0tdddS)NF)fully_use_devices)r r r rrr )r. device_numstagesrrr1rBsz3CrossEntropyLoss._check_and_modify_sharding_contextc Cs|jr||d}||d}|||\}}|||}t|d}||||}| ||t t dt j}|||} | S)Forward processr)r3)gh㈵>)rrrrrr!rrPrrr4rtuple_to_arrayr5r6r) r,rrr8Z input_masksoftmaxrr numerator denominatorr9rrr1r:Is    zCrossEntropyLoss.construct)r;r<r=r>rloggerr r STAND_ALONErr staticmethodrr:r?rrr/r1r s"  rcs(eZdZdZfddZddZZS)ram Calculate the compare loss for reward model. Args: config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **rewards** (Tensor) - Tensor of shape (B, S, 1). Data type must be float16 or float32. The output logits of the backbone. - **loss_mask** (Tensor) - Tensor of shape (B, S, 1). The loss mask of the rewards. - **end_ind** (Tensor) - Tensor of shape (B, ). end index of all tensors. Outputs: Tensor. The corresponding loss. Examples: cstt||j}d}t|_t|_tj dd|_ t d|_ t d|_t ||f||ff|_t ||f||ff|_dS)NrFr)rrr)rrrr r!GatherDgatherdrr r%ri StridedSlicer#slice slice_indr'r(rBrC)r,configr.rr/rr1rts  zCompareLoss.__init__cCsr|jdd}|jd}||d||fd}|||dfd||fd}||d|fd}|||fd|fd} t|| f} | d} t| tj} tj | dd \} } t| tj} | d } t| tj } |} | || }| || }| |d | d }| |d | d }| t| tjd}|t||| }| | || d|}|}|||fS) rrr[r3)rrr)rr)r[r3)r_)r3rr)r]rrr!Concatreshaper4r5float16ryint32r(rrirLr6r sigmoidrCrV)r,ZrewardsrZend_indbsseq_lenZchosen_rewardsZrejected_rewardsZend_ind_chosenZend_ind_rejecttempZ end_ind_finalrZloss_mask_finalZc_truncated_rewardZr_truncated_rewardZchosen_end_scoresZreject_end_scoresZ compare_lenZ temp_lossr9rrr1r:s.     zCompareLoss.construct)r;r<r=r>rr:r?rrr/r1r^s rN)+r> mindsporerrrr! mindspore.opsrrmindspore.commonrr5mindspore.nn.loss.lossrZmindspore.contextrZmindspore.parallelr r rmindspore.parallel._utilsr r r rmindformers.tools.loggerrmindformers.tools.registerrrZ2mindformers.modules.transformer.op_parallel_configr__all__registerLOSSrCellrrrtrrrrrrr1s2           4 )78 T pengcheng.mind-test/mindformers/core/loss/build_loss.py000066400000000000000000000047751471400134000236740ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Loss API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_loss( config: dict = None, default_args: dict = None, module_type: str = 'loss', class_name: str = None, **kwargs): r"""Build loss For MindFormer. Instantiate the loss from MindFormerRegister's registry. Args: config (dict): The task loss's config. Default: None. default_args (dict): The default argument of loss API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'loss'. class_name (str): The class name of loss API. Default: None. Return: The function instance of loss API. Examples: >>> from mindformers import build_loss >>> loss_config = {'type': 'L1Loss'} >>> # 1) use config dict to build loss >>> loss_from_config = build_loss(loss_config) >>> # 2) use class name to build loss >>> loss_class_name = build_loss(class_name='L1Loss') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.LOSS, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_loss(): """ register MindSpore builtin loss class. """ for module_name in dir(nn.loss): if module_name.startswith('__'): continue loss = getattr(nn.loss, module_name) if inspect.isclass(loss): MindFormerRegister.register_cls( loss, MindFormerModuleType.LOSS) register_ms_loss() pengcheng.mind-test/mindformers/core/loss/loss.py000066400000000000000000000412271471400134000225060ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormer Self-Define Loss.""" from mindspore import nn, Tensor from mindspore import ops as P from mindspore.ops import functional as F from mindspore.common import dtype as mstype from mindspore.nn.loss.loss import LossBase from mindspore.context import ParallelMode from mindspore.parallel import set_algo_parameters from mindspore import log as logger from mindspore.parallel._utils import _get_device_num, _get_pipeline_stages, _get_parallel_mode, _is_sharding_propagation from mindformers.tools.logger import _LogActionOnce from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.modules.transformer.op_parallel_config import default_dpmp_config __all__ = ['SoftTargetCrossEntropy', 'MSELoss', 'L1Loss', 'CrossEntropyLoss', 'CompareLoss'] @MindFormerRegister.register(MindFormerModuleType.LOSS) class SoftTargetCrossEntropy(LossBase): """SoftTargetCrossEntropy for MixUp Augment.""" def __init__(self, parallel_config=default_dpmp_config): super(SoftTargetCrossEntropy, self).__init__() dp = parallel_config.data_parallel self.mean_ops = P.ReduceMean(keep_dims=False).shard(((1,),)) self.sum_ops = P.ReduceSum(keep_dims=False).shard(((dp, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.mul1d = P.Mul().shard(((dp, 1), ())) self.log_softmax = P.LogSoftmax().shard(((dp, 1),)) def construct(self, logit, label): logit = P.Cast()(logit, mstype.float32) label = P.Cast()(label, mstype.float32) logit_softmax = self.log_softmax(logit) neg_target = self.mul1d(label, -1) soft_target = self.mul(neg_target, logit_softmax) loss = self.sum_ops(soft_target, -1) return self.mean_ops(loss) @MindFormerRegister.register(MindFormerModuleType.LOSS) class MSELoss(nn.Cell): """MSELoss for parallel.""" def __init__(self, norm_pixel_loss=True, parallel_config=default_dpmp_config): super(MSELoss, self).__init__() dp = parallel_config.data_parallel self.add_loss = P.Add().shard(((dp, 1, 1), ())) self.sub = P.Sub().shard(((dp, 1, 1), (dp, 1, 1))) self.divide = P.RealDiv().shard(((dp, 1, 1), (dp, 1, 1))) self.pow = P.Pow().shard(((dp, 1, 1), ())) self.divide1 = P.RealDiv().shard(((), ())) self.divide2 = P.RealDiv().shard(((dp, 1, 1), ())) self.square = P.Square().shard(((dp, 1, 1),)) self.cast = P.Cast() self.mean1 = P.ReduceMean(keep_dims=True).shard(((dp, 1, 1),)) self.mean2 = P.ReduceMean().shard(((dp, 1, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.sum = P.ReduceSum().shard(((dp, 1,),)) self.sum2 = P.ReduceSum(keep_dims=True).shard(((dp, 1, 1),)) self.norm_pixel_loss = norm_pixel_loss def construct(self, pred, target, mask): """mse loss construct.""" pred = self.cast(pred, mstype.float32) target = self.cast(target, mstype.float32) mask = self.cast(mask, mstype.float32) if self.norm_pixel_loss: mean = self.mean1(target, -1) var = self.variance(target) var = self.add_loss(var, 1e-6) std = self.pow(var, 0.5) sub = self.sub(target, mean) target = self.divide(sub, std) res = self.sub(pred, target) recon_loss = self.square(res) recon_loss = self.mean2(recon_loss, -1) loss_mask = self.mul(recon_loss, mask) loss_sum = self.sum(loss_mask) mask_sum = self.sum(mask) loss = self.divide1(loss_sum, mask_sum) return loss def variance(self, x): """get variance.""" axis = (x.ndim - 1,) x_mean = self.mean1(x, axis) x_sub = self.sub(x, x_mean) x_pow = self.pow(x_sub, 2) x_sum = self.sum2(x_pow, axis) x_var = self.divide2(x_sum, x.shape[-1]) return x_var @MindFormerRegister.register(MindFormerModuleType.LOSS) class L1Loss(LossBase): """L1Loss for parallel.""" def __init__(self, reduction='mean', parallel_config=default_dpmp_config): super(L1Loss, self).__init__() dp = parallel_config.data_parallel self.abs = P.Abs().shard(((dp, 1, 1, 1),)) self.sub = P.Sub().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.mul = P.Mul().shard(((), (dp, 1, 1, 1))) self.reduce_mean = P.ReduceMean().shard(((dp, 1, 1, 1),)) self.reduce_sum = P.ReduceSum().shard(((dp, 1, 1, 1),)) self.cast = P.Cast() self.average = True self.reduce = True if reduction == 'sum': self.average = False if reduction == 'none': self.reduce = False def get_loss(self, x, weights=1.0): """get loss.""" input_dtype = x.dtype x = self.cast(x, mstype.float32) weights = self.cast(weights, mstype.float32) x = self.mul(weights, x) if self.reduce and self.average: x = self.reduce_mean(x, self.get_axis(x)) if self.reduce and not self.average: x = self.reduce_sum(x, self.get_axis(x)) x = self.cast(x, input_dtype) return x def construct(self, logits, labels): """L1Loss construct.""" x_sub = self.sub(logits, labels) x = self.abs(x_sub) return self.get_loss(x) class _Softmax(nn.Cell): """ Calculate the softmax results with given logits. Note: The bprop of the cell is rewritten, just returns the accepted dout as returns. This cell should be used together with _NLLoss, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. Outputs: Tensor. The corresponding softmax results. """ def __init__(self, parallel_config=default_dpmp_config): super(_Softmax, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel # on/off value for onehot, for smooth labeling, modify the off_value self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.sum = P.ReduceSum().shard(((dp, mp),)) self.max = P.ArgMaxWithValue(axis=-1, keep_dims=True).shard( ((dp, mp),)) self.sub = P.Sub().shard(((dp, mp), (dp, 1))) self.exp = P.Exp().shard(((dp, mp),)) self.div = P.RealDiv().shard(((dp, mp), (dp, 1))) self.onehot = P.OneHot().shard(((dp, mp), (), ())) def construct(self, logits, label): """Forward process """ # LogSoftmax for logits over last dimension logits = F.cast(logits, mstype.float32) _, logit_max = self.max(logits) logit_sub = self.sub(logits, logit_max) logit_exp = self.exp(logit_sub) exp_sum = self.sum(logit_exp, -1) exp_sum = P.Reshape()(exp_sum, (F.shape(exp_sum)[0], 1)) softmax_result = self.div(logit_exp, exp_sum) one_hot_label = self.onehot(label, F.shape(logits)[-1], self.on_value, self.off_value) return softmax_result, one_hot_label def bprop(self, logits, label, _, dout): """just return the loss of the dout. Note this should be used together with _NLLLoss""" d_logits = F.cast(dout[0], F.dtype(logits)) return d_logits, F.zeros_like(label) class _NLLLoss(nn.Cell): """ Calculate the NLLLoss results with given softmax results and the label. Note: The bprop of the cell is rewritten. This cell should be used together with _Softmax, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **loss** (Tensor) - Tensor of shape (N, C). Data type is float32. Outputs: Tensor. The corresponding loss results. """ def __init__(self, parallel_config=default_dpmp_config, eps_const=1e-24): super(_NLLLoss, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.repeat_loss = 1 self.eps_const = Tensor(eps_const, mstype.float32) # In auto parallel, there will be a virtual div in the back propagation begins. As we use custom bprop function # we need to eliminate this virtual div by adding a factor "mp". if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL, ParallelMode.SEMI_AUTO_PARALLEL): self.repeat_loss = mp if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.sum = P.ReduceSum() self.mul = P.Mul() self.neg = P.Neg() self.log = P.Log() self.add = P.Add().shard(((dp, mp), ())) else: self.sum = P.ReduceSum().shard(((dp, mp),)) self.mul = P.Mul().shard(((dp, mp), (dp, mp))) self.neg = P.Neg().shard(((dp, mp),)) self.log = P.Log().shard(((dp, mp),)) self.add = P.Add().shard(((dp, mp), ())) def construct(self, softmax_result, one_hot_label): log_softmax_result = self.log(self.add(softmax_result, self.eps_const)) loss = self.mul(log_softmax_result, one_hot_label) loss_unsum = self.neg(loss) loss_reduce = self.sum(loss_unsum, -1) return loss_reduce def bprop(self, softmax_result, one_hot_label, _, dout): """A simplified function. Note this should be used together with _Softmax""" logits = softmax_result - one_hot_label logits = logits * P.ExpandDims()(dout, -1) * self.repeat_loss return logits, F.zeros_like(one_hot_label) @MindFormerRegister.register(MindFormerModuleType.LOSS) class CrossEntropyLoss(nn.Cell): """ Calculate the cross entropy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. - **labels** (Tensor) - Tensor of shape (N, ). The ground truth label of the sample. - **input_mask** (Tensor) - Tensor of shape (N, ). input_mask indicates whether there are padded inputs and for padded inputs it will not be counted into loss. Outputs: Tensor. The corresponding cross entropy loss. Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> from mindformers.core import CrossEntropyLoss >>> loss = CrossEntropyLoss() >>> >>> logits = Tensor(np.array([[3, 5, 6, 9, 12, 33, 42, 12, 32, 72]]), mstype.float32) >>> labels_np = np.array([1]).astype(np.int32) >>> input_mask = Tensor(np.ones(1).astype(np.float32)) >>> labels = Tensor(labels_np) >>> output = loss(logits, labels, input_mask) >>> print(output.shape) (1,) """ @_LogActionOnce(m_logger=logger, key='CrossEntropyLoss', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) def __init__(self, parallel_config=default_dpmp_config, eps_const=1e-24): super(CrossEntropyLoss, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.enable_force_redistribute = False if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL, ParallelMode.SEMI_AUTO_PARALLEL): self.enable_force_redistribute = True self.add = P.Add().shard(((dp, mp), ())).add_prim_attr("keep_alive", True) self.add_label = P.Add().shard(((dp,), ())).add_prim_attr("keep_alive", True) self._check_and_modify_sharding_context(dp) self.sum2 = P.ReduceSum().shard(((1,),)) self.mul2 = P.Mul().shard(((1,), (1,))) self.add2 = P.Add() self.div2 = P.RealDiv() self.relu = P.ReLU().shard(((1,),)) self._softmax = _Softmax(parallel_config) self._nllloss = _NLLLoss(parallel_config, eps_const) @staticmethod def _check_and_modify_sharding_context(dp): device_num = _get_device_num() stages = _get_pipeline_stages() if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and dp * stages != device_num: set_algo_parameters(fully_use_devices=False) def construct(self, logits, label, input_mask): """Forward process""" # The add is used for forcing the redistribution before stepping in sub graphs, when semi/auto parallel enabled. if self.enable_force_redistribute: logits = self.add(logits, 0) label = self.add_label(label, 0) softmax, one_hot_label = self._softmax(logits, label) loss_reduce = self._nllloss(softmax, one_hot_label) # Using input_mask to mask the loss input_mask = P.Reshape()(input_mask, (-1,)) numerator = self.sum2(self.mul2(loss_reduce, input_mask)) denominator = self.add2( self.sum2(input_mask), P.Cast()(F.tuple_to_array((1e-5,)), mstype.float32)) loss = self.div2(numerator, denominator) return loss @MindFormerRegister.register(MindFormerModuleType.LOSS) class CompareLoss(nn.Cell): """ Calculate the compare loss for reward model. Args: config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **rewards** (Tensor) - Tensor of shape (B, S, 1). Data type must be float16 or float32. The output logits of the backbone. - **loss_mask** (Tensor) - Tensor of shape (B, S, 1). The loss mask of the rewards. - **end_ind** (Tensor) - Tensor of shape (B, ). end index of all tensors. Outputs: Tensor. The corresponding loss. Examples: """ def __init__(self, config): super(CompareLoss, self).__init__() dp = config.data_parallel mp = 1 self.gatherd = P.GatherD() self.log = P.Log() self.reduce_sum = P.ReduceSum(keep_dims=False) self.slice = P.StridedSlice().shard(((1, 1),)) self.slice_ind = P.StridedSlice().shard(((1,),)) self.mul = P.Mul().shard(((dp, mp), (dp, mp))) self.sub = P.Sub().shard(((dp, mp), (dp, mp))) def construct(self, rewards, loss_mask, end_ind): """Forward process""" bs = rewards.shape[0] // 2 # a sample has two bs responses seq_len = rewards.shape[-1] chosen_rewards = self.slice(rewards, (0, 0), (bs, seq_len), (1, 1)) rejected_rewards = self.slice(rewards, (bs, 0), (2*bs, seq_len), (1, 1)) end_ind_chosen = self.slice_ind(end_ind, (0,), (bs,), (1,)) end_ind_reject = self.slice_ind(end_ind, (bs,), (2*bs,), (1,)) temp = P.Concat()((end_ind_chosen, end_ind_reject)) temp = temp.reshape((2, -1)) temp = P.Cast()(temp, mstype.float16) end_ind_final, _ = P.max(temp, axis=0) temp = P.Cast()(temp, mstype.float16) end_ind_final = end_ind_final.reshape((-1, 1)) end_ind_final = P.Cast()(end_ind_final, mstype.int32) loss_mask_final = loss_mask c_truncated_reward = self.mul(chosen_rewards, loss_mask_final) r_truncated_reward = self.mul(rejected_rewards, loss_mask_final) chosen_end_scores = self.gatherd(chosen_rewards, 1, end_ind_final-1) reject_end_scores = self.gatherd(rejected_rewards, 1, end_ind_final-1) compare_len = self.reduce_sum(P.cast(loss_mask_final, mstype.float32), -1) temp_loss = -self.log(P.sigmoid(self.sub(c_truncated_reward, r_truncated_reward))) loss = self.reduce_sum(self.mul(temp_loss, loss_mask_final), -1)/compare_len loss = loss.mean() return loss, chosen_end_scores, reject_end_scores pengcheng.mind-test/mindformers/core/lr/000077500000000000000000000000001471400134000206035ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/lr/.ipynb_checkpoints/000077500000000000000000000000001471400134000243745ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/lr/.ipynb_checkpoints/lr_schedule-checkpoint.py000066400000000000000000000353751471400134000314010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/PanGu-%CE%B1/utils.py # https://github.com/huggingface/transformers/blob/main/src/transformers/optimization.py # ============================================================================ """Self-Define LR Schedule.""" import math import numpy as np from mindspore.ops import operations as P import mindspore.common.dtype as mstype from mindspore.nn.learning_rate_schedule import LearningRateSchedule, PolynomialDecayLR, WarmUpLR, CosineDecayLR from mindspore.common.tensor import Tensor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = [ 'LearningRateWiseLayer', 'WarmUpDecayLR', 'ConstantWarmUpLR', 'LinearWithWarmUpLR', 'CosineWithWarmUpLR', 'CosineWithRestartsAndWarmUpLR', 'PolynomialWithWarmUpLR'] @MindFormerRegister.register(MindFormerModuleType.LR) class ConstantWarmUpLR(LearningRateSchedule): """ConstantWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, warmup_lr_init: float = 0., **kwargs): super(ConstantWarmUpLR, self).__init__() warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.one_constant = Tensor(1.0, mstype.float32) self.greater = P.Greater() self.total_steps = kwargs.get('total_steps', None) def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: percent = self.one_constant learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class LinearWithWarmUpLR(LearningRateSchedule): """LinearWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, warmup_lr_init: float = 0.): super(LinearWithWarmUpLR, self).__init__() linear_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.linear_steps = Tensor(linear_steps, mstype.float32) self.greater = P.Greater() self.max = P.Maximum() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: percent = self.max(self.zero_constant, (self.total_steps - global_step) / self.linear_steps) learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class CosineWithWarmUpLR(LearningRateSchedule): """CosineWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, num_cycles: float = 0.5, lr_end: float = 0., warmup_lr_init: float = 0.): super(CosineWithWarmUpLR, self).__init__() cosine_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: progress = (global_step - self.warmup_steps) / self.cosine_steps percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * self.num_cycles * 2.0 * progress))) learning_rate = self.lr_end + (self.learning_rate - self.lr_end) * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class CosineWithRestartsAndWarmUpLR(LearningRateSchedule): """CosineWithRestartsAndWarmUpLR.""" def __init__(self, learning_rate: float, total_steps: int, warmup_steps: int, num_cycles: float = 0.5, lr_end: float = 0., warmup_lr_init: float = 0.): super(CosineWithRestartsAndWarmUpLR, self).__init__() cosine_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.one_constant = Tensor(1.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: progress = (global_step - self.warmup_steps) / self.cosine_steps if self.greater(self.one_constant, progress): percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * ((self.num_cycles * progress) % self.one_constant)))) learning_rate = self.lr_end + (self.learning_rate - self.lr_end) * percent else: return self.zero_constant return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class PolynomialWithWarmUpLR(LearningRateSchedule): """PolynomialWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, lr_end: float = 1e-7, power: float = 1.0, warmup_lr_init: float = 0.): super(PolynomialWithWarmUpLR, self).__init__() decay_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) if not learning_rate > lr_end: raise ValueError(f"lr_end ({lr_end}) must be be smaller than initial lr ({learning_rate})") self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.lr_end = lr_end self.power = power self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.decay_steps = Tensor(decay_steps, mstype.float32) self.greater = P.Greater() self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: lr_range = self.learning_rate - self.lr_end pct_remaining = 1 - (global_step - self.warmup_steps) / self.decay_steps decay = lr_range * pct_remaining ** self.power + self.lr_end percent = decay / self.learning_rate learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class LearningRateWiseLayer(LearningRateSchedule): """LearningRateWiseLayer.""" def __init__(self, base_lr, lr_scale): super(LearningRateWiseLayer, self).__init__() self.base_lr = base_lr self.lr_scale = lr_scale def construct(self, global_step): lr = self.base_lr(global_step) return self.lr_scale * lr @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpDecayLR(LearningRateSchedule): """ Warmup-decay learning rate for Bert network. """ def __init__(self, learning_rate, end_learning_rate, warmup_steps, decay_steps, power=1.0, use_cosine=False): super(WarmUpDecayLR, self).__init__() self.warmup_flag = False if warmup_steps > 0: self.warmup_flag = True self.warmup_lr = WarmUpLR(learning_rate, warmup_steps) self.decay_lr = PolynomialDecayLR(learning_rate, end_learning_rate, decay_steps, power) self.cosine_decay_lr = CosineDecayLR(end_learning_rate, learning_rate, decay_steps) self.warmup_steps = Tensor(np.array([warmup_steps]).astype(np.float32)) self.greater = P.Greater() self.one = Tensor(np.array([1.0]).astype(np.float32)) self.cast = P.Cast() self.use_cosine = use_cosine def construct(self, global_step): """dynamic learning rate""" if not self.use_cosine: decay_lr = self.decay_lr(global_step) else: decay_lr = self.cosine_decay_lr(global_step) if self.warmup_flag: is_warmup = self.cast(self.greater(self.warmup_steps, global_step), mstype.float32) warmup_lr = self.warmup_lr(global_step) lr = (self.one - is_warmup) * decay_lr + is_warmup * warmup_lr else: lr = decay_lr return lr @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpStableDecayLR(LearningRateSchedule): def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, decay_last_steps: int, lr_end: float = 0., warmup_lr_init: float = 0.): super(WarmUpStableDecayLR, self).__init__() warmup_steps = max(1, warmup_steps) decay_steps = max(1, total_steps - decay_last_steps) self.decay_last_steps = decay_last_steps self.decay_start_step_idx = Tensor(decay_steps, mstype.float32) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = Tensor(warmup_lr_init, mstype.float32) self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.greater = P.Greater() self.mul = P.Mul() self.cast = P.Cast() print(f"use WSD scheduler total: {total_steps}, warmup: {warmup_steps}, decay: {decay_last_steps}") def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent elif self.greater(global_step, self.decay_start_step_idx): grad = Tensor((self.lr_end - self.learning_rate), mstype.float32) / self.decay_last_steps learning_rate = (global_step - self.decay_start_step_idx) * grad + Tensor(self.learning_rate, mstype.float32) else: learning_rate = Tensor(self.learning_rate, mstype.float32) return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpCosineWithDecayLR(LearningRateSchedule): def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, decay_last_steps: int, lr_middle: float = 0., lr_end: float = 0., warmup_lr_init: float = 0.): super(WarmUpCosineWithDecayLR, self).__init__() warmup_steps = max(1, warmup_steps) self.warmup_steps = Tensor(warmup_steps, mstype.float32) decay_steps = max(1, total_steps - decay_last_steps) self.decay_last_steps = decay_last_steps self.decay_start_step_idx = Tensor(decay_steps, mstype.float32) self.cosine_steps = Tensor(decay_steps - warmup_steps, mstype.float32) self.zero_constant = Tensor(0.0, mstype.float32) self.num_cycles = 0.5 self.learning_rate = learning_rate self.lr_end = lr_end self.lr_middle = lr_middle self.warmup_lr_init = Tensor(warmup_lr_init, mstype.float32) self.total_steps = Tensor(total_steps, mstype.float32) self.greater = P.Greater() self.mul = P.Mul() self.max = P.Maximum() self.cast = P.Cast() self.cos = P.Cos() self.math_pi = math.pi print(f"use WSD scheduler total: {total_steps}, warmup: {warmup_steps}, decay: {decay_last_steps}, learning_rate: {learning_rate}, lr_middle: {lr_middle}, lr_end: {lr_end}") def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent elif self.greater(self.decay_start_step_idx, global_step): progress = (global_step - self.warmup_steps) / self.cosine_steps percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * self.num_cycles * 2.0 * progress))) learning_rate = self.lr_middle + (self.learning_rate - self.lr_middle) * percent else: grad = (self.lr_end - self.lr_middle) / self.decay_last_steps learning_rate = (global_step - self.decay_start_step_idx) * grad + self.lr_middle return learning_ratepengcheng.mind-test/mindformers/core/lr/__init__.py000066400000000000000000000014421471400134000227150ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers LR API.""" from .build_lr import build_lr from .lr_schedule import * __all__ = [] __all__.extend(lr_schedule.__all__) pengcheng.mind-test/mindformers/core/lr/__pycache__/000077500000000000000000000000001471400134000230135ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/lr/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004461471400134000272070ustar00rootroot00000000000000a ?cf"@s,dZddlmZddlTgZeejdS)zMindFormers LR API.)build_lr)*N)__doc__r lr_schedule__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/core/lr/__init__.pys pengcheng.mind-test/mindformers/core/lr/__pycache__/build_lr.cpython-39.pyc000066400000000000000000000052041471400134000272410ustar00rootroot00000000000000a (gf@sdZddlZddlmZddlmZmZmZddlm Z m Z m Z m Z m Z mZdeeeedd d Zd d Zd dZeedS)zBuild LR Schedule API.N)nn)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConstantWarmUpLRCosineWithWarmUpLRLinearWithWarmUpLRCosineWithRestartsAndWarmUpLRPolynomialWithWarmUpLRWarmUpStableDecayLRlr)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)acBuild lr For MindFormer. Instantiate the lr from MindFormerRegister's registry. Args: config (dict): The task lr's config. Default: None. default_args (dict): The default argument of lr API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'lr'. class_name (str): The class name of lr API. Default: None. Return: The function instance of lr API. Examples: >>> from mindformers import build_lr >>> lr_config = {'type': 'CosineDecayLR', 'max_lr': 0.001, ... 'min_lr': 0., 'decay_steps':10} >>> # 1) use config dict to build lr >>> lr_from_config = build_lr(lr_config) >>> # 2) use class name to build lr >>> lr_class_name = build_lr(class_name='CosineDecayLR', max_lr=0.001, min_lr=0., decay_steps=10) N)r) isinstancedictrrZget_instance_from_cfgrLR get_instance)rrrrkwargsrG/home/ma-user/work/mindformers-baichuan/mindformers/core/lr/build_lr.pybuild_lrsrcCsDttjD]4}|drq ttj|}t|r t|t j q dS)z& register MindSpore builtin LR class. __N) dirrlearning_rate_schedule startswithgetattrinspectisclassr register_clsrr) module_name lr_schedulerrrregister_ms_lr;s   r$cCsptjttjddtjttjddtjttjddtjttjddtjttjddtjt tjdddS) z( register MindFomrers builtin LR class. Zconstant_with_warmup)raliaslinearcosineZcosine_with_restarts polynomialZwarmup_stable_decayN) rr!rrrr rr r r rrrrregister_mf_lrFs$r))NNr N)__doc__r mindsporermindformers.tools.registerrrrr#rrr r r r rstrrr$r)rrrrs   " pengcheng.mind-test/mindformers/core/lr/__pycache__/lr_schedule.cpython-39.pyc000066400000000000000000000213651471400134000277440ustar00rootroot00000000000000a of1@sRdZddlZddlZddlmZddlmm Z ddl m Z m Z mZmZddlmZddlmZmZgdZeejGdd d e ZeejGd d d e ZeejGd d d e ZeejGddde ZeejGddde ZeejGddde ZeejGddde ZeejGddde ZdS)zSelf-Define LR Schedule.N) operations)LearningRateSchedulePolynomialDecayLRWarmUpLR CosineDecayLR)Tensor)MindFormerRegisterMindFormerModuleType)LearningRateWiseLayer WarmUpDecayLRConstantWarmUpLRLinearWithWarmUpLRCosineWithWarmUpLRCosineWithRestartsAndWarmUpLRPolynomialWithWarmUpLRcs4eZdZdZdeeedfdd ZddZZS) r zConstantWarmUpLR.) learning_rate warmup_stepswarmup_lr_initc s\tt|td|}||_||_t|tj|_ tdtj|_ t |_ |dd|_dS)N? total_steps)superr __init__maxrrrmstypefloat32r one_constantPGreatergreatergetr)selfrrrkwargs __class__J/home/ma-user/work/mindformers-baichuan/mindformers/core/lr/lr_schedule.pyr%s  zConstantWarmUpLR.__init__cCs>||j|r*||j}|j|j|}n|j}|j|}|Scompute current step lr.)r rrrrr" global_steppercentrr&r&r' construct/s   zConstantWarmUpLR.construct)r __name__ __module__ __qualname____doc__floatintrr- __classcell__r&r&r$r'r "s r cs6eZdZdZdeeeedfdd ZddZZS) r zLinearWithWarmUpLR.r)rrrrcstt|td||}td|}||_||_t|tj|_ t|tj|_ t|tj|_ t |_t |_tdtj|_t |_dSNrr)rr rrrrrrrrr linear_stepsrrr Maximum zero_constantCastcast)r"rrrrr7r$r&r'r=s   zLinearWithWarmUpLR.__init__cCs`||tj}||j|r8||j}|j|j|}n$||j|j ||j }|j|}|Sr() r;rrr rrrrr9rr7r*r&r&r'r-Ls  zLinearWithWarmUpLR.construct)rr.r&r&r$r'r :s r cs:eZdZdZd eeeeeedfdd ZddZZS) rzCosineWithWarmUpLR.?r)rrr num_cycleslr_endrcstt|td||}td|}||_||_||_t|tj |_ t|tj |_ t|tj |_ ||_ t|_t|_tj|_t|_tdtj |_t|_dSr6)rrrrrr>rrrrrr cosine_stepsr=rrr r8mathpimath_piCoscosr9r:r;)r"rrrr=r>rr?r$r&r'r[s     zCosineWithWarmUpLR.__init__c Cs||tj}||j|r8||j}|j|j|}nR||j|j}||j dd| |j |j d|}|j |j|j |}|S)r)r<rg@)r;rrr rrrr?rr9rDrBr=r>r"r+r,rZprogressr&r&r'r-ns $zCosineWithWarmUpLR.construct)r<rrr.r&r&r$r'rXs rcs:eZdZdZd eeeeeedfdd ZddZZS) rzCosineWithRestartsAndWarmUpLR.r<r)rrrr=r>rcstt|td||}td|}||_||_||_t|tj |_ t|tj |_ t|tj |_ ||_ t|_t|_tj|_t|_tdtj |_tdtj |_t|_dS)Nrrr)rrrrrr>rrrrrrr?r=rrr r8r@rArBrCrDr9rr:r;)r"rrrr=r>rr?r$r&r'rs"    z&CosineWithRestartsAndWarmUpLR.__init__c Cs||tj}||j|r8||j}|j|j|}nj||j|j}||j|r| |j dd| |j |j ||j}|j|j|j|}n|j S|S)r)r<r)r;rrr rrrr?rrr9rDrBr=r>rEr&r&r'r-s "z'CosineWithRestartsAndWarmUpLR.construct)r<rrr.r&r&r$r'r|s rcs:eZdZdZd eeeeeedfdd Zdd ZZS) rzPolynomialWithWarmUpLR.Hz>rr)rrrr>powerrcstt|td||}td|}||ksDtd|d|d||_||_||_||_t |t j |_ t |t j |_ t |t j |_t|_t|_dS)Nrzlr_end (z&) must be be smaller than initial lr ())rrrr ValueErrorrrr>rGrrrrr decay_stepsrrr r:r;)r"rrrr>rGrrJr$r&r'rs  zPolynomialWithWarmUpLR.__init__cCs||tj}||j|r8||j}|j|j|}nH|j|j}d||j|j}|||j |j}||j}|j|}|S)r)r) r;rrr rrrr>rJrG)r"r+r,rZlr_rangeZ pct_remainingdecayr&r&r'r-s    z PolynomialWithWarmUpLR.construct)rFrrr.r&r&r$r'rs rcs(eZdZdZfddZddZZS)r zLearningRateWiseLayer.cstt|||_||_dSN)rr rbase_lrlr_scale)r"rMrNr$r&r'rszLearningRateWiseLayer.__init__cCs||}|j|SrL)rMrN)r"r+lrr&r&r'r-s zLearningRateWiseLayer.constructr/r0r1r2rr-r5r&r&r$r'r s r cs*eZdZdZdfdd ZddZZS) r z6 Warmup-decay learning rate for Bert network. rFcstt|d|_|dkr.d|_t|||_t|||||_t||||_ t t |g t j|_t|_t t dg t j|_t|_||_dS)NFrTr)rr r warmup_flagr warmup_lrrdecay_lrrcosine_decay_lrrnparrayastyperrrrr oner:r; use_cosine)r"rend_learning_raterrJrGrYr$r&r'rs   zWarmUpDecayLR.__init__cCsd|js||}n ||}|jr\|||j|tj}| |}|j ||||}n|}|S)zdynamic learning rate) rYrSrTrQr;r rrrrRrX)r"r+rSZ is_warmuprRrOr&r&r'r-s   zWarmUpDecayLR.construct)rFrPr&r&r$r'r sr cs6eZdZdeeeeeedfdd ZddZZS)WarmUpStableDecayLRr)rrrdecay_last_stepsr>rcstt|td|}td||}||_t|tj|_||_ ||_ t|tj|_ t|tj|_ t|tj|_ t|_t|_t|_td|d|d|dS)Nrzuse WSD scheduler total: z , warmup: z , decay: )rr[rrr\rrrdecay_start_step_idxrr>rrrrrr Mulmulr:r;print)r"rrrr\r>rrJr$r&r'rs    zWarmUpStableDecayLR.__init__cCs||j|r*||j}|j|j|}nT|||jrpt|j|jtj|j }||j|t|jtj}nt|jtj}|Sr() r rrrr]rr>rrr\)r"r+r,rgradr&r&r'r-s zWarmUpStableDecayLR.construct)rr)r/r0r1r3r4rr-r5r&r&r$r'r[s  r[) r2r@numpyrU mindspore.opsrrZmindspore.common.dtypecommondtyper#mindspore.nn.learning_rate_schedulerrrrmindspore.common.tensorrmindformers.tools.registerrr __all__registerLRr r rrrr r r[r&r&r&r's0     # ( "  " pengcheng.mind-test/mindformers/core/lr/build_lr.py000066400000000000000000000076361471400134000227650ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build LR Schedule API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .lr_schedule import ConstantWarmUpLR, CosineWithWarmUpLR, \ LinearWithWarmUpLR, CosineWithRestartsAndWarmUpLR, PolynomialWithWarmUpLR, WarmUpStableDecayLR, WarmUpCosineWithDecayLR def build_lr( config: dict = None, default_args: dict = None, module_type: str = 'lr', class_name: str = None, **kwargs): r"""Build lr For MindFormer. Instantiate the lr from MindFormerRegister's registry. Args: config (dict): The task lr's config. Default: None. default_args (dict): The default argument of lr API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'lr'. class_name (str): The class name of lr API. Default: None. Return: The function instance of lr API. Examples: >>> from mindformers import build_lr >>> lr_config = {'type': 'CosineDecayLR', 'max_lr': 0.001, ... 'min_lr': 0., 'decay_steps':10} >>> # 1) use config dict to build lr >>> lr_from_config = build_lr(lr_config) >>> # 2) use class name to build lr >>> lr_class_name = build_lr(class_name='CosineDecayLR', max_lr=0.001, min_lr=0., decay_steps=10) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.LR, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_lr(): """ register MindSpore builtin LR class. """ for module_name in dir(nn.learning_rate_schedule): if module_name.startswith('__'): continue lr_schedule = getattr(nn.learning_rate_schedule, module_name) if inspect.isclass(lr_schedule): MindFormerRegister.register_cls( lr_schedule, MindFormerModuleType.LR) def register_mf_lr(): """ register MindFomrers builtin LR class. """ # adapt huggingface MindFormerRegister.register_cls( ConstantWarmUpLR, module_type=MindFormerModuleType.LR, alias="constant_with_warmup") MindFormerRegister.register_cls( LinearWithWarmUpLR, module_type=MindFormerModuleType.LR, alias="linear") MindFormerRegister.register_cls( CosineWithWarmUpLR, module_type=MindFormerModuleType.LR, alias="cosine") MindFormerRegister.register_cls( CosineWithRestartsAndWarmUpLR, module_type=MindFormerModuleType.LR, alias="cosine_with_restarts") MindFormerRegister.register_cls( PolynomialWithWarmUpLR, module_type=MindFormerModuleType.LR, alias="polynomial") ############################### build for WSD ############################## MindFormerRegister.register_cls( WarmUpStableDecayLR, module_type=MindFormerModuleType.LR, alias="warmup_stable_decay") MindFormerRegister.register_cls( WarmUpCosineWithDecayLR, module_type=MindFormerModuleType.LR, alias="warmup_consin_decay") register_ms_lr() register_mf_lr() pengcheng.mind-test/mindformers/core/lr/lr_schedule.py000066400000000000000000000357061471400134000234610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/PanGu-%CE%B1/utils.py # https://github.com/huggingface/transformers/blob/main/src/transformers/optimization.py # ============================================================================ """Self-Define LR Schedule.""" import math import numpy as np from mindspore.ops import operations as P import mindspore.common.dtype as mstype from mindspore.nn.learning_rate_schedule import LearningRateSchedule, PolynomialDecayLR, WarmUpLR, CosineDecayLR from mindspore.common.tensor import Tensor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = [ 'LearningRateWiseLayer', 'WarmUpDecayLR', 'ConstantWarmUpLR', 'LinearWithWarmUpLR', 'CosineWithWarmUpLR', 'CosineWithRestartsAndWarmUpLR', 'PolynomialWithWarmUpLR'] @MindFormerRegister.register(MindFormerModuleType.LR) class ConstantWarmUpLR(LearningRateSchedule): """ConstantWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, warmup_lr_init: float = 0., **kwargs): super(ConstantWarmUpLR, self).__init__() warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.one_constant = Tensor(1.0, mstype.float32) self.greater = P.Greater() self.total_steps = kwargs.get('total_steps', None) def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: percent = self.one_constant learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class LinearWithWarmUpLR(LearningRateSchedule): """LinearWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, warmup_lr_init: float = 0.): super(LinearWithWarmUpLR, self).__init__() linear_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.linear_steps = Tensor(linear_steps, mstype.float32) self.greater = P.Greater() self.max = P.Maximum() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: percent = self.max(self.zero_constant, (self.total_steps - global_step) / self.linear_steps) learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class CosineWithWarmUpLR(LearningRateSchedule): """CosineWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, num_cycles: float = 0.5, lr_end: float = 0., warmup_lr_init: float = 0.): super(CosineWithWarmUpLR, self).__init__() cosine_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: progress = (global_step - self.warmup_steps) / self.cosine_steps percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * self.num_cycles * 2.0 * progress))) learning_rate = self.lr_end + (self.learning_rate - self.lr_end) * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpCosineWithDecayLR(LearningRateSchedule): """WarmUpCosineWithDecayLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, decay_last_steps: int, num_cycles: float = 0.5, lr_middle: float = 0., lr_end: float = 0., warmup_lr_init: float = 0.): super(WarmUpCosineWithDecayLR, self).__init__() decay_steps = max(1, total_steps - decay_last_steps) self.decay_last_steps = decay_last_steps self.decay_start_step_idx = Tensor(decay_steps, mstype.float32) cosine_steps = max(1, total_steps - warmup_steps - decay_last_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_middle = lr_middle self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() print(f"Use ConSin WSD scheduler total: {total_steps}, warmup: {warmup_steps}, cosin: {cosine_steps}, decay: {decay_last_steps}") print(f"learning_rate: {learning_rate}, lr_middle: {lr_middle}, lr_end： {lr_end}") def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent elif self.greater(self.decay_start_step_idx, global_step): progress = (global_step - self.warmup_steps) / self.cosine_steps percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * self.num_cycles * 2.0 * progress))) learning_rate = self.lr_middle + (self.learning_rate - self.lr_middle) * percent else: grad = Tensor((self.lr_end - self.lr_middle), mstype.float32) / self.decay_last_steps learning_rate = (global_step - self.decay_start_step_idx) * grad + Tensor(self.lr_middle, mstype.float32) return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class CosineWithRestartsAndWarmUpLR(LearningRateSchedule): """CosineWithRestartsAndWarmUpLR.""" def __init__(self, learning_rate: float, total_steps: int, warmup_steps: int, num_cycles: float = 0.5, lr_end: float = 0., warmup_lr_init: float = 0.): super(CosineWithRestartsAndWarmUpLR, self).__init__() cosine_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.one_constant = Tensor(1.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: progress = (global_step - self.warmup_steps) / self.cosine_steps if self.greater(self.one_constant, progress): percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * ((self.num_cycles * progress) % self.one_constant)))) learning_rate = self.lr_end + (self.learning_rate - self.lr_end) * percent else: return self.zero_constant return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class PolynomialWithWarmUpLR(LearningRateSchedule): """PolynomialWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, lr_end: float = 1e-7, power: float = 1.0, warmup_lr_init: float = 0.): super(PolynomialWithWarmUpLR, self).__init__() decay_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) if not learning_rate > lr_end: raise ValueError(f"lr_end ({lr_end}) must be be smaller than initial lr ({learning_rate})") self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.lr_end = lr_end self.power = power self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.decay_steps = Tensor(decay_steps, mstype.float32) self.greater = P.Greater() self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: lr_range = self.learning_rate - self.lr_end pct_remaining = 1 - (global_step - self.warmup_steps) / self.decay_steps decay = lr_range * pct_remaining ** self.power + self.lr_end percent = decay / self.learning_rate learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class LearningRateWiseLayer(LearningRateSchedule): """LearningRateWiseLayer.""" def __init__(self, base_lr, lr_scale): super(LearningRateWiseLayer, self).__init__() self.base_lr = base_lr self.lr_scale = lr_scale def construct(self, global_step): lr = self.base_lr(global_step) return self.lr_scale * lr @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpDecayLR(LearningRateSchedule): """ Warmup-decay learning rate for Bert network. """ def __init__(self, learning_rate, end_learning_rate, warmup_steps, decay_steps, power=1.0, use_cosine=False): super(WarmUpDecayLR, self).__init__() self.warmup_flag = False if warmup_steps > 0: self.warmup_flag = True self.warmup_lr = WarmUpLR(learning_rate, warmup_steps) self.decay_lr = PolynomialDecayLR(learning_rate, end_learning_rate, decay_steps, power) self.cosine_decay_lr = CosineDecayLR(end_learning_rate, learning_rate, decay_steps) self.warmup_steps = Tensor(np.array([warmup_steps]).astype(np.float32)) self.greater = P.Greater() self.one = Tensor(np.array([1.0]).astype(np.float32)) self.cast = P.Cast() self.use_cosine = use_cosine def construct(self, global_step): """dynamic learning rate""" if not self.use_cosine: decay_lr = self.decay_lr(global_step) else: decay_lr = self.cosine_decay_lr(global_step) if self.warmup_flag: is_warmup = self.cast(self.greater(self.warmup_steps, global_step), mstype.float32) warmup_lr = self.warmup_lr(global_step) lr = (self.one - is_warmup) * decay_lr + is_warmup * warmup_lr else: lr = decay_lr return lr @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpStableDecayLR(LearningRateSchedule): def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, decay_last_steps: int, lr_end: float = 0., warmup_lr_init: float = 0.): super(WarmUpStableDecayLR, self).__init__() warmup_steps = max(1, warmup_steps) decay_steps = max(1, total_steps - decay_last_steps) self.decay_last_steps = decay_last_steps self.decay_start_step_idx = Tensor(decay_steps, mstype.float32) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = Tensor(warmup_lr_init, mstype.float32) self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.greater = P.Greater() self.mul = P.Mul() self.cast = P.Cast() print(f"use WSD scheduler total: {total_steps}, warmup: {warmup_steps}, decay: {decay_last_steps}") def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent elif self.greater(global_step, self.decay_start_step_idx): grad = Tensor((self.lr_end - self.learning_rate), mstype.float32) / self.decay_last_steps learning_rate = (global_step - self.decay_start_step_idx) * grad + Tensor(self.learning_rate, mstype.float32) else: learning_rate = Tensor(self.learning_rate, mstype.float32) return learning_rate pengcheng.mind-test/mindformers/core/metric/000077500000000000000000000000001471400134000214515ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/metric/.ipynb_checkpoints/000077500000000000000000000000001471400134000252425ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/metric/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000014401471400134000314770ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Metric.""" from .build_metric import build_metric from .metric import * __all__ = [] __all__.extend(metric.__all__) pengcheng.mind-test/mindformers/core/metric/.ipynb_checkpoints/build_metric-checkpoint.py000066400000000000000000000051731471400134000324110ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Metric API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_metric( config: dict = None, default_args: dict = None, module_type: str = 'metric', class_name: str = None, **kwargs): r"""Build metric For MindFormer. Instantiate the metric from MindFormerRegister's registry. Args: config (dict, list): The task metric's config. Default: None. default_args (dict): The default argument of metric API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'metric'. class_name (str): The class name of metric API. Default: None. Return: The function instance of metric API. Examples: >>> from mindformers import build_metric >>> metric_config = {'type': 'Accuracy', 'eval_type': 'classification'} >>> # 1) use config dict to build metric >>> metric_from_config = build_metric(metric_config) >>> # 2) use class name to build metric >>> metric_class_name = build_metric(class_name='Accuracy', eval_type='classification') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.METRIC, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_mt(): """ register MindSpore builtin LR class. """ for module_name in dir(nn.metrics): if module_name.startswith('__'): continue ms_metric = getattr(nn.metrics, module_name) if inspect.isclass(ms_metric): MindFormerRegister.register_cls( ms_metric, MindFormerModuleType.METRIC) register_ms_mt() pengcheng.mind-test/mindformers/core/metric/.ipynb_checkpoints/metric-checkpoint.py000066400000000000000000001056111471400134000312300ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/lonePatient/daguan_2019_rank9/blob/master/pydatagrand/train/ner_utils.py # ============================================================================ """MindFormer Self-Define Metric.""" import os import sys import re import collections import json import math import string import shutil import six import jieba import numpy as np from rouge_chinese import Rouge from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction import mindspore.nn as nn import mindspore as ms from mindspore.ops import operations as P from mindspore.communication import get_group_size, get_rank from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models import BasicTokenizer from mindformers.core.loss import CrossEntropyLoss from ...dataset.labels import cluener_labels __all__ = ['EntityScore', 'SQuADMetric', 'PerplexityMetric', 'ADGENMetric', 'PromptAccMetric', 'EmF1Metric'] @MindFormerRegister.register(MindFormerModuleType.METRIC) class EntityScore(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self): super(EntityScore, self).__init__() self.label2id = {label: label_id for label_id, label in enumerate(cluener_labels)} self.id2label = {label_id: label for label, label_id in self.label2id.items()} self.clear() def clear(self): "Initialization." self.origins = [] self.founds = [] self.rights = [] def update(self, *inputs): """Update results for every batch""" batch_logits = inputs[0].asnumpy() batch_label_ids = inputs[1].asnumpy() batch_pred_ids = np.argmax(batch_logits, axis=2).tolist() pred_paths = [[self.id2label[id_] for id_ in pred_ids] for pred_ids in batch_pred_ids] label_paths = [[self.id2label[id_] for id_ in label_ids] for label_ids in batch_label_ids] for label_path, pre_path in zip(label_paths, pred_paths): label_entities = self.get_entities_bios(label_path) pred_entities = self.get_entities_bios(pre_path) self.origins.extend(label_entities) self.founds.extend(pred_entities) self.rights.extend([pred_entity for pred_entity in pred_entities if pred_entity in label_entities]) def eval(self): """Compute final results.""" class_info = {} origin_counter = collections.Counter([x[0] for x in self.origins]) found_counter = collections.Counter([x[0] for x in self.founds]) right_counter = collections.Counter([x[0] for x in self.rights]) for type_, count in origin_counter.items(): origin = count found = found_counter.get(type_, 0) right = right_counter.get(type_, 0) recall, precision, f1 = self.compute(origin, found, right) class_info[type_] = {"precision": round(precision, 4), 'recall': round(recall, 4), 'f1': round(f1, 4)} origin = len(self.origins) found = len(self.founds) right = len(self.rights) recall, precision, f1 = self.compute(origin, found, right) return {"precision": round(precision, 4), 'recall': round(recall, 4), 'f1': round(f1, 4)}, class_info def compute(self, origin, found, right): """Compute f1, precision and recall.""" recall = 0 if origin == 0 else (right / origin) precision = 0 if found == 0 else (right / found) f1 = 0. if recall + precision == 0 else (2 * precision * recall) / (precision + recall) return recall, precision, f1 def get_entities_bios(self, seq): """Get entities from sequence.""" chunks = [] chunk = [-1, -1, -1] for indx, tag in enumerate(seq): if tag.startswith("S-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[2] = indx chunk[0] = tag.split('-')[1] chunks.append(chunk) chunk = [-1, -1, -1] if tag.startswith("B-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[0] = tag.split('-')[1] elif tag.startswith('I-') and chunk[1] != -1: entity_type = tag.split('-')[1] if entity_type == chunk[0]: chunk[2] = indx if indx == len(seq) - 1: chunks.append(chunk) else: if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] return chunks @MindFormerRegister.register(MindFormerModuleType.METRIC) class SQuADMetric(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self, dataset_dir, n_best_size=20, max_answer_len=30, do_lower_case=True, temp_file_dir="./squad_temp"): self.outputs = [] self.temp_file_dir = temp_file_dir temp_examples_file = os.path.join(temp_file_dir, "temp_examples.json") temp_features_file = os.path.join(temp_file_dir, "temp_features.json") self.all_examples = self._load_temp_data(temp_examples_file) self.all_features = self._load_temp_data(temp_features_file) self.dev_file_path = os.path.join(dataset_dir, "dev-v1.1.json") self.basic_tokenizer = BasicTokenizer(do_lower_case) self.n_best_size = n_best_size self.max_answer_len = max_answer_len def clear(self): """Clearing the internal evaluation result.""" return def update(self, *inputs): """Update results for every batch""" ids = inputs[0].asnumpy() start = inputs[1].asnumpy() end = inputs[2].asnumpy() batch_size = len(ids) RawResult = collections.namedtuple("RawResult", ["unique_id", "start_logits", "end_logits"]) for i in range(batch_size): unique_id = int(ids[i]) start_logits = [float(x) for x in start[i].flat] end_logits = [float(x) for x in end[i].flat] self.outputs.append(RawResult(unique_id=unique_id, start_logits=start_logits, end_logits=end_logits)) def eval(self): """Compute final result""" predictions = self._get_predictions() with open(self.dev_file_path) as ds: dataset_json = json.load(ds) dataset = dataset_json['data'] f1 = exact_match = total = 0 for article in dataset: for paragraph in article['paragraphs']: for qa in paragraph['qas']: total += 1 if qa['id'] not in predictions: message = 'Unanswered question ' + qa['id'] + \ ' will receive score 0.' print(message, file=sys.stderr) continue ground_truths = list(map(lambda x: x['text'], qa['answers'])) if not ground_truths: continue prediction = predictions[qa['id']] exact_match += self._metric_max_over_ground_truths( self._exact_match_score, prediction, ground_truths) f1 += self._metric_max_over_ground_truths( self._f1_score, prediction, ground_truths) exact_match = 100.0 * exact_match / total f1 = 100.0 * f1 / total self._remove_temp_data() return {'exact_match': exact_match, 'f1': f1} def _remove_temp_data(self): shutil.rmtree(self.temp_file_dir) def _load_temp_data(self, temp_file_path): with open(temp_file_path, "r", encoding="utf-8") as f: data = [] for line in f.readlines(): data.append(json.loads(line.strip())) return data def _normalize_answer(self, s): """Lower text and remove punctuation, articles and extra whitespace.""" def remove_articles(text): return re.sub(r'\b(a|an|the)\b', ' ', text) def white_space_fix(text): return ' '.join(text.split()) def remove_punc(text): exclude = set(string.punctuation) return ''.join(ch for ch in text if ch not in exclude) def lower(text): return text.lower() return white_space_fix(remove_articles(remove_punc(lower(s)))) def _f1_score(self, prediction, ground_truth): """calculate f1 score""" prediction_tokens = self._normalize_answer(prediction).split() ground_truth_tokens = self._normalize_answer(ground_truth).split() common = collections.Counter(prediction_tokens) & collections.Counter(ground_truth_tokens) num_same = sum(common.values()) if num_same == 0: return 0 precision = 1.0 * num_same / len(prediction_tokens) recall = 1.0 * num_same / len(ground_truth_tokens) f1 = (2 * precision * recall) / (precision + recall) return f1 def _exact_match_score(self, prediction, ground_truth): return self._normalize_answer(prediction) == self._normalize_answer(ground_truth) def _metric_max_over_ground_truths(self, metric_fn, prediction, ground_truths): scores_for_ground_truths = [] for ground_truth in ground_truths: score = metric_fn(prediction, ground_truth) scores_for_ground_truths.append(score) return max(scores_for_ground_truths) def _get_predictions(self): """Get final predictions""" example_index_to_features = collections.defaultdict(list) for feature in self.all_features: example_index_to_features[feature["example_index"]].append(feature) unique_id_to_result = {} for result in self.outputs: unique_id_to_result[result.unique_id] = result all_predictions = collections.OrderedDict() for (example_index, example) in enumerate(self.all_examples): features = example_index_to_features[example_index] prelim_predictions = self._get_prelim_predictions(features, unique_id_to_result) nbest = self._get_nbest(prelim_predictions, features, example) total_scores = [] best_non_null_entry = None for entry in nbest: total_scores.append(entry.start_logit + entry.end_logit) if not best_non_null_entry: if entry.text: best_non_null_entry = entry probs = self._compute_softmax(total_scores) nbest_json = [] for (i, entry) in enumerate(nbest): output = collections.OrderedDict() output["text"] = entry.text output["probability"] = probs[i] output["start_logit"] = entry.start_logit output["end_logit"] = entry.end_logit nbest_json.append(output) assert len(nbest_json) >= 1 all_predictions[example["qas_id"]] = nbest_json[0]["text"] return all_predictions def _get_prelim_predictions(self, features, unique_id_to_result): """get prelim predictions""" _PrelimPrediction = collections.namedtuple( "PrelimPrediction", ["feature_index", "start_index", "end_index", "start_logit", "end_logit"]) prelim_predictions = [] # keep track of the minimum score of null start+end of position 0 for (feature_index, feature) in enumerate(features): if feature["unique_id"] not in unique_id_to_result: continue result = unique_id_to_result[feature["unique_id"]] start_indexes = self._get_best_indexes(result.start_logits) end_indexes = self._get_best_indexes(result.end_logits) # if we could have irrelevant answers, get the min score of irrelevant for start_index in start_indexes: for end_index in end_indexes: # We could hypothetically create invalid predictions, e.g., predict # that the start of the span is in the question. We throw out all # invalid predictions. if start_index >= len(feature["tokens"]): continue if end_index >= len(feature["tokens"]): continue if str(start_index) not in feature["token_to_orig_map"]: continue if str(end_index) not in feature["token_to_orig_map"]: continue if not feature["token_is_max_context"].get(str(start_index), False): continue if end_index < start_index: continue length = end_index - start_index + 1 if length > self.max_answer_len: continue prelim_predictions.append( _PrelimPrediction( feature_index=feature_index, start_index=start_index, end_index=end_index, start_logit=result.start_logits[start_index], end_logit=result.end_logits[end_index])) prelim_predictions = sorted( prelim_predictions, key=lambda x: (x.start_logit + x.end_logit), reverse=True) return prelim_predictions def _get_nbest(self, prelim_predictions, features, example): """get nbest predictions""" _NbestPrediction = collections.namedtuple( "NbestPrediction", ["text", "start_logit", "end_logit"]) seen_predictions = {} nbest = [] for pred in prelim_predictions: if len(nbest) >= self.n_best_size: break feature = features[pred.feature_index] if pred.start_index > 0: # this is a non-null prediction tok_tokens = feature["tokens"][pred.start_index:(pred.end_index + 1)] orig_doc_start = feature["token_to_orig_map"][str(pred.start_index)] orig_doc_end = feature["token_to_orig_map"][str(pred.end_index)] orig_tokens = example["doc_tokens"][orig_doc_start:(orig_doc_end + 1)] tok_text = " ".join(tok_tokens) # De-tokenize WordPieces that have been split off. tok_text = tok_text.replace(" ##", "") tok_text = tok_text.replace("##", "") # Clean whitespace tok_text = tok_text.strip() tok_text = " ".join(tok_text.split()) orig_text = " ".join(orig_tokens) final_text = self._get_final_text(tok_text, orig_text) if final_text in seen_predictions: continue seen_predictions[final_text] = True else: final_text = "" seen_predictions[final_text] = True nbest.append( _NbestPrediction( text=final_text, start_logit=pred.start_logit, end_logit=pred.end_logit)) # In very rare edge cases we could have no valid predictions. So we # just create a nonce prediction in this case to avoid failure. if not nbest: nbest.append(_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0)) assert len(nbest) >= 1 return nbest def _compute_softmax(self, scores): """Compute softmax probability over raw logits.""" if not scores: return [] max_score = None for score in scores: if max_score is None or score > max_score: max_score = score exp_scores = [] total_sum = 0.0 for score in scores: x = math.exp(score - max_score) exp_scores.append(x) total_sum += x probs = [] for score in exp_scores: probs.append(score / total_sum) return probs def _get_final_text(self, pred_text, orig_text): """Project the tokenized prediction back to the original text.""" def _strip_spaces(text): ns_chars = [] ns_to_s_map = collections.OrderedDict() for (i, c) in enumerate(text): if c == " ": continue ns_to_s_map[len(ns_chars)] = i ns_chars.append(c) ns_text = "".join(ns_chars) return (ns_text, ns_to_s_map) tok_text = " ".join(self.basic_tokenizer.tokenize(orig_text)) start_position = tok_text.find(pred_text) if start_position == -1: return orig_text end_position = start_position + len(pred_text) - 1 (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text) (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text) if len(orig_ns_text) != len(tok_ns_text): return orig_text tok_s_to_ns_map = {} for (i, tok_index) in six.iteritems(tok_ns_to_s_map): tok_s_to_ns_map[tok_index] = i orig_start_position = None if start_position in tok_s_to_ns_map: ns_start_position = tok_s_to_ns_map[start_position] if ns_start_position in orig_ns_to_s_map: orig_start_position = orig_ns_to_s_map[ns_start_position] if orig_start_position is None: return orig_text orig_end_position = None if end_position in tok_s_to_ns_map: ns_end_position = tok_s_to_ns_map[end_position] if ns_end_position in orig_ns_to_s_map: orig_end_position = orig_ns_to_s_map[ns_end_position] if orig_end_position is None: return orig_text output_text = orig_text[orig_start_position:(orig_end_position + 1)] return output_text def _get_best_indexes(self, logits): """Get the n-best logits from a list.""" index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True) best_indexes = [] for (i, score) in enumerate(index_and_score): if i >= self.n_best_size: break best_indexes.append(score[0]) return best_indexes @MindFormerRegister.register(MindFormerModuleType.METRIC) class PerplexityMetric(nn.Metric): """Compute the loss and PPL of each entity""" def __init__(self): super(PerplexityMetric, self).__init__() self.num_data = None self.total_loss = None self.loss = CrossEntropyLoss() self.pipeline_stages = ms.get_auto_parallel_context('pipeline_stages') self.pipeline_parallel = self.pipeline_stages > 1 self.rank_id = 0 self.device_num = 1 self.cast = P.Cast() self.reshape = P.Reshape() self.not_equal = P.NotEqual() self.sub = P.Sub() if self.pipeline_parallel: self.rank_id = get_rank() self.device_num = get_group_size() per_stage_device_num = self.device_num // self.pipeline_stages stage_id = self.rank_id // per_stage_device_num self.is_last_stage = (stage_id == self.pipeline_stages - 1) self.parallel_mode = ms.get_auto_parallel_context("parallel_mode") self.full_batch = ms.get_auto_parallel_context("full_batch") self.auto_parallel = self.parallel_mode in ['semi_auto_parallel', 'auto_parallel'] def clear(self): """Clearing the internal evaluation result.""" self.num_data = 0 self.total_loss = 0.0 def update(self, *inputs): """Update results for every batch""" if self.pipeline_parallel: if not self.is_last_stage: return if self.auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) # logits, labels, input_mask = inputs[0], inputs[1], inputs[2] # # input_mask was added 1 in GPT2LMModel to avoid allgather issue in Mindspore1.10 # input_mask = self.sub(input_mask, 1) # batch_size, seq_length, _ = logits.shape # logits = self.reshape(logits[::, :-1, ::], (batch_size * (seq_length - 1), -1)) # labels = self.reshape(labels[::, 1:], (-1,)) # input_mask = self.reshape(input_mask[::, 1:], (-1,)) # loss = self.loss(logits, labels, input_mask) loss = inputs[0] loss = float(loss.asnumpy()) self.total_loss += loss self.num_data += 1 if self.auto_parallel: ms.set_auto_parallel_context(parallel_mode=self.parallel_mode, full_batch=True, pipeline_stages=self.pipeline_stages) else: # logits, labels, input_mask = inputs[0], inputs[1], inputs[2] # batch_size, seq_length, _ = logits.shape # logits = self.reshape(logits[::, :-1, ::], (batch_size * (seq_length - 1), -1)) # labels = self.reshape(labels[::, 1:], (-1,)) # input_mask = self.reshape(input_mask[::, 1:], (-1,)) # loss = self.loss(logits, labels, input_mask) loss = inputs[0] loss = float(loss.asnumpy()) self.total_loss += loss self.num_data += 1 def eval(self): """Compute final result""" if self.pipeline_parallel and not self.is_last_stage: return None avg_loss = float(self.total_loss / self.num_data) result = {"loss": avg_loss, "PPL": math.exp(avg_loss)} if self.pipeline_parallel: print("Average Loss and PPL Metric:", result) return result @MindFormerRegister.register(MindFormerModuleType.METRIC) class ADGENMetric(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self): super(ADGENMetric, self).__init__() self.score_dict = { "rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": [] } def clear(self): self.score_dict = { "rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": [] } def update(self, *inputs): """Update results for every batch""" preds = inputs[0] # list[numpy] labels = inputs[1] # numpy if isinstance(preds, tuple): preds = preds[0] for pred, label in zip(preds, labels): print(f"pred is:\n {pred}\n", f"label is:\n {label}") hypothesis = list(jieba.cut(pred)) reference = list(jieba.cut(label)) rouge = Rouge() scores = rouge.get_scores(' '.join(hypothesis), ' '.join(reference)) result = scores[0] for k, v in result.items(): self.score_dict[k].append(round(v["f"] * 100, 4)) bleu_score = sentence_bleu([list(label)], list(pred), smoothing_function=SmoothingFunction().method3) self.score_dict["bleu-4"].append(round(bleu_score * 100, 4)) def eval(self): """Compute final result""" for k, v in self.score_dict.items(): self.score_dict[k] = float(np.mean(v)) print('metric: ADGENMetric\n' + f'rouge-1: {self.score_dict["rouge-1"]:.4f}\n' + f'rouge-2: {self.score_dict["rouge-2"]:.4f}\n' + f'rouge-l: {self.score_dict["rouge-l"]:.4f}\n' + f'bleu-4: {self.score_dict["bleu-4"]:.4f}') return self.score_dict @MindFormerRegister.register(MindFormerModuleType.METRIC) class PromptAccMetric(nn.Metric): r""" Computes the prompt acc of each entity. The prompt acc is the accuracy of text classification base on building prompt. The accurate index is the index of the prompt which has the minimum perplexity. 1. Build the prompt for this metric is described as follows: 这是关于**体育**的文章：$passage 这是关于**文化**的文章：$passage 2. Computes perplexity of each generated context based on prompt. Perplexity is a measurement about how well a probability distribution or a model predicts a sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows: .. math:: PP(W)=P(w_{1}w_{2}...w_{N})^{-\frac{1}{N}}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}...w_{N})}} Where :math:`w` represents words in corpus. 3. Compute classification result by choosing the index of the prompt which has the minimum perplexity. 4. Count the number of correctly classified and the total number of samples and compute the acc as follows: .. math:: \text{accuracy} =\frac{\text{correct_sample_nums}}{\text{total_sample_nums}} """ def __init__(self): super(PromptAccMetric, self).__init__() self.num_data = None self.total_acc_num = None self.loss = CrossEntropyLoss() self.pipeline_stages = ms.get_auto_parallel_context('pipeline_stages') self.pipeline_parallel = self.pipeline_stages > 1 self.last_card_id = 0 self.rank_id = 0 self.device_num = 1 self.cast = P.Cast() self.reshape = P.Reshape() self.equal = P.Equal() self.softmax = P.Softmax() self.argmin = P.Argmin() self.sum = P.ReduceSum() if self.pipeline_parallel: self.rank_id = get_rank() self.device_num = get_group_size() per_stage_device_num = self.device_num // self.pipeline_stages stage_id = self.rank_id // per_stage_device_num self.is_last_stage = (stage_id == self.pipeline_stages - 1) self.parallel_mode = ms.get_auto_parallel_context("parallel_mode") self.full_batch = ms.get_auto_parallel_context("full_batch") self.auto_parallel = self.parallel_mode in ['semi_auto_parallel', 'auto_parallel'] def clear(self): """Clearing the internal evaluation result.""" self.num_data = 0 self.total_acc_num = 0 def calculate_circle(self, *inputs): """The main calculate logic.""" logits, input_ids, input_mask, labels = inputs[0], inputs[1], inputs[2], inputs[3] batch_size, num_labels, seq_length, _ = logits.shape logits = self.reshape(logits, (batch_size*num_labels, seq_length, -1)) ppl_list = [] for index in range(batch_size*num_labels): sub_logits, sub_tokens, sub_mask_list = logits[index], input_ids[index], input_mask[index] sub_logits = sub_logits[:-1, ::] sub_tokens = sub_tokens[1:] sub_mask_list = sub_mask_list[1:] loss = self.loss(sub_logits, sub_tokens, sub_mask_list) loss = float(loss.asnumpy()) ppl_list.append(loss) # smaller, better ppl_ms = ms.Tensor(ppl_list, dtype=ms.float32) ppl_ms = self.reshape(ppl_ms, (batch_size, num_labels)) ppl_ms = self.cast(self.argmin(ppl_ms), ms.int32) label = self.reshape(labels, (-1,)) cur_acc_num = self.cast(self.equal(ppl_ms, label), ms.float16).sum().asnumpy() self.num_data += batch_size self.total_acc_num += cur_acc_num def update(self, *inputs): """Update results for every batch""" if self.pipeline_parallel: if not self.is_last_stage: return if self.auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) self.calculate_circle(*inputs) if self.auto_parallel: ms.set_auto_parallel_context(parallel_mode=self.parallel_mode, full_batch=True, pipeline_stages=self.pipeline_stages) else: self.calculate_circle(*inputs) print("Current data num is {}, total acc num is {}, ACC is {}".format( self.num_data, self.total_acc_num, "%.3f" % (self.total_acc_num / self.num_data))) return def eval(self): """Compute final result""" if self.pipeline_parallel and not self.is_last_stage: return None acc_rate = float(self.total_acc_num / self.num_data) result = {"Acc": acc_rate} print(f"Acc: {('%.3f' % result.get('Acc', 0))}, total_acc_num: {self.total_acc_num}, " f"total_num: {self.num_data}") return result @MindFormerRegister.register(MindFormerModuleType.METRIC) class EmF1Metric(nn.Metric): """ Compute the Em/F1 scores of examples. Em score is the prediction exact matches the labels except the punctuations. For example, the question is "河南的省会是哪里？" and the label is "郑州市", when prediction is "郑州市", Em score is 100; when prediction is "郑州市。", Em score is 100; when prediction is "郑州", Em score is 0. F1 score is calculated as 2*precision*recall/(precision+recall), the precision and recall are calculated as precision = lcs_length/len(prediction_segment), recall = lcs_length/len(label_segment), lcs_length is the length of the longest common subsequence. """ def __init__(self): super(EmF1Metric, self).__init__() self.gens = None self.labels = None self.metrics = None self.num_data = None def clear(self): """Clearing the internal evaluation result.""" self.gens = [] self.labels = [] self.metrics = { 'Em': 0.0, 'F1': 0.0 } self.num_data = 0 def update(self, *inputs): """Update results for every batch""" gen, label = inputs[0], inputs[1] for i in range(len(gen)): gen[i] = gen[i].strip() gen[i] = gen[i].split("\n")[0] print(f"pred is:\n {gen}\n", f"label is:\n {label}") self.gens.extend(gen) self.labels.extend(label) self.num_data += len(gen) result, current_count = self.evaluate_pairs(gen, label) print("The F1/Em of this example is: ", result) if self.num_data % 10 == 0: result, current_count = self.evaluate_pairs(self.gens, self.labels) print(f"F1 score: {result.get('F1', 0)}, Em score: {result.get('Em', 0)}, current_count: {current_count}") def eval(self): """Compute final result""" result, total_count = self.evaluate_pairs(self.gens, self.labels) print(f"F1 score: {result.get('F1', 0)}, Em score: {result.get('Em', 0)}, total_count: {total_count}") return result def mixed_segmentation(self, in_str, rm_punc=False): """cut input for calculating lcs""" in_str = str(in_str).lower().strip() segs_out = [] temp_str = "" sp_char = ['-', ':', '_', '*', '^', '/', '\\', '~', '`', '+', '=', '，', '。', '：', '？', '！', '“', '”', '；', '’', '《', '》', '……', '·', '、', '「', '」', '（', '）', '－', '～', '『', '』'] for char in in_str: if rm_punc and char in sp_char: continue if re.search(r'[\u4e00-\u9fa5]', char) or char in sp_char: if temp_str != "": ss = list(jieba.cut(temp_str)) segs_out.extend(ss) temp_str = "" segs_out.append(char) else: temp_str += char if temp_str != "": ss = list(jieba.cut(temp_str)) segs_out.extend(ss) return segs_out def remove_punctuation(self, in_str): """remove punctuations in inputs""" in_str = str(in_str).lower().strip() sp_char = ['-', ':', '_', '*', '^', '/', '\\', '~', '`', '+', '=', '，', '。', '：', '？', '！', '“', '”', '；', '’', '《', '》', '……', '·', '、', '「', '」', '（', '）', '－', '～', '『', '』'] out_segs = [] for char in in_str: if char in sp_char: continue else: out_segs.append(char) return ''.join(out_segs) def find_lcs(self, s1, s2): """calculate the length of lcs""" m = [[0 for i in range(len(s2) + 1)] for j in range(len(s1) + 1)] mmax = 0 p = 0 for i in range(len(s1)): for j in range(len(s2)): if s1[i] == s2[j]: m[i + 1][j + 1] = m[i][j] + 1 if m[i + 1][j + 1] > mmax: mmax = m[i + 1][j + 1] p = i + 1 return s1[p - mmax:p], mmax def calc_f1_score(self, answers, prediction): """calculate f1 score""" f1_scores = [] for ans in answers: ans_segs = self.mixed_segmentation(ans, rm_punc=True) prediction_segs = self.mixed_segmentation(prediction, rm_punc=True) _, lcs_len = self.find_lcs(ans_segs, prediction_segs) if lcs_len == 0: f1_scores.append(0) continue precision = 1.0 * lcs_len / len(prediction_segs) recall = 1.0 * lcs_len / len(ans_segs) f1 = (2 * precision * recall) / (precision + recall) f1_scores.append(f1) return max(f1_scores) def calc_em_score(self, answers, prediction): """calculate em score""" em = 0 for ans in answers: ans_ = self.remove_punctuation(ans) prediction_ = self.remove_punctuation(prediction) if ans_ == prediction_: em = 1 break return em def evaluate_pairs(self, pred_, ans_): """calculate metric""" f1 = 0 em = 0 total_count = 0 for (prediction, answer) in zip(pred_, ans_): total_count += 1 f1 += self.calc_f1_score([answer], prediction) em += self.calc_em_score([answer], prediction) if total_count > 0: f1_score = 100.0 * f1 / total_count em_score = 100.0 * em / total_count result = {'F1': f1_score, 'Em': em_score} else: print("total_count is zero") result = {} return result, total_count pengcheng.mind-test/mindformers/core/metric/__init__.py000066400000000000000000000014401471400134000235610ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Metric.""" from .build_metric import build_metric from .metric import * __all__ = [] __all__.extend(metric.__all__) pengcheng.mind-test/mindformers/core/metric/__pycache__/000077500000000000000000000000001471400134000236615ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/metric/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004511471400134000300510ustar00rootroot00000000000000a ?cf @s,dZddlmZddlTgZeejdS)zMindFormers Metric.) build_metric)*N)__doc__rmetric__all__extendrrK/home/ma-user/work/mindformers-baichuan/mindformers/core/metric/__init__.pys pengcheng.mind-test/mindformers/core/metric/__pycache__/build_metric.cpython-39.pyc000066400000000000000000000037551471400134000307660ustar00rootroot00000000000000a ?cf{ @sTdZddlZddlmZddlmZmZmZd eee e dddZ d d Z e dS) zBuild Metric API.N)nn)MindFormerRegisterMindFormerModuleTypeMindFormerConfigmetric)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)agBuild metric For MindFormer. Instantiate the metric from MindFormerRegister's registry. Args: config (dict, list): The task metric's config. Default: None. default_args (dict): The default argument of metric API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'metric'. class_name (str): The class name of metric API. Default: None. Return: The function instance of metric API. Examples: >>> from mindformers import build_metric >>> metric_config = {'type': 'Accuracy', 'eval_type': 'classification'} >>> # 1) use config dict to build metric >>> metric_from_config = build_metric(metric_config) >>> # 2) use class name to build metric >>> metric_class_name = build_metric(class_name='Accuracy', eval_type='classification') N)r) isinstancedictrrget_instance_from_cfgrMETRIC get_instance)rrr r kwargsrO/home/ma-user/work/mindformers-baichuan/mindformers/core/metric/build_metric.py build_metricsrcCsDttjD]4}|drq ttj|}t|r t|t j q dS)z& register MindSpore builtin LR class. __N) dirrmetrics startswithgetattrinspectisclassr register_clsrr) module_nameZ ms_metricrrrregister_ms_mt8s   r)NNrN) __doc__r mindsporermindformers.tools.registerrrrr strrrrrrrs  ! pengcheng.mind-test/mindformers/core/metric/__pycache__/metric.cpython-39.pyc000066400000000000000000000647731471400134000276160ustar00rootroot00000000000000a Qff@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl mZddlmZmZddlmZddlZddlmZddlmZmZddlmZmZddlm Z dd l!m"Z"d d l#m$Z$gd Z%e&ej'Gd ddej(Z)e&ej'Gdddej(Z*e&ej'Gdddej(Z+e&ej'Gdddej(Z,e&ej'Gdddej(Z-e&ej'Gdddej(Z.dS)zMindFormer Self-Define Metric.N)Rouge) sentence_bleuSmoothingFunction) operations)get_group_sizeget_rank)MindFormerRegisterMindFormerModuleType)BasicTokenizer)CrossEntropyLoss)cluener_labels) EntityScore SQuADMetricPerplexityMetric ADGENMetricPromptAccMetric EmF1MetriccsHeZdZdZfddZddZddZdd Zd d Zd d Z Z S)r9Compute the f1, precision and recall score of each entitycsDtt|ddttD|_dd|jD|_|dS)NcSsi|]\}}||qSr).0label_idlabelrrI/home/ma-user/work/mindformers-baichuan/mindformers/core/metric/metric.py 2z(EntityScore.__init__..cSsi|]\}}||qSrr)rrrrrrr3r) superr__init__ enumerater Zlabel2iditemsid2labelclearself __class__rrr0szEntityScore.__init__cCsg|_g|_g|_dS)zInitialization.N)originsfoundsrightsr"rrrr!6szEntityScore.clearc s|d}|d}tj|dd}fdd|D}fdd|D}t||D]N\}}||} jj| j fdd| DqXd S) Update results for every batchr)axiscsg|]}fdd|DqS)csg|]}j|qSrr rid_r"rr Br1EntityScore.update...r)rZpred_idsr"rrr0Brz&EntityScore.update..csg|]}fdd|DqS)csg|]}j|qSrr-r.r"rrr0Crr1r)rZ label_idsr"rrr0Crcsg|]}|vr|qSrr)rZ pred_entity)label_entitiesrrr0JrN) asnumpynpargmaxtolistzipget_entities_biosr&extendr'r() r#inputsZ batch_logitsZbatch_label_idsZbatch_pred_idsZ pred_pathsZ label_pathsZ label_pathZpre_pathZ pred_entitiesr)r2r#rupdate<s      zEntityScore.updatec Csi}tdd|jD}tdd|jD}tdd|jD}|D]Z\}}|}||d}||d} |||| \} } } t| dt| dt| dd||<qNt |j}t |j}t |j} |||| \} } } t| dt| dt| dd|fS)zCompute final results.cSsg|] }|dqSrrrxrrrr0Orz$EntityScore.eval..cSsg|] }|dqSr<rr=rrrr0PrcSsg|] }|dqSr<rr=rrrr0Qrr) precisionrecallf1) collectionsCounterr&r'r(rgetcomputeroundlen) r#Z class_infoZorigin_counterZ found_counterZ right_countertype_countoriginfoundrightrAr@rBrrrevalLs  $   zEntityScore.evalcCsV|dkr dn||}|dkr dn||}||dkr8dnd||||}|||fS)z!Compute f1, precision and recall.rr+r)r#rKrLrMrAr@rBrrrrF^s$zEntityScore.computecCs>g}gd}t|D]"\}}|drz|ddkr>||gd}||d<||d<|dd|d<||gd}|dr|ddkr||gd}||d<|dd|d<q|d r|ddkr|dd}||dkr||d<|t|dkr8||q|ddkr0||gd}q|S) zGet entities from sequence.)rPrPzS-r+rPr*-rzB-zI-)r startswithappendsplitrH)r#seqchunkschunkindxtagZ entity_typerrrr8es8           zEntityScore.get_entities_bios) __name__ __module__ __qualname____doc__rr!r;rNrFr8 __classcell__rrr$rr,s rc@seZdZdZd'ddZdd Zd d Zd d ZddZddZ ddZ ddZ ddZ ddZ ddZddZddZd d!Zd"d#Zd$d%Zd&S)(rrT ./squad_tempcCsjg|_||_tj|d}tj|d}|||_|||_tj|d|_t ||_ ||_ ||_ dS)Nztemp_examples.jsonztemp_features.jsonz dev-v1.1.json) outputs temp_file_dirospathjoin_load_temp_data all_examples all_features dev_file_pathr basic_tokenizer n_best_sizemax_answer_len)r# dataset_dirrlrmZ do_lower_casercZtemp_examples_fileZtemp_features_filerrrrs   zSQuADMetric.__init__cCsdS)(Clearing the internal evaluation result.Nrr"rrrr!szSQuADMetric.clearc Gs|d}|d}|d}t|}tdgd}t|D]N}t||}dd||jD} dd||jD} |j||| | dqDd S) r)rr*r+ RawResult) unique_id start_logits end_logitscSsg|] }t|qSrfloatr=rrrr0rz&SQuADMetric.update..cSsg|] }t|qSrrtr=rrrr0rN) r3rHrC namedtuplerangeintflatrbrS) r#r:idsstartend batch_sizerpirqrrrsrrrr;s      zSQuADMetric.updatec Cs$|}t|j"}t|}|d}Wdn1s:0Yd}}}|D]}|dD]} | dD]} |d7}| d|vrd| dd } t| tjd qltt d d | d } | sql|| d} || |j | | 7}|| |j | | 7}qlq`qTd||}d||}| ||dS)Compute final resultdataNrZ paragraphsZqasr*idzUnanswered question z will receive score 0.)filecSs|dS)Ntextrr>rrrrz"SQuADMetric.eval..answersY@) exact_matchrB)_get_predictionsopenrjjsonloadprintsysstderrlistmap_metric_max_over_ground_truths_exact_match_score _f1_score_remove_temp_data)r# predictionsds dataset_jsondatasetrBrtotalZarticleZ paragraphZqamessage ground_truths predictionrrrrNs:  &         zSQuADMetric.evalcCst|jdSN)shutilrmtreercr"rrrrszSQuADMetric._remove_temp_datacCsXt|ddd6}g}|D]}|t|qWdn1sJ0Y|S)Nrzutf-8)encoding)r readlinesrSrloadsstrip)r#Ztemp_file_pathfrlinerrrrgs  4zSQuADMetric._load_temp_datacCs4dd}dd}dd}dd}|||||S) zALower text and remove punctuation, articles and extra whitespace.cSstdd|S)Nz\b(a|an|the)\b )resubrrrrremove_articlessz6SQuADMetric._normalize_answer..remove_articlescSsd|S)Nr)rfrTrrrrwhite_space_fixsz6SQuADMetric._normalize_answer..white_space_fixcs"ttjdfdd|DS)Nc3s|]}|vr|VqdSrr)rchexcluderr rzESQuADMetric._normalize_answer..remove_punc..)setstring punctuationrfrrrr remove_puncs z2SQuADMetric._normalize_answer..remove_punccSs|Sr)lowerrrrrrsz,SQuADMetric._normalize_answer..lowerr)r#srrrrrrr_normalize_answers zSQuADMetric._normalize_answerc Cs||}||}t|t|@}t|}|dkrHdSd|t|}d|t|}d||||} | S)calculate f1 scorer?r+)rrTrCrDsumvaluesrH) r#r ground_truthZprediction_tokensZground_truth_tokenscommonZnum_samer@rArBrrrrs zSQuADMetric._f1_scorecCs||||kSr)r)r#rrrrrrszSQuADMetric._exact_match_scorecCs*g}|D]}|||}||qt|Sr)rSmax)r#Z metric_fnrrZscores_for_ground_truthsrscorerrrrs   z*SQuADMetric._metric_max_over_ground_truthscCs8tt}|jD]}||d|qi}|jD]}|||j<q2t}t|j D]\}}||}| ||} | | ||} g} d} | D]$} | | j | j | s| jr| } q|| }g}t| D]D\}} t}| j|d<|||d<| j |d<| j |d<||qt|dksJ|dd||d <qT|S) zGet final predictions example_indexNr probability start_logit end_logitr*rZqas_id)rC defaultdictrrirSrbrq OrderedDictrrh_get_prelim_predictions _get_nbestrrr_compute_softmaxrH)r#Zexample_index_to_featuresfeatureunique_id_to_resultresultZall_predictionsrexamplefeaturesprelim_predictionsnbestZ total_scoresZbest_non_null_entryentryprobsZ nbest_jsonr~outputrrrrs<           zSQuADMetric._get_predictionsc Cs&tdgd}g}t|D]\}}|d|vr2q||d}||j}||j} |D]} | D]} | t|dkrxqb| t|dkrqbt| |dvrqbt| |dvrqb|dt| dsqb| | krqb| | d} | |j krqb| ||| | |j| |j| dqbqZqt |d d d d }|S) zget prelim predictionsZPrelimPrediction) feature_index start_index end_indexrrrqtokenstoken_to_orig_mapZtoken_is_max_contextFr*cSs |j|jSr)rrrrrrrSrz5SQuADMetric._get_prelim_predictions..Tkeyreverse) rCrvr_get_best_indexesrrrsrHstrrErmrSsorted) r#rrZ_PrelimPredictionrrrrZ start_indexesZ end_indexesrrlengthrrrr'sV       z#SQuADMetric._get_prelim_predictionscCsRtdgd}i}g}|D]}t||jkr6q"||j}|jdkr|d|j|jd} |dt|j} |dt|j} |d| | d} d| } | d d } | d d } | } d| } d| }| | |}||vrqd ||<n d }d ||<| |||j|jdq|s<| |d dddt|dksNJ|S)zget nbest predictionsZNbestPrediction)rrrrrr*rZ doc_tokensrz ##rz##TemptyrO)rCrvrHrlrrrrrfreplacerrT_get_final_textrSrr)r#rrrZ_NbestPredictionZseen_predictionsrpredrZ tok_tokensZorig_doc_startZ orig_doc_endZ orig_tokenstok_text orig_textZ final_textrrrrXsJ         zSQuADMetric._get_nbestcCs||sgSd}|D]}|dus$||kr|}qg}d}|D]$}t||}||||7}q6g}|D]}|||qd|S)z,Compute softmax probability over raw logits.NrO)mathexprS)r#scores max_scorerZ exp_scoresZ total_sumr>rrrrrs   zSQuADMetric._compute_softmaxcCsdd}d|j|}||}|dkr0|S|t|d}||\}}||\} } t|t| krl|Si} t| D]\} } | | | <qzd}|| vr| |}||vr||}|dur|Sd}|| vr| |}||vr||}|dur|S|||d}|S)z;Project the tokenized prediction back to the original text.cSsPg}t}t|D](\}}|dkr&q||t|<||qd|}||fS)Nrr)rCrrrHrSrf)rZns_charsZ ns_to_s_mapr~cZns_textrrr _strip_spacess   z2SQuADMetric._get_final_text.._strip_spacesrrPr*N)rfrktokenizefindrHsix iteritems)r#Z pred_textrrrstart_positionZ end_positionZ orig_ns_textZorig_ns_to_s_mapZ tok_ns_textZtok_ns_to_s_mapZtok_s_to_ns_mapr~Z tok_indexZorig_start_positionZns_start_positionZorig_end_positionZns_end_positionZ output_textrrrrs:     zSQuADMetric._get_final_textcCsLtt|dddd}g}t|D]$\}}||jkr8qH||dq"|S)z"Get the n-best logits from a list.cSs|dS)Nr*rrrrrrrz/SQuADMetric._get_best_indexes..Trr)rrrlrS)r#logitsZindex_and_scoreZ best_indexesr~rrrrrs zSQuADMetric._get_best_indexesN)r_r`Tra)rZr[r\r]rr!r;rNrrgrrrrrrrrrrrrrrrs&   (114rcs8eZdZdZfddZddZddZdd ZZS) rz'Compute the loss and PPL of each entitycstt|d|_d|_t|_td|_ |j dk|_ d|_ d|_ t |_t |_t |_t |_|j rt|_ t|_ |j |j }|j |}||j dk|_td|_td|_|jdv|_dSNpipeline_stagesr*r parallel_mode full_batch)semi_auto_parallel auto_parallel)rrrnum_data total_lossr lossmsget_auto_parallel_contextrpipeline_parallelrank_id device_numPCastcastReshapereshapeNotEqual not_equalSubrrr is_last_stagerrrr#Zper_stage_device_numstage_idr$rrrs*          zPerplexityMetric.__init__cCsd|_d|_dS)rorrON)rrr"rrrr!szPerplexityMetric.clearcGs|jrr|jsdS|jr&tjjddd|d}t|}|j|7_|j d7_ |jrtj|j d|j dn0|d}t|}|j|7_|j d7_ dS) r)N data_parallelFrrrr*Trrr) rr rrcontextset_auto_parallel_contextrur3rrrr)r#r:rrrrr;s$    zPerplexityMetric.updatecCsT|jr|jsdSt|j|j}|t|d}|jrP|j|jdkrPt d||S)rN)rZPPLr*zAverage Loss and PPL Metric:) rr rurrrrrrr)r#Zavg_lossrrrrrN,s  zPerplexityMetric.eval rZr[r\r]rr!r;rNr^rrr$rrs  *rcs8eZdZdZfddZddZddZdd ZZS) rrcs"tt|ggggd|_dSN)rouge-1rouge-2rouge-lbleu-4)rrr score_dictr"r$rrr;s zADGENMetric.__init__cCsggggd|_dSr)rr"rrrr!Ds zADGENMetric.clearcGs|d}|d}t|tr"|d}t||D]\}}td|dd|tt|}tt|}t}|d |d |} | d} | D]&\} } |j |  t | ddd qtt|gt|tjd } |j d  t | dd q,d S) r)rr* pred is:  label is: rrdr?)Zsmoothing_functionrN) isinstancetupler7rrjiebacutrZ get_scoresrfrrrSrGrrZmethod3)r#r:predslabelsrr hypothesisZ referenceZrougerrkv bleu_scorerrrr;Ls"   zADGENMetric.updatecCs|jD]\}}tt||j|<q tdd|jdddd|jdddd|jd ddd |jd d|jS) rzmetric: ADGENMetric z rouge-1: rz.4frz rouge-2: rz rouge-l: rz bleu-4: r)rrrur4meanr)r#r#r$rrrrNbszADGENMetric.evalrrrr$rr7s  rcs@eZdZdZfddZddZddZdd Zd d ZZ S) ru Computes the prompt acc of each entity. The prompt acc is the accuracy of text classification base on building prompt. The accurate index is the index of the prompt which has the minimum perplexity. 1. Build the prompt for this metric is described as follows: 这是关于**体育**的文章：$passage 这是关于**文化**的文章：$passage 2. Computes perplexity of each generated context based on prompt. Perplexity is a measurement about how well a probability distribution or a model predicts a sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows: .. math:: PP(W)=P(w_{1}w_{2}...w_{N})^{-\frac{1}{N}}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}...w_{N})}} Where :math:`w` represents words in corpus. 3. Compute classification result by choosing the index of the prompt which has the minimum perplexity. 4. Count the number of correctly classified and the total number of samples and compute the acc as follows: .. math:: \text{accuracy} =\frac{\text{correct_sample_nums}}{\text{total_sample_nums}} cstt|d|_d|_t|_td|_ |j dk|_ d|_ d|_ d|_ t|_t|_t|_t|_t|_t|_|j rt|_ t|_ |j |j }|j |}||j dk|_td|_td|_|jdv|_ dSr)!rrrr total_acc_numr rrrrrZ last_card_idrrrrrrrEqualequalSoftmaxsoftmaxArgminargmin ReduceSumrrrr rrrr r$rrrs0            zPromptAccMetric.__init__cCsd|_d|_dS)rorN)rr'r"rrrr!szPromptAccMetric.clearcGsH|d|d|d|df\}}}}|j\}}}} |||||df}g} t||D]p} || || || } } }| ddddf} | dd} |dd}|| | |}t|}| |qXtj| tj d}||||f}| | |tj }||d}| | ||tj}|j|7_|j|7_dS) zThe main calculate logic.rr*r+r rPN)dtype)rP)shaperrwrrur3rSrTensorfloat32rr-int32r)float16rrr')r#r:r input_ids input_maskr!r}Z num_labels seq_length_Zppl_listindexZ sub_logitsZ sub_tokensZ sub_mask_listrZppl_msrZ cur_acc_numrrrcalculate_circles&$     z PromptAccMetric.calculate_circlecGs||jrL|jsdS|jr&tjjddd|j||jrVtj|jd|jdn |j|t d |j |j d|j |j dS) r)Nr Fr Trz6Current data num is {}, total acc num is {}, ACC is {}%.3f) rr rrrrr:rrrformatrr')r#r:rrrr;s   zPromptAccMetric.updatecCsV|jr|jsdSt|j|j}d|i}tdd|ddd|jd|j|S)rNZAcczAcc: r;rz, total_acc_num: z , total_num: )rr rur'rrrE)r#Zacc_raterrrrrNs zPromptAccMetric.eval) rZr[r\r]rr!r:r;rNr^rrr$rrns  rcsjeZdZdZfddZddZddZdd Zdd d Zd dZ ddZ ddZ ddZ ddZ ZS)ru Compute the Em/F1 scores of examples. Em score is the prediction exact matches the labels except the punctuations. For example, the question is "河南的省会是哪里？" and the label is "郑州市", when prediction is "郑州市", Em score is 100; when prediction is "郑州市。", Em score is 100; when prediction is "郑州", Em score is 0. F1 score is calculated as 2*precision*recall/(precision+recall), the precision and recall are calculated as precision = lcs_length/len(prediction_segment), recall = lcs_length/len(label_segment), lcs_length is the length of the longest common subsequence. cs*tt|d|_d|_d|_d|_dSr)rrrgensr!metricsrr"r$rrrs zEmF1Metric.__init__cCs"g|_g|_ddd|_d|_dS)rorO)EmF1rN)r=r!r>rr"rrrr!s zEmF1Metric.clearcGs|d|d}}tt|D]*}||||<||dd||<qtd|dd||j||j||jt|7_| ||\}}td||jddkr| |j|j\}}td| d dd | d dd |d S)r)rr*rrrzThe F1/Em of this example is: F1 score: r@ , Em score: r?z, current_count: N) rwrHrrTrr=r9r!revaluate_pairsrE)r#r:genrr~r current_countrrrr;s    zEmF1Metric.updatecCsB||j|j\}}td|ddd|ddd||S)rrBr@rrCr?z, total_count: )rDr=r!rrE)r#r total_countrrrrNs*zEmF1Metric.evalFcCst|}g}d}gd}|D]^}|r6||vr6q$td|sJ||vrz|dkrntt|}||d}| |q$||7}q$|dkrtt|}|||S)zcut input for calculating lcsr!rQ:r8*^/\~`+=u，u。u：u？u！u“u”u；u’u《u》u……·u、u「u」u（u）u－u～u『u』z[\u4e00-\u9fa5]) rrrrsearchrrrr9rS)r#in_strrm_puncZsegs_outZtemp_strsp_charcharssrrrmixed_segmentations$     zEmF1Metric.mixed_segmentationcCsFt|}gd}g}|D]}||vr0q q ||q d|S)zremove punctuations in inputsrHr)rrrrSrf)r#rTrVZout_segsrWrrrremove_punctuation5s zEmF1Metric.remove_punctuationcsfddtt|dD}d}d}tt|D]z}ttD]h}|||krB|||d||d|d<||d|d|krB||d|d}|d}qBq2|||||fS)zcalculate the length of lcscs&g|]}ddttdDqS)cSsg|]}dqSr<r)rr~rrrr0Erz2EmF1Metric.find_lcs...r*rwrH)rjs2rrr0Erz'EmF1Metric.find_lcs..r*rr[)r#s1r^mmmaxpr~r\rr]rfind_lcsCs  zEmF1Metric.find_lcsc Csg}|D]}|j|dd}|j|dd}|||\}}|dkrL|dqd|t|} d|t|} d| | | | } || qt|S)rT)rUrrr+)rYrcrSrHr) r#rrZ f1_scoresansZans_segsZprediction_segsr8Zlcs_lenr@rArBrrr calc_f1_scoreQs  zEmF1Metric.calc_f1_scorecCs6d}|D](}||}||}||krd}q2q|S)zcalculate em scorerr*)rZ)r#rremrdans_Z prediction_rrr calc_em_scoreas  zEmF1Metric.calc_em_scorec Csd}d}d}t||D]4\}}|d7}|||g|7}|||g|7}q|dkrxd||}d||} || d} n tdi} | |fS)zcalculate metricrr*r)r@r?ztotal_count is zero)r7rerhr) r#Zpred_rgrBrfrGrZanswerf1_scoreZem_scorerrrrrDls   zEmF1Metric.evaluate_pairs)F)rZr[r\r]rr!r;rNrYrZrcrerhrDr^rrr$rrs    r)/r]rdrrrCrrrrrrnumpyr4Z rouge_chineserZnltk.translate.bleu_scorerr mindspore.nnnn mindsporer mindspore.opsrrmindspore.communicationrrmindformers.tools.registerrr mindformers.modelsr Zmindformers.core.lossr Zdataset.labelsr __all__registerMETRICMetricrrrrrrrrrrsH       Y Z W 6 p pengcheng.mind-test/mindformers/core/metric/build_metric.py000066400000000000000000000051731471400134000244730ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Metric API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_metric( config: dict = None, default_args: dict = None, module_type: str = 'metric', class_name: str = None, **kwargs): r"""Build metric For MindFormer. Instantiate the metric from MindFormerRegister's registry. Args: config (dict, list): The task metric's config. Default: None. default_args (dict): The default argument of metric API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'metric'. class_name (str): The class name of metric API. Default: None. Return: The function instance of metric API. Examples: >>> from mindformers import build_metric >>> metric_config = {'type': 'Accuracy', 'eval_type': 'classification'} >>> # 1) use config dict to build metric >>> metric_from_config = build_metric(metric_config) >>> # 2) use class name to build metric >>> metric_class_name = build_metric(class_name='Accuracy', eval_type='classification') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.METRIC, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_mt(): """ register MindSpore builtin LR class. """ for module_name in dir(nn.metrics): if module_name.startswith('__'): continue ms_metric = getattr(nn.metrics, module_name) if inspect.isclass(ms_metric): MindFormerRegister.register_cls( ms_metric, MindFormerModuleType.METRIC) register_ms_mt() pengcheng.mind-test/mindformers/core/metric/metric.py000066400000000000000000001056651471400134000233230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/lonePatient/daguan_2019_rank9/blob/master/pydatagrand/train/ner_utils.py # ============================================================================ """MindFormer Self-Define Metric.""" import os import sys import re import collections import json import math import string import shutil import six import jieba import numpy as np from rouge_chinese import Rouge from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction import mindspore.nn as nn import mindspore as ms from mindspore.ops import operations as P from mindspore.communication import get_group_size, get_rank from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models import BasicTokenizer from mindformers.core.loss import CrossEntropyLoss from ...dataset.labels import cluener_labels __all__ = ['EntityScore', 'SQuADMetric', 'PerplexityMetric', 'ADGENMetric', 'PromptAccMetric', 'EmF1Metric'] @MindFormerRegister.register(MindFormerModuleType.METRIC) class EntityScore(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self): super(EntityScore, self).__init__() self.label2id = {label: label_id for label_id, label in enumerate(cluener_labels)} self.id2label = {label_id: label for label, label_id in self.label2id.items()} self.clear() def clear(self): "Initialization." self.origins = [] self.founds = [] self.rights = [] def update(self, *inputs): """Update results for every batch""" batch_logits = inputs[0].asnumpy() batch_label_ids = inputs[1].asnumpy() batch_pred_ids = np.argmax(batch_logits, axis=2).tolist() pred_paths = [[self.id2label[id_] for id_ in pred_ids] for pred_ids in batch_pred_ids] label_paths = [[self.id2label[id_] for id_ in label_ids] for label_ids in batch_label_ids] for label_path, pre_path in zip(label_paths, pred_paths): label_entities = self.get_entities_bios(label_path) pred_entities = self.get_entities_bios(pre_path) self.origins.extend(label_entities) self.founds.extend(pred_entities) self.rights.extend([pred_entity for pred_entity in pred_entities if pred_entity in label_entities]) def eval(self): """Compute final results.""" class_info = {} origin_counter = collections.Counter([x[0] for x in self.origins]) found_counter = collections.Counter([x[0] for x in self.founds]) right_counter = collections.Counter([x[0] for x in self.rights]) for type_, count in origin_counter.items(): origin = count found = found_counter.get(type_, 0) right = right_counter.get(type_, 0) recall, precision, f1 = self.compute(origin, found, right) class_info[type_] = {"precision": round(precision, 4), 'recall': round(recall, 4), 'f1': round(f1, 4)} origin = len(self.origins) found = len(self.founds) right = len(self.rights) recall, precision, f1 = self.compute(origin, found, right) return {"precision": round(precision, 4), 'recall': round(recall, 4), 'f1': round(f1, 4)}, class_info def compute(self, origin, found, right): """Compute f1, precision and recall.""" recall = 0 if origin == 0 else (right / origin) precision = 0 if found == 0 else (right / found) f1 = 0. if recall + precision == 0 else (2 * precision * recall) / (precision + recall) return recall, precision, f1 def get_entities_bios(self, seq): """Get entities from sequence.""" chunks = [] chunk = [-1, -1, -1] for indx, tag in enumerate(seq): if tag.startswith("S-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[2] = indx chunk[0] = tag.split('-')[1] chunks.append(chunk) chunk = [-1, -1, -1] if tag.startswith("B-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[0] = tag.split('-')[1] elif tag.startswith('I-') and chunk[1] != -1: entity_type = tag.split('-')[1] if entity_type == chunk[0]: chunk[2] = indx if indx == len(seq) - 1: chunks.append(chunk) else: if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] return chunks @MindFormerRegister.register(MindFormerModuleType.METRIC) class SQuADMetric(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self, dataset_dir, n_best_size=20, max_answer_len=30, do_lower_case=True, temp_file_dir="./squad_temp"): self.outputs = [] self.temp_file_dir = temp_file_dir temp_examples_file = os.path.join(temp_file_dir, "temp_examples.json") temp_features_file = os.path.join(temp_file_dir, "temp_features.json") self.all_examples = self._load_temp_data(temp_examples_file) self.all_features = self._load_temp_data(temp_features_file) self.dev_file_path = os.path.join(dataset_dir, "dev-v1.1.json") self.basic_tokenizer = BasicTokenizer(do_lower_case) self.n_best_size = n_best_size self.max_answer_len = max_answer_len def clear(self): """Clearing the internal evaluation result.""" return def update(self, *inputs): """Update results for every batch""" ids = inputs[0].asnumpy() start = inputs[1].asnumpy() end = inputs[2].asnumpy() batch_size = len(ids) RawResult = collections.namedtuple("RawResult", ["unique_id", "start_logits", "end_logits"]) for i in range(batch_size): unique_id = int(ids[i]) start_logits = [float(x) for x in start[i].flat] end_logits = [float(x) for x in end[i].flat] self.outputs.append(RawResult(unique_id=unique_id, start_logits=start_logits, end_logits=end_logits)) def eval(self): """Compute final result""" predictions = self._get_predictions() with open(self.dev_file_path) as ds: dataset_json = json.load(ds) dataset = dataset_json['data'] f1 = exact_match = total = 0 for article in dataset: for paragraph in article['paragraphs']: for qa in paragraph['qas']: total += 1 if qa['id'] not in predictions: message = 'Unanswered question ' + qa['id'] + \ ' will receive score 0.' print(message, file=sys.stderr) continue ground_truths = list(map(lambda x: x['text'], qa['answers'])) if not ground_truths: continue prediction = predictions[qa['id']] exact_match += self._metric_max_over_ground_truths( self._exact_match_score, prediction, ground_truths) f1 += self._metric_max_over_ground_truths( self._f1_score, prediction, ground_truths) exact_match = 100.0 * exact_match / total f1 = 100.0 * f1 / total self._remove_temp_data() return {'exact_match': exact_match, 'f1': f1} def _remove_temp_data(self): shutil.rmtree(self.temp_file_dir) def _load_temp_data(self, temp_file_path): with open(temp_file_path, "r", encoding="utf-8") as f: data = [] for line in f.readlines(): data.append(json.loads(line.strip())) return data def _normalize_answer(self, s): """Lower text and remove punctuation, articles and extra whitespace.""" def remove_articles(text): return re.sub(r'\b(a|an|the)\b', ' ', text) def white_space_fix(text): return ' '.join(text.split()) def remove_punc(text): exclude = set(string.punctuation) return ''.join(ch for ch in text if ch not in exclude) def lower(text): return text.lower() return white_space_fix(remove_articles(remove_punc(lower(s)))) def _f1_score(self, prediction, ground_truth): """calculate f1 score""" prediction_tokens = self._normalize_answer(prediction).split() ground_truth_tokens = self._normalize_answer(ground_truth).split() common = collections.Counter(prediction_tokens) & collections.Counter(ground_truth_tokens) num_same = sum(common.values()) if num_same == 0: return 0 precision = 1.0 * num_same / len(prediction_tokens) recall = 1.0 * num_same / len(ground_truth_tokens) f1 = (2 * precision * recall) / (precision + recall) return f1 def _exact_match_score(self, prediction, ground_truth): return self._normalize_answer(prediction) == self._normalize_answer(ground_truth) def _metric_max_over_ground_truths(self, metric_fn, prediction, ground_truths): scores_for_ground_truths = [] for ground_truth in ground_truths: score = metric_fn(prediction, ground_truth) scores_for_ground_truths.append(score) return max(scores_for_ground_truths) def _get_predictions(self): """Get final predictions""" example_index_to_features = collections.defaultdict(list) for feature in self.all_features: example_index_to_features[feature["example_index"]].append(feature) unique_id_to_result = {} for result in self.outputs: unique_id_to_result[result.unique_id] = result all_predictions = collections.OrderedDict() for (example_index, example) in enumerate(self.all_examples): features = example_index_to_features[example_index] prelim_predictions = self._get_prelim_predictions(features, unique_id_to_result) nbest = self._get_nbest(prelim_predictions, features, example) total_scores = [] best_non_null_entry = None for entry in nbest: total_scores.append(entry.start_logit + entry.end_logit) if not best_non_null_entry: if entry.text: best_non_null_entry = entry probs = self._compute_softmax(total_scores) nbest_json = [] for (i, entry) in enumerate(nbest): output = collections.OrderedDict() output["text"] = entry.text output["probability"] = probs[i] output["start_logit"] = entry.start_logit output["end_logit"] = entry.end_logit nbest_json.append(output) assert len(nbest_json) >= 1 all_predictions[example["qas_id"]] = nbest_json[0]["text"] return all_predictions def _get_prelim_predictions(self, features, unique_id_to_result): """get prelim predictions""" _PrelimPrediction = collections.namedtuple( "PrelimPrediction", ["feature_index", "start_index", "end_index", "start_logit", "end_logit"]) prelim_predictions = [] # keep track of the minimum score of null start+end of position 0 for (feature_index, feature) in enumerate(features): if feature["unique_id"] not in unique_id_to_result: continue result = unique_id_to_result[feature["unique_id"]] start_indexes = self._get_best_indexes(result.start_logits) end_indexes = self._get_best_indexes(result.end_logits) # if we could have irrelevant answers, get the min score of irrelevant for start_index in start_indexes: for end_index in end_indexes: # We could hypothetically create invalid predictions, e.g., predict # that the start of the span is in the question. We throw out all # invalid predictions. if start_index >= len(feature["tokens"]): continue if end_index >= len(feature["tokens"]): continue if str(start_index) not in feature["token_to_orig_map"]: continue if str(end_index) not in feature["token_to_orig_map"]: continue if not feature["token_is_max_context"].get(str(start_index), False): continue if end_index < start_index: continue length = end_index - start_index + 1 if length > self.max_answer_len: continue prelim_predictions.append( _PrelimPrediction( feature_index=feature_index, start_index=start_index, end_index=end_index, start_logit=result.start_logits[start_index], end_logit=result.end_logits[end_index])) prelim_predictions = sorted( prelim_predictions, key=lambda x: (x.start_logit + x.end_logit), reverse=True) return prelim_predictions def _get_nbest(self, prelim_predictions, features, example): """get nbest predictions""" _NbestPrediction = collections.namedtuple( "NbestPrediction", ["text", "start_logit", "end_logit"]) seen_predictions = {} nbest = [] for pred in prelim_predictions: if len(nbest) >= self.n_best_size: break feature = features[pred.feature_index] if pred.start_index > 0: # this is a non-null prediction tok_tokens = feature["tokens"][pred.start_index:(pred.end_index + 1)] orig_doc_start = feature["token_to_orig_map"][str(pred.start_index)] orig_doc_end = feature["token_to_orig_map"][str(pred.end_index)] orig_tokens = example["doc_tokens"][orig_doc_start:(orig_doc_end + 1)] tok_text = " ".join(tok_tokens) # De-tokenize WordPieces that have been split off. tok_text = tok_text.replace(" ##", "") tok_text = tok_text.replace("##", "") # Clean whitespace tok_text = tok_text.strip() tok_text = " ".join(tok_text.split()) orig_text = " ".join(orig_tokens) final_text = self._get_final_text(tok_text, orig_text) if final_text in seen_predictions: continue seen_predictions[final_text] = True else: final_text = "" seen_predictions[final_text] = True nbest.append( _NbestPrediction( text=final_text, start_logit=pred.start_logit, end_logit=pred.end_logit)) # In very rare edge cases we could have no valid predictions. So we # just create a nonce prediction in this case to avoid failure. if not nbest: nbest.append(_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0)) assert len(nbest) >= 1 return nbest def _compute_softmax(self, scores): """Compute softmax probability over raw logits.""" if not scores: return [] max_score = None for score in scores: if max_score is None or score > max_score: max_score = score exp_scores = [] total_sum = 0.0 for score in scores: x = math.exp(score - max_score) exp_scores.append(x) total_sum += x probs = [] for score in exp_scores: probs.append(score / total_sum) return probs def _get_final_text(self, pred_text, orig_text): """Project the tokenized prediction back to the original text.""" def _strip_spaces(text): ns_chars = [] ns_to_s_map = collections.OrderedDict() for (i, c) in enumerate(text): if c == " ": continue ns_to_s_map[len(ns_chars)] = i ns_chars.append(c) ns_text = "".join(ns_chars) return (ns_text, ns_to_s_map) tok_text = " ".join(self.basic_tokenizer.tokenize(orig_text)) start_position = tok_text.find(pred_text) if start_position == -1: return orig_text end_position = start_position + len(pred_text) - 1 (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text) (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text) if len(orig_ns_text) != len(tok_ns_text): return orig_text tok_s_to_ns_map = {} for (i, tok_index) in six.iteritems(tok_ns_to_s_map): tok_s_to_ns_map[tok_index] = i orig_start_position = None if start_position in tok_s_to_ns_map: ns_start_position = tok_s_to_ns_map[start_position] if ns_start_position in orig_ns_to_s_map: orig_start_position = orig_ns_to_s_map[ns_start_position] if orig_start_position is None: return orig_text orig_end_position = None if end_position in tok_s_to_ns_map: ns_end_position = tok_s_to_ns_map[end_position] if ns_end_position in orig_ns_to_s_map: orig_end_position = orig_ns_to_s_map[ns_end_position] if orig_end_position is None: return orig_text output_text = orig_text[orig_start_position:(orig_end_position + 1)] return output_text def _get_best_indexes(self, logits): """Get the n-best logits from a list.""" index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True) best_indexes = [] for (i, score) in enumerate(index_and_score): if i >= self.n_best_size: break best_indexes.append(score[0]) return best_indexes @MindFormerRegister.register(MindFormerModuleType.METRIC) class PerplexityMetric(nn.Metric): """Compute the loss and PPL of each entity""" def __init__(self): super(PerplexityMetric, self).__init__() self.num_data = None self.total_loss = None self.loss = CrossEntropyLoss() self.pipeline_stages = ms.get_auto_parallel_context('pipeline_stages') self.pipeline_parallel = self.pipeline_stages > 1 self.rank_id = 0 self.device_num = 1 self.cast = P.Cast() self.reshape = P.Reshape() self.not_equal = P.NotEqual() self.sub = P.Sub() if self.pipeline_parallel: self.rank_id = get_rank() self.device_num = get_group_size() per_stage_device_num = self.device_num // self.pipeline_stages stage_id = self.rank_id // per_stage_device_num self.is_last_stage = (stage_id == self.pipeline_stages - 1) self.parallel_mode = ms.get_auto_parallel_context("parallel_mode") self.full_batch = ms.get_auto_parallel_context("full_batch") self.auto_parallel = self.parallel_mode in ['semi_auto_parallel', 'auto_parallel'] def clear(self): """Clearing the internal evaluation result.""" self.num_data = 0 self.total_loss = 0.0 def update(self, *inputs): """Update results for every batch""" if self.pipeline_parallel: if not self.is_last_stage: return if self.auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) # logits, labels, input_mask = inputs[0], inputs[1], inputs[2] # # input_mask was added 1 in GPT2LMModel to avoid allgather issue in Mindspore1.10 # input_mask = self.sub(input_mask, 1) # batch_size, seq_length, _ = logits.shape # logits = self.reshape(logits[::, :-1, ::], (batch_size * (seq_length - 1), -1)) # labels = self.reshape(labels[::, 1:], (-1,)) # input_mask = self.reshape(input_mask[::, 1:], (-1,)) # loss = self.loss(logits, labels, input_mask) loss = inputs[0] loss = float(loss.asnumpy()) self.total_loss += loss self.num_data += 1 if self.auto_parallel: ms.set_auto_parallel_context(parallel_mode=self.parallel_mode, full_batch=True, pipeline_stages=self.pipeline_stages) else: # logits, labels, input_mask = inputs[0], inputs[1], inputs[2] # batch_size, seq_length, _ = logits.shape # logits = self.reshape(logits[::, :-1, ::], (batch_size * (seq_length - 1), -1)) # labels = self.reshape(labels[::, 1:], (-1,)) # input_mask = self.reshape(input_mask[::, 1:], (-1,)) # loss = self.loss(logits, labels, input_mask) loss = inputs[0] loss = float(loss.asnumpy()) self.total_loss += loss self.num_data += 1 def eval(self): """Compute final result""" if self.pipeline_parallel and not self.is_last_stage: return None avg_loss = float(self.total_loss / self.num_data) result = {"loss": avg_loss, "PPL": math.exp(avg_loss)} if self.pipeline_parallel and (self.rank_id == (self.device_num - 1)): print("Average Loss and PPL Metric:", result) return result @MindFormerRegister.register(MindFormerModuleType.METRIC) class ADGENMetric(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self): super(ADGENMetric, self).__init__() self.score_dict = { "rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": [] } def clear(self): self.score_dict = { "rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": [] } def update(self, *inputs): """Update results for every batch""" preds = inputs[0] # list[numpy] labels = inputs[1] # numpy if isinstance(preds, tuple): preds = preds[0] for pred, label in zip(preds, labels): print(f"pred is:\n {pred}\n", f"label is:\n {label}") hypothesis = list(jieba.cut(pred)) reference = list(jieba.cut(label)) rouge = Rouge() scores = rouge.get_scores(' '.join(hypothesis), ' '.join(reference)) result = scores[0] for k, v in result.items(): self.score_dict[k].append(round(v["f"] * 100, 4)) bleu_score = sentence_bleu([list(label)], list(pred), smoothing_function=SmoothingFunction().method3) self.score_dict["bleu-4"].append(round(bleu_score * 100, 4)) def eval(self): """Compute final result""" for k, v in self.score_dict.items(): self.score_dict[k] = float(np.mean(v)) print('metric: ADGENMetric\n' + f'rouge-1: {self.score_dict["rouge-1"]:.4f}\n' + f'rouge-2: {self.score_dict["rouge-2"]:.4f}\n' + f'rouge-l: {self.score_dict["rouge-l"]:.4f}\n' + f'bleu-4: {self.score_dict["bleu-4"]:.4f}') return self.score_dict @MindFormerRegister.register(MindFormerModuleType.METRIC) class PromptAccMetric(nn.Metric): r""" Computes the prompt acc of each entity. The prompt acc is the accuracy of text classification base on building prompt. The accurate index is the index of the prompt which has the minimum perplexity. 1. Build the prompt for this metric is described as follows: 这是关于**体育**的文章：$passage 这是关于**文化**的文章：$passage 2. Computes perplexity of each generated context based on prompt. Perplexity is a measurement about how well a probability distribution or a model predicts a sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows: .. math:: PP(W)=P(w_{1}w_{2}...w_{N})^{-\frac{1}{N}}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}...w_{N})}} Where :math:`w` represents words in corpus. 3. Compute classification result by choosing the index of the prompt which has the minimum perplexity. 4. Count the number of correctly classified and the total number of samples and compute the acc as follows: .. math:: \text{accuracy} =\frac{\text{correct_sample_nums}}{\text{total_sample_nums}} """ def __init__(self): super(PromptAccMetric, self).__init__() self.num_data = None self.total_acc_num = None self.loss = CrossEntropyLoss() self.pipeline_stages = ms.get_auto_parallel_context('pipeline_stages') self.pipeline_parallel = self.pipeline_stages > 1 self.last_card_id = 0 self.rank_id = 0 self.device_num = 1 self.cast = P.Cast() self.reshape = P.Reshape() self.equal = P.Equal() self.softmax = P.Softmax() self.argmin = P.Argmin() self.sum = P.ReduceSum() if self.pipeline_parallel: self.rank_id = get_rank() self.device_num = get_group_size() per_stage_device_num = self.device_num // self.pipeline_stages stage_id = self.rank_id // per_stage_device_num self.is_last_stage = (stage_id == self.pipeline_stages - 1) self.parallel_mode = ms.get_auto_parallel_context("parallel_mode") self.full_batch = ms.get_auto_parallel_context("full_batch") self.auto_parallel = self.parallel_mode in ['semi_auto_parallel', 'auto_parallel'] def clear(self): """Clearing the internal evaluation result.""" self.num_data = 0 self.total_acc_num = 0 def calculate_circle(self, *inputs): """The main calculate logic.""" logits, input_ids, input_mask, labels = inputs[0], inputs[1], inputs[2], inputs[3] batch_size, num_labels, seq_length, _ = logits.shape logits = self.reshape(logits, (batch_size*num_labels, seq_length, -1)) ppl_list = [] for index in range(batch_size*num_labels): sub_logits, sub_tokens, sub_mask_list = logits[index], input_ids[index], input_mask[index] sub_logits = sub_logits[:-1, ::] sub_tokens = sub_tokens[1:] sub_mask_list = sub_mask_list[1:] loss = self.loss(sub_logits, sub_tokens, sub_mask_list) loss = float(loss.asnumpy()) ppl_list.append(loss) # smaller, better ppl_ms = ms.Tensor(ppl_list, dtype=ms.float32) ppl_ms = self.reshape(ppl_ms, (batch_size, num_labels)) ppl_ms = self.cast(self.argmin(ppl_ms), ms.int32) label = self.reshape(labels, (-1,)) cur_acc_num = self.cast(self.equal(ppl_ms, label), ms.float16).sum().asnumpy() self.num_data += batch_size self.total_acc_num += cur_acc_num def update(self, *inputs): """Update results for every batch""" if self.pipeline_parallel: if not self.is_last_stage: return if self.auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) self.calculate_circle(*inputs) if self.auto_parallel: ms.set_auto_parallel_context(parallel_mode=self.parallel_mode, full_batch=True, pipeline_stages=self.pipeline_stages) else: self.calculate_circle(*inputs) print("Current data num is {}, total acc num is {}, ACC is {}".format( self.num_data, self.total_acc_num, "%.3f" % (self.total_acc_num / self.num_data))) return def eval(self): """Compute final result""" if self.pipeline_parallel and not self.is_last_stage: return None acc_rate = float(self.total_acc_num / self.num_data) result = {"Acc": acc_rate} print(f"Acc: {('%.3f' % result.get('Acc', 0))}, total_acc_num: {self.total_acc_num}, " f"total_num: {self.num_data}") return result @MindFormerRegister.register(MindFormerModuleType.METRIC) class EmF1Metric(nn.Metric): """ Compute the Em/F1 scores of examples. Em score is the prediction exact matches the labels except the punctuations. For example, the question is "河南的省会是哪里？" and the label is "郑州市", when prediction is "郑州市", Em score is 100; when prediction is "郑州市。", Em score is 100; when prediction is "郑州", Em score is 0. F1 score is calculated as 2*precision*recall/(precision+recall), the precision and recall are calculated as precision = lcs_length/len(prediction_segment), recall = lcs_length/len(label_segment), lcs_length is the length of the longest common subsequence. """ def __init__(self): super(EmF1Metric, self).__init__() self.gens = None self.labels = None self.metrics = None self.num_data = None def clear(self): """Clearing the internal evaluation result.""" self.gens = [] self.labels = [] self.metrics = { 'Em': 0.0, 'F1': 0.0 } self.num_data = 0 def update(self, *inputs): """Update results for every batch""" gen, label = inputs[0], inputs[1] for i in range(len(gen)): gen[i] = gen[i].strip() gen[i] = gen[i].split("\n")[0] print(f"pred is:\n {gen}\n", f"label is:\n {label}") self.gens.extend(gen) self.labels.extend(label) self.num_data += len(gen) result, current_count = self.evaluate_pairs(gen, label) print("The F1/Em of this example is: ", result) if self.num_data % 10 == 0: result, current_count = self.evaluate_pairs(self.gens, self.labels) print(f"F1 score: {result.get('F1', 0)}, Em score: {result.get('Em', 0)}, current_count: {current_count}") def eval(self): """Compute final result""" result, total_count = self.evaluate_pairs(self.gens, self.labels) print(f"F1 score: {result.get('F1', 0)}, Em score: {result.get('Em', 0)}, total_count: {total_count}") return result def mixed_segmentation(self, in_str, rm_punc=False): """cut input for calculating lcs""" in_str = str(in_str).lower().strip() segs_out = [] temp_str = "" sp_char = ['-', ':', '_', '*', '^', '/', '\\', '~', '`', '+', '=', '，', '。', '：', '？', '！', '“', '”', '；', '’', '《', '》', '……', '·', '、', '「', '」', '（', '）', '－', '～', '『', '』'] for char in in_str: if rm_punc and char in sp_char: continue if re.search(r'[\u4e00-\u9fa5]', char) or char in sp_char: if temp_str != "": ss = list(jieba.cut(temp_str)) segs_out.extend(ss) temp_str = "" segs_out.append(char) else: temp_str += char if temp_str != "": ss = list(jieba.cut(temp_str)) segs_out.extend(ss) return segs_out def remove_punctuation(self, in_str): """remove punctuations in inputs""" in_str = str(in_str).lower().strip() sp_char = ['-', ':', '_', '*', '^', '/', '\\', '~', '`', '+', '=', '，', '。', '：', '？', '！', '“', '”', '；', '’', '《', '》', '……', '·', '、', '「', '」', '（', '）', '－', '～', '『', '』'] out_segs = [] for char in in_str: if char in sp_char: continue else: out_segs.append(char) return ''.join(out_segs) def find_lcs(self, s1, s2): """calculate the length of lcs""" m = [[0 for i in range(len(s2) + 1)] for j in range(len(s1) + 1)] mmax = 0 p = 0 for i in range(len(s1)): for j in range(len(s2)): if s1[i] == s2[j]: m[i + 1][j + 1] = m[i][j] + 1 if m[i + 1][j + 1] > mmax: mmax = m[i + 1][j + 1] p = i + 1 return s1[p - mmax:p], mmax def calc_f1_score(self, answers, prediction): """calculate f1 score""" f1_scores = [] for ans in answers: ans_segs = self.mixed_segmentation(ans, rm_punc=True) prediction_segs = self.mixed_segmentation(prediction, rm_punc=True) _, lcs_len = self.find_lcs(ans_segs, prediction_segs) if lcs_len == 0: f1_scores.append(0) continue precision = 1.0 * lcs_len / len(prediction_segs) recall = 1.0 * lcs_len / len(ans_segs) f1 = (2 * precision * recall) / (precision + recall) f1_scores.append(f1) return max(f1_scores) def calc_em_score(self, answers, prediction): """calculate em score""" em = 0 for ans in answers: ans_ = self.remove_punctuation(ans) prediction_ = self.remove_punctuation(prediction) if ans_ == prediction_: em = 1 break return em def evaluate_pairs(self, pred_, ans_): """calculate metric""" f1 = 0 em = 0 total_count = 0 for (prediction, answer) in zip(pred_, ans_): total_count += 1 f1 += self.calc_f1_score([answer], prediction) em += self.calc_em_score([answer], prediction) if total_count > 0: f1_score = 100.0 * f1 / total_count em_score = 100.0 * em / total_count result = {'F1': f1_score, 'Em': em_score} else: print("total_count is zero") result = {} return result, total_count pengcheng.mind-test/mindformers/core/optim/000077500000000000000000000000001471400134000213165ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/optim/.ipynb_checkpoints/000077500000000000000000000000001471400134000251075ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/optim/.ipynb_checkpoints/optim-checkpoint.py000066400000000000000000000551541471400134000307500ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """FusedAdamWeightDecay, a customized Adam for offloading.""" import numpy as np from mindspore import context, nn from mindspore.common import dtype as mstype from mindspore.ops import operations as P from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.common.initializer import initializer from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter, ParameterTuple # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator as validator from mindspore._checkparam import Rel except ImportError: import mindspore._checkparam as validator import mindspore._checkparam as Rel from mindspore.nn.optim.optimizer import Optimizer from mindformers.version_control import fix_optim_global_step_sig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['FusedAdamWeightDecay', 'FP32StateAdamWeightDecay'] _adam_opt = C.MultitypeFuncGraph("adam_opt") _scaler_one = Tensor(1, mstype.int32) op_mul = P.Mul() @_adam_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _fused_update_with_global_norm(opt, global_norm, beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flags, optim_filter): """ Update parameters by FusedAdamWeightDecay. """ success = True if optim_filter: if decay_flags: next_param = opt(param, m, v, lr, beta1, beta2, eps, weight_decay, P.Cast()(gradient, mstype.float16), global_norm) else: next_param = opt(param, m, v, lr, beta1, beta2, eps, 0.0, P.Cast()(gradient, mstype.float16), global_norm) return F.depend(success, next_param) return success @_adam_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _fused_update(opt, beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flags, optim_filter): """ Update parameters by FusedAdamWeightDecay. """ success = True op_cast = P.Cast() if optim_filter: if decay_flags: opt(param, m, v, lr, beta1, beta2, eps, weight_decay, op_cast(gradient, F.dtype(param))) else: opt(param, m, v, lr, beta1, beta2, eps, 0.0, op_cast(gradient, F.dtype(param))) return success @_adam_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _update_run_op(beta1_power, beta2_power, beta1, beta2, eps, lr, weight_decay, param, \ m, v, gradient, decay_flag, optim_filter): """ Update parameters. Args: beta1 (Tensor): The exponential decay rate for the 1st moment estimations. Should be in range (0.0, 1.0). beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0). eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0. lr (Tensor): Learning rate. weight_decay (Number): Weight decay. Should be equal to or greater than 0. param (Tensor): Parameters. m (Tensor): m value of parameters. v (Tensor): v value of parameters. gradient (Tensor): Gradient of parameters. decay_flag (bool): Applies weight decay or not. optim_filter (bool): Applies parameter update or not. Returns: Tensor, the new value of v after updating. """ if optim_filter: # op_mul = P.Mul(), defined output op_square = P.Square() op_sqrt = P.Sqrt() op_cast = P.Cast() op_reshape = P.Reshape() op_shape = P.Shape() param_fp32 = op_cast(param, mstype.float32) m_fp32 = op_cast(m, mstype.float32) v_fp32 = op_cast(v, mstype.float32) gradient_fp32 = op_cast(gradient, mstype.float32) next_m = op_mul(beta1, m_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32) - beta1, gradient_fp32) next_v = op_mul(beta2, v_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32) - beta2, op_square(gradient_fp32)) regulate_m = next_m / (_scaler_one - beta1_power) regulate_v = next_v / (_scaler_one - beta2_power) update = regulate_m / (eps + op_sqrt(regulate_v)) if decay_flag: update = op_mul(weight_decay, param_fp32) + update update_with_lr = op_mul(lr, update) next_param = param_fp32 - op_reshape(update_with_lr, op_shape(param_fp32)) next_param = F.depend(next_param, F.assign(param, op_cast(next_param, F.dtype(param)))) next_param = F.depend(next_param, F.assign(m, op_cast(next_m, F.dtype(m)))) next_param = F.depend(next_param, F.assign(v, op_cast(next_v, F.dtype(v)))) return op_cast(next_param, F.dtype(param)) return gradient def _check_param_value(beta1, beta2, eps, prim_name): """Check the type of inputs.""" validator.check_value_type("beta1", beta1, [float], prim_name) validator.check_value_type("beta2", beta2, [float], prim_name) validator.check_value_type("eps", eps, [float], prim_name) validator.check_float_range(beta1, 0.0, 1.0, Rel.INC_NEITHER, "beta1", prim_name) validator.check_float_range(beta2, 0.0, 1.0, Rel.INC_NEITHER, "beta2", prim_name) validator.check_positive_float(eps, "eps", prim_name) @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class FusedAdamWeightDecay(Optimizer): """ Implements the Adam algorithm to fix the weight decay. It is a complete operator, not a combination of other ops. Note: When separating parameter groups, the weight decay in each group will be applied on the parameters if the weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive. To improve parameter groups performance, the customized order of parameters can be supported. Args: params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated, the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params", "lr", "weight_decay" and "order_params" are the keys can be parsed. - params: Required. The value must be a list of `Parameter`. - lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used. If not, the `learning_rate` in the API will be used. - weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay will be used. If not, the `weight_decay` in the API will be used. - order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters which in the 'order_params' must be in one of group parameters. learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate. When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule, use dynamic learning rate, the i-th learning rate will be calculated during the process of training according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float. Default: 1e-3. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Supported Platforms: ``CPU`` Examples: >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FusedAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FusedAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = Model(net, loss_fn=loss, optimizer=optim) """ def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0, offload=False): super(FusedAdamWeightDecay, self).__init__(learning_rate, params, weight_decay) _check_param_value(beta1, beta2, eps, self.cls_name) self.beta1 = Tensor(np.array([beta1]).astype(np.float32)) self.beta2 = Tensor(np.array([beta2]).astype(np.float32)) self.eps = Tensor(np.array([eps]).astype(np.float32)) self.moments1 = self.clone_state(prefix="adam_m", init='zeros') self.moments2 = self.clone_state(prefix="adam_v", init='zeros') self.opt = P.AdamWeightDecay() if offload: self.opt.add_prim_attr("primitive_target", "CPU") self.version_flag = fix_optim_global_step_sig() def construct(self, gradients): """construct with gradients""" lr = self.get_lr() if self.version_flag: self.assignadd(self.global_step, self.global_step_increase_tensor) if self.is_group: if self.is_group_lr: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps), lr, self.weight_decay, self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) else: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps, lr), self.weight_decay, self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) else: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps, lr, self.weight_decay), self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) if self.use_parallel: self.broadcast_params(optim_result) return optim_result def clone_state(self, prefix, init, forced_dtype=mstype.float32, is_follow=False): r""" Clone the parameters parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method forced_dtype: mstype. The except the dtype to be cloned. If is_follow is True, forced_dtype will be ignored. Default: mstype.float32 is_follow: bool. Is clone the parameters with the original dtype. If is_follow is True, the forced_dtype argument will be ignored. Default: False. """ parameter_tuple = self.parameters new = [] for old_param in parameter_tuple: param_init = init if init is None: param_init = old_param.init cur_dtype = forced_dtype if is_follow: cur_dtype = old_param.dtype new_state = Parameter(initializer(param_init, shape=old_param.shape, dtype=cur_dtype)) new_state.param_info = old_param.param_info.clone() if hasattr(old_param.param_info, "cloned_obj"): old_param.param_info.cloned_obj.append(new_state) else: old_param.param_info.cloned_obj = [new_state] new_state.is_init = False new_state.is_param_ps = old_param.is_param_ps new_state.init_in_server = old_param.init_in_server new_state.cache_enable = old_param.cache_enable new_state.requires_aggr = old_param.requires_aggr if old_param.cache_shape: new_state.cache_shape = old_param.cache_shape new_state.name = prefix + '.' + new_state.name new.append(new_state) return ParameterTuple(new) @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class FP32StateAdamWeightDecay(nn.AdamWeightDecay): r""" This class is almost same with the mindspore's AdamWeightDecay implements, the only difference is the optimizer's state will be always initialized with float32, where the original AdamWeightDecay will initialize the optimizer's state with float16, if the parameters are initialized with fp16. This setting will avoid overflow in training big model using fp16. Args: params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the `params` is a list of `dict`, the string "params", "lr", "weight_decay", and "order_params" are the keys can be parsed. - params: Required. Parameters in current group. The value must be a list of `Parameter`. - lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used. If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported. - weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule to get the weight decay value of current step. - order_params: Optional. When parameters is grouped, this usually is used to maintain the order of parameters that appeared in the network to improve performance. The value should be parameters whose order will be followed in optimizer. If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in one group of `params`. learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3. - float: The fixed learning rate value. Must be equal to or greater than 0. - int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float. - Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied. For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate. - Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate. - LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of LearningRateSchedule with step as the input to get the learning rate of current step. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0. - float: The fixed weight decay value. Must be equal to or greater than 0. - int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float. - Cell: Weight decay is dynamic. During training, the optimizer calls the instance of the Cell with step as the input to get the weight decay value of current step. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Raises: TypeError: If `learning_rate` is not one of int, float, Tensor, Iterable, LearningRateSchedule. TypeError: If element of `parameters` is neither Parameter nor dict. TypeError: If `beta1`, `beta2` or `eps` is not a float. TypeError: If `weight_decay` is neither float nor int. ValueError: If `eps` is less than or equal to 0. ValueError: If `beta1`, `beta2` is not in range (0.0, 1.0). ValueError: If `weight_decay` is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> from mindformers.core.optim import FP32StateAdamWeightDecay >>> >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FP32StateAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FP32StateAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = ms.Model(net, loss_fn=loss, optimizer=optim) """ def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0): super(nn.AdamWeightDecay, self).__init__(learning_rate, params, weight_decay) _check_param_value(beta1, beta2, eps, self.cls_name) self.beta1 = Tensor(np.array([beta1]).astype(np.float32)) self.beta2 = Tensor(np.array([beta2]).astype(np.float32)) self.eps = Tensor(np.array([eps]).astype(np.float32)) self.moments1 = self.clone_state(prefix='adam_m', init='zeros') self.moments2 = self.clone_state(prefix='adam_v', init='zeros') self.fused_opt = P.AdamWeightDecay() if context.get_context("device_target") == "CPU": self.use_fused_opt = True else: self.use_fused_opt = False def clone_state(self, prefix, init): r""" parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method """ parameter_tuple = self.parameters new = [] for old_param in parameter_tuple: new_state = Parameter(initializer(init, shape=old_param.shape, dtype=mstype.float32)) new_state.param_info = old_param.param_info.clone() if hasattr(old_param.param_info, "cloned_obj"): old_param.param_info.cloned_obj.append(new_state) else: old_param.param_info.cloned_obj = [new_state] new_state.is_init = False new_state.set_data(initializer(init, shape=old_param.shape, dtype=mstype.float32)) new_state.name = prefix + '.' + new_state.name new.append(new_state) return ParameterTuple(new) pengcheng.mind-test/mindformers/core/optim/__init__.py000066400000000000000000000014361471400134000234330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Optimizer.""" from .build_optim import build_optim from .optim import * __all__ = [] __all__.extend(optim.__all__) pengcheng.mind-test/mindformers/core/optim/__pycache__/000077500000000000000000000000001471400134000235265ustar00rootroot00000000000000pengcheng.mind-test/mindformers/core/optim/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004511471400134000277160ustar00rootroot00000000000000a ?cf@s,dZddlmZddlTgZeejdS)zMindFormers Optimizer.) build_optim)*N)__doc__roptim__all__extendrrJ/home/ma-user/work/mindformers-baichuan/mindformers/core/optim/__init__.pys pengcheng.mind-test/mindformers/core/optim/__pycache__/build_optim.cpython-39.pyc000066400000000000000000000056221471400134000304730ustar00rootroot00000000000000a ?cf@sdZddlZddlmZddlmZmZmZmZm Z ddl m Z ddl m Z mZmZddlmZmZdeeeed d d Zd dZddZeedS)zBuild Optimizer API.N)nn) AdaFactorAdamWeightDecaySGDAdagradAdam)build_lr)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)FusedAdamWeightDecayFP32StateAdamWeightDecay optimizer)config default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|jdurxt|jtrx|jjdurhtdt|j}||_tj|t j |dStj ||fi|S)aLBuild optim For MindFormer. Instantiate the optim from MindFormerRegister's registry. Args: config (dict): The task optim's config. Default: None. default_args (dict): The default argument of optim API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'optim'. class_name (str): The class name of optim API. Default: None. Return: The function instance of optim API. Examples: >>> from mindformers import build_optim >>> optim_config = {'type': 'AdamWeightDecay', 'weight_decay':0.05} >>> # 1) use config dict to build optim >>> optim_from_config = build_optim(optim_config) >>> # 2) use class name to build optim >>> optim_class_name = build_optim(class_name='AdamWeightDecay', weight_decay=0.05) Nzfoptimizer's learning rate must be LearningRateSchedule type, but the type is not specified, it is None)r) isinstancedictr learning_ratetype ValueErrorrr get_instance_from_cfgr OPTIMIZER get_instance)rrrrkwargs lr_schedulerM/home/ma-user/work/mindformers-baichuan/mindformers/core/optim/build_optim.py build_optims  r cCsDttjD]4}|drq ttj|}t|r t|t j q dS)z- register MindSpore builtin optimizer class. __N) dirroptim startswithgetattrinspectisclassr register_clsr r) module_namer#rrrregister_ms_optimAs   r*cCstjttjddtjttjddtjttjddtjttjddtjttjddtjt tjddtjt tjddd S) z/ register MindFormers builtin optimizer class. Zadamw)ralias adafactorsgdadamZadagradZ fused_adamwZ fp32_adamwN) r r(rr rrrrrr rrrrrregister_mf_optimLs*r/)NNrN)__doc__r& mindsporerZmindspore.nn.optimrrrrrZmindformers.core.lrrmindformers.tools.registerr r r r#r rrstrr r*r/rrrrs    ' pengcheng.mind-test/mindformers/core/optim/__pycache__/optim.cpython-39.pyc000066400000000000000000000450501471400134000273130ustar00rootroot00000000000000a ?cflZ@sdZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZmZzdd lmZdd lmZWn*eyddlmZddlmZYn0dd lmZddlm Z ddl!m"Z"m#Z#ddgZ$e %dZ&edej'Z(e )Z*e&+ddddddddddddd ddZ,e&+dddddddddddd ddZ-e&+ddddddddddddd ddZ.ddZ/e"+e#j0GdddeZ1e"+e#j0Gd ddej2Z3dS)!z7FusedAdamWeightDecay, a customized Adam for offloading.N)contextnn)dtype) operations) composite) functional) initializer)Tensor) ParameterParameterTuple) Validator)Rel) Optimizer)fix_optim_global_step_sig)MindFormerRegisterMindFormerModuleTypeFusedAdamWeightDecayFP32StateAdamWeightDecayZadam_optFunctionr Boolc  Csjd} | rf| r4|||| |||||t| tj| }n&|||| ||||dt| tj| }t| |S| Sz4 Update parameters by FusedAdamWeightDecay. T)PCastmstypefloat16Fdepend)opt global_normbeta1beta2epslr weight_decayparammvgradient decay_flags optim_filtersuccess next_paramr.G/home/ma-user/work/mindformers-baichuan/mindformers/core/optim/optim.py_fused_update_with_global_norm,s r0c Csbd} t} | r^| r:|||||||||| | t| n$||||||||d| | t| | Sr)rrrr)rr!r"r#r$r%r&r'r(r)r*r+r,op_castr.r.r/ _fused_update?s&$r2c  Csx| rtt} t}t}t}t}||tj}||tj}|| tj}|| tj}t||t|t dtj||}t||t|t dtj|| |}|t |}|t |}||||}| rt|||}t||}|||||}t |t |||t |}t |t |||t |}t |t | ||t | }||t |S| S)aT Update parameters. Args: beta1 (Tensor): The exponential decay rate for the 1st moment estimations. Should be in range (0.0, 1.0). beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0). eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0. lr (Tensor): Learning rate. weight_decay (Number): Weight decay. Should be equal to or greater than 0. param (Tensor): Parameters. m (Tensor): m value of parameters. v (Tensor): v value of parameters. gradient (Tensor): Gradient of parameters. decay_flag (bool): Applies weight decay or not. optim_filter (bool): Applies parameter update or not. Returns: Tensor, the new value of v after updating. )?)rSquareSqrtrReshapeShaperfloat32op_mulrtuple_to_array _scaler_onerassignr) beta1_power beta2_powerr!r"r#r$r%r&r'r(r) decay_flagr+ op_squareop_sqrtr1 op_reshapeop_shape param_fp32m_fp32v_fp32 gradient_fp32next_mnext_vZ regulate_mZ regulate_vupdateupdate_with_lrr-r.r.r/_update_run_opPs@          rLcCsttd|tg|td|tg|td|tg|t|ddtjd|t|ddtjd|t|d|dS)zCheck the type of inputs.r!r"r#rr3N) validatorcheck_value_typefloatcheck_float_ranger INC_NEITHERcheck_positive_float)r!r"r# prim_namer.r.r/_check_param_values rTcs:eZdZdZdfdd Zd d Zejdfd d ZZ S)ra_ Implements the Adam algorithm to fix the weight decay. It is a complete operator, not a combination of other ops. Note: When separating parameter groups, the weight decay in each group will be applied on the parameters if the weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive. To improve parameter groups performance, the customized order of parameters can be supported. Args: params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated, the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params", "lr", "weight_decay" and "order_params" are the keys can be parsed. - params: Required. The value must be a list of `Parameter`. - lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used. If not, the `learning_rate` in the API will be used. - weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay will be used. If not, the `weight_decay` in the API will be used. - order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters which in the 'order_params' must be in one of group parameters. learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate. When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule, use dynamic learning rate, the i-th learning rate will be calculated during the process of training according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float. Default: 1e-3. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Supported Platforms: ``CPU`` Examples: >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FusedAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FusedAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = Model(net, loss_fn=loss, optimizer=optim) MbP??+?ư>rFcstt||||t||||jtt|gtj |_ tt|gtj |_ tt|gtj |_ |j ddd|_|j ddd|_t|_|r|jddt|_dS)Nadam_mzerosprefixinitadam_vprimitive_targetCPU)superr__init__rTcls_namer nparrayastyper8r!r"r# clone_statemoments1moments2rAdamWeightDecayr add_prim_attrr version_flag)selfparams learning_rater!r"r#r%offload __class__r.r/rbs zFusedAdamWeightDecay.__init__c Cs|}|jr||j|j|jr|jrh|t t |j |j |j |j||j|j|j|j||j|j }q|t t |j |j |j |j||j|j|j|j||j|j}n<|t t |j |j |j |j||j|j|j|j||j|j}|jr|||S)zconstruct with gradients)get_lrrl assignadd global_stepglobal_step_increase_tensoris_group is_group_lr map_reverserpartial _adam_optrr!r"r#r% _parametersrhrir*r+ use_parallelbroadcast_params)rm gradientsr$ optim_resultr.r.r/ constructs8     zFusedAdamWeightDecay.constructc Cs|j}g}|D]}|}|dur$|j}|} |r2|j} tt||j| d} |j| _t|jdrn|jj | n | g|j_ d| _ |j | _ |j | _ |j| _|j| _|jr|j| _|d| j| _| | qt|S)a7 Clone the parameters parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method forced_dtype: mstype. The except the dtype to be cloned. If is_follow is True, forced_dtype will be ignored. Default: mstype.float32 is_follow: bool. Is clone the parameters with the original dtype. If is_follow is True, the forced_dtype argument will be ignored. Default: False. Nshaper cloned_objF.) parametersr]rr rr param_infoclonehasattrrappendis_init is_param_psinit_in_server cache_enable requires_aggr cache_shapenamer ) rmr\r]Z forced_dtypeZ is_followparameter_tuplenew old_param param_init cur_dtype new_stater.r.r/rg s0     z FusedAdamWeightDecay.clone_state)rUrVrWrXrF) __name__ __module__ __qualname____doc__rbrrr8rg __classcell__r.r.rqr/rs Hcs*eZdZdZd fdd Zd d ZZS) ra# This class is almost same with the mindspore's AdamWeightDecay implements, the only difference is the optimizer's state will be always initialized with float32, where the original AdamWeightDecay will initialize the optimizer's state with float16, if the parameters are initialized with fp16. This setting will avoid overflow in training big model using fp16. Args: params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the `params` is a list of `dict`, the string "params", "lr", "weight_decay", and "order_params" are the keys can be parsed. - params: Required. Parameters in current group. The value must be a list of `Parameter`. - lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used. If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported. - weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule to get the weight decay value of current step. - order_params: Optional. When parameters is grouped, this usually is used to maintain the order of parameters that appeared in the network to improve performance. The value should be parameters whose order will be followed in optimizer. If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in one group of `params`. learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3. - float: The fixed learning rate value. Must be equal to or greater than 0. - int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float. - Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied. For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate. - Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate. - LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of LearningRateSchedule with step as the input to get the learning rate of current step. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0. - float: The fixed weight decay value. Must be equal to or greater than 0. - int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float. - Cell: Weight decay is dynamic. During training, the optimizer calls the instance of the Cell with step as the input to get the weight decay value of current step. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Raises: TypeError: If `learning_rate` is not one of int, float, Tensor, Iterable, LearningRateSchedule. TypeError: If element of `parameters` is neither Parameter nor dict. TypeError: If `beta1`, `beta2` or `eps` is not a float. TypeError: If `weight_decay` is neither float nor int. ValueError: If `eps` is less than or equal to 0. ValueError: If `beta1`, `beta2` is not in range (0.0, 1.0). ValueError: If `weight_decay` is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> from mindformers.core.optim import FP32StateAdamWeightDecay >>> >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FP32StateAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FP32StateAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = ms.Model(net, loss_fn=loss, optimizer=optim) rUrVrWrXrcsttj||||t||||jtt|g tj |_ tt|g tj |_ tt|g tj |_ |jddd|_|jddd|_t|_tddkrd|_nd|_dS) NrYrZr[r^ device_targetr`TF)rarrjrbrTrcr rdrerfr8r!r"r#rgrhrir fused_optr get_context use_fused_opt)rmrnror!r"r#r%rqr.r/rbs z!FP32StateAdamWeightDecay.__init__cCs|j}g}|D]}tt||jtjd}|j|_t|jdrP|jj |n |g|j_ d|_ | t||jtjd|d|j |_ | |qt|S)z parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method rrFr)rr rrrr8rrrrrrset_datarr )rmr\r]rrrrr.r.r/rgs    z$FP32StateAdamWeightDecay.clone_state)rUrVrWrXr)rrrrrbrgrr.r.rqr/r/se)4rnumpyrd mindsporerrmindspore.commonrr mindspore.opsrrrCrrmindspore.common.initializerrmindspore.common.tensorr mindspore.common.parameterr r Zmindspore._checkparamr rMr ImportError _checkparammindspore.nn.optim.optimizerrmindformers.version_controlrmindformers.tools.registerrr__all__MultitypeFuncGraphr{int32r;Mulr9registerr0r2rLrT OPTIMIZERrrjrr.r.r.r/sP                :  pengcheng.mind-test/mindformers/core/optim/build_optim.py000066400000000000000000000102171471400134000242000ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Optimizer API.""" import inspect from mindspore import nn from mindspore.nn.optim import AdaFactor, AdamWeightDecay, SGD, Adagrad, Adam from mindformers.core.lr import build_lr from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .optim import FusedAdamWeightDecay, FP32StateAdamWeightDecay def build_optim( config: dict = None, default_args: dict = None, module_type: str = 'optimizer', class_name: str = None, **kwargs): r"""Build optim For MindFormer. Instantiate the optim from MindFormerRegister's registry. Args: config (dict): The task optim's config. Default: None. default_args (dict): The default argument of optim API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'optim'. class_name (str): The class name of optim API. Default: None. Return: The function instance of optim API. Examples: >>> from mindformers import build_optim >>> optim_config = {'type': 'AdamWeightDecay', 'weight_decay':0.05} >>> # 1) use config dict to build optim >>> optim_from_config = build_optim(optim_config) >>> # 2) use class name to build optim >>> optim_class_name = build_optim(class_name='AdamWeightDecay', weight_decay=0.05) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.learning_rate is not None and isinstance(config.learning_rate, dict): if config.learning_rate.type is None: raise ValueError("optimizer's learning rate must be LearningRateSchedule type, " "but the type is not specified, it is None") lr_schedule = build_lr(config.learning_rate) config.learning_rate = lr_schedule return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.OPTIMIZER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_optim(): """ register MindSpore builtin optimizer class. """ for module_name in dir(nn.optim): if module_name.startswith('__'): continue optim = getattr(nn.optim, module_name) if inspect.isclass(optim): MindFormerRegister.register_cls( optim, MindFormerModuleType.OPTIMIZER) def register_mf_optim(): """ register MindFormers builtin optimizer class. """ # adapt huggingface MindFormerRegister.register_cls( AdamWeightDecay, module_type=MindFormerModuleType.OPTIMIZER, alias="adamw") MindFormerRegister.register_cls( AdaFactor, module_type=MindFormerModuleType.OPTIMIZER, alias="adafactor") MindFormerRegister.register_cls( SGD, module_type=MindFormerModuleType.OPTIMIZER, alias="sgd") MindFormerRegister.register_cls( Adam, module_type=MindFormerModuleType.OPTIMIZER, alias="adam") MindFormerRegister.register_cls( Adagrad, module_type=MindFormerModuleType.OPTIMIZER, alias="adagrad") MindFormerRegister.register_cls( FusedAdamWeightDecay, module_type=MindFormerModuleType.OPTIMIZER, alias="fused_adamw") MindFormerRegister.register_cls( FP32StateAdamWeightDecay, module_type=MindFormerModuleType.OPTIMIZER, alias="fp32_adamw") register_ms_optim() register_mf_optim() pengcheng.mind-test/mindformers/core/optim/optim.py000066400000000000000000000551541471400134000230320ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """FusedAdamWeightDecay, a customized Adam for offloading.""" import numpy as np from mindspore import context, nn from mindspore.common import dtype as mstype from mindspore.ops import operations as P from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.common.initializer import initializer from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter, ParameterTuple # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator as validator from mindspore._checkparam import Rel except ImportError: import mindspore._checkparam as validator import mindspore._checkparam as Rel from mindspore.nn.optim.optimizer import Optimizer from mindformers.version_control import fix_optim_global_step_sig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['FusedAdamWeightDecay', 'FP32StateAdamWeightDecay'] _adam_opt = C.MultitypeFuncGraph("adam_opt") _scaler_one = Tensor(1, mstype.int32) op_mul = P.Mul() @_adam_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _fused_update_with_global_norm(opt, global_norm, beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flags, optim_filter): """ Update parameters by FusedAdamWeightDecay. """ success = True if optim_filter: if decay_flags: next_param = opt(param, m, v, lr, beta1, beta2, eps, weight_decay, P.Cast()(gradient, mstype.float16), global_norm) else: next_param = opt(param, m, v, lr, beta1, beta2, eps, 0.0, P.Cast()(gradient, mstype.float16), global_norm) return F.depend(success, next_param) return success @_adam_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _fused_update(opt, beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flags, optim_filter): """ Update parameters by FusedAdamWeightDecay. """ success = True op_cast = P.Cast() if optim_filter: if decay_flags: opt(param, m, v, lr, beta1, beta2, eps, weight_decay, op_cast(gradient, F.dtype(param))) else: opt(param, m, v, lr, beta1, beta2, eps, 0.0, op_cast(gradient, F.dtype(param))) return success @_adam_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _update_run_op(beta1_power, beta2_power, beta1, beta2, eps, lr, weight_decay, param, \ m, v, gradient, decay_flag, optim_filter): """ Update parameters. Args: beta1 (Tensor): The exponential decay rate for the 1st moment estimations. Should be in range (0.0, 1.0). beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0). eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0. lr (Tensor): Learning rate. weight_decay (Number): Weight decay. Should be equal to or greater than 0. param (Tensor): Parameters. m (Tensor): m value of parameters. v (Tensor): v value of parameters. gradient (Tensor): Gradient of parameters. decay_flag (bool): Applies weight decay or not. optim_filter (bool): Applies parameter update or not. Returns: Tensor, the new value of v after updating. """ if optim_filter: # op_mul = P.Mul(), defined output op_square = P.Square() op_sqrt = P.Sqrt() op_cast = P.Cast() op_reshape = P.Reshape() op_shape = P.Shape() param_fp32 = op_cast(param, mstype.float32) m_fp32 = op_cast(m, mstype.float32) v_fp32 = op_cast(v, mstype.float32) gradient_fp32 = op_cast(gradient, mstype.float32) next_m = op_mul(beta1, m_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32) - beta1, gradient_fp32) next_v = op_mul(beta2, v_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32) - beta2, op_square(gradient_fp32)) regulate_m = next_m / (_scaler_one - beta1_power) regulate_v = next_v / (_scaler_one - beta2_power) update = regulate_m / (eps + op_sqrt(regulate_v)) if decay_flag: update = op_mul(weight_decay, param_fp32) + update update_with_lr = op_mul(lr, update) next_param = param_fp32 - op_reshape(update_with_lr, op_shape(param_fp32)) next_param = F.depend(next_param, F.assign(param, op_cast(next_param, F.dtype(param)))) next_param = F.depend(next_param, F.assign(m, op_cast(next_m, F.dtype(m)))) next_param = F.depend(next_param, F.assign(v, op_cast(next_v, F.dtype(v)))) return op_cast(next_param, F.dtype(param)) return gradient def _check_param_value(beta1, beta2, eps, prim_name): """Check the type of inputs.""" validator.check_value_type("beta1", beta1, [float], prim_name) validator.check_value_type("beta2", beta2, [float], prim_name) validator.check_value_type("eps", eps, [float], prim_name) validator.check_float_range(beta1, 0.0, 1.0, Rel.INC_NEITHER, "beta1", prim_name) validator.check_float_range(beta2, 0.0, 1.0, Rel.INC_NEITHER, "beta2", prim_name) validator.check_positive_float(eps, "eps", prim_name) @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class FusedAdamWeightDecay(Optimizer): """ Implements the Adam algorithm to fix the weight decay. It is a complete operator, not a combination of other ops. Note: When separating parameter groups, the weight decay in each group will be applied on the parameters if the weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive. To improve parameter groups performance, the customized order of parameters can be supported. Args: params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated, the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params", "lr", "weight_decay" and "order_params" are the keys can be parsed. - params: Required. The value must be a list of `Parameter`. - lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used. If not, the `learning_rate` in the API will be used. - weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay will be used. If not, the `weight_decay` in the API will be used. - order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters which in the 'order_params' must be in one of group parameters. learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate. When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule, use dynamic learning rate, the i-th learning rate will be calculated during the process of training according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float. Default: 1e-3. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Supported Platforms: ``CPU`` Examples: >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FusedAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FusedAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = Model(net, loss_fn=loss, optimizer=optim) """ def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0, offload=False): super(FusedAdamWeightDecay, self).__init__(learning_rate, params, weight_decay) _check_param_value(beta1, beta2, eps, self.cls_name) self.beta1 = Tensor(np.array([beta1]).astype(np.float32)) self.beta2 = Tensor(np.array([beta2]).astype(np.float32)) self.eps = Tensor(np.array([eps]).astype(np.float32)) self.moments1 = self.clone_state(prefix="adam_m", init='zeros') self.moments2 = self.clone_state(prefix="adam_v", init='zeros') self.opt = P.AdamWeightDecay() if offload: self.opt.add_prim_attr("primitive_target", "CPU") self.version_flag = fix_optim_global_step_sig() def construct(self, gradients): """construct with gradients""" lr = self.get_lr() if self.version_flag: self.assignadd(self.global_step, self.global_step_increase_tensor) if self.is_group: if self.is_group_lr: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps), lr, self.weight_decay, self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) else: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps, lr), self.weight_decay, self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) else: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps, lr, self.weight_decay), self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) if self.use_parallel: self.broadcast_params(optim_result) return optim_result def clone_state(self, prefix, init, forced_dtype=mstype.float32, is_follow=False): r""" Clone the parameters parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method forced_dtype: mstype. The except the dtype to be cloned. If is_follow is True, forced_dtype will be ignored. Default: mstype.float32 is_follow: bool. Is clone the parameters with the original dtype. If is_follow is True, the forced_dtype argument will be ignored. Default: False. """ parameter_tuple = self.parameters new = [] for old_param in parameter_tuple: param_init = init if init is None: param_init = old_param.init cur_dtype = forced_dtype if is_follow: cur_dtype = old_param.dtype new_state = Parameter(initializer(param_init, shape=old_param.shape, dtype=cur_dtype)) new_state.param_info = old_param.param_info.clone() if hasattr(old_param.param_info, "cloned_obj"): old_param.param_info.cloned_obj.append(new_state) else: old_param.param_info.cloned_obj = [new_state] new_state.is_init = False new_state.is_param_ps = old_param.is_param_ps new_state.init_in_server = old_param.init_in_server new_state.cache_enable = old_param.cache_enable new_state.requires_aggr = old_param.requires_aggr if old_param.cache_shape: new_state.cache_shape = old_param.cache_shape new_state.name = prefix + '.' + new_state.name new.append(new_state) return ParameterTuple(new) @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class FP32StateAdamWeightDecay(nn.AdamWeightDecay): r""" This class is almost same with the mindspore's AdamWeightDecay implements, the only difference is the optimizer's state will be always initialized with float32, where the original AdamWeightDecay will initialize the optimizer's state with float16, if the parameters are initialized with fp16. This setting will avoid overflow in training big model using fp16. Args: params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the `params` is a list of `dict`, the string "params", "lr", "weight_decay", and "order_params" are the keys can be parsed. - params: Required. Parameters in current group. The value must be a list of `Parameter`. - lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used. If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported. - weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule to get the weight decay value of current step. - order_params: Optional. When parameters is grouped, this usually is used to maintain the order of parameters that appeared in the network to improve performance. The value should be parameters whose order will be followed in optimizer. If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in one group of `params`. learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3. - float: The fixed learning rate value. Must be equal to or greater than 0. - int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float. - Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied. For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate. - Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate. - LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of LearningRateSchedule with step as the input to get the learning rate of current step. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0. - float: The fixed weight decay value. Must be equal to or greater than 0. - int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float. - Cell: Weight decay is dynamic. During training, the optimizer calls the instance of the Cell with step as the input to get the weight decay value of current step. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Raises: TypeError: If `learning_rate` is not one of int, float, Tensor, Iterable, LearningRateSchedule. TypeError: If element of `parameters` is neither Parameter nor dict. TypeError: If `beta1`, `beta2` or `eps` is not a float. TypeError: If `weight_decay` is neither float nor int. ValueError: If `eps` is less than or equal to 0. ValueError: If `beta1`, `beta2` is not in range (0.0, 1.0). ValueError: If `weight_decay` is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> from mindformers.core.optim import FP32StateAdamWeightDecay >>> >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FP32StateAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FP32StateAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = ms.Model(net, loss_fn=loss, optimizer=optim) """ def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0): super(nn.AdamWeightDecay, self).__init__(learning_rate, params, weight_decay) _check_param_value(beta1, beta2, eps, self.cls_name) self.beta1 = Tensor(np.array([beta1]).astype(np.float32)) self.beta2 = Tensor(np.array([beta2]).astype(np.float32)) self.eps = Tensor(np.array([eps]).astype(np.float32)) self.moments1 = self.clone_state(prefix='adam_m', init='zeros') self.moments2 = self.clone_state(prefix='adam_v', init='zeros') self.fused_opt = P.AdamWeightDecay() if context.get_context("device_target") == "CPU": self.use_fused_opt = True else: self.use_fused_opt = False def clone_state(self, prefix, init): r""" parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method """ parameter_tuple = self.parameters new = [] for old_param in parameter_tuple: new_state = Parameter(initializer(init, shape=old_param.shape, dtype=mstype.float32)) new_state.param_info = old_param.param_info.clone() if hasattr(old_param.param_info, "cloned_obj"): old_param.param_info.cloned_obj.append(new_state) else: old_param.param_info.cloned_obj = [new_state] new_state.is_init = False new_state.set_data(initializer(init, shape=old_param.shape, dtype=mstype.float32)) new_state.name = prefix + '.' + new_state.name new.append(new_state) return ParameterTuple(new) pengcheng.mind-test/mindformers/core/parallel_config.py000066400000000000000000000047231471400134000236670ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Parallel Config Init.""" from mindformers.modules.transformer.moe import default_moe_config, MoEConfig from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.tools.logger import logger default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) def build_parallel_config(config): """Build context config.""" if config.moe_config: if not isinstance(config.moe_config, MoEConfig): logger.info("initial moe_config from dict: %s", config.moe_config) config.moe_config = MoEConfig(**config.moe_config) else: config.moe_config = default_moe_config if config.recompute_config: if not isinstance(config.recompute_config, TransformerRecomputeConfig): logger.info("initial recompute_config from dict: %s", config.recompute_config) config.recompute_config = TransformerRecomputeConfig(**config.recompute_config) else: config.recompute_config = default_recompute_config if config.parallel_config: if not isinstance(config.parallel_config, TransformerOpParallelConfig): logger.info("initial parallel_config from dict: %s", config.parallel_config) if config.parallel_config.pipeline_stage > 1: logger.info("pipeline_stage = %s > 1, vocab_emd_dp will be reset to False.", config.parallel_config.pipeline_stage) config.parallel_config.vocab_emb_dp = False config.parallel_config = TransformerOpParallelConfig(recompute=config.recompute_config, **config.parallel_config) else: config.parallel_config = default_parallel_config pengcheng.mind-test/mindformers/dataset/000077500000000000000000000000001471400134000206635ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/.ipynb_checkpoints/000077500000000000000000000000001471400134000244545ustar00rootroot00000000000000causal_language_model_dataset-checkpoint.py000066400000000000000000000247021471400134000351010ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/.ipynb_checkpoints# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Causal Image Modeling Dataset.""" import os import copy import re import numpy as np import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.models.build_tokenizer import build_tokenizer from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset def get_input_data_batch_slice_map(input_ids, eos_token_id, dis, rank_id: int = 0): """ Generate position_id and attention_mask according to input_ids considering eos reset Args: input_ids: the input token ids eos_token_id: the id for dis: the slice value for each rank rank_id: the current rank id Returns: batch_input_ids: the input token ids batch_position_ids: the position ids cosidering eos reset batch_attention_mask: the attention mask considering eos reset """ rank = int(rank_id) input_ids = input_ids[rank*dis: (rank + 1)*dis] seq_length = input_ids.shape[1] - 1 # Initialize position_ids and attention_mask batch_input_ids = input_ids batch_position_ids = np.ones((dis, seq_length)) batch_attention_mask = np.ones((dis, seq_length, seq_length)) # Loop through batches for bs_i in range(len(input_ids)): # Get normal position_ids and attention_mask local_ids = input_ids[bs_i] batch_attention_mask[bs_i] = np.tril(np.ones(shape=(seq_length, seq_length))) batch_position_ids[bs_i] = np.arange(seq_length) # Find the index of eos_index = batch_position_ids[bs_i, local_ids[:-1] == eos_token_id].astype(np.int32) prev_index = 0 for i in range(eos_index.size): # Reset position_ids and attention_mask considering index = eos_index[i] batch_attention_mask[bs_i, (index + 1):, :(index + 1)] = 0 batch_position_ids[bs_i, (index + 1):] -= (index + 1 - prev_index) prev_index = index + 1 return batch_input_ids, batch_position_ids, batch_attention_mask @MindFormerRegister.register(MindFormerModuleType.DATASET) class CausalLanguageModelDataset(BaseDataset): """Causal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import CausalLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='CausalLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = CausalLanguageModelDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Causal Language Model Dataset.") rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) cls.init_dataset_config(dataset_config) rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config.rank_id = rank_id dataset_config.device_num = device_num if dataset_config.data_loader.type != "MindDataset" and \ dataset_config.data_loader.type != "TFRecordDataset": dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) type_cast_op = C.TypeCast(mstype.int32) if dataset_config.eos_reset: if cls._is_semi_full_batch() or cls._is_data_parallel(): rank_id = 0 dis = dataset_config.batch_size else: # Each card slice a small batch from the full batch dis = dataset_config.batch_size // device_num if dataset_config.batch_size % device_num != 0: raise ValueError( f"batch size {dataset_config.batch_size} should be a multiple of device number {device_num}." " You should change the args: per_batch_size.") dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns) map_func = lambda input_ids: get_input_data_batch_slice_map(input_ids, eos_token_id=dataset_config.eos_token_id, rank_id=rank_id, dis=dis) dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns) dataset = dataset.project(columns=dataset_config.output_columns) for input_arg in dataset_config.output_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) else: dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.input_columns) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) dataset = dataset.repeat(dataset_config.repeat) return dataset @classmethod def _prepare_for_model(cls, dataset, dataset_config): """Preprocess data for gpt2 model""" tokenizer_config = dataset_config.tokenizer tokenizer = build_tokenizer(tokenizer_config) max_length = tokenizer_config.max_length def map_func(input_data): input_data = input_data.tolist() input_ids = tokenizer(input_data, padding='max_length', max_length=max_length, truncation=True, add_special_tokens=False) return input_ids.get('input_ids') dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.input_columns) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id}) dataset = cls._prepare_for_model(dataset, dataset_config) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" dataset_files = [] mind_compile = re.compile("mindrecord") if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if (re.findall(mind_compile, file) and not file.endswith(".db")) or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if re.findall(mind_compile, data_dir) or data_dir.endswith(".tfrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id, 'columns_list': dataset_config.input_columns}) return dataset pengcheng.mind-test/mindformers/dataset/__init__.py000066400000000000000000000047641471400134000230070ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Dataset.""" from .dataloader import * from .mask import * from .transforms import * from .sampler import * from .build_dataset import build_dataset from .dataloader.build_dataloader import build_dataset_loader from .mask.build_mask import build_mask from .sampler.build_sampler import build_sampler from .transforms.build_transforms import build_transforms from .base_dataset import BaseDataset from .mim_dataset import MIMDataset from .img_cls_dataset import ImageCLSDataset from .contrastive_language_image_pretrain_dataset import ContrastiveLanguageImagePretrainDataset from .zero_shot_image_classification_dataset import ZeroShotImageClassificationDataset from .token_classification_dataset import TokenClassificationDataset from .mask_language_model_dataset import MaskLanguageModelDataset from .translation_dataset import TranslationDataset from .image_text_retrieval_dataset import ImageToTextRetrievalDataset from .text_classification_dataset import TextClassificationDataset from .question_answering_dataset import QuestionAnsweringDataset from .causal_language_model_dataset import CausalLanguageModelDataset from .reward_model_dataset import RewardModelDataset from .utils import check_dataset_config from .keyword_gen_dataset import KeyWordGenDataset __all__ = ['MIMDataset', 'ImageCLSDataset', 'BaseDataset', 'check_dataset_config', 'MaskLanguageModelDataset', 'ContrastiveLanguageImagePretrainDataset', 'ZeroShotImageClassificationDataset', 'ImageToTextRetrievalDataset', 'TextClassificationDataset', 'TokenClassificationDataset', 'QuestionAnsweringDataset', 'CausalLanguageModelDataset', 'KeyWordGenDataset', 'RewardModelDataset'] __all__.extend(dataloader.__all__) __all__.extend(mask.__all__) __all__.extend(transforms.__all__) __all__.extend(sampler.__all__) __all__.extend(translation_dataset.__all__) pengcheng.mind-test/mindformers/dataset/__pycache__/000077500000000000000000000000001471400134000230735ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/__pycache__/__init__.cpython-39.pyc000066400000000000000000000034011471400134000272610ustar00rootroot00000000000000a ?cf @s\dZddlTddlTddlTddlTddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+gdZ,e,-ej,e,-ej,e,-ej,e,-ej,e,-ej,dS)zMindFormers Dataset.)*) build_dataset)build_dataset_loader) build_mask) build_sampler)build_transforms) BaseDataset) MIMDataset)ImageCLSDataset)'ContrastiveLanguageImagePretrainDataset)"ZeroShotImageClassificationDataset)TokenClassificationDataset)MaskLanguageModelDataset)TranslationDataset)ImageToTextRetrievalDataset)TextClassificationDataset)QuestionAnsweringDataset)CausalLanguageModelDataset)RewardModelDataset)check_dataset_config)KeyWordGenDataset)r r rrrr r rrr rrrrN).__doc__Z dataloadermask transformssamplerrZdataloader.build_dataloaderrZmask.build_maskrZsampler.build_samplerrZtransforms.build_transformsrZ base_datasetrZ mim_datasetr Zimg_cls_datasetr Z+contrastive_language_image_pretrain_datasetr Z&zero_shot_image_classification_datasetr Ztoken_classification_datasetr Zmask_language_model_datasetrZtranslation_datasetrZimage_text_retrieval_datasetrZtext_classification_datasetrZquestion_answering_datasetrZcausal_language_model_datasetrZreward_model_datasetrutilsrZkeyword_gen_datasetr__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/dataset/__init__.pys<                        pengcheng.mind-test/mindformers/dataset/__pycache__/base_dataset.cpython-39.pyc000066400000000000000000000041671471400134000301530ustar00rootroot00000000000000a ?cf @s2dZddlZddlZddlmZGdddZdS) Base Dataset.Nc@sPeZdZdZdedddZeddZedd Zed d Z ed d Z dS) BaseDatasetrNdataset_configcCs ||_dS)Nr)selfrrK/home/ma-user/work/mindformers-baichuan/mindformers/dataset/base_dataset.py__init__szBaseDataset.__init__cCs|durtdtj|jtj|jtj|j|j r|j rNt dt j |jddt j|jd|_tjjd|jdtj|jdS)zInit dataset config.Nzdataset_config cannot be None!znMindSpore's AutoTune is enabled, so Profile cannot be enabled,now Profile's flag is True, please set to False!T)exist_okZautotune)filepath_prefix) ValueErrordsconfigset_seedseedset_prefetch_size prefetch_sizeset_numa_enable numa_enableZ auto_tuneprofileEnvironmentErrorosmakedirsr pathjoinset_enable_autotuneset_autotune_intervalZautotune_per_step)clsrrrrinit_dataset_configszBaseDataset.init_dataset_configcCs|rd}d}||fS)z3Check device num and rank id in auto parallel mode.N)_is_semi_full_batch)rrank_id device_numrrr_check_device_rank_for_parallel/sz+BaseDataset._check_device_rank_for_parallelcCstjddvotjdS)N parallel_mode)semi_auto_parallel auto_parallel full_batch)mscontextget_auto_parallel_contextrrrrr7s zBaseDataset._is_semi_full_batchcCstjdtjjjkS)Nr#)r'r(r) ParallelMode DATA_PARALLELr*rrr_is_data_parallel<szBaseDataset._is_data_parallel)N) __name__ __module__ __qualname____doc__dictr classmethodrr"rr-rrrrrs   r)r1r mindsporer'mindspore.datasetdatasetr rrrrrs pengcheng.mind-test/mindformers/dataset/__pycache__/build_dataset.cpython-39.pyc000066400000000000000000000034771471400134000303430ustar00rootroot00000000000000a ?cf @s2dZddlmZmZmZdeeeedddZdS) zBuild Dataset API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfigNdataset)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)aCBuild dataset For MindFormer. Instantiate the dataset from MindFormerRegister's registry. Args: config (dict): The task dataset's config. Default: None. default_args (dict): The default argument of dataset API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset'. class_name (str): The class name of dataset API. Default: None. Return: The function instance of dataset API. Examples: >>> from mindformers import build_dataset >>> from mindformers.dataset import check_dataset_config >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_class_name = build_dataset(class_name='ImageCLSDataset', dataset_config=config.train_dataset_task) N)r) isinstancedictrrget_instance_from_cfgrDATASET get_instance)rrrr kwargsrL/home/ma-user/work/mindformers-baichuan/mindformers/dataset/build_dataset.py build_datasetsr)NNrN)__doc__mindformers.tools.registerrrrr strrrrrrspengcheng.mind-test/mindformers/dataset/__pycache__/causal_language_model_dataset.cpython-39.pyc000066400000000000000000000160471471400134000335340ustar00rootroot00000000000000a of*@sdZddlZddlZddlZddlZddlmmZ ddl m m m ZddlmZmZddlmZddlmZddlmZddlmZdd lmZded d d ZeejGd ddeZ dS)zCausal Image Modeling Dataset.N)MindFormerRegisterMindFormerModuleType)logger)build_tokenizer)get_dataset_map)build_dataset_loader) BaseDataset)rank_idc Cs"t|}||||d|}|jdd}|}t||f}t|||f}tt|D]} || } ttj||fd|| <t||| <|| | dd|kftj } d} t| j D]V} | | }d|| |ddd|df<|| |ddf|d| 8<|d} qq\|||fS)a Generate position_id and attention_mask according to input_ids considering eos reset Args: input_ids: the input token ids eos_token_id: the id for dis: the slice value for each rank rank_id: the current rank id Returns: batch_input_ids: the input token ids batch_position_ids: the position ids cosidering eos reset batch_attention_mask: the attention mask considering eos reset r)shapeNr) intr nponesrangelentrilarangeastypeint32size) input_ids eos_token_iddisr rank seq_lengthZbatch_input_idsZbatch_position_idsZbatch_attention_maskZbs_iZ local_idsZ eos_index prev_indexiindexr\/home/ma-user/work/mindformers-baichuan/mindformers/dataset/causal_language_model_dataset.pyget_input_data_batch_slice_maps$ $ r!c@sDeZdZdZd edddZeddZedd Zed d Z dS) CausalLanguageModelDatasetaCausal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import CausalLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='CausalLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = CausalLanguageModelDataset(config.train_dataset_task.dataset_config) N)dataset_configcstdttddttdd}t|||\}_ |_ j j dkr~j j dkr~| }n |}ttj}jrN|s|rdjn0j|j|dkrtd jd |d |jjjjd }fd d}t||jjd}|jjd}jD]}t|||d}q6nD|jjjjjd}|jjd}jD]}t|||d}q||j}|S)Nz)Now Create Causal Language Model Dataset.RANK_ID0 RANK_SIZE1 MindDatasetTFRecordDatasetrz batch size z' should be a multiple of device number z-. You should change the args: per_batch_size.)drop_remainderoutput_columnscst|jdS)N)rr r)r!r)rr#rr rr sz4CausalLanguageModelDataset.__new__.. input_columnsr+)columns)r/)r*r+num_parallel_workers) rinfor osgetenvcopydeepcopyinit_dataset_config_check_device_rank_for_parallelr device_num data_loadertype_process_raw_text_data_process_mindrecord_dataCTypeCastmstyperZ eos_reset_is_semi_full_batch_is_data_parallel batch_size ValueErrorbatchr*r/rr+projectr1repeat)clsr#r9dataset type_cast_opmap_func input_argrr,r __new__`sb              z"CausalLanguageModelDataset.__new__cs:|j}t||jfdd}t|||j|jd}|S)zPreprocess data for gpt2 modelcs$|}|dddd}|dS)N max_lengthTF)paddingrN truncationadd_special_tokensr)tolistget) input_datarrN tokenizerrr rKs  z?CausalLanguageModelDataset._prepare_for_model..map_funcr.)rVrrNrr/)rHrIr#tokenizer_configrKrrUr _prepare_for_modelsz-CausalLanguageModelDataset._prepare_for_modelcCs6|jd}t|j||j|jdd}|||}|S)zProcess the text data dataset_dir)rY num_shardsshard_id default_args)r:poprr9r rX)rHr#rYrIrrr r<s  z1CausalLanguageModelDataset._process_raw_text_datac Cs:g}td}|jjr|jd}tj|rt|D]H\}}}|D]8}t ||r`| drj| drF| tj ||qFq8| qt ||s| dr|}n:|jjr|jj}t|ttfrt|}ntd|jdtd|jt|j||j|j|jdd } |jr6| |j} td |jd | S) zProcess the mindrecord data mindrecordrYz.dbz .tfrecordz>data_loader must contain dataset_dir or dataset_files,but get .zbuild dataset with : ) dataset_filesrZr[ columns_listr\zskip z samples)recompiler:rYr^r3pathisdirwalkfindallendswithappendjoinsortra isinstancelisttuplerDprintrr9r r/consumed_samplesskip) rHr#raZ mind_compiledata_dirr_ffilerIrrr r=s>        z3CausalLanguageModelDataset._process_mindrecord_data)N) __name__ __module__ __qualname____doc__dictrM classmethodrXr<r=rrrr r"Fs9  r")r)!r{r3r5rcnumpyrZmindspore.common.dtypecommondtyper@Z)mindspore.dataset.transforms.c_transformsrI transforms c_transformsr>mindformers.tools.registerrrmindformers.tools.loggerrZ"mindformers.models.build_tokenizerrmindformers.version_controlr dataloaderr base_datasetr r r!registerDATASETr"rrrr s     ( contrastive_language_image_pretrain_dataset.cpython-39.pyc000066400000000000000000000067361471400134000364400ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/__pycache__a ?cf@sdZddlZddlmZddlmZddlmZddlm Z ddl m Z d d l m Z d d lmZd d lmZmZeejGd dde ZdS)z,Contrastive Language Image Pretrain Dataset.N)get_dataset_map)build_dataset_loader)build_transforms) build_sampler) BaseDataset)logger)build_tokenizer)MindFormerRegisterMindFormerModuleTypec@s eZdZdZdedddZdS)'ContrastiveLanguageImagePretrainDataseta^ Contrastive Language Image Pretrain Dataset API. output image and text columns Args: dataset_config (dict): Config for dataset. Returns: A dataset for ContrastiveLanguageImagePretrainTrainer. Examples: >>> import os >>> from mindformers import MindFormerBook, MindFormerConfig, build_dataset >>> project_path = MindFormerBook.get_project_path() >>> config_path = os.path.join(project_path, "configs", "clip", >>> "run_clip_vit_b_32_pretrain_flickr8k.yaml") >>> config = MindFormerConfig(config_path) Note: Put flickr8k dataset to ./checkpoint_download The detailed data setting could refer to ./configs/clip/clip.md >>> config.train_dataset_task.dataset_config.batch_size = 1 >>> dataset = build_dataset(config.train_dataset_task) >>> for item in dataset: >>> print(item) >>> break [Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[4.99690473e-001, 6.74871564e-001, ... 3.68304640e-001, 2.36918822e-001], [7.91658998e-001, 7.62462139e-001, ... -2.01033935e-001, -1.13443382e-001], ... [-5.98575652e-001, -6.12795711e-001, ... 1.47755420e+000, 1.46333420e+000], [-3.85274649e-001, -6.27015769e-001, ... 1.42067397e+000, 1.43489408e+000], [-7.97656536e-001, -1.01095748e+000, ... 9.37191546e-001, 9.08751369e-001]]]]), Tensor(shape=[1, 77], dtype=Int32, value= [[49406, 1237, 18250 ... 0, 0, 0]])] N)dataset_configc Cstd||ttdd}ttdd}t|j||dd}t|j }t |j }t|j d|id}t |j}|d ur||}|d urt||d |j|jd }|d urt||d |j|jd }|j|j|j|jd }||j}|S)z new methodz7Now Create Contrastive Language Image Pretrain Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_args tokenizerNimage) input_columnsnum_parallel_workerspython_multiprocessingtext)drop_remainderr)r infoinit_dataset_configintosgetenvr data_loaderr transformsr rtext_transformsrsampler use_samplerrrrbatch batch_sizerrepeat) clsrrank_id device_numdatasetr#rr$r%r.j/home/ma-user/work/mindformers-baichuan/mindformers/dataset/contrastive_language_image_pretrain_dataset.py__new__?sB        z/ContrastiveLanguageImagePretrainDataset.__new__)N)__name__ __module__ __qualname____doc__dictr0r.r.r.r/r s#r )r4r mindformers.version_controlr dataloaderrr#rr%r base_datasetrtoolsr Zmodels.build_tokenizerr Ztools.registerr r registerDATASETr r.r.r.r/s        pengcheng.mind-test/mindformers/dataset/__pycache__/image_text_retrieval_dataset.cpython-39.pyc000066400000000000000000000035671471400134000334470ustar00rootroot00000000000000a ?cf` @sdZddlZddlmmZddlmZddl mm m Z ddl mZmZddlmZddlmZddlmZddl mZeejGd d d eZdS) zImage-text Retrieval Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map) BaseDataset)build_transformsc@s eZdZdZdedddZdS)ImageToTextRetrievalDatasetzBImage-text Retrieval Dataset for filip fine-tuning and evaluation.N)dataset_configcCstd||ttdd}ttdd}tj|jj |jj ||d}t |j }|durzt |||j|jddgd}ttj}t ||ddgd }|dd g}|j|j|jd }|S) Nz(Now Create Image-text Retrieval Dataset.RANK_ID0 RANK_SIZE1)shuffle num_shardsshard_idimage)num_parallel_workerspython_multiprocessing input_columnsoutput_columns)rrtoken)drop_remainder)rinfoinit_dataset_configintosgetenvds MindDataset data_loader dataset_dirrr transformsrrrCTypeCastmstypefloat32projectbatch batch_sizer)clsr rank_id device_numdatasetr" type_cast_opr/[/home/ma-user/work/mindformers-baichuan/mindformers/dataset/image_text_retrieval_dataset.py__new__s0     z#ImageToTextRetrievalDataset.__new__)N)__name__ __module__ __qualname____doc__dictr1r/r/r/r0r sr )r5rZmindspore.common.dtypecommondtyper%mindspore.datasetr-rZ)mindspore.dataset.transforms.c_transformsr" c_transformsr#mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr base_datasetrrregisterDATASETr r/r/r/r0s      pengcheng.mind-test/mindformers/dataset/__pycache__/img_cls_dataset.cpython-39.pyc000066400000000000000000000066721471400134000306610ustar00rootroot00000000000000a ?cf@sdZddlZddlmmmZddlmm Z ddl m Z m Z ddlmZddlmZddlmZddlmZdd lmZdd lmZe e jGd d d eZdS) zImage Classification Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader)build_transforms) build_sampler) BaseDatasetc@s eZdZdZdedddZdS)ImageCLSDataseta Image Classification Dataset API. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import ImageCLSDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['image_classification']['vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/image_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='ImageCLSDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = ImageCLSDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cs*td||ttdd}ttdd}t|j||dd}t|j }t |j }t tj}|durz||}|durt|||jd|j|jd }t|||jd |jd }|j|j|j|jd }|js |jdur tfd di|j}t|||j|j|jd}|j|jd}||j}|S)Nz(Now Create Image Classification Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_argsr) input_columnsnum_parallel_workerspython_multiprocessingr)rr)drop_remainderr class_nameMixup)routput_columnsr)columns)rinfoinit_dataset_configintosgetenvr data_loaderr transformsr samplerCTypeCastmstypeint32 use_samplerrrrrbatch batch_sizerdo_evalmixup_oprprojectrepeat) clsr rank_id device_numdatasetr"r#Z type_cast_opr,r3N/home/ma-user/work/mindformers-baichuan/mindformers/dataset/img_cls_dataset.py__new__:sF         zImageCLSDataset.__new__)N)__name__ __module__ __qualname____doc__dictr5r3r3r3r4r sr )r9rZ)mindspore.dataset.transforms.c_transformsr2r" c_transformsr$Zmindspore.common.dtypecommondtyper&mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderrrr#r base_datasetr registerDATASETr r3r3r3r4s       pengcheng.mind-test/mindformers/dataset/__pycache__/keyword_gen_dataset.cpython-39.pyc000066400000000000000000000227211471400134000315520ustar00rootroot00000000000000a ?cf ;@sdZddlZddlZddlmmZddlmm m Z ddl Z ddlmZddlmZddlmZddlmZddlmZddlmZmZdd lmZeejGd d d eZdS) zKeyword Generation Dataset.N) BaseTokenizer) BaseDataset)build_dataset_loader)build_tokenizer)logger)MindFormerModuleTypeMindFormerRegister)get_dataset_mapc@seZdZdZdedddZeddZedd Zed d Z ed d Z eddZ eddZ eddZ edddZedddZed ddZdS)!KeyWordGenDatasetaKeyword generation dataset. Examples: >>> from mindformers.dataset.dataloader.adgen_dataloader import ADGenDataLoader >>> from mindformers.dataset import build_dataset >>> from mindformers import MindFormerConfig >>> cfg = MindFormerConfig("./configs/glm/run_glm_6b_finetune.yaml") >>> dataset = build_dataset(cfg.eval_dataset_task) >>> for item in dataset.create_dict_iterator(): >>> print(item) >>> break N)dataset_configcCstd||ttdd}ttdd}|||\}}||_||_t |j t rf|j |_ n t |j |_ |j |_ |j|_|j|_|j|j|_|jj|_|jj|_|jjdkr||}n ||}|j|j|j|jd}||j}ttj}|j D]}t!|||d}q|S) Nz&Now Create Keyword Generation Dataset.RANK_ID0 RANK_SIZE1 MindDataset)drop_remaindernum_parallel_workers) input_columns)"rinfoinit_dataset_configintosgetenv_check_device_rank_for_parallelrank_id device_num isinstance tokenizerrrignore_pad_token_for_lossmax_source_lengthmax_target_lengthmax_seq_length data_loaderphaseversiontype_process_raw_text_data_process_mindrecord_databatch batch_sizerrrepeatCTypeCastmstypeint32rr )clsr rrdataset type_cast_op input_argr3R/home/ma-user/work/mindformers-baichuan/mindformers/dataset/keyword_gen_dataset.py__new__/s8             zKeyWordGenDataset.__new__c s|j}|jr|jnd}td||dkr@|jddg}|jn|jgd}|jddg}ddg}fd d }fd d } |d krt||||d}|j |d}|dkrt|| ||d}|j |d}|S)z/Maps the tokenizer on the source and the outputz1Start tokenize on the dataset using tokenizer: %s input_idslabels)r8r9 position_idsattention_maskpromptanswercs ||SNr3r<r=)train_dataset_functionr3r4train_dataset_funcfsz.train_dataset_funccs ||Sr>r3r?)eval_dataset_functionr3r4eval_dataset_funcisz;KeyWordGenDataset._tokenizer_map..eval_dataset_functrain)routput_columns)columnseval) r#r$rrtrain_dataset_functionv2eval_dataset_functionv2r@rBr project) r/r0tokenizer_configr#r$Ztrain_output_columnsrZeval_output_columnsrArCr3)rBr@r4_tokenizer_mapQs:     z KeyWordGenDataset._tokenizer_mapcCs8|jd}t|j||j|jdd}|||j}|S)zProcess the text data dataset_dir)rM num_shardsshard_id default_args)r"poprrrrLr)r/r rMr0r3r3r4r&zs z(KeyWordGenDataset._process_raw_text_datac Cst|}g}|jjr|jd}tj|rvt|D]2\}}}|D]"}| drF| tj ||qFq8| q| dr|}n:|jj r|jj }t|ttfrt|}ntd|jdtd|jt|j||j|j|jdd}|S)zProcess the mindrecord datarMz .mindrecordz>data_loader must contain dataset_dir or dataset_files,but get .z&Using args %s to instance the dataset.) dataset_filesrNrO columns_listrP)copydeepcopyr"rMrRrpathisdirwalkendswithappendjoinsortrTrlisttuple ValueErrorrrrrrr) r/r rTdata_dirr_ffiler0r3r3r4r's8        z*KeyWordGenDataset._process_mindrecord_datac s4||}}jj|dd}jj|dd}t|jdkrV|djd}t|jdkrz|djd}j||}|jj}|d}dg|||dd}j t|} |jj g| }|jj g| d}j rfdd|D} t |} t |} ||| | fS) generates train datasetFtextadd_special_tokensr6Nr7cs g|]}|jjkr|ndqSrkr pad_token_id.0lr/r3r4 z.)tolistrencodelenrr build_inputs_with_special_tokensindex bos_token_idr!rnrcreate_position_idsnparray get_masks) r/r<r= prompt_ids answer_idsr8context_lengthZ mask_positionlabelpad_lenr:r;r3rrr4r@s&z(KeyWordGenDataset.train_dataset_functionc s8jjd}||}}d}j||}jj|djd}jj|djd}t|jdkr|djd}t|jdkr|djd}t|}||jjg}jjg||ddjjg} |t|} |jjg| }| jjg| d} j r0fdd| D} || fS) rgr6NTrirj max_lengthFr7cs g|]}|jjkr|ndqSrlrmrorrr3r4rsrtz>KeyWordGenDataset.train_dataset_functionv2..) rr rur build_promptrvrw eos_token_idrnr) r/r<r=r!historyrrrr8r9rr3rrr4rHs&$ z*KeyWordGenDataset.train_dataset_functionv2cCs||}}d}|j||}t||jdkrH|d|jd}t||jdkrl|d|jd}|jj|d|jd}|jj|d|jd}|jt|}||jjg|}|jt|}||jjg|}||fS)generates eval datasetNr6Tr)rurrrwrr rvrn)r/r<r=rr8rrr3r3r4rIsz)KeyWordGenDataset.eval_dataset_functionv2cCs||}}t||jdkr6|d|jd}t||jdkrZ|d|jd}|jj|dd}|jj|dd}|jt|}||jjg|}|jt|}||jjg|}||fS)rr7NTrh)rurwrr rrvrn)r/r<r=r8rrr3r3r4rBsz'KeyWordGenDataset.eval_dataset_functioncCs|jd}|tj|tjd}t||}t|dddf}ttj||ftjd}|D]}d|ddd|f<q^t| tj }| tj}t |d}|S)zgenerate mask from input idrshapedtypeNrr6) rr|onesr.equalargwheretrilfloat32 logical_notastypebool_ expand_dims)r/r8rz seq_lengthmaskcontext_lengthsr;rr3r3r4r~ s    zKeyWordGenDataset.get_masksTc s|jd|durdg}|tjtjd}t||}t|dddf}|rtjtjd}t|D]\} } || || d<qjfdd|D} tj | dd  } tj || fdd }n:tjtjd}t|D] \} } || s|| || d<q|S) z6generate position ids from input id and mask positionsrNFrrrc s:g|]2}ttj|tjdtj|tjddfqS)rr6)r| concatenatezerosint64arange)rprrr3r4rs/s z6KeyWordGenDataset.get_position_ids..)axis) rr|rr.rrrr enumeratestacksqueeze) r/r8mask_positions use_gmasksrzZposition_encoding_2drrr:irZblock_position_idsr3rr4get_position_idss(   z"KeyWordGenDataset.get_position_idsc CsX|jd}|}|tj|tjd}t||}t|dddf}|j|||d}|S)z#generate position ids from input idrrNr)rr)rr|rr.rrr) r/r8Zgmask_token_idrseqsrrrr:r3r3r4r{<s  z%KeyWordGenDataset.create_position_ids)N)r)NrT)r)__name__ __module__ __qualname____doc__dictr5 classmethodrLr&r'r@rHrIrBr~rr{r3r3r3r4r s0 " (        r ) rrVrZmindspore.common.dtypecommonrr-Z)mindspore.dataset.transforms.c_transformsr0 transforms c_transformsr+numpyr|!mindformers.models.base_tokenizerrZ mindformers.dataset.base_datasetrZmindformers.dataset.dataloaderrZ"mindformers.models.build_tokenizerrmindformers.tools.loggerrmindformers.tools.registerrrmindformers.version_controlr registerDATASETr r3r3r3r4s       pengcheng.mind-test/mindformers/dataset/__pycache__/labels.cpython-39.pyc000066400000000000000000000301311471400134000267640ustar00rootroot00000000000000a ?cf@?@sTdZgdZgdZgdZddgZddgZddgZgdZeeeeeeed Zd S) zlabels of imagenet(ZtenchZgoldfishzgreat white sharkz tiger sharkZ hammerheadz electric rayZstingrayZcockhenZostrichZ bramblingZ goldfinchz house finchZjuncozindigo buntingZrobinZbulbulZjayZmagpieZ chickadeez water ouzelZkitez bald eagleZvulturezgreat grey owlzEuropean fire salamanderz common newtZeftzspotted salamanderZaxolotlZbullfrogz tree frogz tailed frogZ loggerheadzleatherback turtlez mud turtleZterrapinz box turtlez banded geckoz common iguanazAmerican chameleonZwhiptailZagamazfrilled lizardzalligator lizardz Gila monsterz green lizardzAfrican chameleonz Komodo dragonzAfrican crocodilezAmerican alligatorZ triceratopsz thunder snakezringneck snakez hognose snakez green snakez king snakez garter snakez water snakez vine snakez night snakezboa constrictorz rock pythonz Indian cobraz green mambaz sea snakez horned viperZ diamondbackZ sidewinder trilobiteZ harvestmanscorpionzblack and gold garden spiderz barn spiderz garden spiderz black widowZ tarantulaz wolf spidertickZ centipedez black grouseZ ptarmiganz ruffed grousezprairie chickenZpeacockZquailZ partridgez African greyZmacawzsulphur-crested cockatooZlorikeetZcoucalz bee eaterZhornbillZ hummingbirdZjacamarZtoucanZdrakezred-breasted merganserZgoosez black swanZtuskerZechidnaplatypusZwallabyZkoalaZwombatZ jellyfishz sea anemonez brain coralZflatwormZnematodeZconchZsnailslugzsea slugZchitonzchambered nautiluszDungeness crabz rock crabz fiddler crabz king crabzAmerican lobsterz spiny lobstercrayfishz hermit crabZisopodz white storkz black storkZ spoonbillflamingozlittle blue heronzAmerican egretZbitterncraneZlimpkinzEuropean gallinulez American cootZbustardzruddy turnstonezred-backed sandpiperZredshankZ dowitcherZ oystercatcherZpelicanz king penguinZ albatrossz grey whalez killer whaleZdugongzsea lionZ ChihuahuazJapanese spanielz Maltese dogZPekinesezShih-TzuzBlenheim spanielZpapillonz toy terrierzRhodesian ridgebackz Afghan houndZbassetZbeagleZ bloodhoundZbluetickzblack-and-tan coonhoundz Walker houndzEnglish foxhoundZredboneZborzoizIrish wolfhoundzItalian greyhoundZwhippetz Ibizan houndzNorwegian elkhoundZ otterhoundZSalukizScottish deerhoundZ WeimaranerzStaffordshire bullterrierzAmerican Staffordshire terrierzBedlington terrierzBorder terrierzKerry blue terrierz Irish terrierzNorfolk terrierzNorwich terrierzYorkshire terrierzwire-haired fox terrierzLakeland terrierzSealyham terrierZAiredaleZcairnzAustralian terrierzDandie Dinmontz Boston bullzminiature schnauzerzgiant schnauzerzstandard schnauzerzScotch terrierzTibetan terrierz silky terrierzsoft-coated wheaten terrierzWest Highland white terrierZLhasazflat-coated retrieverzcurly-coated retrieverzgolden retrieverzLabrador retrieverzChesapeake Bay retrieverzGerman short-haired pointerZvizslazEnglish setterz Irish setterz Gordon setterzBrittany spanielZclumberzEnglish springerzWelsh springer spanielzcocker spanielzSussex spanielzIrish water spanielZkuvaszZ schipperkeZ groenendaelZmalinoisZbriardZkelpieZkomondorzOld English sheepdogzShetland sheepdogZcolliez Border colliezBouvier des FlandresZ RottweilerzGerman shepherdZDobermanzminiature pinscherzGreater Swiss Mountain dogzBernese mountain dogZ AppenzellerZ EntleBucherboxerz bull mastiffzTibetan mastiffzFrench bulldogz Great Danez Saint Bernardz Eskimo dogZmalamutezSiberian husky dalmatianZ affenpinscherZbasenjiZpugZLeonbergZ NewfoundlandzGreat PyreneesZSamoyedZ PomeranianZchowZkeeshondzBrabancon griffonZPembrokeZCardiganz toy poodlezminiature poodlezstandard poodlezMexican hairlessz timber wolfz white wolfzred wolfZcoyoteZdingoZdholezAfrican hunting dogZhyenazred foxzkit foxz Arctic foxzgrey foxZtabbyz tiger catz Persian catz Siamese catz Egyptian catZcougarZlynxZleopardz snow leopardZjaguarZlionZtigerZcheetahz brown bearzAmerican black bearzice bearz sloth bearZmongooseZmeerkatz tiger beetleZladybugz ground beetlezlong-horned beetlez leaf beetlez dung beetlezrhinoceros beetleZweevilZflyZbeeantZ grasshopperZcricketz walking stickZ cockroachZmantisZcicadaZ leafhopperZlacewing dragonflyZ damselflyZadmiralZringletZmonarchzcabbage butterflyzsulphur butterflyZlycaenidstarfishz sea urchinz sea cucumberz wood rabbithareZAngoraZhamsterZ porcupinez fox squirrelZmarmotbeaverz guinea pigZsorrelZzebraZhogz wild boarZwarthogZ hippopotamusoxz water buffaloZbisonZramZbighornZibexZ hartebeestZimpalaZgazellez Arabian camelllamaZweaselZminkZpolecatzblack-footed ferretZotterZskunkZbadgerZ armadillozthree-toed slothZ orangutanZgorillaZ chimpanzeeZgibbonZsiamangZguenonZpatasZbaboonZmacaqueZlangurZcolobuszproboscis monkeyZmarmosetZcapuchinz howler monkeyZtitiz spider monkeyzsquirrel monkeyzMadagascar catZindrizIndian elephantzAfrican elephantz lesser pandaz giant pandaZ barracoutaZeelZcohoz rock beautyz anemone fishZsturgeonZgarZlionfishZpufferZabacusZabayaz academic gown accordionzacoustic guitarzaircraft carrierZairlinerZairshipZaltarZ ambulanceZ amphibianz analog clockZapiaryZapronZashcanz assault rifleZbackpackZbakeryz balance beamZballoonZ ballpointzBand AidZbanjoZ bannisterZbarbellz barber chairZ barbershopZbarnZ barometerbarrelZbarrowZbaseballZ basketballZbassinetZbassoonz bathing capz bath towelZbathtubz beach wagonZbeaconZbeakerZbearskinz beer bottlez beer glassz bell cotebibzbicycle-built-for-twoZbikiniZbinderZ binocularsZ birdhouseZ boathouseZbobsledzbolo tieZbonnetZbookcaseZbookshopZ bottlecapZbowzbow tieZbrassZ brassiereZ breakwaterZ breastplateZbroombucketZbucklezbulletproof vestz bullet trainz butcher shopcabZcaldronZcandlecannonZcanoez can openerZcardiganz car mirrorZcarouselzcarpenter's kitZcartonz car wheelz cash machineZcassettezcassette playerZcastleZ catamaranz CD playerZcellozcellular telephonechainzchainlink fencez chain mailz chain sawZchestZ chiffonierZchimez china cabinetzChristmas stockingZchurchZcinemaZcleaverzcliff dwellingZcloakZclogzcocktail shakerz coffee mugZ coffeepotZcoilzcombination lockzcomputer keyboardZ confectioneryzcontainer shipZ convertibleZ corkscrewZcornetz cowboy bootz cowboy hatZcradler z crash helmetcrateZcribz Crock Potz croquet ballZcrutchZcuirassZdamZdeskzdesktop computerzdial telephoneZdiaperz digital clockz digital watchz dining tableZdishragZ dishwasherz disk brakeZdockZdogsledZdomeZdoormatzdrilling platformZdrumZ drumstickZdumbbellz Dutch ovenz electric fanzelectric guitarzelectric locomotivezentertainment centerZenvelopezespresso makerz face powderz feather boafileZfireboatz fire enginez fire screenZflagpoleZflutez folding chairzfootball helmetZforkliftZfountainz fountain penz four-posterz freight carz French hornz frying panzfur coatz garbage truckZgasmaskzgas pumpZgobletzgo-kartz golf ballZgolfcartZgondolaZgongZgownz grand pianoZ greenhouseZgrillez grocery storeZ guillotinez hair slidez hair sprayz half trackZhammerZhamperz hand blowerzhand-held computerZ handkerchiefz hard discZ harmonicaZharpZ harvesterZhatchetZholsterz home theaterZ honeycombhookZ hoopskirtzhorizontal barz horse cartZ hourglassZiPodZironzjack-o'-lanternZjeanZjeepZjerseyz jigsaw puzzleZ jinrikishaZjoystickZkimonozknee padknotzlab coatZladleZ lampshadelaptopz lawn mowerzlens capz letter openerlibraryZlifeboatZlighterZ limousineZlinerZlipstickZLoaferZlotionZ loudspeakerZloupeZ lumbermillzmagnetic compassZmailbagZmailboxmaillotr z manhole coverZmaracaZmarimbamaskZ matchstickZmaypoleZmazez measuring cupzmedicine chestZmegalithZ microphoneZ microwavezmilitary uniformzmilk canZminibusZ miniskirtZminivanZmissileZmittenz mixing bowlz mobile homezModel TZmodemZ monasterymonitorZmopedZmortarZ mortarboardZmosquez mosquito netz motor scooterz mountain bikez mountain tentZmouseZ mousetrapz moving vanZmuzzleZnailz neck braceZnecklaceZnipplenotebookZobeliskZoboeZocarinaZodometerz oil filterZorganZ oscilloscopeZ overskirtZoxcartz oxygen maskpacketpaddleZ paddlewheelZpadlockZ paintbrushZpajamaZpalaceZpanpipez paper towelZ parachutez parallel barsz park benchz parking meterz passenger carZpatioz pay-phoneZpedestalz pencil boxzpencil sharpenerZperfumez Petri dishZ photocopierpickZ pickelhaubez picket fenceZpickupZpierz piggy bankz pill bottleZpillowzping-pong ballZpinwheelZpirateZpitcherplaneZ planetariumz plastic bagz plate rackplowZplungerzPolaroid camerapolez police vanZponchoz pool tablez pop bottlepotzpotter's wheelz power drillz prayer rugprinterZprisonZ projectileZ projectorZpuckz punching bagZpurseZquillZquiltZracerZracketZradiatorZradiozradio telescopez rain barrelzrecreational vehicleZreelz reflex cameraZ refrigeratorzremote control restaurantrevolverZriflez rocking chairZ rotisseriez rubber eraserz rugby ballrulez running shoesafez safety pinZ saltshakerZsandalZsarongsaxZscabbardscalez school busschoonerZ scoreboardscreenZscrewZ screwdriverz seat beltzsewing machineZshieldz shoe shopZshojizshopping basketz shopping cartZshovelz shower capzshower curtainskizski maskz sleeping bagz slide rulez sliding doorslotZsnorkelZ snowmobileZsnowplowzsoap dispenserz soccer ballsockz solar dishZsombreroz soup bowlz space barz space heaterz space shuttleZspatulaZ speedboatz spider webZspindlez sports carZ spotlightstagezsteam locomotivezsteel arch bridgez steel drumZ stethoscopeZstolez stone wallZ stopwatchZstoveZstrainerZ streetcarZ stretcherz studio couchZstupaZ submarineZsuitZsundialZsunglassZ sunglassesZ sunscreenzsuspension bridgeZswabZ sweatshirtzswimming trunksZswingswitchZsyringez table lampZtankz tape playerteapotZteddyZ televisionz tennis ballZthatchztheater curtainZthimbleZthresherZthronez tile roofZtoasterz tobacco shopz toilet seattorchz totem polez tow truckZtoyshopZtractorz trailer truckZtrayz trench coatZtricycleZtrimaranZtripodztriumphal archZ trolleybusZtromboneZtubZ turnstileztypewriter keyboardumbrellaZunicycleZuprightZvacuumZvaseZvaultZvelvetzvending machineZvestmentZviaductZviolinZ volleyballz waffle ironz wall clockZwalletZwardrobeZwarplaneZ washbasinZwasherz water bottlez water jugz water towerz whiskey jugZwhistleZwigz window screenz window shadez Windsor tiez wine bottleZwingwokz wooden spoonZwoolz worm fenceZwreckZyawlZyurtzweb sitez comic bookzcrossword puzzlez street signz traffic lightz book jacketmenuZplateZ guacamoleZconsommezhot potZtriflez ice creamz ice lollyz French loafZbagelZpretzelZ cheeseburgerZhotdogz mashed potatoz head cabbageZbroccoliZ cauliflowerZzucchinizspaghetti squashz acorn squashzbutternut squashZcucumber artichokez bell pepperZcardoonZmushroomz Granny Smith strawberryZorangeZlemonfigZ pineappleZbananaZ jackfruitz custard appleZ pomegranateZhayZ carbonarazchocolate sauceZdoughz meat loafpizzaZpotpieZburritozred wineZespressocupZeggnogZalpZbubbleZcliffz coral reefZgeyserZlakesideZ promontoryZsandbarZseashoreZvalleyZvolcanoZ ballplayerZgroomz scuba diverZrapeseedZdaisyzyellow lady's slipperZcornZacornZhipZbuckeyez coral fungusZagaricZ gyromitraZ stinkhornZ earthstarzhen-of-the-woodsZboleteZearz toilet tissue)Oz B-addresszB-bookz B-companyzB-gamez B-governmentzB-moviezB-namezB-organizationz B-positionzB-scenez I-addresszI-bookz I-companyzI-gamez I-governmentzI-moviezI-namezI-organizationz I-positionzI-scenez S-addresszS-bookz S-companyzS-gamez S-governmentzS-moviezS-namezS-organizationz S-positionzS-scene)Z contradiction entailmentZneutralZ unacceptableZ acceptablenegativepositive)ZworldZsportsbusinesszsci/tech)imagenetZcluenerZmnliZcolaZsst2ZimdbZag_newsN) __doc__Zimagenet_labelscluener_labelsZ mnli_labelsZ cola_labelsZ sst2_labelsZ imdb_labelsZag_news_labelslabelsrLrLE/home/ma-user/work/mindformers-baichuan/mindformers/dataset/labels.pys Npengcheng.mind-test/mindformers/dataset/__pycache__/mask_language_model_dataset.cpython-39.pyc000066400000000000000000000075621471400134000332210ustar00rootroot00000000000000a ?cf@sdZddlZddlZddlmmZddlmm m Z ddl m Z mZddlmZddlmZddlmZddlmZe ejGd d d eZdS) zMasked Image Modeling Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader) BaseDatasetc@s eZdZdZdedddZdS)MaskLanguageModelDataseta Bert pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MaskLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['fill_mask']['bert_tiny_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/bert.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MaskLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MaskLanguageModelDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cstdttdd}ttdd}|||||\}}t|}|j j dksn|j j dksnt dg}|j j r|j d }tj|rt|D]<\}}}|D],} | d s| d r|tj|| qq|n|d s|d r|}n>|j jr*|j j}t|ttfrdata_loader must contain dataset_dir or dataset_files,but get .) dataset_files num_shardsshard_id) default_args)drop_remainderoutput_columnsnum_parallel_workers)columns) input_columns)'rinfointosgetenvinit_dataset_config_check_device_rank_for_parallelcopydeepcopy data_loadertypeNotImplementedErrorrpoppathisdirwalkendswithappendjoinsortr isinstancelisttuple ValueErrorrbatch batch_sizerrrprojectrepeatCTypeCastmstypeint32r) clsr rank_id device_numrdata_dirr_ffiledataset type_cast_op input_argrFZ/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mask_language_model_dataset.py__new__7sX              z MaskLanguageModelDataset.__new__)N)__name__ __module__ __qualname____doc__dictrHrFrFrFrGr sr )rLrr"Zmindspore.common.dtypecommondtyper9Z)mindspore.dataset.transforms.c_transformsrC transforms c_transformsr7mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderr base_datasetrregisterDATASETr rFrFrFrGs     pengcheng.mind-test/mindformers/dataset/__pycache__/mim_dataset.cpython-39.pyc000066400000000000000000000062401471400134000300150ustar00rootroot00000000000000a ?cf@sdZddlZddlmZmZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZeejGd d d eZdS)zMasked Image Modeling Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader) build_mask)build_transforms) build_sampler) BaseDatasetc@s eZdZdZdedddZdS) MIMDatasetap Masked Image Modeling Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MIMDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['masked_image_modeling']['mae_vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/mae.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MIMDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MIMDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cstd||ttdd}ttdd}t|j||dd}t|j }t |j }t |j }|durx||}|dur|jD]}t||||j|jd}q|durt|||j|j|j|jd }|j|jd }|j|j|j|jd }||j}|S) Nz)Now Create Masked Image Modeling Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_args) input_columnsnum_parallel_workerspython_multiprocessing)routput_columnsrr)columns)drop_remainderr)rinfoinit_dataset_configintosgetenvr data_loaderr transformsr mask_policyr sampler use_samplerrrrrrprojectbatch batch_sizerrepeat) clsr rank_id device_numdatasetr!maskr#columnr/J/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mim_dataset.py__new__8sB         zMIMDataset.__new__)N)__name__ __module__ __qualname____doc__dictr1r/r/r/r0r sr )r5rmindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderrr-rr!r r#r base_datasetr registerDATASETr r/r/r/r0s        pengcheng.mind-test/mindformers/dataset/__pycache__/question_answering_dataset.cpython-39.pyc000066400000000000000000000065641471400134000331700ustar00rootroot00000000000000a ?cf@sdZddlZddlmmZddlmmm Z ddl m Z m Z ddlmZddlmZddlmZdd lmZdd lmZdd lmZe e jGd d d eZdS)zQuestion Answering Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_tokenizer) build_sampler)build_dataset_loader) BaseDatasetc@s eZdZdZdedddZdS)QuestionAnsweringDataseta Question Answering Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import QuestionAnsweringDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['question_answering']['qa_bert_base_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/question_answering.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='QuestionAnsweringDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = QuestionAnsweringDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cstd||ttdd}ttdd}t|j}t|j j |j j ||j j |j j |j||d}t|j}|dur||}|j|j|j|j|jd}||j}ttj}|jD]}t|||d }q|S) z new methodz&Now Create Question Answering Dataset.RANK_ID0 RANK_SIZE1) class_name dataset_dir tokenizerstage column_namesnum_parallel_workers num_shardsshard_idN)drop_remainderoutput_columnsr) input_columns)rinfoinit_dataset_configintosgetenvrrr data_loadertyperrrrr sampler use_samplerbatch batch_sizerrrepeatCTypeCastmstypeint32r) clsr rank_id device_numrdatasetr$ type_cast_op input_argr3Y/home/ma-user/work/mindformers-baichuan/mindformers/dataset/question_answering_dataset.py__new__9s6        z QuestionAnsweringDataset.__new__)N)__name__ __module__ __qualname____doc__dictr5r3r3r3r4r sr )r9r Zmindspore.common.dtypecommondtyper+Z)mindspore.dataset.transforms.c_transformsr0 transforms c_transformsr)mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlrZmodels.build_tokenizerrr$r dataloaderr base_datasetr registerDATASETr r3r3r3r4s       pengcheng.mind-test/mindformers/dataset/__pycache__/reward_model_dataset.cpython-39.pyc000066400000000000000000000151511471400134000317000ustar00rootroot00000000000000a ?cfr(@sdZddlZddlZddlZddlZddlmmZ ddl m m m ZddlmZmZddlmZddlmZddlmZddlmZdd lmZdeeed d d ZeejGdddeZ dS)zCausal Image Modeling Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_tokenizer)build_dataset_loader) BaseDataset)rankdispad_idc Cst|}||||d|}||||d|}||||d|}||||d|}||||d|}||||d|}t||f} t||f} t||f}tj| | kdd}| || ||fS)z^ Generate position_id and attention_mask according to input_ids considering eos reset r)axis)intnp concatenatesum) chosen_input_idschosen_attention_maskrejected_input_idsrejected_attention_mask position_id loss_maskend_indr r r input_idsattention_maskrS/home/ma-user/work/mindformers-baichuan/mindformers/dataset/reward_model_dataset.pyget_input_data_batch_slice_mapsrc@sDeZdZdZd edddZeddZedd Zed d Z dS) RewardModelDatasetazCausal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import RewardModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='RewardModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = RewardModelDataset(config.train_dataset_task.dataset_config) N)dataset_configcstdtd|ttddttdd}t|}||td| |\}|_ ||_ |j j dkr|j j d kr||}n ||}td |s|rd |jn0|j||j|d krtd |jd |dtd|j|j|j|jd}|jdur$|jndtdfdd}t|||j|jd}|j|jd}ttj}ttj}t|d|d}t|d|d}t|d|d}t|d|d}t|d|d}| |j }|S)Nz Now Create Reward Model Dataset.r RANK_ID0 RANK_SIZE1z!Now Create Reward Model Dataset1. MindDatasetTFRecordDatasetz"Now Create Reward Model Dataset1.5rz batch size z' should be a multiple of device number z-. You should change the args: per_batch_size.z"Now Create Reward Model Dataset1.6)drop_remainderoutput_columnsr z!Now Create Reward Model Dataset2.c st|||||||d S)N)r r )r)rrrrrrrr r rank_idrros  z,RewardModelDataset.__new__.. input_columnsr()columnsr)r- operationsrrrr)!rinfoprintrosgetenvcopydeepcopyinit_dataset_config_check_device_rank_for_parallelr* device_num data_loadertype_process_raw_text_data_process_mindrecord_data_is_semi_full_batch_is_data_parallel batch_size ValueErrorbatchr'r- pad_token_idrr(projectCTypeCastmstypeint32float16repeat)clsr r8datasetmap_func type_cast_opZtype_cast_op_floatrr)r__new__Ms\                zRewardModelDataset.__new__cs:|j}t||jfdd}t|||j|jd}|S)zPreprocess data for gpt2 modelcs$|}|dddd}|dS)N max_lengthTF)paddingrO truncationadd_special_tokensr)tolistget) input_datarrO tokenizerrrrLs  z7RewardModelDataset._prepare_for_model..map_funcr,)rWrrOrr-)rJrKr tokenizer_configrLrrVr_prepare_for_modelsz%RewardModelDataset._prepare_for_modelcCs6|jd}t|j||j|jdd}|||}|S)zProcess the text data dataset_dir)rZ num_shardsshard_id default_args)r9poprr8r*rY)rJr rZrKrrrr;s  z)RewardModelDataset._process_raw_text_datac Csg}td}|jjr|jd}tj|rt|D]>\}}}|D].}t ||s`| drF| tj ||qFq8| qt ||s| dr|}n:|jjr|jj}t|ttfrt|}ntd|jdt|j||j|j|jdd} | S)zProcess the mindrecord dataz mindrecord0*$rZz .tfrecordz>data_loader must contain dataset_dir or dataset_files,but get .) dataset_filesr[r\ columns_listr])recompiler9rZr_r2pathisdirwalkfindallendswithappendjoinsortra isinstancelisttupler@rr8r*r-) rJr ra mind_compiledata_dirr_ffilerKrrrr<s6      z+RewardModelDataset._process_mindrecord_data)N) __name__ __module__ __qualname____doc__dictrN classmethodrYr;r<rrrrr3s9  r)rrr )!ryr2r4rcnumpyrZmindspore.common.dtypecommondtyperFZ)mindspore.dataset.transforms.c_transformsrK transforms c_transformsrDmindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlrZ"mindformers.models.build_tokenizerr dataloaderr base_datasetr rrregisterDATASETrrrrrs$       pengcheng.mind-test/mindformers/dataset/__pycache__/text_classification_dataset.cpython-39.pyc000066400000000000000000000070221471400134000332710ustar00rootroot00000000000000a ?cf*@sdZddlZddlZddlmmZddlmm m Z ddl m Z mZddlmZddlmZddlmZddlmZe ejGd d d eZdS) zText Classification Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader) BaseDatasetc@s eZdZdZdedddZdS)TextClassificationDataseta Text Classification dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TextClassificationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_classification']['txtcls_bert_base_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/text_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TextClassificationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TextClassificationDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cs`td||ttdd}ttdd}t|}d|jvr|jj rg}|jj }tj |rt |D]<\}}}|D],} | ds| dr|tj || qqrq| ds| dr||n t|jj}|jd t|j|||d d } | j|j|j|j|jd } | j|jd } | |j} ttj} |jD]} t| | | d} qF| S)Nz'Now Create Text Classification Dataset.RANK_ID0 RANK_SIZE1Z data_filesz .mindrecordz .tfrecord dataset_dir) dataset_files num_shardsshard_id) default_args)drop_remainderoutput_columnsnum_parallel_workers)columns) operations input_columns) rinfoinit_dataset_configintosgetenvcopydeepcopy data_loaderrpathisdirwalkendswithappendjoinlistrpoprbatch batch_sizerrrprojectrepeatCTypeCastmstypeint32r) clsr rank_id device_numrdata_dirr_ffiledataset type_cast_op input_argr=Z/home/ma-user/work/mindformers-baichuan/mindformers/dataset/text_classification_dataset.py__new__6sF           z!TextClassificationDataset.__new__)N)__name__ __module__ __qualname____doc__dictr?r=r=r=r>r sr )rCrrZmindspore.common.dtypecommondtyper0Z)mindspore.dataset.transforms.c_transformsr: transforms c_transformsr.mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderr base_datasetrregisterDATASETr r=r=r=r>s     pengcheng.mind-test/mindformers/dataset/__pycache__/token_classification_dataset.cpython-39.pyc000066400000000000000000000065411471400134000334320ustar00rootroot00000000000000a ?cf@sdZddlZddlmZmZddlmZddlmZddl m Z dd l m Z dd l mZdd lmZdd lmZeejGd ddeZdS)zToken classification Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader)build_tokenizer)build_transforms) build_sampler) BaseDatasetc@s eZdZdZdedddZdS)TokenClassificationDataseta Token classification Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TokenClassificationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['token_classification']['tokcls_bert_base_chinese'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/token_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TokenClassificationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TokenClassificationDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cstd||ttdd}ttdd}t|j||dd}t|j }t |j d|id}t |j }t |j}|d ur||}|d urt||j||j|j|jd }|d urt||jd ||j|jd }|j|j|j|jd }||j}|S)z new methodz(Now Create Token classification Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_args tokenizerN) input_columns operationsoutput_columnsnum_parallel_workerspython_multiprocessingr)rrrr)drop_remainderr)rinfoinit_dataset_configintosgetenvr data_loaderr rr text_transformslabel_transformsr sampler use_samplerrrrrrbatch batch_sizerrepeat) clsrrank_id device_numdatasetrr#r$r%r.[/home/ma-user/work/mindformers-baichuan/mindformers/dataset/token_classification_dataset.py__new__8sH        z"TokenClassificationDataset.__new__)N)__name__ __module__ __qualname____doc__dictr0r.r.r.r/r sr )r4r mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderrZmodels.build_tokenizerr transformsr r%r base_datasetr registerDATASETr r.r.r.r/s        pengcheng.mind-test/mindformers/dataset/__pycache__/translation_dataset.cpython-39.pyc000066400000000000000000000125031471400134000315700ustar00rootroot00000000000000a ?cf@sdZddlZddlZddlZddlmmZddl m m m Z ddlmZmZddlmZddlmZddlmZddlmZd d lmZd gZeejGd d d eZdS) z T5 Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader) BaseDataset) AutoTokenizerTranslationDatasetc@sDeZdZdZd edddZeddZedd Zed d Z dS) r afTranslation dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TranslationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['translation']['t5_small'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/t5.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TranslationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TranslationDataset(config.train_dataset_task.dataset_config) N)dataset_configcCstd|||jjdkr,||}n ||}|j|j|j |j d}| |j }t tj}|jD]}t|||d}qj|S)NzNow Create Translation Dataset. MindDataset)drop_remaindernum_parallel_workers) operations input_columns)rinfoinit_dataset_config data_loadertype_process_raw_text_data_process_mindrecord_databatch batch_sizerrrepeatCTypeCastmstypeint32rr)clsr dataset type_cast_op input_argr#R/home/ma-user/work/mindformers-baichuan/mindformers/dataset/translation_dataset.py__new__:s        zTranslationDataset.__new__csht|j|j|j|jtd|fdd}t||ddggdd}|j gdd}|S) z/Maps the tokenizer on the source and the outputz1Start tokenize on the dataset using tokenizer: %scs|}t|tr|}|ddd}|}t|trH|}|ddd}t|dtj}t|dtj}t|dtj}|||fS)N max_lengthT)paddingr& truncation input_idsattention_mask)tolist isinstancebytesdecodenparrayrfloat32)srctgtoutputZ tgt_outputr)r*labelsprefixsrc_max_lengthtgt_max_length tokenizerr#r$pad_max_functionTs  z;TranslationDataset._tokenizer_map..pad_max_functionsourcetarget)r)r*r5)routput_columns)columns) r from_pretrainedrr7r8r9rrrproject)rr Ztokenizer_configr;r#r6r$_tokenizer_mapKs  z!TranslationDataset._tokenizer_mapcCsTttdd}ttdd}|jd}t|j|||dd}|||j}|S)zProcess the text dataRANK_ID0 RANK_SIZE1 dataset_dir)rG num_shardsshard_id default_args)intosgetenvrpoprrBr:)rr rank_id device_numrGr r#r#r$rjs z)TranslationDataset._process_raw_text_datac Cs ttdd}ttdd}t|}g}|jjr|jd}tj |rt |D]2\}}}|D]"} | drf| tj || qfqX|q| dr|}n:|jjr|jj}t|ttfrt|}ntd|jdtd |jt|j||||jd d } | S) zProcess the mindrecord datarCrDrErFrGz .mindrecordz>data_loader must contain dataset_dir or dataset_files,but get .z&Using args %s to instance the dataset.) dataset_filesrHrI columns_listrJ)rLrMrNcopydeepcopyrrGrOpathisdirwalkendswithappendjoinsortrSr,listtuple ValueErrorrrrr) rr rPrQrSdata_dirr_ffiler r#r#r$rws:        z+TranslationDataset._process_mindrecord_data)N) __name__ __module__ __qualname____doc__dictr% classmethodrBrrr#r#r#r$r s  )rirMrUnumpyr/Zmindspore.common.dtypecommondtyperZ)mindspore.dataset.transforms.c_transformsr transforms c_transformsrmindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderr base_datasetr auto_classr __all__registerDATASETr r#r#r#r$s      pengcheng.mind-test/mindformers/dataset/__pycache__/utils.cpython-39.pyc000066400000000000000000000014041471400134000266630ustar00rootroot00000000000000a ?cf@sdZddZdS)zDataset Utils.cCs|jdurpd|j_|j|j_|j|j_|j|j_|j|j_|j|j_|jj|j_|jj rf|jj |jj _ |j|j _ |j durd|j _|j|j _|j|j _|j|j _|j|j _|j|j _|jj|j _|j |j_ dS)zCheck dataset config.NFT) train_datasetdo_evalseed auto_tunefilepath_prefixautotune_per_stepprofile runner_config batch_sizemixup_op num_classesZtrain_dataset_taskdataset_config eval_datasetZeval_dataset_task)configrD/home/ma-user/work/mindformers-baichuan/mindformers/dataset/utils.pycheck_dataset_configs(               rN)__doc__rrrrrszero_shot_image_classification_dataset.cpython-39.pyc000066400000000000000000000073511471400134000354110ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/__pycache__a ?cf@sdZddlZddlmZddlmZddlmZddlm Z dd l m Z dd l m Z mZdd lmZe ejGd d d e ZdS)z'Zero Shot Image Classification Dataset.N)get_dataset_map)build_dataset_loader)build_transforms) BaseDataset)logger)MindFormerRegisterMindFormerModuleType)build_tokenizerc@s eZdZdZdedddZdS)"ZeroShotImageClassificationDataseta Zero Shot Image Classification Dataset API. output image, text, and label columns Args: dataset_config (dict): Config for dataset. Returns: A dataset for ZeroShotImageClassificationTrainer. Examples: >>> import os >>> from mindformers import MindFormerBook, MindFormerConfig, build_dataset >>> project_path = MindFormerBook.get_project_path() >>> config_path = os.path.join(project_path, "configs", "clip", >>> "run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml") >>> config = MindFormerConfig(config_path) Note: Put cifar100 dataset to ./ The detailed data setting could refer to ./configs/clip/clip.md >>> config.eval_dataset_task.dataset_config.batch_size = 1 >>> dataset = build_dataset(config.eval_dataset_task) >>> for item in dataset: >>> print(item) >>> break [Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], [1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], [1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], ... [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000], [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000], [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000]]]]), Tensor(shape=[1, 100, 77], dtype=Int32, value= [[[49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], ... [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0]]]), Tensor(shape=[1], dtype=Int32, value= [49])] N)dataset_configcCstd||ttdd}ttdd}t|j||dd}t|j }t |j }t|j d|id}|d urt |d ||j|jd }|d urt |d ||j|jd }|j|j|j|jd }||j}|S)z New methodz2Now Create Zero Shot Image Classification Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_args tokenizerNimage) input_columns operationsnum_parallel_workerspython_multiprocessingtext)drop_remainderr)rinfoinit_dataset_configintosgetenvr data_loaderr transformsr rtext_transformsrrrbatch batch_sizerrepeat)clsr rank_id device_numdatasetr#rr$r,e/home/ma-user/work/mindformers-baichuan/mindformers/dataset/zero_shot_image_classification_dataset.py__new__Fs@      z*ZeroShotImageClassificationDataset.__new__)N)__name__ __module__ __qualname____doc__dictr.r,r,r,r-r s+r )r2r mindformers.version_controlr dataloaderrr#r base_datasetrtoolsrZtools.registerr r Zmodels.build_tokenizerr registerDATASETr r,r,r,r-s       pengcheng.mind-test/mindformers/dataset/base_dataset.py000066400000000000000000000050161471400134000236560ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Base Dataset.""" import os import mindspore as ms import mindspore.dataset as ds class BaseDataset: """Base Dataset.""" def __init__(self, dataset_config: dict = None): self.dataset_config = dataset_config @classmethod def init_dataset_config(cls, dataset_config): """Init dataset config.""" if dataset_config is None: raise ValueError("dataset_config cannot be None!") ds.config.set_seed(dataset_config.seed) ds.config.set_prefetch_size(dataset_config.prefetch_size) ds.config.set_numa_enable(dataset_config.numa_enable) if dataset_config.auto_tune: if dataset_config.profile: raise EnvironmentError( "MindSpore's AutoTune is enabled, so Profile cannot be enabled," "now Profile's flag is True, please set to False!") os.makedirs(dataset_config.filepath_prefix, exist_ok=True) dataset_config.filepath_prefix = os.path.join(dataset_config.filepath_prefix, "autotune") ds.config.set_enable_autotune(True, filepath_prefix=dataset_config.filepath_prefix) ds.config.set_autotune_interval(dataset_config.autotune_per_step) @classmethod def _check_device_rank_for_parallel(cls, rank_id, device_num): """Check device num and rank id in auto parallel mode.""" if cls._is_semi_full_batch(): rank_id = None device_num = None return rank_id, device_num @classmethod def _is_semi_full_batch(cls): return ((ms.context.get_auto_parallel_context("parallel_mode") in ['semi_auto_parallel', 'auto_parallel']) and ms.context.get_auto_parallel_context("full_batch")) @classmethod def _is_data_parallel(cls): return ms.context.get_auto_parallel_context("parallel_mode") == ms.context.ParallelMode.DATA_PARALLEL pengcheng.mind-test/mindformers/dataset/build_dataset.py000066400000000000000000000046561471400134000240540ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Dataset API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_dataset( config: dict = None, default_args: dict = None, module_type: str = 'dataset', class_name: str = None, **kwargs): r"""Build dataset For MindFormer. Instantiate the dataset from MindFormerRegister's registry. Args: config (dict): The task dataset's config. Default: None. default_args (dict): The default argument of dataset API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset'. class_name (str): The class name of dataset API. Default: None. Return: The function instance of dataset API. Examples: >>> from mindformers import build_dataset >>> from mindformers.dataset import check_dataset_config >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_class_name = build_dataset(class_name='ImageCLSDataset', dataset_config=config.train_dataset_task) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.DATASET, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-test/mindformers/dataset/causal_language_model_dataset.py000066400000000000000000000252731471400134000272460ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Causal Image Modeling Dataset.""" import os import copy import re import numpy as np import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.models.build_tokenizer import build_tokenizer from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset def get_input_data_batch_slice_map(input_ids, eos_token_id, dis, rank_id: int = 0): """ Generate position_id and attention_mask according to input_ids considering eos reset Args: input_ids: the input token ids eos_token_id: the id for dis: the slice value for each rank rank_id: the current rank id Returns: batch_input_ids: the input token ids batch_position_ids: the position ids cosidering eos reset batch_attention_mask: the attention mask considering eos reset """ rank = int(rank_id) input_ids = input_ids[rank*dis: (rank + 1)*dis] seq_length = input_ids.shape[1] - 1 # Initialize position_ids and attention_mask batch_input_ids = input_ids batch_position_ids = np.ones((dis, seq_length)) batch_attention_mask = np.ones((dis, seq_length, seq_length)) # Loop through batches for bs_i in range(len(input_ids)): # Get normal position_ids and attention_mask local_ids = input_ids[bs_i] batch_attention_mask[bs_i] = np.tril(np.ones(shape=(seq_length, seq_length))) batch_position_ids[bs_i] = np.arange(seq_length) # Find the index of eos_index = batch_position_ids[bs_i, local_ids[:-1] == eos_token_id].astype(np.int32) prev_index = 0 for i in range(eos_index.size): # Reset position_ids and attention_mask considering index = eos_index[i] batch_attention_mask[bs_i, (index + 1):, :(index + 1)] = 0 batch_position_ids[bs_i, (index + 1):] -= (index + 1 - prev_index) prev_index = index + 1 return batch_input_ids, batch_position_ids, batch_attention_mask @MindFormerRegister.register(MindFormerModuleType.DATASET) class CausalLanguageModelDataset(BaseDataset): """Causal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import CausalLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='CausalLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = CausalLanguageModelDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Causal Language Model Dataset.") rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) cls.init_dataset_config(dataset_config) rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config.rank_id = rank_id dataset_config.device_num = device_num if dataset_config.data_loader.type != "MindDataset" and \ dataset_config.data_loader.type != "TFRecordDataset": dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) type_cast_op = C.TypeCast(mstype.int32) if dataset_config.eos_reset: if cls._is_semi_full_batch() or cls._is_data_parallel(): rank_id = 0 dis = dataset_config.batch_size else: # Each card slice a small batch from the full batch dis = dataset_config.batch_size // device_num if dataset_config.batch_size % device_num != 0: raise ValueError( f"batch size {dataset_config.batch_size} should be a multiple of device number {device_num}." " You should change the args: per_batch_size.") dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns) map_func = lambda input_ids: get_input_data_batch_slice_map(input_ids, eos_token_id=dataset_config.eos_token_id, rank_id=rank_id, dis=dis) dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns) dataset = dataset.project(columns=dataset_config.output_columns) for input_arg in dataset_config.output_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) else: dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.input_columns) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) dataset = dataset.repeat(dataset_config.repeat) return dataset @classmethod def _prepare_for_model(cls, dataset, dataset_config): """Preprocess data for gpt2 model""" tokenizer_config = dataset_config.tokenizer tokenizer = build_tokenizer(tokenizer_config) max_length = tokenizer_config.max_length def map_func(input_data): input_data = input_data.tolist() input_ids = tokenizer(input_data, padding='max_length', max_length=max_length, truncation=True, add_special_tokens=False) return input_ids.get('input_ids') dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.input_columns) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id}) dataset = cls._prepare_for_model(dataset, dataset_config) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" dataset_files = [] mind_compile = re.compile("mindrecord") if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if (re.findall(mind_compile, file) and not file.endswith(".db")) or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if re.findall(mind_compile, data_dir) or data_dir.endswith(".tfrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") print(f"build dataset with : {dataset_config.data_loader}") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id, 'columns_list': dataset_config.input_columns}) if dataset_config.consumed_samples: dataset = dataset.skip(dataset_config.consumed_samples) print(f"skip {dataset_config.consumed_samples} samples") return dataset pengcheng.mind-test/mindformers/dataset/contrastive_language_image_pretrain_dataset.py000066400000000000000000000112171471400134000322160ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Contrastive Language Image Pretrain Dataset.""" import os from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .transforms import build_transforms from .sampler import build_sampler from .base_dataset import BaseDataset from ..tools import logger from ..models.build_tokenizer import build_tokenizer from ..tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET) class ContrastiveLanguageImagePretrainDataset(BaseDataset): r""" Contrastive Language Image Pretrain Dataset API. output image and text columns Args: dataset_config (dict): Config for dataset. Returns: A dataset for ContrastiveLanguageImagePretrainTrainer. Examples: >>> import os >>> from mindformers import MindFormerBook, MindFormerConfig, build_dataset >>> project_path = MindFormerBook.get_project_path() >>> config_path = os.path.join(project_path, "configs", "clip", >>> "run_clip_vit_b_32_pretrain_flickr8k.yaml") >>> config = MindFormerConfig(config_path) Note: Put flickr8k dataset to ./checkpoint_download The detailed data setting could refer to ./configs/clip/clip.md >>> config.train_dataset_task.dataset_config.batch_size = 1 >>> dataset = build_dataset(config.train_dataset_task) >>> for item in dataset: >>> print(item) >>> break [Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[4.99690473e-001, 6.74871564e-001, ... 3.68304640e-001, 2.36918822e-001], [7.91658998e-001, 7.62462139e-001, ... -2.01033935e-001, -1.13443382e-001], ... [-5.98575652e-001, -6.12795711e-001, ... 1.47755420e+000, 1.46333420e+000], [-3.85274649e-001, -6.27015769e-001, ... 1.42067397e+000, 1.43489408e+000], [-7.97656536e-001, -1.01095748e+000, ... 9.37191546e-001, 9.08751369e-001]]]]), Tensor(shape=[1, 77], dtype=Int32, value= [[49406, 1237, 18250 ... 0, 0, 0]])] """ def __new__(cls, dataset_config: dict = None): """new method""" logger.info("Now Create Contrastive Language Image Pretrain Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) tokenizer = build_tokenizer(dataset_config.tokenizer) text_transforms = build_transforms(dataset_config.text_transforms, default_args={"tokenizer": tokenizer}) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) if transforms is not None: dataset = get_dataset_map(dataset, transforms, input_columns="image", num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if text_transforms is not None: dataset = get_dataset_map(dataset, text_transforms, input_columns="text", num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-test/mindformers/dataset/dataloader/000077500000000000000000000000001471400134000227635ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/dataloader/__init__.py000066400000000000000000000024031471400134000250730ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers DataLoader.""" from .build_dataloader import build_dataset_loader from .flickr8k_dataloader import Flickr8kDataLoader from .multi_image_cap_dataloader import MultiImgCapDataLoader from .cifar100_dataloader import Cifar100DataLoader from .wmt16_dataloader import WMT16DataLoader from .cluener_dataloader import CLUENERDataLoader from .squad_dataloader import SQuADDataLoader from .adgen_dataloader import ADGenDataLoader __all__ = ['Flickr8kDataLoader', 'Cifar100DataLoader', 'WMT16DataLoader', 'CLUENERDataLoader', 'SQuADDataLoader', 'ADGenDataLoader', 'MultiImgCapDataLoader'] pengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/000077500000000000000000000000001471400134000251735ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/__init__.cpython-39.pyc000066400000000000000000000013621471400134000313650ustar00rootroot00000000000000a ?cf@spdZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZgd Zd S) zMindFormers DataLoader.)build_dataset_loader)Flickr8kDataLoader)MultiImgCapDataLoader)Cifar100DataLoader)WMT16DataLoader)CLUENERDataLoader)SQuADDataLoader)ADGenDataLoader)rrrrrr rN)__doc__Zbuild_dataloaderrZflickr8k_dataloaderrZmulti_image_cap_dataloaderrZcifar100_dataloaderrZwmt16_dataloaderrZcluener_dataloaderrZsquad_dataloaderrZadgen_dataloaderr __all__r r R/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/__init__.pys        pengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/adgen_dataloader.cpython-39.pyc000066400000000000000000000077601471400134000330740ustar00rootroot00000000000000a ?cf`@shdZddlZddlZddlmZddlmZddlmZm Z e ej GdddZ Gdd d Z dS) zADGen DataLoaderN)GeneratorDataset)logger)MindFormerModuleTypeMindFormerRegisterc@seZdZdZddZdS)ADGenDataLoaderzADGen Dataloaderc Kstj|st|d|dvr*tdddg}t|ttfs\tdt|dt ||D] }t|t s`tdt|q`t|ttfrt |d krtd t|dt || d d t |||}d |d|d} t | | dd t||fd|i|S)a ADGen Dataloader API. Args: dataset_dir: The directory to ADGen dataset. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 6 phase: The supported key words are in ["train", "dev"] Return: A GeneratorDataset for ADGen dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import ADGenDataLoader >>> from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer >>> data_loader = ADGenDataLoader("./ADGen/") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break  is not existed.)trainevalz!phase should be in train or eval.promptanswerzJcolumn_names should be a tuple or a list of string with length 2, but got z , length z8the item type of column_names should be string, but got zLorigin_columns should be a tuple or a list of string with length 2, but got NoneNz[DATASET] shuffle status is z , phase is .versionshuffle)ospathisfile ValueError isinstancetuplelist TypeErrortypelenstrpop ADGenDatasetrinfor) cls dataset_dirphaserorigin_columnskwargs column_namesnameZ adgen_datasetrr&Z/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/adgen_dataloader.py__new__s8       zADGenDataLoader.__new__N)__name__ __module__ __qualname____doc__r(r&r&r&r'rsrc@s*eZdZdZd ddZddZddZd S) rz ADGen Datasetrc Cstj|st|d||_||_i}g}g}|d|_|d|_t|jZ}|D]0}| t ||j| t ||jqV|||j<|||j<Wdn1s0Y||_ dS)aE ADGen Dataset Args: dataset_dir (str): The directory to ADGen dataset. phase (str): The supported key words are in ["train", "dev"] Return: A iterable dataset for ADGen dataset Raises: ValueError: Error input for dataset_dir, phase. rrN) rrrrr r! prompt_columnresponse_columnopenappendjsonloadsexamples) selfr r"r!r4Z content_list summary_listfpliner&r&r'__init__Zs      (zADGenDataset.__init__cCst|j|jS)zGet the size of dataset)rr4r.)r5r&r&r'__len__{szADGenDataset.__len__cCs*|j|j||j|j|}}||fS)zReturn input data for model)r4r.r/)r5ir r r&r&r' __getitem__s"zADGenDataset.__getitem__N)r)r)r*r+r,r9r:r<r&r&r&r'rWs !r)r,r2rmindspore.datasetrmindformers.tools.loggerrmindformers.tools.registerrrregisterDATASET_LOADERrrr&r&r&r's   =pengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/build_dataloader.cpython-39.pyc000066400000000000000000000043341471400134000331070ustar00rootroot00000000000000a ?cfr @sTdZddlZddlmZddlmZmZmZd e e e e dddZ d d Z e dS) zBuild DataLoader API.N)dataset)MindFormerRegisterMindFormerModuleTypeMindFormerConfigdataset_loader)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)a!Build dataset loader For MindFormer. Instantiate the dataset loader from MindFormerRegister's registry. Args: config (dict): The task dataset loader's config. Default: None. default_args (dict): The default argument of dataset loader API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset loader'. class_name (str): The class name of dataset loader API. Default: None. Return: The function instance of dataset loader API. Examples: >>> from mindformers.dataset.dataloader import build_dataset_loader >>> dataset_loader_config = {'type': 'ImageFolderDataset', 'dataset_dir': './imagenet/train'} >>> # 1) use config dict to build dataset loader >>> dataset_loader_from_config = build_dataset_loader(dataset_loader_config) >>> # 2) use class name to build dataset loader >>> dataset_loader_class_name = build_dataset_loader( ... class_name='ImageFolderDataset', dataset_dir='./imagenet/train') N)r) isinstancedictrrget_instance_from_cfgrDATASET_LOADER get_instance)rrr r kwargsrZ/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/build_dataloader.pybuild_dataset_loadersrcCs@ttD]2}|drqtt|}t|rt|tj qdS)z2 register MindSpore builtin dataset loader class. __N) dirds startswithgetattrinspectisclassr register_clsrr) module_namerrrrregister_ms_dataset_loader9s    r)NNrN) __doc__r mindsporerrmindformers.tools.registerrrrr strrrrrrrs  " pengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/cifar100_dataloader.cpython-39.pyc000066400000000000000000000141141471400134000333120ustar00rootroot00000000000000a ?cf@s|dZddlZddlZddlmZmZmZmZddlZ ddl m Z ddl m Z mZe ejGdddZGd d d ZdS) zCifar100 DataLoader.N)OptionalUnionListTuple)GeneratorDataset)MindFormerRegisterMindFormerModuleTypec@sVeZdZdZgdZd eeeeee efeeee ee eedd d Z dS) Cifar100DataLoaderz2Cifar100 Dataloader with class name as text columnimagetextlabelNtrainTFThis is a photo of {}.) dataset_dir column_namesstage fine_labelshufflehypothesis_templatec Kstj|st|d|dur(|j}t|ttfsHtdt |t |dkrftdt |t t |D](}t||t srtdt ||qr| ddt||||} t| ||d} t| d | j| S) a Cifar100 Dataloader API. Args: dataset_dir (str): The dataset directory, such as "/home/desktop/cifar-100-python" column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 3. stage (Optional[str]): The supported key words are in ["train","test", "all"] fine_label (Optional[bool]): True for 100 classes, False for 10 classes. shuffle (Optional[bool]): Shuffle the samples. hypothesis_template (Optional[str]): Prompt template for class label. Return: A GeneratorDataset for Cifar100 dataset Raises: ValueError: Error input for dataset_dir and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import Cifar100DataLoader >>> dataloader = Cifar100DataLoader("./cifar-100-python") >>> dataloader = dataloader.batch(1) >>> for item in dataloader: >>> print(item) >>> break [Tensor(shape=[1, 32, 32, 3], dtype=UInt8, value= [[[[255, 255, 255], [255, 255, 255], [255, 255, 255], ... [195, 205, 193], [212, 224, 204], [182, 194, 167]]]]]), Tensor(shape=[1, 100], dtype=String, value= [['This is a photo of apple.', 'This is a photo of aquarium_fish.', 'This is a photo of baby.', 'This is a photo of bear.', ... 'This is a photo of woman.', 'This is a photo of worm.']]), Tensor(shape=[1], dtype=Int32, value= [19])]  is not existed.NzJcolumn_names should be a tuple or a list of string with length 3, but got rz0the length of column_names should be 3, but got z8the item type of column_names should be string, but got None)r label_names)ospathisdir ValueError_default_column_names isinstancetuplelist TypeErrortypelenrangestrpopCifar100DataSetrsetattrr) clsrrrrrrkwargsindexZcifar100_datasetcifar100_dataloaderr.]/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/cifar100_dataloader.py__new__s,,    zCifar100DataLoader.__new__)NrTFr) __name__ __module__ __qualname____doc__rr&rrrrboolr0r.r.r.r/r s r c@sBeZdZdZd eeeeeeedddZdd Zd d Z d S)r(z Cifar100 dataSet with class namerTr)rrrrcCsftj|st|d|dvr*tdtj|d}t|d}tj|dd}Wdn1sf0Y|r~|d }n|d }tt |D]} ||  || <q||_ g} tt |D]} | | || q|d vrtj|d } t| d}tj|dd} Wdn1s 0Y|r:| d } n| d} | d}|jd}||dddf}|dddd}||_| |_| |_|dvr8tj|d}t|d}tj|dd}Wdn1s0Y|r|d }n|d}|d}|jd}||dddf}|dddd}||_||_| |_|dkrbt||g|_| ||_| |_dS)a" Cifar100 Dataset Args: dataset_dir (str): The dataset directory, such as "/home/desktop/cifar-100-python". stage (Optional[str]): The supported key words are in ["train", "test", "all"]. fine_label (Optional[bool]): True for 100 classes, False for 10 classes. hypothesis_template (Optional[str]): Prompt template for class label. Return: A iterable dataset for Cifar100 dataset. Raises: ValueError: Unsupported values for stage. r)rtestallz>unsupported stage, stage should be in ["train", "test", "all"]metarbbytes)encodingNsfine_label_namesscoarse_label_names)rr7rs fine_labelss coarse_labelssdatarr )r6r7r6r7)rrrrjoinopenpickleloadr%r$decoderappendformatshapereshape transposer rr np row_stack)selfrrrrZ meta_filefoZ meta_dictrr, hypothesis train_fileZ train_dictZ train_labelZ train_imageZ train_num test_fileZ test_dictZ test_labelZ test_imageZtest_numr.r.r/__init__es`  ,   .    .    zCifar100DataSet.__init__cCs|j||j|j|fS)getitemr )rKitemr.r.r/ __getitem__szCifar100DataSet.__getitem__cCs |jjdS)r$r)r rF)rKr.r.r/__len__szCifar100DataSet.__len__N)rTr) r1r2r3r4r&rr5rPrSrTr.r.r.r/r(cs  Pr()r4rrAtypingrrrrnumpyrImindspore.datasetrZtools.registerrr registerDATASET_LOADERr r(r.r.r.r/s  Hpengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/cluener_dataloader.cpython-39.pyc000066400000000000000000000140611471400134000334430ustar00rootroot00000000000000a ?cf1@sdZddlZddlZddlmZmZmZmZddlm Z ddl m Z m Z ddl mZe e jGd d d ZGd d d ZdS) zCLUENER DataLoader.N)OptionalUnionListTuple)GeneratorDataset)MindFormerRegisterMindFormerModuleType)cluener_labelsc@sDeZdZdZddgZd eeeeee efeedddZ dS) CLUENERDataLoaderzCLUENER Dataloadertextlabel_idNtrain) dataset_dir column_namesstagecKstj|st|d|dur(|j}t|ttfsHtdt |t |dkrftdt |t|dt rt|dt stdt |dd t |d| d dt ||}t||S) u CLUENER Dataloader API. Args: dataset_dir (str): The directory to cluener dataset. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "test", "del", "all"] Return: A GeneratorDataset for cluener dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import CLUENERDataLoader >>> data_loader = CLUENERDataLoader("./cluener/") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[], dtype=String, value= '虚幻引擎3动作游戏《黑光》新作公布'), Tensor(shape=[49], dtype=Int64, value= [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 15, 15, 0, 0, 0, 0, 0, 5, 15, 15, 15, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])]  is not existed.NzJcolumn_names should be a tuple or a list of string with length 2, but got r z0the length of column_names should be 2, but got rz8the item type of column_names should be string, but got z and None)ospathisdir ValueError_default_column_names isinstancetuplelist TypeErrortypelenstrpopCLUENERDataSetr)clsrrrkwargsZcluener_datasetr&\/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/cluener_dataloader.py__new__s*      zCLUENERDataLoader.__new__)Nr) __name__ __module__ __qualname____doc__rr!rrrrr(r&r&r&r'r sr c@sZeZdZdZdddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ dS)r#zCLUENER DataSetrcCstj|st|d||_ddttD|_g|_g|_ |dkrbtj |d}| |nL|dkrtj |d}| |n*|dkrtj |d }| |ntd d S) aS CLUENER Dataset Args: dataset_dir (str): The directory to cluener dataset. stage (str): The supported key words are in ["train", "dev", "test"] Return: A iterable dataset for cluener dataset Raises: ValueError: Error input for dataset_dir, stage. rcSsi|]\}}||qSr&r&).0rlabelr&r&r' iz+CLUENERDataSet.__init__..rz train.jsondevzdev.jsontestz test.jsonzunsupported stage.N)rrrrr enumerater label2idtexts label_idsjoin_get_train_examples_get_dev_examples_get_test_examples)selfrrtrain_data_path dev_data_pathtest_data_pathr&r&r'__init__Ws     zCLUENERDataSet.__init__cCs|j|}|j|}||fSN)r5r6)r;itemr rr&r&r' __getitem__|s  zCLUENERDataSet.__getitem__cCs t|jSr@)r r5)r;r&r&r'__len__szCLUENERDataSet.__len__cCs|||S)zGet train examples._create_examples _read_json)r;r<r&r&r'r8sz"CLUENERDataSet._get_train_examplescCs|||S)zGet dev examples.rD)r;r=r&r&r'r9sz CLUENERDataSet._get_dev_examplescCs|||S)zGet test examples.rD)r;r>r&r&r'r:sz!CLUENERDataSet._get_test_examplescCs\t|ddd:}g}|D]}|t|q|WdS1sN0YdS)z!Reads a tab separated value file.rzutf-8)encodingN)open readlinesappendjsonloadsstrip)r; input_fileflinesliner&r&r'rFs  zCLUENERDataSet._read_jsonc Cs|dD]\}}|D]|\}}|D]n}|d|dkrNd|||d<q(d|||d<d|||d<t|dd|dD]}d|||<qq(qq |S)zGenerate labelr.rrzS-zB-zI-)itemsrange) r;rRr.lwords_indicesindexjr&r&r'generate_labelszCLUENERDataSet.generate_labelcCsn|D]d}|d}dgt|}d|vr2|||}g}|D]}||j|q:|j||j|qdS)zCreate Example.r Or.N)r r[rKr4r5r6)r;rQrRr r.rZ token_labelr&r&r'rEs  zCLUENERDataSet._create_examplesN)r) r)r*r+r,r?rBrCr8r9r:rFr[rEr&r&r&r'r#Us %r#)r,rrLtypingrrrrmindspore.datasetrZtools.registerrr labelsr registerDATASET_LOADERr r#r&r&r&r's   9pengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/flickr8k_dataloader.cpython-39.pyc000066400000000000000000000136201471400134000335230ustar00rootroot00000000000000a ?cf@sdZddlZddlmZddlmZmZmZmZddl m Z ddl m Z ddl mZmZeejGd d d ZGd d d ZdS) zFlickr8k DataLoader.N) defaultdict)OptionalUnionListTuple)GeneratorDataset) load_image)MindFormerRegisterMindFormerModuleTypec@sDeZdZdZddgZd eeeeee efeedddZ dS) Flickr8kDataLoaderzFlicker8k DataloaderimagetextNtrain) dataset_dir column_namesstagecKstj|d}tj|dd}tj|s8t|dtj|sRt|d|dur`|j}t|ttfst dt |t |dkrtdt |t|dt rt|d t std t |dd t |d | d dt|||}t||fi|S) a Flicker8k Dataloader API. Args: dataset_dir (str): The directory. Flickr_8k.devImages.txt, and Flickr8k.token.txt column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "test", "del", "all"] Return: A GeneratorDataset for Flickr8k dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import Flickr8kDataLoader >>> data_loader = Flickr8kDataLoader("./Flickr8k") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[1, 378, 500, 3], dtype=UInt8, value= [[[[ 89, 46, 55], [ 89, 62, 55], [ 83, 59, 49], ... [161, 152, 135], [162, 156, 142], [159, 162, 145]]]]), Tensor(shape=[1, 5], dtype=String, value= [['A woman and child sitting on a bench beside a wooden bear carving .', 'A woman and little girl are sitting on a wooden bench beside a wooden bear .', 'A woman holds a little girl next to a large wooden carving of a bear .', 'A woman with a young girl pose with bear statues in front of a store .', 'A young girl being held by an older woman wearing bluejeans sitting on the lap of a bear statue near the entrance to a wooden building .']])] Z Flickr8k_textZFlickr8k_Dataset is not existed.NzJcolumn_names should be a tuple or a list of string with length 2, but got z0the length of column_names should be 2, but got rz8the item type of column_names should be string, but got z and None)ospathjoinisdir ValueError_default_column_names isinstancetuplelist TypeErrortypelenstrpopFlickr8kDataSetr)clsrrrkwargsannotation_dirZflick8k_datasetr)]/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/flickr8k_dataloader.py__new__s2*       zFlickr8kDataLoader.__new__)Nr) __name__ __module__ __qualname____doc__rr#rrrrr+r)r)r)r*r sr c@s*eZdZdZd ddZddZddZd S) r%zFlickr8k DataSetrcCstj|st|dtj|s4t|d||_|dkrtj|d}t|ddd}|}Wdn1s0Yn|dkrtj|d }t|ddd}|}Wdn1s0Yn~|d kr4tj|d }t|ddd}|}Wdn1s(0Yn(|d krTd dt |D}ntdtj|d} t| ddd}|} Wdn1s0Yt t } | D]:} | dd} | |vr| dd}| |  |q||_| |_dS)a Flickr8k Dataset Args: dataset_dir (str): The directory to images annotation_dir (str): The directory to Flickr_8k.trainImages.txt, Flickr_8k.testImages.txt, Flickr_8k.devImages.txt, and Flickr8k.token.txt stage (str): The supported key words are in ["train", "test", "dev", "all"] Return: A iterable dataset for Flickr8k dataset Raises: ValueError: Error input for dataset_dir, annotation_dir, and stage. rrzFlickr_8k.trainImages.txtrzutf-8)encodingNtestzFlickr_8k.testImages.txtdevzFlickr_8k.devImages.txtallcSsg|]}|dr|qS)z.jpg)endswith).0filer)r)r* z,Flickr8kDataSet.__init__..zunsupported stage.zFlickr8k.token.txt#r )rrrrrropenread splitlineslistdirrrsplitappend image_names dataset_dict)selfrr(rZ train_filer7rCZ test_fileZdev_fileannotation_fileZannotation_listrDline image_name image_annor)r)r*__init__fs>  ,, . , zFlickr8kDataSet.__init__cCs4|j|}tj|j|}t|}|j|}||fSN)rCrrrrr rD)rEitemrH image_pathr rIr)r)r* __getitem__s   zFlickr8kDataSet.__getitem__cCs t|jSrK)r"rC)rEr)r)r*__len__szFlickr8kDataSet.__len__N)r)r,r-r.r/rJrNrOr)r)r)r*r%ds 7r%)r/r collectionsrtypingrrrrmindspore.datasetrZtools.image_toolsr Ztools.registerr r registerDATASET_LOADERr r%r)r)r)r*s    Imulti_image_cap_dataloader.cpython-39.pyc000066400000000000000000000117461471400134000350550ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/dataloader/__pycache__a ?cf@sdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl mZddlmZmZeejGdd d ZGd d d ZdS) zMultiImageCapPairs DataLoader.N) OrderedDict)OptionalUnionListTuple)GeneratorDataset)Image)MindFormerRegisterMindFormerModuleTypec @s\eZdZdZddgZd eeeeeeeeee efeeee ee ddd Z dS) MultiImgCapDataLoaderz!Multiple Image-Caption DataloaderimagetextNtrainF) dataset_dirannotation_files image_dirs column_namesstage repeat_imagesshufflec Kst|t|krtdt|D]@\} } tj||| || <tj|| s t|| dq t|D]@\} } tj||| || <tj|| sjt|| dqj|dur|j}| ddt ||||} |dkrt | |fd|i|St | ||dS)a2 MultiImgCapDataLoader Dataloader API. Args: dataset_dir (str): The directory which is the parent dir of these datasets. annotation_files list[str]: the list of files contains annotations. image_dirs list[str]: the list of dir contains images. (one-to-one matching to ' annotation_files') column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "eval"] repeat_images (Optional[bool]): whether repeat image when it has multiple corresponding captions. shuffle (Optional[bool]): whether to shuffle the dataset. Return: A GeneratorDataset for loading multiple image-caption datasets Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. z=the number of image_dirs should be equal to annotation_files!z is not existed.NNonerr)r) len ValueError enumerateospathjoinisfileisdir_default_column_namespopMultiImgCapDataSetr) clsrrrrrrrkwargsi_Zmulticap_datasetr&d/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/multi_image_cap_dataloader.py__new__!s@ zMultiImgCapDataLoader.__new__)NrFF) __name__ __module__ __qualname____doc__rstrrrrrboolr(r&r&r&r'r sr c@s2eZdZdZd ddZddZdd Zd d Zd S)r!aMultiImgCapDataSet API. Args: image_dirs (str): The directory which contains images. annotation_files list[str]: the list of files contains annotations. stage (Optional[str]): The supported key words are in ["train", "eval"] repeat_images (Optional[bool]): whether repeat image when it has multiple corresponding captions. Return: A Dataset for loading multiple image-caption datasets rFc CsDg|_|dvr8t|D]p\}}t|ddd}t|}Wdn1sN0Y|D]} tj||| d| d<q\|j|q|dkr@i|_ i|_ |rg}d} t|jD]r\} } g|j | <t| dD]R\}} |j |  | | |j | <|rt | }|d| i| || d 7} qq|r@||_ntd dS) N)revalrzutf-8)encodingr r/rcaptionzunsupported stage.) annotationropenjsonloadrrrextendZtxt2imgZimg2txtappendcopydeepcopyupdater)selfrrrrr$annotation_filefileZnew_annotationZnew_annZtxt_idZimg_idannr2tempr&r&r'__init__ns: (        zMultiImgCapDataSet.__init__cCs*|j|}t|dd}||dfS)Nr RGBr2)r4rr5convert)r=indexr@r r&r&r' __getitem__s zMultiImgCapDataSet.__getitem__cCs t|jS)N)rr4)r=r&r&r'__len__szMultiImgCapDataSet.__len__cCs0|||j|}}t|d|d|ddS)z~display item Args: index (int): index Returns: out (OrderedDict): item info r r2r)r?r2r )r4r)r=rEsampler@r&r&r' display_items zMultiImgCapDataSet.display_itemN)rF)r)r*r+r,rBrFrGrIr&r&r&r'r!`s %r!)r,r collectionsrtypingrrrrr6r:mindspore.datasetrPILrmindformers.tools.registerr r registerDATASET_LOADERr r!r&r&r&r's    Cpengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/squad_dataloader.cpython-39.pyc000066400000000000000000000336531471400134000331330ustar00rootroot00000000000000a ?cfG]@sdZddlZddlZddlZddlmZddlmZmZe ej GdddZ GdddZ d d Z d d Zd dZddZddZGdddZGdddZdS)zSQuAD DataLoaderN)GeneratorDataset)MindFormerRegisterMindFormerModuleTypec@s"eZdZdZgdZd dd ZdS) SQuADDataLoaderzSQuAD Dataloader) input_ids input_mask token_type_idZstart_positionsZ end_positions unique_idNtrain@c Kstj|st|d|dvr*td|dur8|j}t|ttfsXtdt |t |dkrvtdt ||D] } t| t sztdt | qz| d dt ||||||} t| |fi|S) a SQuAD Dataloader API. Args: dataset_dir: The directory to SQuAD dataset. tokenizer: a tokenizer for text processing. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 6 stage: The supported key words are in ["train", "dev"] max_question_len: The maximum number of tokens for the question, Questions longer than this will be truncated to this length. max_seq_len: Maximum sequence length. doc_stride: When splitting up a long document into chunks, how much stride to take between chunks. Return: A GeneratorDataset for SQuAD dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import SQuADDataLoader >>> from mindformers.models import BertTokenizer >>> bert_tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased') >>> data_loader = SQuADDataLoader("./squad/", bert_tokenizer) >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[24, 384], dtype=Int32, value= [[ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], ... [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0]]), Tensor(shape=[24, 384], dtype=Int32, value= [[1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], ... [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0]]), Tensor(shape=[24, 384], dtype=Int32, value= [[0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], ... [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0]]), Tensor(shape=[24], dtype=Int32, value= [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24 ]), Tensor(shape=[24], dtype=Int32, value= [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24 ]), Tensor(shape=[24], dtype=Int32, value= [1000000029, 1000000020, 1000000023, 1000000012, 1000000024, 1000000004, 1000000006, 1000000003, 1000000017, 1000000022, 1000000028, 1000000007, 1000000005, 1000000027, 1000000014, 1000000015, 1000000002, 1000000025, 1000000011, 1000000008, 1000000021, 1000000010, 1000000019, 1000000016])]  is not existed.)r devz stage should be in train or dev.NzJcolumn_names should be a tuple or a list of string with length 7, but got z0the length of column_names should be 6, but got z8the item type of column_names should be string, but got None)ospathisdir ValueError_default_column_names isinstancetuplelist TypeErrortypelenstrpop SQuADDatasetr) cls dataset_dir tokenizer column_namesstagemax_question_len max_seq_len doc_stridekwargsnameZ squad_datasetr*Z/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/squad_dataloader.py__new__s0A      zSQuADDataLoader.__new__)Nr r r r )__name__ __module__ __qualname____doc__rr,r*r*r*r+rs rc@sJeZdZdZdddZd d Zd d Zd dZddZddZ ddZ dS)rz SQuAD Datasetr r r r ./squad_tempc Cstj|st|d||_||_||_||_||_|dkrbd|_ tj |d}| ||_ n2|dkrd|_ tj |d} | | |_ ntdt|j |j|j|j|j|j |_|dkr||d S) a3 SQuAd Dataset Args: dataset_dir (str): The directory to SQuAd dataset. tokenizer (Tokenizer): A tokenizer for text processing. stage (str): The supported key words are in ["train", "dev"] max_question_len (int): The maximum number of tokens for the question, Questions longer than this will be truncated to this length. max_seq_len (int): Maximum sequence length. doc_stride (int): When splitting up a long document into chunks, how much stride to take between chunks. temp_file_dir (str): Save temporary files for SQuAD dataset. Return: A iterable dataset for SQuAd dataset Raises: ValueError: Error input for dataset_dir, stage. rr Tztrain-v1.1.jsonrFz dev-v1.1.jsonzunsupported stage.N)rrrrr!r"r%r&r' is_trainingjoin_get_train_examplesexamples_get_dev_examplesconvert_examples_to_featuresinput_features _save_eval_examples_and_features) selfr!r"r$r%r&r' temp_file_dirtrain_data_path dev_data_pathr*r*r+__init__}s,  zSQuADDataset.__init__c Cs>|j|}|j}|j}|j}|j}|j}|j}||||||fS)zReturn input data for model)r8rrrstart_position end_positionr ) r:itemfeaturerrrr?r@r r*r*r+ __getitem__s zSQuADDataset.__getitem__cCs t|jS)zGet the size of dataset)rr8)r:r*r*r+__len__szSQuADDataset.__len__cCs ||S)zGet train examples._read_squad_examples)r:r<r*r*r+r4sz SQuADDataset._get_train_examplescCs ||S)zGet dev examples.rE)r:r=r*r*r+r6szSQuADDataset._get_dev_examplescCstj|ddtj|d}tj|d}t|ddd2}|jD]}|t|j dq@Wd n1sp0Yt|ddd2}|j D]}|t|j dqWd n1s0Yd S) z)Save examples and features for evaluationT)exist_okztemp_examples.jsonztemp_features.jsonwzutf-8)encoding N) rmakedirsrr3openr5writejsondumps__dict__r8)r:r;temp_examples_filetemp_features_filefexamplerBr*r*r+r9s 6 z-SQuADDataset._save_eval_examples_and_featuresc Cst|d}t|d}Wdn1s.0Yg}|D]}|d}|dD]}|d}|dD]~} | d} | d } d} d} g}| d d }|s|jr| d d }|d} |d} n| d }t| | || | |||d}||qhqTq@|S)z3Read a SQuAD json file into a list of SquadExample.rdataNtitle paragraphscontextqasidquestion is_impossibleFanswersrtextZ answer_start)qas_id question_text context_text answer_textstart_position_characterrWr]r^)rLrNloadgetr2 SquadExampleappend)r:input_file_pathreader input_datar5entryrW paragraphrbqar`rardrcr^r]answerrTr*r*r+rFs@ ,      z!SQuADDataset._read_squad_examplesN)r r r r r1) r-r.r/r0r>rCrDr4r6r9rFr*r*r*r+r{s 2 rc#Csg}d}t|D]\}} || j} t| |kr>| d|} ggg} } } t| jD]>\}}| t| ||}|D]}| || |q|qXd\}}|r| jrd}d}|r| js| | j}| jt| jdkr| | jdd}n t| d}t || ||| j \}}|t| d}t || |}t|D]n\}}gg}}ii}}|d|d| D]}|||dqn|d|dt |j D]P}|j|}| ||t|<t|||}||t|<|| ||dq|d|d||}dgt|}t||krV|d|d|dq&t|||t| | j|\} }!t|||||||||| |!| jd }"||"|d7}q8q|S) zConvert examples to featuresiʚ;r)NNz[CLS]z[SEP] r example_indexdoc_span_indextokenstoken_to_orig_maptoken_is_max_contextrrrr?r@r]) enumeratetokenizerar doc_tokensrhr]r?r@_improve_answer_spanrc_get_doc_spansrangelengthstart_check_is_max_contextconvert_tokens_to_ids_get_positions InputFeatures)#r5r"r&r%r'r2r8r rtrTZ query_tokensZtok_to_orig_indexZorig_to_tok_indexall_doc_tokensitoken sub_tokensZ sub_tokentok_start_positiontok_end_positionmax_tokens_for_doc doc_spansrudoc_spanrvrrwrxZsplit_token_indexZis_max_contextrrr?r@rBr*r*r+r7s                          r7c Csd\}}|rl|sl|j}|j|jd} d} ||kr:|| ks>d} | rLd\}}n |d} ||| }||| }|r||r|d\}}||fS)z#Get start position and end position)rprprqFTrrrr) rrrZquery_tokens_lengthr]r2r?r@Z doc_startZdoc_endZ out_of_spanZ doc_offsetr*r*r+r`s"   rcCsttdddg}g}d}|t|krpt||}||kr<|}||||d||t|kr`qp|t||7}q|S)z Get doc spanZDocSpanrrrr) collections namedtuplerrhmin)r'rrZ_DocSpanr start_offsetrr*r*r+r}ys  r}c Cspd||}t||dD]H}t||ddD]2}d|||d}||kr2||fSq2q||fS)zFReturns tokenized answer spans that better match the annotated answer. rqrp)r3rzr~) r"r{Z input_startZ input_endZorig_answer_textZtok_answer_text new_startnew_endZ text_spanr*r*r+r|sr|c Csd}d}t|D]l\}}|j|jd}||jkr4q||kr>q||j}||} t|| d|j} |dust| |kr| }|}q||kS)z:Check if this is the 'max context' doc span for the token.Nrqg{Gz?)ryrrr) rZcur_span_indexposition best_scoreZbest_span_indexZ span_indexrendZnum_left_contextZnum_right_contextscorer*r*r+rs  rc@s"eZdZdZdddZddZdS) rgaS A single training/test example for the Squad dataset, as loaded from disk. Args: qas_id: The example's unique identifier question_text: The question string context_text: The context string answer_text: The answer string start_position_character: The character position of the start of the answer title: The title of the example answers: None by default, this is used during evaluation. Holds answers as well as their start positions. is_impossible: False by default, set to True if the example has no possible answer. NFc Cs||_||_||_||_||_||_||_d\|_|_g} g} d} |jD]J} | | r\d} n$| rl| | n| d| 7<d} | t | dqH| |_ | |_ |dur|s| ||_| t|t |dt | d|_dS)NrTrpFrq)r`rarbrcrWr]r^r?r@_is_whitespacerhrr{char_to_word_offsetr) r:r`rarbrcrdrWr^r]r{rZprev_is_whitespacecr*r*r+r>s4       zSquadExample.__init__cCs4|dks,|dks,|dks,|dks,t|dkr0dSdS)z%Check whether character is whitespacer  rJi/ TF)ord)r:rr*r*r+r s,zSquadExample._is_whitespace)NF)r-r.r/r0r>rr*r*r*r+rgs  .rgc@seZdZdZdddZdS)rz!A single set of features of data.rpFc CsL||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ dS)Nrs) r:r rtrurvrwrxrrrr?r@r]r*r*r+r>s zInputFeatures.__init__N)rprpF)r-r.r/r0r>r*r*r*r+rs  r)r0rrNrmindspore.datasetrmindformers.tools.registerrrregisterDATASET_LOADERrrr7rr}r|rrgrr*r*r*r+s   ab#%Bpengcheng.mind-test/mindformers/dataset/dataloader/__pycache__/wmt16_dataloader.cpython-39.pyc000066400000000000000000000072421471400134000327670ustar00rootroot00000000000000a ?cf@shdZddlZddlmZddlmZddlmZmZe ej GdddZ d d Z Gd d d Z dS) zWMT16 DataLoader.N)GeneratorDataset)logger)MindFormerRegisterMindFormerModuleTypec@s"eZdZdZddgZd dd ZdS) WMT16DataLoaderzWMT16 DataloadersourcetargetNtrainrcCstj|st|d|dur6|j}td|j|rZt|tt fsZt dt |t |dkrxtdt |t|dt rt|dt std t |dd t |dt||}t||||d S) uZ WMT16 Dataloader API Args: dataset_dir: the directory to dataset column_names: the output column names, a tuple or a list of string with length 2 stage: the supported `option` are in ["train"、"test"、"del"、"all"] Return: a GeneratorDataset for WMT16 dataset  is not existed.NzZThe column_names to the WMT16DataLoader is None, so assign it with default_column_names %szJcolumn_names should be a tuple or a list of string with length 2, but got z0the length of column_names should be 2, but got rr z8the item type of column_names should be string, but got z and ) num_shardsshard_id)ospathisdir ValueError_default_column_namesrinfo isinstancetuplelist TypeErrortypelenstr WMT16DataSetr)cls dataset_dir column_namesstagerrdatasetr#Z/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/wmt16_dataloader.py__new__s.     zWMT16DataLoader.__new__)Nr r r)__name__ __module__ __qualname____doc__rr%r#r#r#r$rsrcCsPt|4}g}|D]}|}|r||qWdn1sB0Y|S)z&Read the text files and return a list.N)openstripappend) train_filefpdataliner#r#r$ read_textAs *r1c@s*eZdZdZd ddZddZddZd S) rz WMT16 DataSetr csztjstd|_||_t}fdd}tdddkrZ||||<ndD]}||||<q^||_ dS) u WMT16DataSet Dataset Args: dataset_dir: the directory to wmt16 dataset stage: the supported key word are in ["train"、"test"、"del"、"all"] Return: an iterable dataset for wmt16 dataset r csJtj|d}t|}tj|d}t|}tt||}|S)Nz.sourcez.target)rrjoinr1rzip) cur_stage source_pathsrcZ target_sourcetgtZ src_tgt_pairrr#r$read_and_add_to_stageas z4WMT16DataSet.__init__..read_and_add_to_stagez,Start to read the raw data from the disk %s.r!all)r devtestN) rrrrrr!dictrr dataset_dict)selfrr!r>r9itemr#r8r$__init__Os   zWMT16DataSet.__init__cCs|j|j|SN)r>r!)r?r@r#r#r$ __getitem__rszWMT16DataSet.__getitem__cCst|j|jSrB)rr>r!)r?r#r#r$__len__uszWMT16DataSet.__len__N)r )r&r'r(r)rArCrDr#r#r#r$rLs #r)r)rmindspore.datasetrmindformers.toolsrZtools.registerrrregisterDATASET_LOADERrr1rr#r#r#r$s   ( pengcheng.mind-test/mindformers/dataset/dataloader/adgen_dataloader.py000066400000000000000000000115401471400134000265740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ADGen DataLoader""" import json import os from mindspore.dataset import GeneratorDataset from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerModuleType, MindFormerRegister @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class ADGenDataLoader: """ADGen Dataloader""" def __new__(cls, dataset_dir, phase, shuffle, origin_columns, **kwargs): r""" ADGen Dataloader API. Args: dataset_dir: The directory to ADGen dataset. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 6 phase: The supported key words are in ["train", "dev"] Return: A GeneratorDataset for ADGen dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import ADGenDataLoader >>> from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer >>> data_loader = ADGenDataLoader("./ADGen/") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break """ if not os.path.isfile(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if phase not in ["train", "eval"]: raise ValueError(f"phase should be in train or eval.") column_names = ["prompt", "answer"] # verify dataset column names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 2, but got {type(column_names)}, length {len(column_names)}") for name in column_names: if not isinstance(name, str): raise ValueError(f"the item type of column_names should be string," f" but got {type(name)}") # verify origin column names if not isinstance(origin_columns, (tuple, list)) or len(origin_columns) != 2: raise TypeError(f"origin_columns should be a tuple or a list" f" of string with length 2, but got {type(origin_columns)}, length {len(origin_columns)}") kwargs.pop("None", None) adgen_dataset = ADGenDataset(dataset_dir, origin_columns, phase) info = f"[DATASET] shuffle status is {shuffle}, phase is {phase}." logger.info(info) kwargs.pop("version", None) return GeneratorDataset(adgen_dataset, column_names, shuffle=shuffle, **kwargs) class ADGenDataset: """ADGen Dataset""" def __init__(self, dataset_dir, origin_columns, phase="train"): r""" ADGen Dataset Args: dataset_dir (str): The directory to ADGen dataset. phase (str): The supported key words are in ["train", "dev"] Return: A iterable dataset for ADGen dataset Raises: ValueError: Error input for dataset_dir, phase. """ if not os.path.isfile(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") self.dataset_dir = dataset_dir self.phase = phase examples = {} content_list = [] summary_list = [] self.prompt_column = origin_columns[0] self.response_column = origin_columns[1] with open(self.dataset_dir) as fp: for line in fp: content_list.append(json.loads(line)[self.prompt_column]) summary_list.append(json.loads(line)[self.response_column]) examples[self.prompt_column] = content_list examples[self.response_column] = summary_list self.examples = examples def __len__(self): """Get the size of dataset""" return len(self.examples[self.prompt_column]) def __getitem__(self, i): """Return input data for model""" prompt, answer = self.examples[self.prompt_column][i], self.examples[self.response_column][i] return prompt, answer pengcheng.mind-test/mindformers/dataset/dataloader/build_dataloader.py000066400000000000000000000055621471400134000266240ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build DataLoader API.""" import inspect from mindspore import dataset as ds from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_dataset_loader( config: dict = None, default_args: dict = None, module_type: str = 'dataset_loader', class_name: str = None, **kwargs): r"""Build dataset loader For MindFormer. Instantiate the dataset loader from MindFormerRegister's registry. Args: config (dict): The task dataset loader's config. Default: None. default_args (dict): The default argument of dataset loader API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset loader'. class_name (str): The class name of dataset loader API. Default: None. Return: The function instance of dataset loader API. Examples: >>> from mindformers.dataset.dataloader import build_dataset_loader >>> dataset_loader_config = {'type': 'ImageFolderDataset', 'dataset_dir': './imagenet/train'} >>> # 1) use config dict to build dataset loader >>> dataset_loader_from_config = build_dataset_loader(dataset_loader_config) >>> # 2) use class name to build dataset loader >>> dataset_loader_class_name = build_dataset_loader( ... class_name='ImageFolderDataset', dataset_dir='./imagenet/train') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.DATASET_LOADER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_dataset_loader(): """ register MindSpore builtin dataset loader class. """ for module_name in dir(ds): if module_name.startswith('__'): continue dataset = getattr(ds, module_name) if inspect.isclass(dataset): MindFormerRegister.register_cls( dataset, MindFormerModuleType.DATASET_LOADER) register_ms_dataset_loader() pengcheng.mind-test/mindformers/dataset/dataloader/cifar100_dataloader.py000066400000000000000000000173251471400134000270320ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Cifar100 DataLoader.""" import os import pickle from typing import Optional, Union, List, Tuple import numpy as np from mindspore.dataset import GeneratorDataset from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class Cifar100DataLoader: """Cifar100 Dataloader with class name as text column""" _default_column_names = ["image", "text", "label"] def __new__(cls, dataset_dir: str, column_names: Optional[Union[List[str], Tuple[str]]] = None, stage: Optional[str] = "train", fine_label: Optional[bool] = True, shuffle: Optional[bool] = False, hypothesis_template: Optional[str] = "This is a photo of {}.", **kwargs): r""" Cifar100 Dataloader API. Args: dataset_dir (str): The dataset directory, such as "/home/desktop/cifar-100-python" column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 3. stage (Optional[str]): The supported key words are in ["train","test", "all"] fine_label (Optional[bool]): True for 100 classes, False for 10 classes. shuffle (Optional[bool]): Shuffle the samples. hypothesis_template (Optional[str]): Prompt template for class label. Return: A GeneratorDataset for Cifar100 dataset Raises: ValueError: Error input for dataset_dir and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import Cifar100DataLoader >>> dataloader = Cifar100DataLoader("./cifar-100-python") >>> dataloader = dataloader.batch(1) >>> for item in dataloader: >>> print(item) >>> break [Tensor(shape=[1, 32, 32, 3], dtype=UInt8, value= [[[[255, 255, 255], [255, 255, 255], [255, 255, 255], ... [195, 205, 193], [212, 224, 204], [182, 194, 167]]]]]), Tensor(shape=[1, 100], dtype=String, value= [['This is a photo of apple.', 'This is a photo of aquarium_fish.', 'This is a photo of baby.', 'This is a photo of bear.', ... 'This is a photo of woman.', 'This is a photo of worm.']]), Tensor(shape=[1], dtype=Int32, value= [19])] """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if column_names is None: column_names = cls._default_column_names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 3, but got {type(column_names)}") if len(column_names) != 3: raise ValueError(f"the length of column_names should be 3," f" but got {len(column_names)}") for index in range(len(column_names)): if not isinstance(column_names[index], str): raise ValueError(f"the item type of column_names should be string," f" but got {type(column_names[index])}") kwargs.pop("None", None) cifar100_dataset = Cifar100DataSet(dataset_dir, stage, fine_label, hypothesis_template) cifar100_dataloader = GeneratorDataset(cifar100_dataset, column_names, shuffle=shuffle) setattr(cifar100_dataloader, "label_names", cifar100_dataset.label_names) return cifar100_dataloader class Cifar100DataSet: """Cifar100 dataSet with class name""" def __init__(self, dataset_dir: str, stage: Optional[str] = "train", fine_label: Optional[bool] = True, hypothesis_template: Optional[str] = "This is a photo of {}."): r""" Cifar100 Dataset Args: dataset_dir (str): The dataset directory, such as "/home/desktop/cifar-100-python". stage (Optional[str]): The supported key words are in ["train", "test", "all"]. fine_label (Optional[bool]): True for 100 classes, False for 10 classes. hypothesis_template (Optional[str]): Prompt template for class label. Return: A iterable dataset for Cifar100 dataset. Raises: ValueError: Unsupported values for stage. """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if stage not in ["train", "test", "all"]: raise ValueError("unsupported stage," " stage should be in [\"train\", \"test\", \"all\"]") meta_file = os.path.join(dataset_dir, "meta") with open(meta_file, 'rb') as fo: meta_dict = pickle.load(fo, encoding="bytes") if fine_label: label_names = meta_dict[b'fine_label_names'] else: label_names = meta_dict[b'coarse_label_names'] for index in range(len(label_names)): label_names[index] = label_names[index].decode() self.label_names = label_names hypothesis = [] for index in range(len(label_names)): hypothesis.append(hypothesis_template.format(label_names[index])) if stage in ["train", "all"]: train_file = os.path.join(dataset_dir, "train") with open(train_file, 'rb') as fo: train_dict = pickle.load(fo, encoding="bytes") if fine_label: train_label = train_dict[b"fine_labels"] else: train_label = train_dict[b"coarse_labels"] train_image = train_dict[b"data"] train_num = train_image.shape[0] train_image = train_image.reshape((train_num, 3, 32, 32)) train_image = train_image.transpose(0, 2, 3, 1) self.image = train_image self.label = train_label self.text = hypothesis if stage in ["test", "all"]: test_file = os.path.join(dataset_dir, "test") with open(test_file, 'rb') as fo: test_dict = pickle.load(fo, encoding="bytes") if fine_label: test_label = test_dict[b"fine_labels"] else: test_label = test_dict[b"coarse_labels"] test_image = test_dict[b"data"] test_num = test_image.shape[0] test_image = test_image.reshape((test_num, 3, 32, 32)) test_image = test_image.transpose(0, 2, 3, 1) self.image = test_image self.label = test_label self.text = hypothesis if stage == "all": self.image = np.row_stack([train_image, test_image]) self.label = train_label + test_label self.text = hypothesis def __getitem__(self, item): """getitem""" return self.image[item], self.text, self.label[item] def __len__(self): """len""" return self.image.shape[0] pengcheng.mind-test/mindformers/dataset/dataloader/cluener_dataloader.py000066400000000000000000000154611471400134000271610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://gitee.com/mindspore/models/blob/master/official/nlp/Bert/src/generate_mindrecord/generate_cluener_mindrecord.py # ============================================================================ """CLUENER DataLoader.""" import os import json from typing import Optional, Union, List, Tuple from mindspore.dataset import GeneratorDataset from ...tools.register import MindFormerRegister, MindFormerModuleType from ..labels import cluener_labels @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class CLUENERDataLoader: """CLUENER Dataloader""" _default_column_names = ["text", "label_id"] def __new__(cls, dataset_dir: str, column_names: Optional[Union[List[str], Tuple[str]]] = None, stage: Optional[str] = "train", **kwargs): r""" CLUENER Dataloader API. Args: dataset_dir (str): The directory to cluener dataset. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "test", "del", "all"] Return: A GeneratorDataset for cluener dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import CLUENERDataLoader >>> data_loader = CLUENERDataLoader("./cluener/") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[], dtype=String, value= '虚幻引擎3动作游戏《黑光》新作公布'), Tensor(shape=[49], dtype=Int64, value= [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 15, 15, 0, 0, 0, 0, 0, 5, 15, 15, 15, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])] """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if column_names is None: column_names = cls._default_column_names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 2, but got {type(column_names)}") if len(column_names) != 2: raise ValueError(f"the length of column_names should be 2," f" but got {len(column_names)}") if not isinstance(column_names[0], str) or not isinstance(column_names[1], str): raise ValueError(f"the item type of column_names should be string," f" but got {type(column_names[0])} and {type(column_names[1])}") kwargs.pop("None", None) cluener_dataset = CLUENERDataSet(dataset_dir, stage) return GeneratorDataset(cluener_dataset, column_names) class CLUENERDataSet: """CLUENER DataSet""" def __init__(self, dataset_dir, stage="train"): r""" CLUENER Dataset Args: dataset_dir (str): The directory to cluener dataset. stage (str): The supported key words are in ["train", "dev", "test"] Return: A iterable dataset for cluener dataset Raises: ValueError: Error input for dataset_dir, stage. """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") self.dataset_dir = dataset_dir self.label2id = {label: label_id for label_id, label in enumerate(cluener_labels)} self.texts = [] self.label_ids = [] if stage == "train": train_data_path = os.path.join(dataset_dir, "train.json") self._get_train_examples(train_data_path) elif stage == "dev": dev_data_path = os.path.join(dataset_dir, "dev.json") self._get_dev_examples(dev_data_path) elif stage == "test": test_data_path = os.path.join(dataset_dir, "test.json") self._get_test_examples(test_data_path) else: raise ValueError("unsupported stage.") def __getitem__(self, item): text = self.texts[item] label_id = self.label_ids[item] return text, label_id def __len__(self): return len(self.texts) def _get_train_examples(self, train_data_path): """Get train examples.""" return self._create_examples(self._read_json(train_data_path)) def _get_dev_examples(self, dev_data_path): """Get dev examples.""" return self._create_examples(self._read_json(dev_data_path)) def _get_test_examples(self, test_data_path): """Get test examples.""" return self._create_examples(self._read_json(test_data_path)) def _read_json(self, input_file): """Reads a tab separated value file.""" with open(input_file, "r", encoding="utf-8") as f: lines = [] for line in f.readlines(): lines.append(json.loads(line.strip())) return lines def generate_label(self, line, label): """Generate label""" for l, words in line['label'].items(): for _, indices in words.items(): for index in indices: if index[0] == index[1]: label[index[0]] = 'S-' + l else: label[index[0]] = 'B-' + l label[index[1]] = 'I-' + l for j in range(index[0] + 1, index[1]): label[j] = 'I-' + l return label def _create_examples(self, lines): """Create Example.""" for line in lines: text = line['text'] label = ['O'] * len(text) if 'label' in line: label = self.generate_label(line, label) label_id = [] for token_label in label: label_id.append(self.label2id[token_label]) self.texts.append(text) self.label_ids.append(label_id) pengcheng.mind-test/mindformers/dataset/dataloader/flickr8k_dataloader.py000066400000000000000000000156731471400134000272460ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Flickr8k DataLoader.""" import os from collections import defaultdict from typing import Optional, Union, List, Tuple from mindspore.dataset import GeneratorDataset from ...tools.image_tools import load_image from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class Flickr8kDataLoader: """Flicker8k Dataloader""" _default_column_names = ["image", "text"] def __new__(cls, dataset_dir: str, column_names: Optional[Union[List[str], Tuple[str]]] = None, stage: Optional[str] = "train", **kwargs): r""" Flicker8k Dataloader API. Args: dataset_dir (str): The directory. Flickr_8k.devImages.txt, and Flickr8k.token.txt column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "test", "del", "all"] Return: A GeneratorDataset for Flickr8k dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import Flickr8kDataLoader >>> data_loader = Flickr8kDataLoader("./Flickr8k") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[1, 378, 500, 3], dtype=UInt8, value= [[[[ 89, 46, 55], [ 89, 62, 55], [ 83, 59, 49], ... [161, 152, 135], [162, 156, 142], [159, 162, 145]]]]), Tensor(shape=[1, 5], dtype=String, value= [['A woman and child sitting on a bench beside a wooden bear carving .', 'A woman and little girl are sitting on a wooden bench beside a wooden bear .', 'A woman holds a little girl next to a large wooden carving of a bear .', 'A woman with a young girl pose with bear statues in front of a store .', 'A young girl being held by an older woman wearing bluejeans sitting on the lap of a bear statue near the entrance to a wooden building .']])] """ annotation_dir = os.path.join(dataset_dir, "Flickr8k_text") dataset_dir = os.path.join(dataset_dir, "Flickr8k_Dataset", "Flickr8k_Dataset") if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if not os.path.isdir(annotation_dir): raise ValueError(f"{annotation_dir} is not existed.") if column_names is None: column_names = cls._default_column_names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 2, but got {type(column_names)}") if len(column_names) != 2: raise ValueError(f"the length of column_names should be 2," f" but got {len(column_names)}") if not isinstance(column_names[0], str) or not isinstance(column_names[1], str): raise ValueError(f"the item type of column_names should be string," f" but got {type(column_names[0])} and {type(column_names[1])}") kwargs.pop("None", None) flick8k_dataset = Flickr8kDataSet(dataset_dir, annotation_dir, stage) return GeneratorDataset(flick8k_dataset, column_names, **kwargs) class Flickr8kDataSet: """Flickr8k DataSet""" def __init__(self, dataset_dir, annotation_dir, stage="train"): r""" Flickr8k Dataset Args: dataset_dir (str): The directory to images annotation_dir (str): The directory to Flickr_8k.trainImages.txt, Flickr_8k.testImages.txt, Flickr_8k.devImages.txt, and Flickr8k.token.txt stage (str): The supported key words are in ["train", "test", "dev", "all"] Return: A iterable dataset for Flickr8k dataset Raises: ValueError: Error input for dataset_dir, annotation_dir, and stage. """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if not os.path.isdir(annotation_dir): raise ValueError(f"{annotation_dir} is not existed.") self.dataset_dir = dataset_dir if stage == "train": train_file = os.path.join(annotation_dir, "Flickr_8k.trainImages.txt") with open(train_file, 'r', encoding='utf-8') as file: image_names = file.read().splitlines() elif stage == "test": test_file = os.path.join(annotation_dir, "Flickr_8k.testImages.txt") with open(test_file, 'r', encoding='utf-8') as file: image_names = file.read().splitlines() elif stage == "dev": dev_file = os.path.join(annotation_dir, "Flickr_8k.devImages.txt") with open(dev_file, 'r', encoding='utf-8') as file: image_names = file.read().splitlines() elif stage == "all": image_names = [file for file in os.listdir(dataset_dir) if file.endswith(".jpg")] else: raise ValueError("unsupported stage.") annotation_file = os.path.join(annotation_dir, "Flickr8k.token.txt") with open(annotation_file, 'r', encoding='utf-8') as file: annotation_list = file.read().splitlines() dataset_dict = defaultdict(list) for line in annotation_list: image_name = line.split("#")[0] if image_name in image_names: image_anno = line.split("\t")[-1] dataset_dict[image_name].append(image_anno) self.image_names = image_names self.dataset_dict = dataset_dict def __getitem__(self, item): image_name = self.image_names[item] image_path = os.path.join(self.dataset_dir, image_name) image = load_image(image_path) image_anno = self.dataset_dict[image_name] return image, image_anno def __len__(self): return len(self.image_names) pengcheng.mind-test/mindformers/dataset/dataloader/multi_image_cap_dataloader.py000066400000000000000000000150211471400134000306330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MultiImageCapPairs DataLoader.""" import os from collections import OrderedDict from typing import Optional, Union, List, Tuple import json import copy from mindspore.dataset import GeneratorDataset from PIL import Image from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class MultiImgCapDataLoader: """Multiple Image-Caption Dataloader""" _default_column_names = ["image", "text"] def __new__(cls, dataset_dir: str, annotation_files: List[str], image_dirs: List[str], column_names: Optional[Union[List[str], Tuple[str]]] = None, stage: Optional[str] = "train", repeat_images: Optional[bool] = False, shuffle: Optional[bool] = False, **kwargs): r""" MultiImgCapDataLoader Dataloader API. Args: dataset_dir (str): The directory which is the parent dir of these datasets. annotation_files list[str]: the list of files contains annotations. image_dirs list[str]: the list of dir contains images. (one-to-one matching to ' annotation_files') column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "eval"] repeat_images (Optional[bool]): whether repeat image when it has multiple corresponding captions. shuffle (Optional[bool]): whether to shuffle the dataset. Return: A GeneratorDataset for loading multiple image-caption datasets Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. """ if len(image_dirs) != len(annotation_files): raise ValueError( "the number of image_dirs should be equal to annotation_files!" ) for i, _ in enumerate(annotation_files): annotation_files[i] = os.path.join(dataset_dir, annotation_files[i]) if not os.path.isfile(annotation_files[i]): raise ValueError(f"{annotation_files[i]} is not existed.") for i, _ in enumerate(image_dirs): image_dirs[i] = os.path.join(dataset_dir, image_dirs[i]) if not os.path.isdir(image_dirs[i]): raise ValueError(f"{image_dirs[i]} is not existed.") if column_names is None: column_names = cls._default_column_names kwargs.pop("None", None) multicap_dataset = MultiImgCapDataSet(image_dirs, annotation_files, stage, repeat_images) if stage == 'train': return GeneratorDataset(multicap_dataset, column_names, shuffle=shuffle, **kwargs) return GeneratorDataset(multicap_dataset, column_names, shuffle=shuffle) class MultiImgCapDataSet: """MultiImgCapDataSet API. Args: image_dirs (str): The directory which contains images. annotation_files list[str]: the list of files contains annotations. stage (Optional[str]): The supported key words are in ["train", "eval"] repeat_images (Optional[bool]): whether repeat image when it has multiple corresponding captions. Return: A Dataset for loading multiple image-caption datasets """ def __init__(self, image_dirs, annotation_files, stage="train", repeat_images=False): self.annotation = [] if stage in ("train", "eval"): for i, annotation_file in enumerate(annotation_files): with open(annotation_file, 'r', encoding='utf-8') as file: new_annotation = json.load(file) for new_ann in new_annotation: new_ann["image"] = os.path.join(image_dirs[i], new_ann["image"]) self.annotation.extend(new_annotation) if stage == "eval": self.txt2img = {} self.img2txt = {} if repeat_images: new_annotation = [] txt_id = 0 for img_id, ann in enumerate(self.annotation): self.img2txt[img_id] = [] for i, caption in enumerate(ann["caption"]): self.img2txt[img_id].append(txt_id) self.txt2img[txt_id] = img_id if repeat_images: temp = copy.deepcopy(ann) temp.update({"caption": caption}) new_annotation.append(temp) txt_id += 1 if repeat_images: self.annotation = new_annotation else: raise ValueError("unsupported stage.") def __getitem__(self, index): ann = self.annotation[index] image = Image.open(ann["image"]).convert("RGB") return image, ann["caption"] def __len__(self): return len(self.annotation) def display_item(self, index): """display item Args: index (int): index Returns: out (OrderedDict): item info """ sample, ann = self[index], self.annotation[index] return OrderedDict({ "file": ann["image"], "caption": ann["caption"], "image": sample[0], }) pengcheng.mind-test/mindformers/dataset/dataloader/squad_dataloader.py000066400000000000000000000565071471400134000266470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SQuAD DataLoader""" import os import json import collections from mindspore.dataset import GeneratorDataset from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class SQuADDataLoader: """SQuAD Dataloader""" _default_column_names = ["input_ids", "input_mask", "token_type_id", "start_positions", "end_positions", "unique_id"] def __new__(cls, dataset_dir, tokenizer, column_names=None, stage="train", max_question_len=64, max_seq_len=384, doc_stride=128, **kwargs): r""" SQuAD Dataloader API. Args: dataset_dir: The directory to SQuAD dataset. tokenizer: a tokenizer for text processing. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 6 stage: The supported key words are in ["train", "dev"] max_question_len: The maximum number of tokens for the question, Questions longer than this will be truncated to this length. max_seq_len: Maximum sequence length. doc_stride: When splitting up a long document into chunks, how much stride to take between chunks. Return: A GeneratorDataset for SQuAD dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import SQuADDataLoader >>> from mindformers.models import BertTokenizer >>> bert_tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased') >>> data_loader = SQuADDataLoader("./squad/", bert_tokenizer) >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[24, 384], dtype=Int32, value= [[ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], ... [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0]]), Tensor(shape=[24, 384], dtype=Int32, value= [[1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], ... [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0]]), Tensor(shape=[24, 384], dtype=Int32, value= [[0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], ... [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0]]), Tensor(shape=[24], dtype=Int32, value= [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24 ]), Tensor(shape=[24], dtype=Int32, value= [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24 ]), Tensor(shape=[24], dtype=Int32, value= [1000000029, 1000000020, 1000000023, 1000000012, 1000000024, 1000000004, 1000000006, 1000000003, 1000000017, 1000000022, 1000000028, 1000000007, 1000000005, 1000000027, 1000000014, 1000000015, 1000000002, 1000000025, 1000000011, 1000000008, 1000000021, 1000000010, 1000000019, 1000000016])] """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if stage not in ["train", "dev"]: raise ValueError(f"stage should be in train or dev.") if column_names is None: column_names = cls._default_column_names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 7, but got {type(column_names)}") if len(column_names) != 6: raise ValueError(f"the length of column_names should be 6," f" but got {len(column_names)}") for name in column_names: if not isinstance(name, str): raise ValueError(f"the item type of column_names should be string," f" but got {type(name)}") kwargs.pop("None", None) squad_dataset = SQuADDataset(dataset_dir, tokenizer, stage, max_question_len, max_seq_len, doc_stride) return GeneratorDataset(squad_dataset, column_names, **kwargs) class SQuADDataset: """SQuAD Dataset""" def __init__(self, dataset_dir, tokenizer, stage="train", max_question_len=64, max_seq_len=384, doc_stride=128, temp_file_dir="./squad_temp"): r""" SQuAd Dataset Args: dataset_dir (str): The directory to SQuAd dataset. tokenizer (Tokenizer): A tokenizer for text processing. stage (str): The supported key words are in ["train", "dev"] max_question_len (int): The maximum number of tokens for the question, Questions longer than this will be truncated to this length. max_seq_len (int): Maximum sequence length. doc_stride (int): When splitting up a long document into chunks, how much stride to take between chunks. temp_file_dir (str): Save temporary files for SQuAD dataset. Return: A iterable dataset for SQuAd dataset Raises: ValueError: Error input for dataset_dir, stage. """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") self.dataset_dir = dataset_dir self.tokenizer = tokenizer self.max_question_len = max_question_len self.max_seq_len = max_seq_len self.doc_stride = doc_stride if stage == "train": self.is_training = True train_data_path = os.path.join(dataset_dir, "train-v1.1.json") self.examples = self._get_train_examples(train_data_path) elif stage == "dev": self.is_training = False dev_data_path = os.path.join(dataset_dir, "dev-v1.1.json") self.examples = self._get_dev_examples(dev_data_path) else: raise ValueError("unsupported stage.") self.input_features = convert_examples_to_features(self.examples, self.tokenizer, self.max_seq_len, self.max_question_len, self.doc_stride, self.is_training) if stage == "dev": self._save_eval_examples_and_features(temp_file_dir) def __getitem__(self, item): """Return input data for model""" feature = self.input_features[item] input_ids = feature.input_ids input_mask = feature.input_mask token_type_id = feature.token_type_id start_position = feature.start_position end_position = feature.end_position unique_id = feature.unique_id return input_ids, input_mask, token_type_id, start_position, end_position, unique_id def __len__(self): """Get the size of dataset""" return len(self.input_features) def _get_train_examples(self, train_data_path): """Get train examples.""" return self._read_squad_examples(train_data_path) def _get_dev_examples(self, dev_data_path): """Get dev examples.""" return self._read_squad_examples(dev_data_path) def _save_eval_examples_and_features(self, temp_file_dir): """Save examples and features for evaluation""" os.makedirs(temp_file_dir, exist_ok=True) temp_examples_file = os.path.join(temp_file_dir, "temp_examples.json") temp_features_file = os.path.join(temp_file_dir, "temp_features.json") with open(temp_examples_file, 'w', encoding='utf-8') as f: for example in self.examples: f.write(json.dumps(example.__dict__) + '\n') with open(temp_features_file, 'w', encoding='utf-8') as f: for feature in self.input_features: f.write(json.dumps(feature.__dict__) + '\n') def _read_squad_examples(self, input_file_path): """Read a SQuAD json file into a list of SquadExample.""" with open(input_file_path, "r") as reader: input_data = json.load(reader)["data"] examples = [] for entry in input_data: title = entry["title"] for paragraph in entry["paragraphs"]: context_text = paragraph["context"] for qa in paragraph["qas"]: qas_id = qa["id"] question_text = qa["question"] start_position_character = None answer_text = None answers = [] is_impossible = qa.get("is_impossible", False) if not is_impossible: if self.is_training: answer = qa["answers"][0] answer_text = answer["text"] start_position_character = answer["answer_start"] else: answers = qa["answers"] example = SquadExample( qas_id=qas_id, question_text=question_text, context_text=context_text, answer_text=answer_text, start_position_character=start_position_character, title=title, is_impossible=is_impossible, answers=answers, ) examples.append(example) return examples def convert_examples_to_features(examples, tokenizer, max_seq_len, max_question_len, doc_stride, is_training): """Convert examples to features""" input_features = [] unique_id = 1000000000 for (example_index, example) in enumerate(examples): query_tokens = tokenizer.tokenize(example.question_text) if len(query_tokens) > max_question_len: query_tokens = query_tokens[0: max_question_len] tok_to_orig_index, orig_to_tok_index, all_doc_tokens = [], [], [] for (i, token) in enumerate(example.doc_tokens): orig_to_tok_index.append(len(all_doc_tokens)) sub_tokens = tokenizer.tokenize(token) for sub_token in sub_tokens: tok_to_orig_index.append(i) all_doc_tokens.append(sub_token) tok_start_position, tok_end_position = None, None if is_training and example.is_impossible: tok_start_position = -1 tok_end_position = -1 if is_training and not example.is_impossible: tok_start_position = orig_to_tok_index[example.start_position] if example.end_position < len(example.doc_tokens) - 1: tok_end_position = orig_to_tok_index[example.end_position + 1] - 1 else: tok_end_position = len(all_doc_tokens) - 1 (tok_start_position, tok_end_position) = _improve_answer_span(tokenizer, all_doc_tokens, tok_start_position, tok_end_position, example.answer_text) # The -3 accounts for [CLS], [SEP] and [SEP] max_tokens_for_doc = max_seq_len - len(query_tokens) - 3 # We can have documents that are longer than the maximum sequence length. # To deal with this we do a sliding window approach, where we take chunks # of the up to our max length with a stride of `doc_stride`. doc_spans = _get_doc_spans(doc_stride, all_doc_tokens, max_tokens_for_doc) for (doc_span_index, doc_span) in enumerate(doc_spans): tokens, token_type_id = [], [] token_to_orig_map, token_is_max_context = {}, {} tokens.append("[CLS]") token_type_id.append(0) for token in query_tokens: tokens.append(token) token_type_id.append(0) tokens.append("[SEP]") token_type_id.append(0) for i in range(doc_span.length): split_token_index = doc_span.start + i token_to_orig_map[len(tokens)] = tok_to_orig_index[split_token_index] is_max_context = _check_is_max_context(doc_spans, doc_span_index, split_token_index) token_is_max_context[len(tokens)] = is_max_context tokens.append(all_doc_tokens[split_token_index]) token_type_id.append(1) tokens.append("[SEP]") token_type_id.append(1) input_ids = tokenizer.convert_tokens_to_ids(tokens) # The mask has 1 for real tokens and 0 for padding tokens. Only real # tokens are attended to. input_mask = [1] * len(input_ids) # Zero-pad up to the sequence length. while len(input_ids) < max_seq_len: input_ids.append(0) input_mask.append(0) token_type_id.append(0) start_position, end_position = _get_positions(doc_span, tok_start_position, tok_end_position, len(query_tokens), example.is_impossible, is_training) feature = InputFeatures( unique_id=unique_id, example_index=example_index, doc_span_index=doc_span_index, tokens=tokens, token_to_orig_map=token_to_orig_map, token_is_max_context=token_is_max_context, input_ids=input_ids, input_mask=input_mask, token_type_id=token_type_id, start_position=start_position, end_position=end_position, is_impossible=example.is_impossible) input_features.append(feature) unique_id += 1 return input_features def _get_positions(doc_span, tok_start_position, tok_end_position, query_tokens_length, is_impossible, is_training): """Get start position and end position""" start_position, end_position = -1, -1 if is_training and not is_impossible: # For training, if our document chunk does not contain an annotation # we throw it out, since there is nothing to predict. doc_start = doc_span.start doc_end = doc_span.start + doc_span.length - 1 out_of_span = False if not (tok_start_position >= doc_start and tok_end_position <= doc_end): out_of_span = True if out_of_span: start_position, end_position = 0, 0 else: doc_offset = query_tokens_length + 2 start_position = tok_start_position - doc_start + doc_offset end_position = tok_end_position - doc_start + doc_offset if is_training and is_impossible: start_position, end_position = 0, 0 return start_position, end_position def _get_doc_spans(doc_stride, all_doc_tokens, max_tokens_for_doc): """Get doc span""" _DocSpan = collections.namedtuple("DocSpan", ["start", "length"]) doc_spans = [] start_offset = 0 while start_offset < len(all_doc_tokens): length = len(all_doc_tokens) - start_offset if length > max_tokens_for_doc: length = max_tokens_for_doc doc_spans.append(_DocSpan(start=start_offset, length=length)) if start_offset + length == len(all_doc_tokens): break start_offset += min(length, doc_stride) return doc_spans def _improve_answer_span(tokenizer, doc_tokens, input_start, input_end, orig_answer_text): """Returns tokenized answer spans that better match the annotated answer.""" # The SQuAD annotations are character based. We first project them to # whitespace-tokenized words. But then after WordPiece tokenization, we can # often find a "better match". For example: # # Question: What year was John Smith born? # Context: The leader was John Smith (1895-1943). # Answer: 1895 # # The original whitespace-tokenized answer will be "(1895-1943).". However # after tokenization, our tokens will be "( 1895 - 1943 ) .". So we can match # the exact answer, 1895. # # However, this is not always possible. Consider the following: # # Question: What country is the top exporter of electornics? # Context: The Japanese electronics industry is the lagest in the world. # Answer: Japan # # In this case, the annotator chose "Japan" as a character sub-span of # the word "Japanese". Since our WordPiece tokenizer does not split # "Japanese", we just use "Japanese" as the annotation. This is fairly rare # in SQuAD, but does happen. tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text)) for new_start in range(input_start, input_end + 1): for new_end in range(input_end, new_start - 1, -1): text_span = " ".join(doc_tokens[new_start:(new_end + 1)]) if text_span == tok_answer_text: return (new_start, new_end) return (input_start, input_end) def _check_is_max_context(doc_spans, cur_span_index, position): """Check if this is the 'max context' doc span for the token.""" # Because of the sliding window approach taken to scoring documents, a single # token can appear in multiple documents. E.g. # Doc: the man went to the store and bought a gallon of milk # Span A: the man went to the # Span B: to the store and bought # Span C: and bought a gallon of # ... # # Now the word 'bought' will have two scores from spans B and C. We only # want to consider the score with "maximum context", which we define as # the *minimum* of its left and right context (the *sum* of left and # right context will always be the same, of course). # # In the example the maximum context for 'bought' would be span C since # it has 1 left context and 3 right context, while span B has 4 left context # and 0 right context. best_score = None best_span_index = None for (span_index, doc_span) in enumerate(doc_spans): end = doc_span.start + doc_span.length - 1 if position < doc_span.start: continue if position > end: continue num_left_context = position - doc_span.start num_right_context = end - position score = min(num_left_context, num_right_context) + 0.01 * doc_span.length if best_score is None or score > best_score: best_score = score best_span_index = span_index return cur_span_index == best_span_index class SquadExample: """ A single training/test example for the Squad dataset, as loaded from disk. Args: qas_id: The example's unique identifier question_text: The question string context_text: The context string answer_text: The answer string start_position_character: The character position of the start of the answer title: The title of the example answers: None by default, this is used during evaluation. Holds answers as well as their start positions. is_impossible: False by default, set to True if the example has no possible answer. """ def __init__(self, qas_id, question_text, context_text, answer_text, start_position_character, title, answers=None, is_impossible=False): self.qas_id = qas_id self.question_text = question_text self.context_text = context_text self.answer_text = answer_text self.title = title self.is_impossible = is_impossible self.answers = answers self.start_position, self.end_position = 0, 0 doc_tokens = [] char_to_word_offset = [] prev_is_whitespace = True # Split on whitespace so that different tokens may be attributed to their original position. for c in self.context_text: if self._is_whitespace(c): prev_is_whitespace = True else: if prev_is_whitespace: doc_tokens.append(c) else: doc_tokens[-1] += c prev_is_whitespace = False char_to_word_offset.append(len(doc_tokens) - 1) self.doc_tokens = doc_tokens self.char_to_word_offset = char_to_word_offset # Start and end positions only has a value during evaluation. if start_position_character is not None and not is_impossible: self.start_position = char_to_word_offset[start_position_character] self.end_position = char_to_word_offset[ min(start_position_character + len(answer_text) - 1, len(char_to_word_offset) - 1) ] def _is_whitespace(self, c): """Check whether character is whitespace""" if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F: return True return False class InputFeatures: """A single set of features of data.""" def __init__(self, unique_id, example_index, doc_span_index, tokens, token_to_orig_map, token_is_max_context, input_ids, input_mask, token_type_id, start_position=-1, end_position=-1, is_impossible=False): self.unique_id = unique_id self.example_index = example_index self.doc_span_index = doc_span_index self.tokens = tokens self.token_to_orig_map = token_to_orig_map self.token_is_max_context = token_is_max_context self.input_ids = input_ids self.input_mask = input_mask self.token_type_id = token_type_id self.start_position = start_position self.end_position = end_position self.is_impossible = is_impossible pengcheng.mind-test/mindformers/dataset/dataloader/wmt16_dataloader.py000066400000000000000000000104101471400134000264670ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """WMT16 DataLoader.""" import os from mindspore.dataset import GeneratorDataset from mindformers.tools import logger from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class WMT16DataLoader: """WMT16 Dataloader""" _default_column_names = ["source", "target"] def __new__(cls, dataset_dir, column_names=None, stage="train", num_shards=1, shard_id=0): """ WMT16 Dataloader API Args: dataset_dir: the directory to dataset column_names: the output column names, a tuple or a list of string with length 2 stage: the supported `option` are in ["train"、"test"、"del"、"all"] Return: a GeneratorDataset for WMT16 dataset """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if column_names is None: column_names = cls._default_column_names logger.info("The column_names to the WMT16DataLoader is None, so assign it with default_column_names %s", cls._default_column_names) if column_names and not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 2, but got {type(column_names)}") if len(column_names) != 2: raise ValueError(f"the length of column_names should be 2," f" but got {len(column_names)}") if not isinstance(column_names[0], str) or not isinstance(column_names[1], str): raise ValueError(f"the item type of column_names should be string," f" but got {type(column_names[0])} and {type(column_names[1])}") dataset = WMT16DataSet(dataset_dir, stage) return GeneratorDataset(dataset, column_names, num_shards=num_shards, shard_id=shard_id) def read_text(train_file): """Read the text files and return a list.""" with open(train_file) as fp: data = [] for line in fp: line = line.strip() if line: data.append(line) return data class WMT16DataSet: """WMT16 DataSet""" def __init__(self, dataset_dir, stage="train"): """ WMT16DataSet Dataset Args: dataset_dir: the directory to wmt16 dataset stage: the supported key word are in ["train"、"test"、"del"、"all"] Return: an iterable dataset for wmt16 dataset """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") self.dataset_dir = dataset_dir self.stage = stage dataset_dict = dict() def read_and_add_to_stage(cur_stage): source_path = os.path.join(dataset_dir, f'{cur_stage}.source') src = read_text(source_path) target_source = os.path.join(dataset_dir, f'{cur_stage}.target') tgt = read_text(target_source) src_tgt_pair = list(zip(src, tgt)) return src_tgt_pair logger.info("Start to read the raw data from the disk %s.", dataset_dir) if 'stage' != 'all': dataset_dict[stage] = read_and_add_to_stage(stage) else: for item in ['train', 'dev', 'test']: dataset_dict[stage] = read_and_add_to_stage(item) self.dataset_dict = dataset_dict def __getitem__(self, item): return self.dataset_dict[self.stage][item] def __len__(self): return len(self.dataset_dict[self.stage]) pengcheng.mind-test/mindformers/dataset/image_text_retrieval_dataset.py000066400000000000000000000051401471400134000271450ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image-text Retrieval Dataset.""" import os import mindspore.common.dtype as mstype import mindspore.dataset as ds import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .base_dataset import BaseDataset from .transforms import build_transforms @MindFormerRegister.register(MindFormerModuleType.DATASET) class ImageToTextRetrievalDataset(BaseDataset): """Image-text Retrieval Dataset for filip fine-tuning and evaluation.""" def __new__(cls, dataset_config: dict = None): logger.info("Now Create Image-text Retrieval Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = ds.MindDataset(dataset_config.data_loader.dataset_dir, shuffle=dataset_config.data_loader.shuffle, num_shards=device_num, shard_id=rank_id) transforms = build_transforms(dataset_config.transforms) if transforms is not None: dataset = get_dataset_map(dataset, transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing, input_columns="image", output_columns=['image']) type_cast_op = C.TypeCast(mstype.float32) dataset = get_dataset_map(dataset, type_cast_op, input_columns="image", output_columns=['image']) dataset = dataset.project(["image", "token"]) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder) return dataset pengcheng.mind-test/mindformers/dataset/img_cls_dataset.py000066400000000000000000000112541471400134000243620ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Dataset.""" import os import mindspore.dataset.transforms.c_transforms as C import mindspore.common.dtype as mstype from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .transforms import build_transforms from .sampler import build_sampler from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class ImageCLSDataset(BaseDataset): """ Image Classification Dataset API. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import ImageCLSDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['image_classification']['vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/image_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='ImageCLSDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = ImageCLSDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Image Classification Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) sampler = build_sampler(dataset_config.sampler) type_cast_op = C.TypeCast(mstype.int32) if sampler is not None: dataset = dataset.use_sampler(sampler) if transforms is not None: dataset = get_dataset_map(dataset, transforms, input_columns=dataset_config.input_columns[0], num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = get_dataset_map(dataset, type_cast_op, input_columns=dataset_config.input_columns[1], num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) if not dataset_config.do_eval and dataset_config.mixup_op is not None: mixup_op = build_transforms(class_name="Mixup", **dataset_config.mixup_op) dataset = get_dataset_map(dataset, mixup_op, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.output_columns) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-test/mindformers/dataset/keyword_gen_dataset.py000066400000000000000000000354141471400134000252660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Keyword Generation Dataset.""" import copy import os import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C import numpy as np from mindformers.models.base_tokenizer import BaseTokenizer from mindformers.dataset.base_dataset import BaseDataset from mindformers.dataset.dataloader import build_dataset_loader from mindformers.models.build_tokenizer import build_tokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerModuleType, MindFormerRegister from mindformers.version_control import get_dataset_map @MindFormerRegister.register(MindFormerModuleType.DATASET) class KeyWordGenDataset(BaseDataset): """Keyword generation dataset. Examples: >>> from mindformers.dataset.dataloader.adgen_dataloader import ADGenDataLoader >>> from mindformers.dataset import build_dataset >>> from mindformers import MindFormerConfig >>> cfg = MindFormerConfig("./configs/glm/run_glm_6b_finetune.yaml") >>> dataset = build_dataset(cfg.eval_dataset_task) >>> for item in dataset.create_dict_iterator(): >>> print(item) >>> break """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Keyword Generation Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config.rank_id = rank_id dataset_config.device_num = device_num if isinstance(dataset_config.tokenizer, BaseTokenizer): cls.tokenizer = dataset_config.tokenizer else: cls.tokenizer = build_tokenizer(dataset_config.tokenizer) cls.ignore_pad_token_for_loss = dataset_config.ignore_pad_token_for_loss cls.max_source_length = dataset_config.max_source_length cls.max_target_length = dataset_config.max_target_length cls.max_seq_length = cls.max_source_length + cls.max_target_length cls.phase = dataset_config.data_loader.phase cls.version = dataset_config.data_loader.version if dataset_config.data_loader.type != 'MindDataset': dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) return dataset @classmethod def _tokenizer_map(cls, dataset, tokenizer_config): """Maps the tokenizer on the source and the output""" phase = cls.phase version = cls.version if cls.version else 1 logger.info("Start tokenize on the dataset using tokenizer: %s", tokenizer_config) if version == 2: train_dataset_function = cls.train_dataset_functionv2 train_output_columns = ["input_ids", "labels"] eval_dataset_function = cls.eval_dataset_functionv2 else: train_dataset_function = cls.train_dataset_function train_output_columns = ["input_ids", "labels", "position_ids", "attention_mask"] eval_dataset_function = cls.eval_dataset_function input_columns = ["prompt", "answer"] eval_output_columns = ["input_ids", "labels"] # Avoid to_json error when summary monitor is opened def train_dataset_func(prompt, answer): return train_dataset_function(prompt, answer) def eval_dataset_func(prompt, answer): return eval_dataset_function(prompt, answer) if phase == "train": dataset = get_dataset_map(dataset, train_dataset_func, input_columns=input_columns, output_columns=train_output_columns) dataset = dataset.project(columns=train_output_columns) if phase == "eval": dataset = get_dataset_map(dataset, eval_dataset_func, input_columns=input_columns, output_columns=eval_output_columns) dataset = dataset.project(columns=eval_output_columns) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id}) dataset = cls._tokenizer_map(dataset, dataset_config.tokenizer) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" dataset_config = copy.deepcopy(dataset_config) dataset_files = [] if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if file.endswith(".mindrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if data_dir.endswith(".mindrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") logger.info("Using args %s to instance the dataset.", dataset_config.data_loader) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id, 'columns_list': dataset_config.input_columns}) return dataset @classmethod def train_dataset_function(cls, prompt, answer): """generates train dataset""" prompt, answer = prompt.tolist(), answer.tolist() prompt_ids = cls.tokenizer.encode(text=prompt, add_special_tokens=False) answer_ids = cls.tokenizer.encode(text=answer, add_special_tokens=False) if len(prompt_ids) > cls.max_source_length - 1: prompt_ids = prompt_ids[: cls.max_source_length - 1] if len(answer_ids) > cls.max_target_length - 2: answer_ids = answer_ids[: cls.max_target_length - 2] input_ids = cls.tokenizer.build_inputs_with_special_tokens(prompt_ids, answer_ids) context_length = input_ids.index(cls.tokenizer.bos_token_id) mask_position = context_length - 1 label = [-100] * context_length + input_ids[mask_position + 2:] # +1 for logits shift pad_len = cls.max_seq_length - len(input_ids) input_ids = input_ids + [cls.tokenizer.pad_token_id] * pad_len label = label + [cls.tokenizer.pad_token_id] * (pad_len + 1) # +1 for logits shift if cls.ignore_pad_token_for_loss: label = [(l if l != cls.tokenizer.pad_token_id else -100) for l in label] position_ids = cls.create_position_ids(np.array(input_ids)) attention_mask = cls.get_masks(np.array(input_ids)) return input_ids, label, position_ids, attention_mask @classmethod def train_dataset_functionv2(cls, prompt, answer): """generates train dataset""" max_seq_length = cls.max_source_length + cls.max_target_length + 1 prompt, answer = prompt.tolist(), answer.tolist() history = None prompt = cls.tokenizer.build_prompt(prompt, history) prompt_ids = cls.tokenizer.encode(text=prompt, add_special_tokens=True, max_length=cls.max_source_length) answer_ids = cls.tokenizer.encode(text=answer, add_special_tokens=False, max_length=cls.max_target_length) if len(prompt_ids) > cls.max_source_length - 1: prompt_ids = prompt_ids[: cls.max_source_length - 1] if len(answer_ids) > cls.max_target_length - 2: answer_ids = answer_ids[: cls.max_target_length - 2] context_length = len(prompt_ids) input_ids = prompt_ids + answer_ids + [cls.tokenizer.eos_token_id] labels = [cls.tokenizer.pad_token_id] * context_length + answer_ids[1:] + [cls.tokenizer.eos_token_id] pad_len = max_seq_length - len(input_ids) input_ids = input_ids + [cls.tokenizer.pad_token_id] * pad_len labels = labels + [cls.tokenizer.pad_token_id] * (pad_len + 1) # +1 for logits shift if cls.ignore_pad_token_for_loss: labels = [(l if l != cls.tokenizer.pad_token_id else -100) for l in labels] return input_ids, labels @classmethod def eval_dataset_functionv2(cls, prompt, answer): """generates eval dataset""" prompt, answer = prompt.tolist(), answer.tolist() history = None prompt = cls.tokenizer.build_prompt(prompt, history) if len(prompt) > cls.max_source_length - 1: prompt = prompt[: cls.max_source_length - 1] if len(answer) > cls.max_target_length - 1: answer = answer[: cls.max_target_length - 1] input_ids = cls.tokenizer.encode(text=prompt, add_special_tokens=True, max_length=cls.max_source_length) label = cls.tokenizer.encode(text=answer, add_special_tokens=True, max_length=cls.max_target_length) pad_len = cls.max_source_length - len(input_ids) input_ids = input_ids + [cls.tokenizer.pad_token_id] * pad_len pad_len = cls.max_target_length - len(label) label = label + [cls.tokenizer.pad_token_id] * pad_len return input_ids, label @classmethod def eval_dataset_function(cls, prompt, answer): """generates eval dataset""" prompt, answer = prompt.tolist(), answer.tolist() if len(prompt) > cls.max_source_length - 2: prompt = prompt[: cls.max_source_length - 2] if len(answer) > cls.max_target_length - 2: answer = answer[: cls.max_target_length - 2] input_ids = cls.tokenizer.encode(text=prompt, add_special_tokens=True) label = cls.tokenizer.encode(text=answer, add_special_tokens=True) pad_len = cls.max_source_length - len(input_ids) input_ids = input_ids + [cls.tokenizer.pad_token_id] * pad_len pad_len = cls.max_target_length - len(label) label = label + [cls.tokenizer.pad_token_id] * pad_len return input_ids, label @classmethod def get_masks(cls, input_ids, bos_token_id=130004): """generate mask from input id""" seq_length = input_ids.shape[0] mask = bos_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(input_ids, mask) # 要求input_ids中有且仅有一个bos_token_id context_lengths = np.argwhere(mask)[:, -1] attention_mask = np.tril(np.ones((seq_length, seq_length), dtype=np.float32)) for context_length in context_lengths: attention_mask[:, :context_length] = 1 attention_mask = np.logical_not(attention_mask.astype(np.bool_)) attention_mask = attention_mask.astype(np.float32) attention_mask = np.expand_dims(attention_mask, 0) return attention_mask @classmethod def get_position_ids(cls, input_ids, mask_positions, use_gmasks=None, bos_token_id=130004, position_encoding_2d=True): """generate position ids from input id and mask positions""" seq_length = input_ids.shape[0] if use_gmasks is None: use_gmasks = [False] mask = bos_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(input_ids, mask) # 要求input_ids中有且仅有一个bos_token_id context_lengths = np.argwhere(mask)[:, -1] if position_encoding_2d: position_ids = np.arange(seq_length, dtype=np.int64) for i, context_length in enumerate(context_lengths): position_ids[context_length:] = mask_positions[i] block_position_ids = [np.concatenate(( np.zeros(context_length, dtype=np.int64), np.arange(seq_length - context_length, dtype=np.int64) + 1 )) for context_length in context_lengths] block_position_ids = np.stack(block_position_ids, axis=0).squeeze() position_ids = np.stack((position_ids, block_position_ids), axis=0) else: position_ids = np.arange(seq_length, dtype=np.int64) for i, context_length in enumerate(context_lengths): if not use_gmasks[i]: position_ids[context_length:] = mask_positions[i] return position_ids @classmethod def create_position_ids(cls, input_ids, gmask_token_id=130001): """generate position ids from input id""" seq_length = input_ids.shape[0] seqs = input_ids # 要求input_ids中, 每行有且仅有一个gMASK use_gmasks = gmask_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(seqs, use_gmasks) mask_positions = np.argwhere(mask)[:, -1] position_ids = cls.get_position_ids(input_ids, mask_positions=mask_positions, use_gmasks=use_gmasks) return position_ids pengcheng.mind-test/mindformers/dataset/labels.py000066400000000000000000000375001471400134000225040ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # This file was refer to project: # https://huggingface.co/WinKawaks/vit-tiny-patch16-224/blob/main/config.json # ============================================================================ """labels of imagenet""" imagenet_labels = [ "tench", "goldfish", "great white shark", "tiger shark", "hammerhead", "electric ray", "stingray", "cock", "hen", "ostrich", "brambling", "goldfinch", "house finch", "junco", "indigo bunting", "robin", "bulbul", "jay", "magpie", "chickadee", "water ouzel", "kite", "bald eagle", "vulture", "great grey owl", "European fire salamander", "common newt", "eft", "spotted salamander", "axolotl", "bullfrog", "tree frog", "tailed frog", "loggerhead", "leatherback turtle", "mud turtle", "terrapin", "box turtle", "banded gecko", "common iguana", "American chameleon", "whiptail", "agama", "frilled lizard", "alligator lizard", "Gila monster", "green lizard", "African chameleon", "Komodo dragon", "African crocodile", "American alligator", "triceratops", "thunder snake", "ringneck snake", "hognose snake", "green snake", "king snake", "garter snake", "water snake", "vine snake", "night snake", "boa constrictor", "rock python", "Indian cobra", "green mamba", "sea snake", "horned viper", "diamondback", "sidewinder", "trilobite", "harvestman", "scorpion", "black and gold garden spider", "barn spider", "garden spider", "black widow", "tarantula", "wolf spider", "tick", "centipede", "black grouse", "ptarmigan", "ruffed grouse", "prairie chicken", "peacock", "quail", "partridge", "African grey", "macaw", "sulphur-crested cockatoo", "lorikeet", "coucal", "bee eater", "hornbill", "hummingbird", "jacamar", "toucan", "drake", "red-breasted merganser", "goose", "black swan", "tusker", "echidna", "platypus", "wallaby", "koala", "wombat", "jellyfish", "sea anemone", "brain coral", "flatworm", "nematode", "conch", "snail", "slug", "sea slug", "chiton", "chambered nautilus", "Dungeness crab", "rock crab", "fiddler crab", "king crab", "American lobster", "spiny lobster", "crayfish", "hermit crab", "isopod", "white stork", "black stork", "spoonbill", "flamingo", "little blue heron", "American egret", "bittern", "crane", "limpkin", "European gallinule", "American coot", "bustard", "ruddy turnstone", "red-backed sandpiper", "redshank", "dowitcher", "oystercatcher", "pelican", "king penguin", "albatross", "grey whale", "killer whale", "dugong", "sea lion", "Chihuahua", "Japanese spaniel", "Maltese dog", "Pekinese", "Shih-Tzu", "Blenheim spaniel", "papillon", "toy terrier", "Rhodesian ridgeback", "Afghan hound", "basset", "beagle", "bloodhound", "bluetick", "black-and-tan coonhound", "Walker hound", "English foxhound", "redbone", "borzoi", "Irish wolfhound", "Italian greyhound", "whippet", "Ibizan hound", "Norwegian elkhound", "otterhound", "Saluki", "Scottish deerhound", "Weimaraner", "Staffordshire bullterrier", "American Staffordshire terrier", "Bedlington terrier", "Border terrier", "Kerry blue terrier", "Irish terrier", "Norfolk terrier", "Norwich terrier", "Yorkshire terrier", "wire-haired fox terrier", "Lakeland terrier", "Sealyham terrier", "Airedale", "cairn", "Australian terrier", "Dandie Dinmont", "Boston bull", "miniature schnauzer", "giant schnauzer", "standard schnauzer", "Scotch terrier", "Tibetan terrier", "silky terrier", "soft-coated wheaten terrier", "West Highland white terrier", "Lhasa", "flat-coated retriever", "curly-coated retriever", "golden retriever", "Labrador retriever", "Chesapeake Bay retriever", "German short-haired pointer", "vizsla", "English setter", "Irish setter", "Gordon setter", "Brittany spaniel", "clumber", "English springer", "Welsh springer spaniel", "cocker spaniel", "Sussex spaniel", "Irish water spaniel", "kuvasz", "schipperke", "groenendael", "malinois", "briard", "kelpie", "komondor", "Old English sheepdog", "Shetland sheepdog", "collie", "Border collie", "Bouvier des Flandres", "Rottweiler", "German shepherd", "Doberman", "miniature pinscher", "Greater Swiss Mountain dog", "Bernese mountain dog", "Appenzeller", "EntleBucher", "boxer", "bull mastiff", "Tibetan mastiff", "French bulldog", "Great Dane", "Saint Bernard", "Eskimo dog", "malamute", "Siberian husky", "dalmatian", "affenpinscher", "basenji", "pug", "Leonberg", "Newfoundland", "Great Pyrenees", "Samoyed", "Pomeranian", "chow", "keeshond", "Brabancon griffon", "Pembroke", "Cardigan", "toy poodle", "miniature poodle", "standard poodle", "Mexican hairless", "timber wolf", "white wolf", "red wolf", "coyote", "dingo", "dhole", "African hunting dog", "hyena", "red fox", "kit fox", "Arctic fox", "grey fox", "tabby", "tiger cat", "Persian cat", "Siamese cat", "Egyptian cat", "cougar", "lynx", "leopard", "snow leopard", "jaguar", "lion", "tiger", "cheetah", "brown bear", "American black bear", "ice bear", "sloth bear", "mongoose", "meerkat", "tiger beetle", "ladybug", "ground beetle", "long-horned beetle", "leaf beetle", "dung beetle", "rhinoceros beetle", "weevil", "fly", "bee", "ant", "grasshopper", "cricket", "walking stick", "cockroach", "mantis", "cicada", "leafhopper", "lacewing", "dragonfly", "damselfly", "admiral", "ringlet", "monarch", "cabbage butterfly", "sulphur butterfly", "lycaenid", "starfish", "sea urchin", "sea cucumber", "wood rabbit", "hare", "Angora", "hamster", "porcupine", "fox squirrel", "marmot", "beaver", "guinea pig", "sorrel", "zebra", "hog", "wild boar", "warthog", "hippopotamus", "ox", "water buffalo", "bison", "ram", "bighorn", "ibex", "hartebeest", "impala", "gazelle", "Arabian camel", "llama", "weasel", "mink", "polecat", "black-footed ferret", "otter", "skunk", "badger", "armadillo", "three-toed sloth", "orangutan", "gorilla", "chimpanzee", "gibbon", "siamang", "guenon", "patas", "baboon", "macaque", "langur", "colobus", "proboscis monkey", "marmoset", "capuchin", "howler monkey", "titi", "spider monkey", "squirrel monkey", "Madagascar cat", "indri", "Indian elephant", "African elephant", "lesser panda", "giant panda", "barracouta", "eel", "coho", "rock beauty", "anemone fish", "sturgeon", "gar", "lionfish", "puffer", "abacus", "abaya", "academic gown", "accordion", "acoustic guitar", "aircraft carrier", "airliner", "airship", "altar", "ambulance", "amphibian", "analog clock", "apiary", "apron", "ashcan", "assault rifle", "backpack", "bakery", "balance beam", "balloon", "ballpoint", "Band Aid", "banjo", "bannister", "barbell", "barber chair", "barbershop", "barn", "barometer", "barrel", "barrow", "baseball", "basketball", "bassinet", "bassoon", "bathing cap", "bath towel", "bathtub", "beach wagon", "beacon", "beaker", "bearskin", "beer bottle", "beer glass", "bell cote", "bib", "bicycle-built-for-two", "bikini", "binder", "binoculars", "birdhouse", "boathouse", "bobsled", "bolo tie", "bonnet", "bookcase", "bookshop", "bottlecap", "bow", "bow tie", "brass", "brassiere", "breakwater", "breastplate", "broom", "bucket", "buckle", "bulletproof vest", "bullet train", "butcher shop", "cab", "caldron", "candle", "cannon", "canoe", "can opener", "cardigan", "car mirror", "carousel", "carpenter's kit", "carton", "car wheel", "cash machine", "cassette", "cassette player", "castle", "catamaran", "CD player", "cello", "cellular telephone", "chain", "chainlink fence", "chain mail", "chain saw", "chest", "chiffonier", "chime", "china cabinet", "Christmas stocking", "church", "cinema", "cleaver", "cliff dwelling", "cloak", "clog", "cocktail shaker", "coffee mug", "coffeepot", "coil", "combination lock", "computer keyboard", "confectionery", "container ship", "convertible", "corkscrew", "cornet", "cowboy boot", "cowboy hat", "cradle", "crane", "crash helmet", "crate", "crib", "Crock Pot", "croquet ball", "crutch", "cuirass", "dam", "desk", "desktop computer", "dial telephone", "diaper", "digital clock", "digital watch", "dining table", "dishrag", "dishwasher", "disk brake", "dock", "dogsled", "dome", "doormat", "drilling platform", "drum", "drumstick", "dumbbell", "Dutch oven", "electric fan", "electric guitar", "electric locomotive", "entertainment center", "envelope", "espresso maker", "face powder", "feather boa", "file", "fireboat", "fire engine", "fire screen", "flagpole", "flute", "folding chair", "football helmet", "forklift", "fountain", "fountain pen", "four-poster", "freight car", "French horn", "frying pan", "fur coat", "garbage truck", "gasmask", "gas pump", "goblet", "go-kart", "golf ball", "golfcart", "gondola", "gong", "gown", "grand piano", "greenhouse", "grille", "grocery store", "guillotine", "hair slide", "hair spray", "half track", "hammer", "hamper", "hand blower", "hand-held computer", "handkerchief", "hard disc", "harmonica", "harp", "harvester", "hatchet", "holster", "home theater", "honeycomb", "hook", "hoopskirt", "horizontal bar", "horse cart", "hourglass", "iPod", "iron", "jack-o'-lantern", "jean", "jeep", "jersey", "jigsaw puzzle", "jinrikisha", "joystick", "kimono", "knee pad", "knot", "lab coat", "ladle", "lampshade", "laptop", "lawn mower", "lens cap", "letter opener", "library", "lifeboat", "lighter", "limousine", "liner", "lipstick", "Loafer", "lotion", "loudspeaker", "loupe", "lumbermill", "magnetic compass", "mailbag", "mailbox", "maillot", "maillot", "manhole cover", "maraca", "marimba", "mask", "matchstick", "maypole", "maze", "measuring cup", "medicine chest", "megalith", "microphone", "microwave", "military uniform", "milk can", "minibus", "miniskirt", "minivan", "missile", "mitten", "mixing bowl", "mobile home", "Model T", "modem", "monastery", "monitor", "moped", "mortar", "mortarboard", "mosque", "mosquito net", "motor scooter", "mountain bike", "mountain tent", "mouse", "mousetrap", "moving van", "muzzle", "nail", "neck brace", "necklace", "nipple", "notebook", "obelisk", "oboe", "ocarina", "odometer", "oil filter", "organ", "oscilloscope", "overskirt", "oxcart", "oxygen mask", "packet", "paddle", "paddlewheel", "padlock", "paintbrush", "pajama", "palace", "panpipe", "paper towel", "parachute", "parallel bars", "park bench", "parking meter", "passenger car", "patio", "pay-phone", "pedestal", "pencil box", "pencil sharpener", "perfume", "Petri dish", "photocopier", "pick", "pickelhaube", "picket fence", "pickup", "pier", "piggy bank", "pill bottle", "pillow", "ping-pong ball", "pinwheel", "pirate", "pitcher", "plane", "planetarium", "plastic bag", "plate rack", "plow", "plunger", "Polaroid camera", "pole", "police van", "poncho", "pool table", "pop bottle", "pot", "potter's wheel", "power drill", "prayer rug", "printer", "prison", "projectile", "projector", "puck", "punching bag", "purse", "quill", "quilt", "racer", "racket", "radiator", "radio", "radio telescope", "rain barrel", "recreational vehicle", "reel", "reflex camera", "refrigerator", "remote control", "restaurant", "revolver", "rifle", "rocking chair", "rotisserie", "rubber eraser", "rugby ball", "rule", "running shoe", "safe", "safety pin", "saltshaker", "sandal", "sarong", "sax", "scabbard", "scale", "school bus", "schooner", "scoreboard", "screen", "screw", "screwdriver", "seat belt", "sewing machine", "shield", "shoe shop", "shoji", "shopping basket", "shopping cart", "shovel", "shower cap", "shower curtain", "ski", "ski mask", "sleeping bag", "slide rule", "sliding door", "slot", "snorkel", "snowmobile", "snowplow", "soap dispenser", "soccer ball", "sock", "solar dish", "sombrero", "soup bowl", "space bar", "space heater", "space shuttle", "spatula", "speedboat", "spider web", "spindle", "sports car", "spotlight", "stage", "steam locomotive", "steel arch bridge", "steel drum", "stethoscope", "stole", "stone wall", "stopwatch", "stove", "strainer", "streetcar", "stretcher", "studio couch", "stupa", "submarine", "suit", "sundial", "sunglass", "sunglasses", "sunscreen", "suspension bridge", "swab", "sweatshirt", "swimming trunks", "swing", "switch", "syringe", "table lamp", "tank", "tape player", "teapot", "teddy", "television", "tennis ball", "thatch", "theater curtain", "thimble", "thresher", "throne", "tile roof", "toaster", "tobacco shop", "toilet seat", "torch", "totem pole", "tow truck", "toyshop", "tractor", "trailer truck", "tray", "trench coat", "tricycle", "trimaran", "tripod", "triumphal arch", "trolleybus", "trombone", "tub", "turnstile", "typewriter keyboard", "umbrella", "unicycle", "upright", "vacuum", "vase", "vault", "velvet", "vending machine", "vestment", "viaduct", "violin", "volleyball", "waffle iron", "wall clock", "wallet", "wardrobe", "warplane", "washbasin", "washer", "water bottle", "water jug", "water tower", "whiskey jug", "whistle", "wig", "window screen", "window shade", "Windsor tie", "wine bottle", "wing", "wok", "wooden spoon", "wool", "worm fence", "wreck", "yawl", "yurt", "web site", "comic book", "crossword puzzle", "street sign", "traffic light", "book jacket", "menu", "plate", "guacamole", "consomme", "hot pot", "trifle", "ice cream", "ice lolly", "French loaf", "bagel", "pretzel", "cheeseburger", "hotdog", "mashed potato", "head cabbage", "broccoli", "cauliflower", "zucchini", "spaghetti squash", "acorn squash", "butternut squash", "cucumber", "artichoke", "bell pepper", "cardoon", "mushroom", "Granny Smith", "strawberry", "orange", "lemon", "fig", "pineapple", "banana", "jackfruit", "custard apple", "pomegranate", "hay", "carbonara", "chocolate sauce", "dough", "meat loaf", "pizza", "potpie", "burrito", "red wine", "espresso", "cup", "eggnog", "alp", "bubble", "cliff", "coral reef", "geyser", "lakeside", "promontory", "sandbar", "seashore", "valley", "volcano", "ballplayer", "groom", "scuba diver", "rapeseed", "daisy", "yellow lady's slipper", "corn", "acorn", "hip", "buckeye", "coral fungus", "agaric", "gyromitra", "stinkhorn", "earthstar", "hen-of-the-woods", "bolete", "ear", "toilet tissue", ] """labels of cluener""" cluener_labels = ["O", "B-address", "B-book", "B-company", "B-game", "B-government", "B-movie", "B-name", "B-organization", "B-position", "B-scene", "I-address", "I-book", "I-company", "I-game", "I-government", "I-movie", "I-name", "I-organization", "I-position", "I-scene", "S-address", "S-book", "S-company", "S-game", "S-government", "S-movie", "S-name", "S-organization", "S-position", "S-scene"] """labels of mnli""" mnli_labels = ["contradiction", "entailment", "neutral"] """labels of cola""" cola_labels = ["unacceptable", "acceptable"] """labels of sst2""" sst2_labels = ["negative", "positive"] """labels of imdb""" imdb_labels = ["negative", "positive"] """labels of ag news""" ag_news_labels = ["world", "sports", "business", "sci/tech"] # labels dict labels = {"imagenet": imagenet_labels, "cluener": cluener_labels, "mnli": mnli_labels, "cola": cola_labels, "sst2": sst2_labels, "imdb": imdb_labels, "ag_news": ag_news_labels} pengcheng.mind-test/mindformers/dataset/mask/000077500000000000000000000000001471400134000216165ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/mask/__init__.py000066400000000000000000000014501471400134000237270ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Mask API.""" from .build_mask import build_mask from .vision_mask import * __all__ = [] __all__.extend(vision_mask.__all__) pengcheng.mind-test/mindformers/dataset/mask/__pycache__/000077500000000000000000000000001471400134000240265ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/mask/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004571471400134000302240ustar00rootroot00000000000000a ?cf(@s,dZddlmZddlTgZeejdS)zMindFormers Mask API.) build_mask)*N)__doc__rZ vision_mask__all__extendrrL/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mask/__init__.pys pengcheng.mind-test/mindformers/dataset/mask/__pycache__/build_mask.cpython-39.pyc000066400000000000000000000030471471400134000305750ustar00rootroot00000000000000a ?cf@s2dZddlmZmZmZdeeeedddZdS) zBuild Mask API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfigN mask_policy)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)a'Build mask For MindFormer. Instantiate the mask from MindFormerRegister's registry. Args: config (dict): The task mask's config. Default: None. default_args (dict): The default argument of mask API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'mask'. class_name (str): The class name of mask API. Default: None. Return: The function instance of mask API. Examples: >>> from mindformers import build_mask >>> mask_config = {'type': 'SimMask', 'input_size': 224} >>> # 1) use config dict to build mask >>> mask_from_config = build_mask(mask_config) >>> # 2) use class name to build mask >>> mask_class_name = build_mask(class_name='SimMask', input_size=224) N)r) isinstancedictrrget_instance_from_cfgr MASK_POLICY get_instance)rrrr kwargsrN/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mask/build_mask.py build_masksr)NNrN)__doc__mindformers.tools.registerrrrr strrrrrrspengcheng.mind-test/mindformers/dataset/mask/__pycache__/vision_mask.cpython-39.pyc000066400000000000000000000057031471400134000310060ustar00rootroot00000000000000a ?cf@spdZddlZddlmZddlmZmZddgZe ej Gdddej Z e ej Gdddej Z dS) zSelf-Define Vision Mask Policy.N) py_transforms)MindFormerRegisterMindFormerModuleTypeSimMaskMaeMaskcs2eZdZdZd fdd Zdd Zd d ZZS) rzSimMIM Mask Policy. 333333?cstt|||_||_||_||_|j|jdkrNtd|jd|j|j|jdkrvtd|jd|j|j|j|_|j|j|_ |jd|_ t t |j |j|_dS)Nrz;input_size % mask_patch_size must be 0, but get input_size z and mask_patch_size zFmask_patch_size % model_patch_size must be 0, but get mask_patch_size z and model_patch_size )superr__init__ input_sizemask_patch_sizemodel_patch_size mask_ratio ValueError rand_sizescale token_countintnpceil mask_count)selfrrrr __class__O/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mask/vision_mask.pyr s&  zSimMask.__init__cCshtj|jd|j}tj|jtjd}d||<||j|jf}|j |j ddj |j dd}||fS)N)dtyperaxis) rrandom permutationrrzerosint32reshaperrepeatr)rimgZmask_idxmaskrrr__call__0s zSimMask.__call__cCsdS)NzMask generator for simmin arch.rrrrr__repr__9szSimMask.__repr__)rrr r __name__ __module__ __qualname____doc__r r+r- __classcell__rrrrrs cs2eZdZdZd fdd ZddZd d ZZS) rzMAE Mask Policy.rr ?csRtt|d|kr"dks,ntd||d|_td||j|_dS)Nrr zLmasking ratio must be kept between 0 and 1, but get mask_ratio {mask_ratio}.r )r rr r num_patchesrkeep_num)rr patch_sizerrrrr @s zMaeMask.__init__cCs|tjtjj|jfdddtj}tj|ddtj}t|jftj}d|d|j<|d|j}||||f}|S)N)sizerr!) rargsortr#uniformr5astyper&onesr6)rZimgsZ rand_indicesZ ids_restorer*Z unmask_indexoutrrrr+Is zMaeMask.__call__cCsdS)NzMask generator for mae arch.rr,rrrr-SszMaeMask.__repr__)rr r4r.rrrrr=s  )r2numpyrZmindspore.dataset.transformsrmindformers.tools.registerrr__all__register MASK_POLICYPyTensorOperationrrrrrrs  # pengcheng.mind-test/mindformers/dataset/mask/build_mask.py000066400000000000000000000042241471400134000243040ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Mask API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_mask( config: dict = None, default_args: dict = None, module_type: str = 'mask_policy', class_name: str = None, **kwargs): r"""Build mask For MindFormer. Instantiate the mask from MindFormerRegister's registry. Args: config (dict): The task mask's config. Default: None. default_args (dict): The default argument of mask API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'mask'. class_name (str): The class name of mask API. Default: None. Return: The function instance of mask API. Examples: >>> from mindformers import build_mask >>> mask_config = {'type': 'SimMask', 'input_size': 224} >>> # 1) use config dict to build mask >>> mask_from_config = build_mask(mask_config) >>> # 2) use class name to build mask >>> mask_class_name = build_mask(class_name='SimMask', input_size=224) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.MASK_POLICY, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-test/mindformers/dataset/mask/vision_mask.py000066400000000000000000000072601471400134000245170ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/adapter-hub/adapter-transformers/blob/master/examples/pytorch/image-pretraining/run_mim.py # ============================================================================ """Self-Define Vision Mask Policy.""" import numpy as np from mindspore.dataset.transforms import py_transforms from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['SimMask', 'MaeMask'] @MindFormerRegister.register(MindFormerModuleType.MASK_POLICY) class SimMask(py_transforms.PyTensorOperation): """SimMIM Mask Policy.""" def __init__(self, input_size=192, mask_patch_size=32, model_patch_size=4, mask_ratio=0.6): super(SimMask, self).__init__() self.input_size = input_size self.mask_patch_size = mask_patch_size self.model_patch_size = model_patch_size self.mask_ratio = mask_ratio if self.input_size % self.mask_patch_size != 0: raise ValueError(f"input_size % mask_patch_size must be 0, but get input_size {self.input_size} and " f"mask_patch_size {self.mask_patch_size}") if self.mask_patch_size % self.model_patch_size != 0: raise ValueError(f"mask_patch_size % model_patch_size must be 0, but get mask_patch_size " f"{self.mask_patch_size} and model_patch_size {self.model_patch_size}") self.rand_size = self.input_size // self.mask_patch_size self.scale = self.mask_patch_size // self.model_patch_size self.token_count = self.rand_size ** 2 self.mask_count = int(np.ceil(self.token_count * self.mask_ratio)) def __call__(self, img): mask_idx = np.random.permutation(self.token_count)[:self.mask_count] mask = np.zeros(self.token_count, dtype=np.int32) mask[mask_idx] = 1 mask = mask.reshape((self.rand_size, self.rand_size)) mask = mask.repeat(self.scale, axis=0).repeat(self.scale, axis=1) return img, mask def __repr__(self): return "Mask generator for simmin arch." @MindFormerRegister.register(MindFormerModuleType.MASK_POLICY) class MaeMask(py_transforms.PyTensorOperation): """MAE Mask Policy.""" def __init__(self, input_size=192, patch_size=4, mask_ratio=0.75): super(MaeMask, self).__init__() if not 0 < mask_ratio < 1: raise ValueError('masking ratio must be kept between 0 and 1, but get mask_ratio {mask_ratio}.') # seq_length self.num_patches = (input_size // patch_size) ** 2 # seq masked number self.keep_num = int((1 - mask_ratio) * self.num_patches) def __call__(self, imgs): rand_indices = np.argsort( np.random.uniform(size=(self.num_patches,)), axis=0).astype(np.int32) ids_restore = np.argsort(rand_indices, axis=0).astype(np.int32) mask = np.ones((self.num_patches,)).astype(np.int32) mask[:self.keep_num] = 0 unmask_index = rand_indices[:self.keep_num] out = (imgs, mask, ids_restore, unmask_index,) return out def __repr__(self): return "Mask generator for mae arch." pengcheng.mind-test/mindformers/dataset/mask_language_model_dataset.py000066400000000000000000000122171471400134000267230ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Dataset.""" import os import copy import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class MaskLanguageModelDataset(BaseDataset): """ Bert pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MaskLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['fill_mask']['bert_tiny_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/bert.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MaskLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MaskLanguageModelDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Masked Image Modeling Dataset.") rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) cls.init_dataset_config(dataset_config) rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config = copy.deepcopy(dataset_config) if not (dataset_config.data_loader.type == 'MindDataset' or dataset_config.data_loader.type == 'TFRecordDataset'): raise NotImplementedError("Now, Causal Language Modeling Dataset only supports " "MindSpore's MindDataset and TFRecordDataset two data loading modes") dataset_files = [] if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if file.endswith(".mindrecord") or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if data_dir.endswith(".mindrecord") or data_dir.endswith(".tfrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': device_num, 'shard_id': rank_id}) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.input_columns) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) return dataset pengcheng.mind-test/mindformers/dataset/mim_dataset.py000066400000000000000000000106201471400134000235230ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Dataset.""" import os from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .mask import build_mask from .transforms import build_transforms from .sampler import build_sampler from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class MIMDataset(BaseDataset): """ Masked Image Modeling Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MIMDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['masked_image_modeling']['mae_vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/mae.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MIMDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MIMDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Masked Image Modeling Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) mask = build_mask(dataset_config.mask_policy) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) if transforms is not None: for column in dataset_config.input_columns: dataset = get_dataset_map(dataset, transforms, input_columns=column, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if mask is not None: dataset = get_dataset_map(dataset, mask, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.project(columns=dataset_config.output_columns) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-test/mindformers/dataset/question_answering_dataset.py000066400000000000000000000102111471400134000266610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Question Answering Dataset.""" import os import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from ..models.build_tokenizer import build_tokenizer from .sampler import build_sampler from .dataloader import build_dataset_loader from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class QuestionAnsweringDataset(BaseDataset): """ Question Answering Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import QuestionAnsweringDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['question_answering']['qa_bert_base_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/question_answering.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='QuestionAnsweringDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = QuestionAnsweringDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): """new method""" logger.info("Now Create Question Answering Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) tokenizer = build_tokenizer(dataset_config.tokenizer) dataset = build_dataset_loader( class_name=dataset_config.data_loader.type, dataset_dir=dataset_config.data_loader.dataset_dir, tokenizer=tokenizer, stage=dataset_config.data_loader.stage, column_names=dataset_config.data_loader.column_names, num_parallel_workers=dataset_config.num_parallel_workers, num_shards=device_num, shard_id=rank_id ) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) return dataset pengcheng.mind-test/mindformers/dataset/reward_model_dataset.py000066400000000000000000000241621471400134000254130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Causal Image Modeling Dataset.""" import os import copy import re import numpy as np import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from mindformers.models.build_tokenizer import build_tokenizer from .dataloader import build_dataset_loader from .base_dataset import BaseDataset def get_input_data_batch_slice_map(chosen_input_ids, chosen_attention_mask, rejected_input_ids, rejected_attention_mask, position_id, loss_mask, end_ind, rank: int = 0, dis: int = 1, pad_id: int = 2): """ Generate position_id and attention_mask according to input_ids considering eos reset """ rank = int(rank) chosen_input_ids = chosen_input_ids[rank*dis: (rank + 1)*dis] rejected_input_ids = rejected_input_ids[rank*dis: (rank + 1)*dis] chosen_attention_mask = chosen_attention_mask[rank*dis: (rank + 1)*dis] rejected_attention_mask = rejected_attention_mask[rank*dis: (rank + 1)*dis] position_id = position_id[rank*dis: (rank + 1)*dis] loss_mask = loss_mask[rank*dis: (rank + 1)*dis] input_ids = np.concatenate((chosen_input_ids, rejected_input_ids)) attention_mask = np.concatenate((chosen_attention_mask, rejected_attention_mask)) position_id = np.concatenate((position_id, position_id)) end_ind = np.sum(input_ids != pad_id, axis=1) return input_ids, position_id, attention_mask, loss_mask, end_ind @MindFormerRegister.register(MindFormerModuleType.DATASET) class RewardModelDataset(BaseDataset): """Causal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import RewardModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='RewardModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = RewardModelDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Reward Model Dataset.") print("dataset_config", dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) cls.init_dataset_config(dataset_config) logger.info("Now Create Reward Model Dataset1.") rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config.rank_id = rank_id dataset_config.device_num = device_num if dataset_config.data_loader.type != "MindDataset" and \ dataset_config.data_loader.type != "TFRecordDataset": dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) logger.info("Now Create Reward Model Dataset1.5") # type_cast_op = C.TypeCast(mstype.int32) if cls._is_semi_full_batch() or cls._is_data_parallel(): rank_id = 0 dis = dataset_config.batch_size else: # Each card slice a small batch from the full batch dis = dataset_config.batch_size // device_num if dataset_config.batch_size % device_num != 0: raise ValueError( f"batch size {dataset_config.batch_size} should be a multiple of device number {device_num}." " You should change the args: per_batch_size.") logger.info("Now Create Reward Model Dataset1.6") dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns) pad_id = dataset_config.pad_token_id if dataset_config.pad_token_id is not None else 2 logger.info("Now Create Reward Model Dataset2.") map_func = (lambda chosen_input_ids, chosen_attention_mask, rejected_input_ids, rejected_attention_mask, position_id, loss_mask, end_ind: \ get_input_data_batch_slice_map(chosen_input_ids, chosen_attention_mask, rejected_input_ids, rejected_attention_mask, position_id, loss_mask, end_ind, rank_id, dis=dis, pad_id=pad_id)) dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns) dataset = dataset.project(columns=dataset_config.output_columns) type_cast_op = C.TypeCast(mstype.int32) type_cast_op_float = C.TypeCast(mstype.float16) dataset = get_dataset_map(dataset, input_columns="input_ids", operations=type_cast_op) dataset = get_dataset_map(dataset, input_columns="position_id", operations=type_cast_op) dataset = get_dataset_map(dataset, input_columns="attention_mask", operations=type_cast_op_float) dataset = get_dataset_map(dataset, input_columns="loss_mask", operations=type_cast_op_float) dataset = get_dataset_map(dataset, input_columns="end_ind", operations=type_cast_op) dataset = dataset.repeat(dataset_config.repeat) return dataset @classmethod def _prepare_for_model(cls, dataset, dataset_config): """Preprocess data for gpt2 model""" tokenizer_config = dataset_config.tokenizer tokenizer = build_tokenizer(tokenizer_config) max_length = tokenizer_config.max_length def map_func(input_data): input_data = input_data.tolist() input_ids = tokenizer(input_data, padding='max_length', max_length=max_length, truncation=True, add_special_tokens=False) return input_ids.get('input_ids') dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.input_columns) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id}) dataset = cls._prepare_for_model(dataset, dataset_config) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" dataset_files = [] mind_compile = re.compile("mindrecord0*$") if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if re.findall(mind_compile, file) or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if re.findall(mind_compile, data_dir) or data_dir.endswith(".tfrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id, 'columns_list': dataset_config.input_columns}) return dataset pengcheng.mind-test/mindformers/dataset/sampler/000077500000000000000000000000001471400134000223265ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/sampler/__init__.py000066400000000000000000000013621471400134000244410ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Sampler API.""" from .build_sampler import build_sampler __all__ = [] pengcheng.mind-test/mindformers/dataset/sampler/__pycache__/000077500000000000000000000000001471400134000245365ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/sampler/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004061471400134000307260ustar00rootroot00000000000000a ?cf@sdZddlmZgZdS)zMindFormers Sampler API.) build_samplerN)__doc__r__all__rrO/home/ma-user/work/mindformers-baichuan/mindformers/dataset/sampler/__init__.pys pengcheng.mind-test/mindformers/dataset/sampler/__pycache__/build_sampler.cpython-39.pyc000066400000000000000000000040211471400134000320060ustar00rootroot00000000000000a ?cf @sTdZddlZddlmZddlmZmZmZd e e e e dddZ d d Z e dS) zBuild Sampler API.N)samplers)MindFormerRegisterMindFormerModuleTypeMindFormerConfigdataset_sampler)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)aiBuild sampler For MindFormer. Instantiate the sampler from MindFormerRegister's registry. Args: config (dict): The task sampler's config. Default: None. default_args (dict): The default argument of sampler API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'sampler'. class_name (str): The class name of sampler API. Default: None. Return: The function instance of sampler API. Examples: >>> from mindformers import build_sampler >>> sampler_config = {'type': 'RandomSampler', 'replacement': False} >>> # 1) use config dict to build sampler >>> sampler_from_config = build_sampler(sampler_config) >>> # 2) use class name to build sampler >>> sampler_class_name = build_sampler(class_name='RandomSampler', replacement=False) N)r) isinstancedictrrget_instance_from_cfgrDATASET_SAMPLER get_instance)rrr r kwargsrT/home/ma-user/work/mindformers-baichuan/mindformers/dataset/sampler/build_sampler.py build_samplersrcCs@ttD]2}|drqtt|}t|rt|tj qdS)z. register MindSpore builtin transforms class. __N) dirsp startswithgetattrinspectisclassr register_clsrr) module_namerrrrregister_ms_samplers8s     r)NNrN) __doc__rmindspore.datasetrrmindformers.tools.registerrrrr strrrrrrrs  ! pengcheng.mind-test/mindformers/dataset/sampler/build_sampler.py000066400000000000000000000052371471400134000255310ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Sampler API.""" import inspect from mindspore.dataset import samplers as sp from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_sampler( config: dict = None, default_args: dict = None, module_type: str = 'dataset_sampler', class_name: str = None, **kwargs): r"""Build sampler For MindFormer. Instantiate the sampler from MindFormerRegister's registry. Args: config (dict): The task sampler's config. Default: None. default_args (dict): The default argument of sampler API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'sampler'. class_name (str): The class name of sampler API. Default: None. Return: The function instance of sampler API. Examples: >>> from mindformers import build_sampler >>> sampler_config = {'type': 'RandomSampler', 'replacement': False} >>> # 1) use config dict to build sampler >>> sampler_from_config = build_sampler(sampler_config) >>> # 2) use class name to build sampler >>> sampler_class_name = build_sampler(class_name='RandomSampler', replacement=False) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.DATASET_SAMPLER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_samplers(): """ register MindSpore builtin transforms class. """ for module_name in dir(sp): if module_name.startswith('__'): continue samplers = getattr(sp, module_name) if inspect.isclass(samplers): MindFormerRegister.register_cls(samplers, MindFormerModuleType.DATASET_SAMPLER) register_ms_samplers() pengcheng.mind-test/mindformers/dataset/text_classification_dataset.py000066400000000000000000000110521471400134000270000ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Classification Dataset.""" import os import copy import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class TextClassificationDataset(BaseDataset): """ Text Classification dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TextClassificationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_classification']['txtcls_bert_base_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/text_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TextClassificationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TextClassificationDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Text Classification Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) if "data_files" not in dataset_config.data_loader \ and dataset_config.data_loader.dataset_dir: dataset_files = [] data_dir = dataset_config.data_loader.dataset_dir if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if file.endswith(".mindrecord") or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) else: if data_dir.endswith(".mindrecord") or data_dir.endswith(".tfrecord"): dataset_files.append(data_dir) else: dataset_files = list(dataset_config.data_loader.dataset_files) dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': device_num, 'shard_id': rank_id}) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.input_columns) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, operations=type_cast_op, input_columns=input_arg) return dataset pengcheng.mind-test/mindformers/dataset/token_classification_dataset.py000066400000000000000000000113271471400134000271410ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Token classification Dataset.""" import os from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from ..models.build_tokenizer import build_tokenizer from .transforms import build_transforms from .sampler import build_sampler from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class TokenClassificationDataset(BaseDataset): """ Token classification Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TokenClassificationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['token_classification']['tokcls_bert_base_chinese'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/token_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TokenClassificationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TokenClassificationDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): """new method""" logger.info("Now Create Token classification Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) tokenizer = build_tokenizer(dataset_config.tokenizer) text_transforms = build_transforms(dataset_config.text_transforms, default_args={"tokenizer": tokenizer}) label_transforms = build_transforms(dataset_config.label_transforms) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) if text_transforms is not None: dataset = get_dataset_map(dataset, input_columns=dataset_config.input_columns, operations=text_transforms, output_columns=dataset_config.output_columns, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if label_transforms is not None: dataset = get_dataset_map(dataset, input_columns=dataset_config.input_columns[1], operations=label_transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-test/mindformers/dataset/transforms/000077500000000000000000000000001471400134000230615ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/transforms/__init__.py000066400000000000000000000022501471400134000251710ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Transforms API.""" from .build_transforms import build_transforms from .vision_transforms import * from .text_transforms import * from .mixup import Mixup from .auto_augment import rand_augment_transform, auto_augment_transform, augment_and_mix_transform from .random_erasing import RandomErasing __all__ = ['Mixup', 'rand_augment_transform', 'auto_augment_transform', 'augment_and_mix_transform', 'RandomErasing'] __all__.extend(vision_transforms.__all__) __all__.extend(text_transforms.__all__) pengcheng.mind-test/mindformers/dataset/transforms/__pycache__/000077500000000000000000000000001471400134000252715ustar00rootroot00000000000000pengcheng.mind-test/mindformers/dataset/transforms/__pycache__/__init__.cpython-39.pyc000066400000000000000000000011461471400134000314630ustar00rootroot00000000000000a ?cf@spdZddlmZddlTddlTddlmZddlmZmZm Z ddl m Z gdZ e ej e ej dS) zMindFormers Transforms API.)build_transforms)*)Mixup)rand_augment_transformauto_augment_transformaugment_and_mix_transform) RandomErasing)rrrrrN)__doc__rZvision_transformsZtext_transformsZmixuprZ auto_augmentrrrZrandom_erasingr__all__extendr r R/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/__init__.pys    pengcheng.mind-test/mindformers/dataset/transforms/__pycache__/auto_augment.cpython-39.pyc000066400000000000000000000627011471400134000324200ustar00rootroot00000000000000a ?cf@sdZddlZddlZddlZddlZddlZddlmZmZm Z ddl m Z ddl m Z mZeddejddd DZd Zd Zed ed ZejejfZdd ddZddZddZddZGdddZGdddZeZ ddde j!e j"e j#e j$e j%e j&e j'e j(e j)e j(e j)e j(e j)e j(e j)e j*e j*e j+e j+e j,e j,dZ-eZ.e.j/e.j0e.j1e.j2e.j3e.j3e.j3e.j4e.j4e.j5e.j6e.j6e.j7e.j7e.j8e.j8e.j9e.j9e.j:e.j;e.je.j?dZ@GdddZAddZBdd ZCd!d"ZDd#d$ZEdDd&d'ZFGd(d)d)ZGe HejId*d+ZJgd,ZKgd-ZLd.d/d/d0d0d1d1d1d2d2d2d2d2ddd3ZMdEd4d5ZNdFd7d8ZOGd9d:d:e ZPe HejId;d<ZQgd=ZRdGd>d?ZSGd@dAdAZTe HejIdBdCZUdS)Hz2AutoAugment, RandAugment, and AugMix for MindSporeN)ImageImageOps ImageEnhance)PyTensorOperation)MindFormerRegisterMindFormerModuleTypecCsg|] }t|qS)int).0xrrV/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/auto_augment.py r .)rrg$@)translate_constimg_mean)nearestlinearcubiccCsF|dkrtj}n.|dkr(tj}n|dkr.RGB)Lri)rangemodelenpoint)r;r<rgrbr]lutr!rrfr solarize_adds  zPolicyHelper.solarize_addcKs|dkr |n t||}|S)z posterize of auto_augment policy)r posterize)r;r<Z bits_to_keepr]r!rrr rrszPolicyHelper.posterizecKst||S)zcontrast of auto_augment policy)rContrastenhancer;r<r=r]rrr contrastszPolicyHelper.contrastcKst||S)zcolor of auto_augment policy)rColorrtrurrr colorszPolicyHelper.colorcKst||S)z!brightness of auto_augment policy)r Brightnessrtrurrr brightnessszPolicyHelper.brightnesscKst||S)z sharpness of auto_augment policy)r Sharpnessrtrurrr sharpnessszPolicyHelper.sharpnessN)r)__name__ __module__ __qualname____doc__r>r?rBrCrFrGrJr^r_r`rarprrrvrxrzr|rrrr r4Ls$ r4c@sxeZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ ddZ ddZddZddZdS) LevelHelperz Level helper cCstdkr| S|S)zWith 50% prob, negate the value?)r()r;vrrr _randomly_negateszLevelHelper._randomly_negatecCs||td}|fS)z&_randomly_negate of auto_augment levelg>@r _MAX_LEVELr;level_rrr rotate_level_to_argszLevelHelper.rotate_level_to_argcCs|tddfS)z*enhance_level_to_arg of auto_augment levelg?皙?)rrrrr enhance_level_to_argsz LevelHelper.enhance_level_to_argcCsd||td}|fS)z5enhance_increasing_level_to_arg of auto_augment level?g?rrrrr enhance_increasing_level_to_argsz+LevelHelper.enhance_increasing_level_to_argcCs||td}|fS)z(shear_level_to_arg of auto_augment level333333?rrrrr shear_level_to_argszLevelHelper.shear_level_to_argcCs$|d}||tt|}|fS)z0translate_abs_level_to_arg of auto_augment levelr)rrfloat)r;rhparamsrrrr translate_abs_level_to_argsz&LevelHelper.translate_abs_level_to_argcCs$|dd}||t|}|fS)z0translate_rel_level_to_arg of auto_augment level translate_pctg?)r+rr)r;rrrrrr translate_rel_level_to_args z&LevelHelper.translate_rel_level_to_argcCst|tdfS)z,posterize_level_to_arg of auto_augment levelr rrrrr posterize_level_to_argsz"LevelHelper.posterize_level_to_argcCsd|||dfS)z7posterize_increasing_level_to_arg of auto_augment levelrr)rr;rrrrr !posterize_increasing_level_to_argsz-LevelHelper.posterize_increasing_level_to_argcCst|tddfS)z5posterize_original_level_to_arg of auto_augment levelrrrrrr posterize_original_level_to_argsz+LevelHelper.posterize_original_level_to_argcCst|tdfS)z+solarize_level_to_arg of auto_augment levelrhrrrrr solarize_level_to_argsz!LevelHelper.solarize_level_to_argcCsd|||dfS)z6solarize_increasing_level_to_arg of auto_augment levelrhr)rrrrr solarize_increasing_level_to_argsz,LevelHelper.solarize_increasing_level_to_argcCst|tdfS)z/solarize_add_level_to_arg of auto_augment levelnrrrrr solarize_add_level_to_argsz%LevelHelper.solarize_add_level_to_argN)r}r~rrrrrrrrrrrrrrrrrrr rsr) AutoContrastEqualizeInvertRotate PosterizePosterizeIncreasingPosterizeOriginalSolarizeSolarizeIncreasing SolarizeAddrwColorIncreasingrsContrastIncreasingryBrightnessIncreasingr{SharpnessIncreasingShearXShearYZ TranslateXZ TranslateY TranslateXRel TranslateYRelc@s"eZdZdZdddZddZdS) AugmentOpr NcCsx|pt}t||_t||_||_||_||_t d|vrHt |dnt d|vrZ|dnt d|_ |jdd|_dS)Nrr-)r/r# magnitude_stdr)_HPARAMS_DEFAULT NAME_TO_OPaug_fn LEVEL_TO_ARGlevel_fnprob magnitudecopyrdictr'_FILL_RANDOM_INTERPOLATIONr,r+r)r;namerrrrrr __init__Fs    zAugmentOp.__init__cCs|jdkrt|jkr|S|j}|jr\|jtdkrDtd|}n|jdkr\t||j}ttt d|}|j dur| ||j }nt }|j |g|Ri|jS)z apply augmentrinfrN)rr(rrruniformgaussrdrmaxrrr'rr,)r;r<rZ level_argsrrr __call__Ys  zAugmentOp.__call__)rrNr}r~rrrrrrrr rCs rcsddgddgddgddgd d gd d gd dgddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gdd.gd/d0gg}fd1d2|D}|S)3auto_augment_policy_v0r皙?r5rrrrw皙? r333333?rrwrr5rrrqrrrrrrrrrrrrw皙?rrrrqrrrqrrrrrrrwrr5rrrrrrrrrrrrrrrrwrrrrr)rrrrrrrrqrwrrrrrrrrrrrrrrrrIrrrrrrrrwrrrwrrrrrrrrrrIrqrrrrrrrrrrrr)rrrrrrrrr5rwrrrrr1csg|]}fdd|DqS)csg|]}t|diqSrrr rUrrr r rz5auto_augment_policy_v0...rr sprrr r rz*auto_augment_policy_v0..rrpolicypcrrr rks8rcsddgddgddgddgd d gd d gd dgddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gdd.gd/d0gg}fd1d2|D}|S)3auto_augment_policy_v0rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr)rrrrrrrcsg|]}fdd|DqS)csg|]}t|diqSrrrrrr r rz6auto_augment_policy_v0r...rrrrr r rz+auto_augment_policy_v0r..rrrrr rs8rcsddgddgddgddgd d gd d gd dgddgddgddgd dgddgddgddgddgd dgddgd d!gd"d#gddgd d gddgddgddgddgg}fd$d%|D}|S)&auto_augment_policy_original)rrrqrrrrrr1rrr1rr)rrr)rrrrrrrrrrrqrrrrrr)rrr1rrrrrrrrq)rrrrwrrrrrrrrrrIrrrrrrrqrrsrrqrwrrrwrrqrrrr{rrrrrqrrr1rcsg|]}fdd|DqS)csg|]}t|diqSrrrrrr r rz;auto_augment_policy_original...rrrrr r rz0auto_augment_policy_original..rrrrr rs8rcsddgddgddgddgd d gd d gd dgddgddgddgd dgddgddgddgddgd dgddgd d!gd"d#gddgd d gddgddgddgddgg}fd$d%|D}|S)&auto_augment_policy_originalr)rrrqr r r rr)rrr)rrrrrr r rr)rrr1rrrrrrrrrrrrrrrrrrrrcsg|]}fdd|DqS)csg|]}t|diqSrrrrrr r rz..rrrrr r rz1auto_augment_policy_originalr..rrrrr rs8rv0cCsb|pt}|dkrt|}nD|dkr,t|}n2|dkr>t|}n |dkrPt|}ntd||S)auto_augment_policyoriginalZ originalrr Zv0rzUnknown AA policy )rrrrr ValueError)rrr!rrr r!s    r!c@s eZdZdZddZddZdS) AutoAugmentcCs ||_dSN)r)r;rrrr rszAutoAugment.__init__cCs"t|j}|D] }||}q|S)zapply autoaugment)r(r)r)r;r<Z sub_policyoprrr r s  zAutoAugment.__call__Nrrrrr r$sr$cCs|d}|d}|dd}|D]N}td|}t|dkr"|dd\}}|dkrh|dt|q"td q"tt||d S) a_ Create a AutoAugment transform :param config_str: String defining configuration of auto augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the AutoAugment policy (one of 'v0', 'v0r', 'original', 'originalr'). The remaining sections, not order specific determine 'mstd' - float std deviation of magnitude noise applied Ex 'original-mstd0.5' results in AutoAugment with original policy, magnitude_std 0.5 :param hparams: Other hparams (kwargs) for the AutoAugmentation scheme :return: A MindSpore compatible Transform -rr5N(\d.*)rmstdrz"Unknown AutoAugment config sectionr)splitrerm setdefaultrr#r$r!) config_strrconfigZ policy_namerWcskeyvalrrr auto_augment_transforms     r2)rrrrrrrrwrsryr{rrrr)rrrrrrrrrrrrrrrrrrg?g{Gzt?)rrrrrrwr{rrrrsryrrrcsF|pt}|dkrtd|tfdd|D}|t|}|S)_select_rand_weightsrz2only one set of weights currently, get weight_idx csg|] }|qSrr)r kZ rand_weightsrr r prz(_select_rand_weights..)_RAND_TRANSFORMSr#_RAND_CHOICE_WEIGHTS_0npsum) weight_idx transformsprobsrr5r r3jsr3rcs$pt|pt}fdd|DS)rand_augment_opscsg|]}t|ddqS)rrrrrr rrrrr r ysz$rand_augment_ops..)rr6rrr;rr@r r=us  r=cs*eZdZdZdfdd ZddZZS) RandAugmentrNcs$tt|||_||_||_dSr%)superrBrops num_layerschoice_weights)r;rDrErF __class__rr rszRandAugment.__init__cCs6tjj|j|j|jdu|jd}|D] }||}q$|S)N)replacep)r8r(r)rDrErF)r;r<rDr&rrr rs  zRandAugment.__call__)rN)r}r~rrrr __classcell__rrrGr rB}srBc Cst}d}d}t}|d}|ddkr8td|d|dd}|D]}td|}t|dkrH|dd\} } | d kr|d t| qH| d krt| rt }qH| d krt | }qH| d krt | }qH| dkrt | }qHtdqHt |||d} |durdnt |} t | || dS)a Create a RandAugment transform :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining sections, not order specific determine 'm' - integer magnitude of rand augment 'n' - integer num layers (number of transform ops selected per image) 'w' - integer probabiliy weight index (index of a set of weights to influence choice of op) 'mstd' - float std deviation of magnitude noise applied 'inc' - integer (bool), use augmentations that increase in severity with magnitude (default: 0) Ex 'rand-m9-n3-mstd0.5' results in RandAugment with magnitude 9, num_layers 3, magnitude_std 0.5 'rand-mstd1-w0' results in magnitude_std 1.0, weights 0, default magnitude of 10 and num_layers 2 :param hparams: Other hparams (kwargs) for the RandAugmentation scheme :return: A MindSpore compatible Transform rNr'rrandzconfig[0] is not rand, get r5r(r)rincmnwz"Unknown RandAugment config sectionrA)rF)rr6r*r#r+rmr,rbool_RAND_INCREASING_TRANSFORMSr r=r3rB) r-rrrEr:r;r.rWr/r0r1Zra_opsrFrrr rand_augment_transforms6         rS) rrrrrrrrrrrrrcs$pt|pt}fdd|DS) augmix_opscsg|]}t|ddqS)rr>rr?r@rr r szaugmix_ops..)r_AUGMIX_TRANSFORMSrArr@r rTs  rTc@s:eZdZdZdddZdd Zd d Zd d ZddZdS) AugMixAugmentz AugMix Transform Adapted and improved from impl here From paper: 'AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty - https://arxiv.org/abs/1912.02781 rrFcCs"||_||_||_||_||_dSr%)rDalphawidthdepthblended)r;rDrXrYrZr[rrr rs zAugMixAugment.__init__cCs\||}d}g}|dddD]"}||}|d|9}||qtj|dddtjdS)z&_calc_blended_weights of AugMixAugmentrNrWr5dtype)appendr8arrayfloat32)r;wsrNZcumpZrwsrPrXrrr _calc_blended_weightss  z#AugMixAugment._calc_blended_weightsc Csx|}|||}|D]Z}|jdkr,|jn tjdd}tjj|j|dd}|} |D] } | | } qVt || |}q|S)z_apply_blended of AugMixAugmentrr5rTrI) rrbrZr8r(randintr)rDrblend) r;r<mixing_weightsrNZimg_origrarPrZrDimg_augr&rrr _apply_blendeds  zAugMixAugment._apply_blendedc Cs|jd|jdt|f}tj|tjd}|D]d}|jdkrF|jn tjdd}tjj |j |dd}|} |D] } | | } qp||tj | tjd7}q2tj |dd|dt |tj}t |||S) z_apply_basic of AugMixAugmentrr5r\rTrcgo@)out)r8rmgetbandsr8zerosr`rZr(rdr)rDasarrayclipr fromarrayastypeuint8re) r;r<rfrN img_shapemixedmwrZrDrgr&rrr _apply_basics zAugMixAugment._apply_basiccCs\ttj|jg|j}ttj|j|j}|jrJ||||}n| |||}|S)zAugMixAugment apply) r8r`r( dirichletrXrYbetar[rhrt)r;r<rfrNrrrrr rs zAugMixAugment.__call__N)rrrWF) r}r~rrrrbrhrtrrrrr rVs   rVc Cs d}d}d}d}d}td|d<|d}|dd krHtd |d|d d }|D]}td |} t| dkrX| d d\} } | dkr|dt| qX| dkrt| }qX| dkrt| }qX| dkrt| }qX| dkrt| }qX| dkrt| }qXtdqXt||d} t | ||||dS)a Create AugMix MindSpore transform :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining sections, not order specific determine 'm' - integer magnitude (severity) of augmentation mix (default: 3) 'w' - integer width of augmentation chain (default: 3) 'd' - integer depth of augmentation chain (-1 is random [1, 3], default: -1) 'b' - integer (bool), blend each branch of chain into end result without a final blend, less CPU (default: 0) 'mstd' - float std deviation of magnitude noise applied (default: 0) Ex 'augmix-m5-w4-d2' results in AugMix with severity 5, chain width 4, chain depth 2 :param hparams: Other hparams (kwargs) for the Augmentation transforms :return: A MindSpore compatible Transform rrWrFrrr'rZaugmixzconfig[0] is not augmix, get r5Nr(rr)rNrPrXrUrVzUnknown AugMix config section)rr)rXrYrZr[) rr*r#r+rmr,r rQrTrV) r-rrrYrZrXr[r.rWr/r0r1rDrrr augment_and_mix_transform#s:             rw)r N)rN)rNN)rNN)VrrKr(r+numpyr8PILrrrZ#mindspore.dataset.vision.transformsrmindformers.tools.registerrrr' __version__r*r2rrrrrrrr*r"r.r3r4rlhrrrrrrrrrrrrrphr^r`r_rJrrrarprxrvrzr|r>r?rFrGrBrCrrrrrrr!r$register TRANSFORMSr2r6rRr7r3r=rBrSrUrTrVrwrrrr s "  hS("#!"      3 G pengcheng.mind-test/mindformers/dataset/transforms/__pycache__/build_transforms.cpython-39.pyc000066400000000000000000000044321471400134000333020ustar00rootroot00000000000000a ?cf$ @sddZddlZddlmZddlmZddlmZm Z m Z d e e ge e e ddd Zd d ZedS) zBuild Transforms API.N) transforms)vision)MindFormerRegisterMindFormerModuleTypeMindFormerConfigr)config default_args module_type class_namec Ks|dur|durdS|durt|tr>t|ts>tfi|}|}t|ts^tj|tj|dSg}|D]}t|tj}||qf|Stj ||fi|S)a~Build transform For MindFormer. Instantiate the transform from MindFormerRegister's registry. Args: config (dict, list): The task transform's config. Default: None. default_args (dict): The default argument of transform API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'transform'. class_name (str): The class name of transform API. Default: None. Return: The function instance of transform API. Examples: >>> from mindformers import build_transforms >>> transform_config = [{'type': 'Decode'}, {'type': 'Resize', 'size': 256}] >>> # 1) use config dict to build transform >>> transform_from_config = build_transforms(transform_config) >>> # 2) use class name to build transform >>> transform_class_name = build_transforms(class_name='Decode') N)r) isinstancedictrlistrget_instance_from_cfgr TRANSFORMSappend get_instance) rrr r kwargsZcfg_transformsr transformZ transform_oprZ/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/build_transforms.pybuild_transformss$  rcCsntttjttjD]P}|dr(qttj|drDttj|dn ttj|}t|rt |t j qdS)z,register MindSpore builtin transforms class.__N) setdirvsrtf startswithgetattrinspectisclassr register_clsrr) module_namerrrrregister_ms_transformsAs    r")NNrN)__doc__rmindspore.datasetrrrrmindformers.tools.registerrrrr r strrr"rrrrs   )pengcheng.mind-test/mindformers/dataset/transforms/__pycache__/mixup.cpython-39.pyc000066400000000000000000000205621471400134000310710ustar00rootroot00000000000000a ?cf,@sdZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZdd d Zdd dZdddZdddZdddZeejGddde ZdS)zMixup and Cutmix for MindSporeN)Tensordtype)ops)PyTensorOperation)logger)MindFormerRegisterMindFormerModuleType?cCs0|d}t||}tj|||tjd}|S)zone hot to label)a_mina_maxr)reshapenpeyeclipfloat32)x num_classeson_value off_valuerO/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/mixup.pyone_hots rcCsR||}d||}t||||d}ttj|dd|||d}|||d|S) mixup_targetr )rrraxis)rrflip)targetrlam smoothingrry1y2rrrr!s  rcCstd|}|dd\}}t||t||}}t||t||} } tjjd| || |d} tjjd| || |d} t| |dd|} t| |dd|}t| |dd|}t| |dd|}| |||fS)a Standard CutMix bounding-box Generates a random square bbox based on lambda value. This impl includes support for enforcing a border margin as percent of bbox dimensions. Args: img_shape (tuple): Image shape as tuple lam (float): Cutmix lambda value margin (float): Percentage of bbox dimension to enforce as margin (reduce amount of box outside image) count (int): Number of bbox to generate Nrsize)rsqrtintrandomrandintr) img_shaper margincountratioimg_himg_wcut_hcut_wZmargin_yZmargin_xcycxylyhxlxhrrr rand_bbox*s r;c Cst|dkrtd|dd\}}tjjt||dt||d|d}tjjt||dt||d|d}tjjd|||d}tjjd|||d}||} ||} || || fS)a Min-Max CutMix bounding-box Inspired by Darknet cutmix impl, generates a random rectangular bbox based on min/max percent values applied to each dimension of the input image. Typical defaults for minmax are usually in the .2-.3 for min and .8-.9 range for max. Args: img_shape (tuple): Image shape as tuple minmax (tuple or list): Min and max bbox ratios (as percent of image size) count (int): Number of bbox to generate r(zlen(minmax) is not 2. r%Nrr$r&)len ValueErrorrr+r,r*) r-minmaxr/r1r2r3r4r7r9yuxurrrrand_bbox_minmaxBs **rATc Cs~|dur t|||d\}}}}nt|||d\}}}}|sB|durn||||} d| t|d|d}||||f|fS)z0 Generate bbox and apply lambda correction. N)r/r r%r )rAr;float) r-r ratio_minmax correct_lamr/r7r?r9r@Z bbox_arearrrcutmix_bbox_and_lamZs rEc sReZdZdZdfd d Zd d ZddZddZddZddZ ddZ Z S)Mixupas Mixup/Cutmix that applies different params to each element or whole batch Args: mixup_alpha (float): mixup alpha value, mixup is active if > 0. cutmix_alpha (float): cutmix alpha value, cutmix is active if > 0. cutmix_minmax (List[float]): cutmix min/max image ratio, cutmix is active and uses this vs alpha if not None. prob (float): probability of applying mixup or cutmix per batch or element switch_prob (float): probability of switching to cutmix instead of mixup when both are active mode (str): how to apply mixup/cutmix params (per 'batch', 'pair' (pair of elements), 'elem' (element) correct_lam (bool): apply lambda correction when cutmix bbox clipped by image borders label_smoothing (float): apply label smoothing to the mixed target tensor num_classes (int): number of classes for target r r N?batchT皙?c sttt|||_||_||_|jdurFt|jdkr@tdd|_||_||_ ||_ | |_ ||_ ||_ d|_dS)Nr(zlen(cutmix_minmax) is not 2.r T)superrF__init__ mixup_alpha cutmix_alpha cutmix_minmaxr<r=mix_prob switch_problabel_smoothingrmoderD mixup_enabled) selfrMrNrOprobrQrSrDrRr __class__rrrLws zMixup.__init__c Cs tj|tjd}tj|tjd}|jr|jdkr|jdkrtj ||j k}t |tjj |j|j|dtjj |j|j|d}n\|jdkrtjj |j|j|d}n:|jdkrtj|tjd}tjj |j|j|d}nt dt tj ||jk|tj|}||fS)_params_per_elemrr r&ROne of mixup_alpha > 0., cutmix_alpha > 0., cutmix_minmax not None should be true.)ronesrzerosboolrTrMrNr+randrQwherebetar=rPastype)rU batch_sizer use_cutmixlam_mixrrrrYs$  $zMixup._params_per_elemcCsd}d}|jrtj|jkr|jdkrl|jdkrltj|jk}|rXtj|j|jntj|j|j}nH|jdkrtj|j|j}n*|jdkrd}tj|j|j}nt dt |}||fS)_params_per_batchr Fr TrZ) rTrr+r^rPrMrNrQr`r=rB)rUr rcrdrrrres   zMixup._params_per_batchc Cst|}||\}}|}t|D]}||d}||}|dkr&||rt||j||j|jd\\} } } } }||dd| | | | f||dd| | | | f<|||<q&|||||d|||<q&t t |t j ddS) _mix_elemr$r rCrDNr) r<rYclonerangerEshaperOrDP ExpandDimsrmstyper rUrrbZ lam_batchrcZx_origijr r7r8r9r:rrrrfs  8 "zMixup._mix_elemc Csjt|}||d\}}|}t|dD]}||d}||}|dkr.||rt||j||j|jd\\} } } } }||dd| | | | f||dd| | | | f<||dd| | | | f||dd| | | | f<|||<q.|||||d|||<|||||d|||<q.t ||dddf}t t |t jddS) _mix_pairr(r$r rgNr r)r<rYrhrirErjrOrDr concatenaterkrlrrmrrnrrrrqs$ 88  "zMixup._mix_pairc Cs|\}}|dkrdS|rt|j||j|jd\\}}}}}tj|dddddd||||f|dddd||||f<n&tj|ddd|}||9}||7}|S) _mix_batchr rgrrN)rerErjrOrDrr) rUrr rcr7r8r9r:Z x_flippedrrrrss HzMixup._mix_batchcCst|ddkr>t|dkr4|dd}tdn td|jdkrT||}n |jd krj||}n ||}t||j ||j }| t j | t j fS) z Mixup applyr(rr$Nr zBatch size is odd. When using mixup, batch size should be even.The last data in batch has been dropped to use mixip.you can set "drop_remainder" true in dataset config manually.zTBatch size is 1.If error occurs, please set "drop_remainder" true in dataset config.elempair)r<rwarningerrorrSrfrqrsrrrRrarr)rUrrr rrr__call__s         zMixup.__call__) r r Nr rGrHTrIrJ) __name__ __module__ __qualname____doc__rLrYrerfrqrsrx __classcell__rrrWrrFgsrF)r r )r r )r N)N)NTN)r|numpyr mindsporerrrmrrkZ#mindspore.dataset.vision.transformsrmindformers.tools.loggerrmindformers.tools.registerrr rrr;rArEregister TRANSFORMSrFrrrr s         pengcheng.mind-test/mindformers/dataset/transforms/__pycache__/random_erasing.cpython-39.pyc000066400000000000000000000074201471400134000327150ustar00rootroot00000000000000a ?cf@sfdZddlZddlZddlZddlmZddlmZm Z ej fddZ e e j GdddeZdS) z%Random Erasing (Cutout) for MindSporeN)PyTensorOperation)MindFormerRegisterMindFormerModuleTypecCsZ|rtjj|d|}n<|r>tjj|dddfd|}ntj|dddf|d}|S) _get_pixels)sizerdtype)nprandomnormalastypezeros) per_pixel rand_color patch_sizer funcrX/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/random_erasing.pyrs  rc s2eZdZdZdfd d Zd d ZddZZS) RandomErasingaY Randomly selects a rectangle region in an image and erases its pixels. 'Random Erasing Data Augmentation' by Zhong et al. See https://arxiv.org/pdf/1708.04896.pdf This variant of RandomErasing is intended to be applied to either a batch or single image tensor after it has been normalized by dataset mean and std. Args: probability: Probability that the Random Erasing operation will be performed. min_area: Minimum percentage of erased area wrt input image area. max_area: Maximum percentage of erased area wrt input image area. min_aspect: Minimum aspect ratio of erased area. mode: pixel color mode, one of 'const', 'rand', or 'pixel' 'const' - erase block is constant color of 0 for all channels 'rand' - erase block is same per-channel random (normal) color 'pixel' - erase block is per-pixel random (normal) color max_count: maximum number of erasing blocks per image, area per box is scaled by count. per-image count is randomly chosen between 1 and this value. ?{Gz?UUUUUU?333333?Nconstrrc stt|||_||_||_|p*d|}t|t|f|_||_ |pN||_ | |_ | }d|_ d|_|dkr|d|_ n$|dkrd|_n|r|dkrtddS)NrFrandTpixelrz mode is const)superr__init__ probabilitymin_areamax_areamathloglog_aspect_ratio min_count max_count num_splitslowerrr ValueError) selfrr r!Z min_aspectZ max_aspectmoder%r&r' __class__rrr4s$   zRandomErasing.__init__c Cst|jkrn||}|j|jkr,|jnt|j|j}t|D]}tdD]}t|j|j||} t tj|j } t t t | | } t t t | | } | |krP| |krPtd|| } td|| }t|j|j|| | f|d|dd| | | ||| f<qDqPqD|S)_erase rrN)r rr%r&randintrangeuniformr r!r"expr$introundsqrtrrr)r*imgchanimg_himg_wr areacount_ target_area aspect_ratiohwtopleftrrrr.Js*  $zRandomErasing._erasec Cst|jdkr*|j|g|j|jR}n\t|}|j\}}}}|jdkrV||jnd}t||D] }|||||||j||<qd|S)zRandomErasing applyrr)lenshaper.r r zeros_liker'r1) r*xoutput batch_sizer8r9r: batch_startirrr__call__as zRandomErasing.__call__) rrrrNrrNr)__name__ __module__ __qualname____doc__rr.rM __classcell__rrr,rrs r)rQr"r numpyr Z#mindspore.dataset.vision.transformsrmindformers.tools.registerrrfloat32rregister TRANSFORMSrrrrr s  pengcheng.mind-test/mindformers/dataset/transforms/__pycache__/text_transforms.cpython-39.pyc000066400000000000000000000116741471400134000331750ustar00rootroot00000000000000a ?cfW@sdZddlZddlZddlmZmZgdZeej GdddZ eej Gdd d Z eej Gd d d Z eej Gd d d Z eej GdddZdS)zText Transforms.N)MindFormerRegisterMindFormerModuleType)RandomChoiceTokenizerForwardTokenizerForwardTokenizeWithLabel LabelPaddingCaptionTransformc@s"eZdZdZd ddZddZd S) rzRandom Choice Tokenizer ForwardM max_lengthcCs||_||_||_||_dSN)r padding tokenizer random_seed)selfrr rrrY/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/text_transforms.py__init__sz%RandomChoiceTokenizerForward.__init__cCsd|}tj|jtjt|}|jt||t rH|| ddn|||j |j dd}|S)Nutf-8ignorer r input_ids) tolistnprandomseedrchoicelenr isinstancebytesdecoder r)rtextZ text_listindextoken_idrrr__call__%s z%RandomChoiceTokenizerForward.__call__N)r r r __name__ __module__ __qualname____doc__rr%rrrrrs rc@s"eZdZdZd ddZddZdS) rzTokenizer Forwardr r cCs||_||_||_dSr rr rrrr rrrrr5szTokenizerForward.__init__cCsht|tjr|n|}tt|D]&}t||tr$||dd||<q$|j||j |j dd}|S) call methodrrrr) rrndarrayrrangerr r!rr r)rr"ir$rrrr%:szTokenizerForward.__call__N)r r r&rrrrr2s rc@s"eZdZdZd ddZddZdS) rzTokenizer With Labelr cCs||_||_||_dSr r+r,rrrrHszTokenizeWithLabel.__init__cGs|d}|d}t|tjr$|n|}t|tr>|dd}|j||j|jd}tj |dtj d}tj |dtj d}tj |d tj d}||||fS) r-rrrrrdtypetoken_type_idsattention_mask) rrr.rr r!rr rarrayint32)rinputsr"label_idoutputrr5r6rrrr%Ms  zTokenizeWithLabel.__call__N)r1r r&rrrrrEs rc@s"eZdZdZdddZddZdS) rz Label PaddingrcCs||_||_dSr )r padding_value)rr r<rrrr`szLabelPadding.__init__cCst||jdkr$|d|jd}g}||jtt|D]}|||q@t||jkrp||jqTtj|tjd}|S)r-r2Nr3)rr appendr<r/rr7r8)rr:Z pad_label_idr0rrrr%ds zLabelPadding.__call__N)rr&rrrrr]s rc@s*eZdZdZddd Zd d Zd d ZdS)r z\ Caption Transform, preprocess captions and tokenize it, align with torch impl. 2 r r Tc Cs4||_||_||_||_||_||_||_||_dSr )rprompt max_wordsr rr truncationadd_special_tokens) rrrArBr rrrCrDrrrr}szCaptionTransform.__init__cCsH|jdkr:g}t|jdD]}||||q|S||}|S)Nr2r)ndimr/shaper= pre_caption)rcaptionZ caption_listr0rrrrr%s  zCaptionTransform.__call__cCst|}|j|}tdd|}tdd|}|d}|d}|d}t||j krrd |d|j }|j ||j |j |j|jd}tj|dtjd}|S) z Caption preprocessing removes any punctuationmarks except commas, tailing spaces and transform sentence into lower case. z([.!\"()*#:;~]) z\s{2,} N)r rrCrDrr3)strrAresublowerrstripstripsplitrrBjoinrr rrCrDrr7r8)rrHZ caption_wordsr;rrrrrGs.     zCaptionTransform.pre_captionN)r>r?r@r r TT)r'r(r)r*rr%rGrrrrr ws   r )r*rLnumpyrZtools.registerrr__all__register TRANSFORMSrrrrr rrrrs     pengcheng.mind-test/mindformers/dataset/transforms/__pycache__/vision_transforms.cpython-39.pyc000066400000000000000000000261231471400134000335130ustar00rootroot00000000000000a ?cf/@sBdZddlZddlmZddlmZddlZddl m Z ddl m Z m Z gdZe je je je je jdZGd d d ZGd d d ZGd ddZGdddZGdddZGdddZe e jGdddejjZe e jGdddejjZe e jGdddejj Z e e jGdddejj!Z!dS)z% transform methods for vision models N)Image)vision)Inter)MindFormerRegisterMindFormerModuleType) BatchResize BCHW2BHWC BatchPILizeBatchNormalizeBatchCenterCrop BatchToTensorRandomCropDecodeResizeRandomResizedCropResizeRandomHorizontalFlip)nearest antialiaslinearcubicbicubicc@seZdZdZddZdS)ra Transform a batch of image from CHW to HWC. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the channel should be (bz, c, h, w) or (c, h, w). for list, the item should be PIL.Image or numpy.array (c, h, w). Return: transformed image batch: for numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for PIL.Image input, it is returned directly. cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrZ| ddddSt|jdkrv| dddSt dt|jt|t j r|St d t |d d S) zthe call functioncsg|] }|qSr.0itemselfr[/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/vision_transforms.py >z&BCHW2BHWC.__call__..r2the rank of image_batch should be 3 or 4, but got the type  of image_batch is unsupported.N) isinstancemsTensorasnumpylistnpndarraylenshape transpose ValueErrorr TypeErrortyper image_batchrrr__call__8s    zBCHW2BHWC.__call__N__name__ __module__ __qualname____doc__r5rrrrr*s rc@s"eZdZdZdddZddZdS) rz Resize a batch of image to the given shape. Args: image_resolution (int): the target size. interpolation (str): interpolate method, default is 'cubic'. rcCs t||_t||j|_dSN) INTERPOLATIONget interpolationrrresize)rimage_resolutionr>rrr__init__Ts zBatchResize.__init__cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrbt fdd|DSt|jdkrz |St dt|jt|t j r |St dt|dd S) a The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the shape should be (bz, h, w, c) or (h, w, c). for list, the item should be PIL.Image or numpy.array (h, w, c). Returns: resized image batch: for numpy or tensor input, return a numpy array; for PIL.Image input, it returns PIL.Image. csg|]}|qSr)r?rrrrrirz(BatchResize.__call__..rcs$g|]}|tjddfqSr;)r?r+newaxisrrrrrlsr!r#r$r%N)r&r'r(r)r*r+r,r-r. row_stackr?r0rr1r2r3rrrr5Xs"      zBatchResize.__call__N)rr7r8r9r:rAr5rrrrrKs rc@s eZdZdZddZddZdS)r zv CenterCrop a batch of image to the given shape. Args: image_resolution (int): the target size. cCst||_dSr;)r CenterCropcrop)rr@rrrrAszBatchCenterCrop.__init__cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrbt fdd|DSt|jdkrz |St dt|jt|t j r |St dt|dd S) ab The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the shape should be (bz, h, w, c) or (h, w, c). for list, the item should be PIL.Image or numpy.array (h, w, c). Returns: center cropped image batch: for numpy or tensor input, return a numpy array, the shape is (bz, image_resolution, image_resolution, c) or (image_resolution, image_resolution, c); for PIL.Image input, it is returned with shape (image_resolution, image_resolution). csg|]}|qSr)rFrrrrrrz,BatchCenterCrop.__call__..rcs$g|]}|tjddfqSr;)rFr+rBrrrrrsr!r#r$r%N)r&r'r(r)r*r+r,r-r.rCrFr0rr1r2r3rrrr5s"      zBatchCenterCrop.__call__NrDrrrrr wsr c@s eZdZdZddZddZdS)r z9Transform a batch of image to tensor and scale to (0, 1).cCstjj|_dSr;)r'datasetrToTensortotensorrrrrrAszBatchToTensor.__init__cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrbt fdd|DSt|jdkrz |St dt|jt|t j r |St dt|dd S) a1 The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be PIL.Image or numpy.array. Returns: return a tensor or a list of tensor. csg|]}|qSr)rIrrrrrrz*BatchToTensor.__call__..rcs$g|]}|tjddfqSr;)rIr+rBrrrrrsr!r#r$r%N)r&r'r(r)r*r+r,r-r.rCrIr0rr1r2r3rrrr5s"      zBatchToTensor.__call__NrDrrrrr sr c@s"eZdZdZd ddZddZd S) r zNormalize a batch of image.g3<4'?gwg M?gy{ ?gB91?gwt.?g U?FcCstj|||d|_dS)N)meanstdis_hwc)r Normalize normalize)rrLrMrNrrrrAszBatchNormalize.__init__cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrT |St|jdkrzt fdd|DSt dt|jt dt |dd S) a The forward process. Args: image_batch (tensor, numpy.array, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be numpy.array. Returns: return a tensor or a list of tensor. csg|]}|qSr)rPrrrrrrz+BatchNormalize.__call__..r!rcs$g|]}|tjddfqSr;)rPr+rBrrrrrsr#r$r%N)r&r'r(r)r*r+r,r-r.rPrCr0r1r2r3rrrr5s    zBatchNormalize.__call__N)rJrKFrDrrrrr s  r c@seZdZdZddZdS)r z-transform a batch of image to PIL.Image list.cCst|tjr|St|tr<|D]}t|tjstdq|St|tjrP|}t|tjrt |j dkrxdd|DSt |j dkrt | tj Stdt |j tddS) a The forward process. Args: image_batch (tensor, numpy.array, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be PIL.Image. Returns: return a tensor or a list of tensor. z_unsupported type in list, when the image_batch is a list, the item in list should be PIL.Image.rcSsg|]}t|tjqSr)r fromarrayastyper+uint8rrrrr rz(BatchPILize.__call__..r!r#zunsupported input type.N)r&rr*r1r'r(r)r+r,r-r.rQrRrSr0)rr4rrrrr5s$      zBatchPILize.__call__Nr6rrrrr sr cs"eZdZdZdfdd ZZS) r !wrapper of RandomCropDecodeResizeg{Gz?g?g?gUUUUUU?r csB||_||_||_t||_||_tt| ||||j|dSr;) sizescaleratior<r=r> max_attemptssuperr rArrXrYrZr>r[ __class__rrrAs  zRandomCropDecodeResize.__init__)rUrVrrWr7r8r9r:rA __classcell__rrr^rr sr cs"eZdZdZdfdd ZZS) rrTrUrVrrWcsB||_||_||_t||_||_tt| ||||j|dSr;) rXrYrZr<r=r>r[r\rrAr]r^rrrA%s  zRandomResizedCrop.__init__)rUrVrrWr`rrr^rr!srcs"eZdZdZdfdd ZZS)rzwrapper of Resizercs0||_t||_d|_tt|||jdS)NF)rXr<r=r>randomr\rrA)rrXr>r^rrrA3s zResize.__init__)rr`rrr^rr/srcs"eZdZdZdfdd ZZS)rzwrapper of RandomHorizontalFlip?cs||_tt||dSr;)probr\rrA)rrdr^rrrA>szRandomHorizontalFlip.__init__)rcr`rrr^rr:sr)"r:numpyr+PILrmindspore.datasetr mindsporer'Zmindspore.dataset.vision.utilsrmindformers.tools.registerrr__all__NEAREST ANTIALIASLINEARPILCUBICBICUBICr<rrr r r r register TRANSFORMS transformsr rrrrrrrs6   !,,#&'    pengcheng.mind-test/mindformers/dataset/transforms/auto_augment.py000066400000000000000000001012011471400134000261160ustar00rootroot00000000000000# AutoAugment, RandAugment, and AugMix for MindSpore # # This code implements the searched ImageNet policies with various tweaks and improvements and # does not include any of the search code. # # AA and RA Implementation adapted from: # https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py, Apache 2.0 # # AugMix adapted from: # https://github.com/google-research/augmix, Apache 2.0 # # Hacked together by / Copyright 2020 Ross Wightman # # has been modified to adapt mindspore and mindformers # ============================================================================ """AutoAugment, RandAugment, and AugMix for MindSpore""" import math import random import re import numpy as np import PIL from PIL import Image, ImageOps, ImageEnhance from mindspore.dataset.vision.transforms import PyTensorOperation from mindformers.tools.register import MindFormerRegister, MindFormerModuleType _PIL_VER = tuple([int(x) for x in PIL.__version__.split('.')[:2]]) _FILL = (128, 128, 128) # This signifies the max integer that the controller RNN could predict for the # augmentation scheme. _MAX_LEVEL = 10. _HPARAMS_DEFAULT = dict( translate_const=250, img_mean=_FILL, ) _RANDOM_INTERPOLATION = (Image.BILINEAR, Image.BICUBIC) INTERPOLATION = {'nearest': 0, 'linear': 2, 'cubic': 3} def pil_interp(method): """Interpolation method selection""" if method.lower() == 'bicubic': func = Image.BICUBIC elif method.lower() == 'hamming': func = Image.HAMMING elif method.lower() == 'lanczos': func = Image.LANCZOS else: func = Image.BILINEAR return func def _interpolation(kwargs): """_interpolation of auto_augment""" interpolation = kwargs.pop('resample', 'linear') interpolation = random.choice(interpolation) \ if isinstance(interpolation, (list, tuple)) else interpolation return INTERPOLATION.get(interpolation) def _check_args_tf(kwargs): """_check_args_tf of auto_augment""" if 'fillcolor' in kwargs and _PIL_VER < (5, 0): kwargs.pop('fillcolor') kwargs['resample'] = _interpolation(kwargs) class PolicyHelper: """ auto_augment Policy helper """ def shear_x(self, img, factor, **kwargs): """shear_x of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, factor, 0, 0, 1, 0), **kwargs) def shear_y(self, img, factor, **kwargs): """shear_y of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, 0, factor, 1, 0), **kwargs) def translate_x_rel(self, img, pct, **kwargs): """translate_x_rel of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, pct * img.size[0], 0, 1, 0), **kwargs) def translate_y_rel(self, img, pct, **kwargs): """translate_y_rel of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, 0, 0, 1, pct * img.size[1]), **kwargs) def translate_x_abs(self, img, pixels, **kwargs): """translate_x_abs of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, pixels, 0, 1, 0), **kwargs) def translate_y_abs(self, img, pixels, **kwargs): """translate_y_abs of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, 0, 0, 1, pixels), **kwargs) def rotate(self, img, degrees, **kwargs): """rotate of auto_augment policy""" _check_args_tf(kwargs) if _PIL_VER >= (5, 2): func = img.rotate(degrees, **kwargs) elif _PIL_VER >= (5, 0): post_trans = (0, 0) rot_center = (img.size[0] / 2.0, img.size[1] / 2.0) angle = -math.radians(degrees) cos_val = round(math.cos(angle), 15) sin_val = round(math.sin(angle), 15) sin_val_neg = round(-math.sin(angle), 15) matrix = [cos_val, sin_val, 0.0, sin_val_neg, cos_val, 0.0] x = -rot_center[0] - post_trans[0] y = -rot_center[1] - post_trans[1] (a, b, c, d, e, f) = matrix matrix[2] = a * x + b * y + c matrix[5] = d * x + e * y + f matrix[2] += rot_center[0] matrix[5] += rot_center[1] func = img.transform(img.size, Image.AFFINE, matrix, **kwargs) else: func = img.rotate(degrees, resample=kwargs['resample']) return func def auto_contrast(self, img, **__): """auto_contrast of auto_augment policy""" return ImageOps.autocontrast(img) def invert(self, img, **__): """invert of auto_augment policy""" return ImageOps.invert(img) def equalize(self, img, **__): """equalize of auto_augment policy""" return ImageOps.equalize(img) def solarize(self, img, thresh, **__): """solarize of auto_augment policy""" return ImageOps.solarize(img, thresh) def solarize_add(self, img, add, thresh=128, **__): """solarize_add of auto_augment policy""" lut = [(min(255, i + add) if i < thresh else i) for i in range(256)] if img.mode == "RGB" and len(lut) == 256: lut = lut + lut + lut func = img.point(lut) if img.mode in ("L", "RGB") else img return func def posterize(self, img, bits_to_keep, **__): """posterize of auto_augment policy""" func = img if bits_to_keep >= 8 else ImageOps.posterize(img, bits_to_keep) return func def contrast(self, img, factor, **__): """contrast of auto_augment policy""" return ImageEnhance.Contrast(img).enhance(factor) def color(self, img, factor, **__): """color of auto_augment policy""" return ImageEnhance.Color(img).enhance(factor) def brightness(self, img, factor, **__): """brightness of auto_augment policy""" return ImageEnhance.Brightness(img).enhance(factor) def sharpness(self, img, factor, **__): """sharpness of auto_augment policy""" return ImageEnhance.Sharpness(img).enhance(factor) class LevelHelper: """Level helper """ def _randomly_negate(self, v): """With 50% prob, negate the value""" return -v if random.random() > 0.5 else v def rotate_level_to_arg(self, level, _): """_randomly_negate of auto_augment level""" # range [-30, 30] level = self._randomly_negate((level / _MAX_LEVEL) * 30.) return (level,) def enhance_level_to_arg(self, level, _): """enhance_level_to_arg of auto_augment level""" # range [0.1, 1.9] return ((level / _MAX_LEVEL) * 1.8 + 0.1,) def enhance_increasing_level_to_arg(self, level, _): """enhance_increasing_level_to_arg of auto_augment level""" # the 'no change' level is 1.0, moving away from that towards 0. or 2.0 increases the enhancement blend # range [0.1, 1.9] level = 1.0 + self._randomly_negate((level / _MAX_LEVEL) * .9) return (level,) def shear_level_to_arg(self, level, _): """shear_level_to_arg of auto_augment level""" # range [-0.3, 0.3] level = self._randomly_negate((level / _MAX_LEVEL) * 0.3) return (level,) def translate_abs_level_to_arg(self, level, hparams): """translate_abs_level_to_arg of auto_augment level""" translate_const = hparams['translate_const'] level = self._randomly_negate((level / _MAX_LEVEL) * float(translate_const)) return (level,) def translate_rel_level_to_arg(self, level, hparams): """translate_rel_level_to_arg of auto_augment level""" # default range [-0.45, 0.45] translate_pct = hparams.get('translate_pct', 0.45) level = self._randomly_negate((level / _MAX_LEVEL) * translate_pct) return (level,) def posterize_level_to_arg(self, level, _): """posterize_level_to_arg of auto_augment level""" # As per Tensorflow TPU EfficientNet impl # range [0, 4], 'keep 0 up to 4 MSB of original image' # intensity/severity of augmentation decreases with level return (int((level / _MAX_LEVEL) * 4),) def posterize_increasing_level_to_arg(self, level, hparams): """posterize_increasing_level_to_arg of auto_augment level""" # As per Tensorflow models research and UDA impl # range [4, 0], 'keep 4 down to 0 MSB of original image', # intensity/severity of augmentation increases with level return (4 - self.posterize_level_to_arg(level, hparams)[0],) def posterize_original_level_to_arg(self, level, _): """posterize_original_level_to_arg of auto_augment level""" # As per original AutoAugment paper description # range [4, 8], 'keep 4 up to 8 MSB of image' # intensity/severity of augmentation decreases with level return (int((level / _MAX_LEVEL) * 4) + 4,) def solarize_level_to_arg(self, level, _): """solarize_level_to_arg of auto_augment level""" # range [0, 256] # intensity/severity of augmentation decreases with return (int((level / _MAX_LEVEL) * 256),) def solarize_increasing_level_to_arg(self, level, hparams): """solarize_increasing_level_to_arg of auto_augment level""" # range [0, 256] # intensity/severity of augmentation increases with level return (256 - self.solarize_level_to_arg(level, hparams)[0],) def solarize_add_level_to_arg(self, level, _): """solarize_add_level_to_arg of auto_augment level""" # range [0, 110] return (int((level / _MAX_LEVEL) * 110),) lh = LevelHelper() LEVEL_TO_ARG = { 'AutoContrast': None, 'Equalize': None, 'Invert': None, 'Rotate': lh.rotate_level_to_arg, # There are several variations of the posterize level scaling in various Tensorflow/Google repositories/papers 'Posterize': lh.posterize_level_to_arg, 'PosterizeIncreasing': lh.posterize_increasing_level_to_arg, 'PosterizeOriginal': lh.posterize_original_level_to_arg, 'Solarize': lh.solarize_level_to_arg, 'SolarizeIncreasing': lh.solarize_increasing_level_to_arg, 'SolarizeAdd': lh.solarize_add_level_to_arg, 'Color': lh.enhance_level_to_arg, 'ColorIncreasing': lh.enhance_increasing_level_to_arg, 'Contrast': lh.enhance_level_to_arg, 'ContrastIncreasing': lh.enhance_increasing_level_to_arg, 'Brightness': lh.enhance_level_to_arg, 'BrightnessIncreasing': lh.enhance_increasing_level_to_arg, 'Sharpness': lh.enhance_level_to_arg, 'SharpnessIncreasing': lh.enhance_increasing_level_to_arg, 'ShearX': lh.shear_level_to_arg, 'ShearY': lh.shear_level_to_arg, 'TranslateX': lh.translate_abs_level_to_arg, 'TranslateY': lh.translate_abs_level_to_arg, 'TranslateXRel': lh.translate_rel_level_to_arg, 'TranslateYRel': lh.translate_rel_level_to_arg, } ph = PolicyHelper() NAME_TO_OP = { 'AutoContrast': ph.auto_contrast, 'Equalize': ph.equalize, 'Invert': ph.invert, 'Rotate': ph.rotate, 'Posterize': ph.posterize, 'PosterizeIncreasing': ph.posterize, 'PosterizeOriginal': ph.posterize, 'Solarize': ph.solarize, 'SolarizeIncreasing': ph.solarize, 'SolarizeAdd': ph.solarize_add, 'Color': ph.color, 'ColorIncreasing': ph.color, 'Contrast': ph.contrast, 'ContrastIncreasing': ph.contrast, 'Brightness': ph.brightness, 'BrightnessIncreasing': ph.brightness, 'Sharpness': ph.sharpness, 'SharpnessIncreasing': ph.sharpness, 'ShearX': ph.shear_x, 'ShearY': ph.shear_y, 'TranslateX': ph.translate_x_abs, 'TranslateY': ph.translate_y_abs, 'TranslateXRel': ph.translate_x_rel, 'TranslateYRel': ph.translate_y_rel, } class AugmentOp: """AugmentOp""" def __init__(self, name, prob=0.5, magnitude=10, hparams=None): hparams = hparams or _HPARAMS_DEFAULT self.aug_fn = NAME_TO_OP[name] self.level_fn = LEVEL_TO_ARG[name] self.prob = prob self.magnitude = magnitude self.hparams = hparams.copy() self.kwargs = dict( fillcolor=tuple(hparams['img_mean']) if 'img_mean' in hparams else _FILL, resample=hparams['interpolation'] if 'interpolation' in hparams else _RANDOM_INTERPOLATION, ) # If magnitude_std is > 0, we introduce some randomness # in the usually fixed policy and sample magnitude from a normal distribution # with mean `magnitude` and std-dev of `magnitude_std`. # NOTE This is my own hack, being tested, not in papers or reference impls. # If magnitude_std is inf, we sample magnitude from a uniform distribution self.magnitude_std = self.hparams.get('magnitude_std', 0) def __call__(self, img): """apply augment""" if self.prob < 1.0 and random.random() > self.prob: return img magnitude = self.magnitude if self.magnitude_std: if self.magnitude_std == float('inf'): magnitude = random.uniform(0, magnitude) elif self.magnitude_std > 0: magnitude = random.gauss(magnitude, self.magnitude_std) magnitude = min(_MAX_LEVEL, max(0, magnitude)) # clip to valid range if self.level_fn is not None: level_args = self.level_fn(magnitude, self.hparams) else: level_args = tuple() return self.aug_fn(img, *level_args, **self.kwargs) def auto_augment_policy_v0(hparams): """auto_augment_policy_v0""" # ImageNet v0 policy from TPU EfficientNet impl, cannot find a paper reference. policy = [ [('Equalize', 0.8, 1), ('ShearY', 0.8, 4)], [('Color', 0.4, 9), ('Equalize', 0.6, 3)], [('Color', 0.4, 1), ('Rotate', 0.6, 8)], [('Solarize', 0.8, 3), ('Equalize', 0.4, 7)], [('Solarize', 0.4, 2), ('Solarize', 0.6, 2)], [('Color', 0.2, 0), ('Equalize', 0.8, 8)], [('Equalize', 0.4, 8), ('SolarizeAdd', 0.8, 3)], [('ShearX', 0.2, 9), ('Rotate', 0.6, 8)], [('Color', 0.6, 1), ('Equalize', 1.0, 2)], [('Invert', 0.4, 9), ('Rotate', 0.6, 0)], [('Equalize', 1.0, 9), ('ShearY', 0.6, 3)], [('Color', 0.4, 7), ('Equalize', 0.6, 0)], [('Posterize', 0.4, 6), ('AutoContrast', 0.4, 7)], [('Solarize', 0.6, 8), ('Color', 0.6, 9)], [('Solarize', 0.2, 4), ('Rotate', 0.8, 9)], [('Rotate', 1.0, 7), ('TranslateYRel', 0.8, 9)], [('ShearX', 0.0, 0), ('Solarize', 0.8, 4)], [('ShearY', 0.8, 0), ('Color', 0.6, 4)], [('Color', 1.0, 0), ('Rotate', 0.6, 2)], [('Equalize', 0.8, 4), ('Equalize', 0.0, 8)], [('Equalize', 1.0, 4), ('AutoContrast', 0.6, 2)], [('ShearY', 0.4, 7), ('SolarizeAdd', 0.6, 7)], [('Posterize', 0.8, 2), ('Solarize', 0.6, 10)], # This results in black image with Tpu posterize [('Solarize', 0.6, 8), ('Equalize', 0.6, 1)], [('Color', 0.8, 6), ('Rotate', 0.4, 5)], ] pc = [[AugmentOp(*a, hparams=hparams) for a in sp] for sp in policy] return pc def auto_augment_policy_v0r(hparams): """auto_augment_policy_v0r""" # ImageNet v0 policy from TPU EfficientNet impl, with variation of Posterize used # in Google research implementation (number of bits discarded increases with magnitude) policy = [ [('Equalize', 0.8, 1), ('ShearY', 0.8, 4)], [('Color', 0.4, 9), ('Equalize', 0.6, 3)], [('Color', 0.4, 1), ('Rotate', 0.6, 8)], [('Solarize', 0.8, 3), ('Equalize', 0.4, 7)], [('Solarize', 0.4, 2), ('Solarize', 0.6, 2)], [('Color', 0.2, 0), ('Equalize', 0.8, 8)], [('Equalize', 0.4, 8), ('SolarizeAdd', 0.8, 3)], [('ShearX', 0.2, 9), ('Rotate', 0.6, 8)], [('Color', 0.6, 1), ('Equalize', 1.0, 2)], [('Invert', 0.4, 9), ('Rotate', 0.6, 0)], [('Equalize', 1.0, 9), ('ShearY', 0.6, 3)], [('Color', 0.4, 7), ('Equalize', 0.6, 0)], [('PosterizeIncreasing', 0.4, 6), ('AutoContrast', 0.4, 7)], [('Solarize', 0.6, 8), ('Color', 0.6, 9)], [('Solarize', 0.2, 4), ('Rotate', 0.8, 9)], [('Rotate', 1.0, 7), ('TranslateYRel', 0.8, 9)], [('ShearX', 0.0, 0), ('Solarize', 0.8, 4)], [('ShearY', 0.8, 0), ('Color', 0.6, 4)], [('Color', 1.0, 0), ('Rotate', 0.6, 2)], [('Equalize', 0.8, 4), ('Equalize', 0.0, 8)], [('Equalize', 1.0, 4), ('AutoContrast', 0.6, 2)], [('ShearY', 0.4, 7), ('SolarizeAdd', 0.6, 7)], [('PosterizeIncreasing', 0.8, 2), ('Solarize', 0.6, 10)], [('Solarize', 0.6, 8), ('Equalize', 0.6, 1)], [('Color', 0.8, 6), ('Rotate', 0.4, 5)], ] pc = [[AugmentOp(*a, hparams=hparams) for a in sp] for sp in policy] return pc def auto_augment_policy_original(hparams): """auto_augment_policy_original""" policy = [ [('PosterizeOriginal', 0.4, 8), ('Rotate', 0.6, 9)], [('Solarize', 0.6, 5), ('AutoContrast', 0.6, 5)], [('Equalize', 0.8, 8), ('Equalize', 0.6, 3)], [('PosterizeOriginal', 0.6, 7), ('PosterizeOriginal', 0.6, 6)], [('Equalize', 0.4, 7), ('Solarize', 0.2, 4)], [('Equalize', 0.4, 4), ('Rotate', 0.8, 8)], [('Solarize', 0.6, 3), ('Equalize', 0.6, 7)], [('PosterizeOriginal', 0.8, 5), ('Equalize', 1.0, 2)], [('Rotate', 0.2, 3), ('Solarize', 0.6, 8)], [('Equalize', 0.6, 8), ('PosterizeOriginal', 0.4, 6)], [('Rotate', 0.8, 8), ('Color', 0.4, 0)], [('Rotate', 0.4, 9), ('Equalize', 0.6, 2)], [('Equalize', 0.0, 7), ('Equalize', 0.8, 8)], [('Invert', 0.6, 4), ('Equalize', 1.0, 8)], [('Color', 0.6, 4), ('Contrast', 1.0, 8)], [('Rotate', 0.8, 8), ('Color', 1.0, 2)], [('Color', 0.8, 8), ('Solarize', 0.8, 7)], [('Sharpness', 0.4, 7), ('Invert', 0.6, 8)], [('ShearX', 0.6, 5), ('Equalize', 1.0, 9)], [('Color', 0.4, 0), ('Equalize', 0.6, 3)], [('Equalize', 0.4, 7), ('Solarize', 0.2, 4)], [('Solarize', 0.6, 5), ('AutoContrast', 0.6, 5)], [('Invert', 0.6, 4), ('Equalize', 1.0, 8)], [('Color', 0.6, 4), ('Contrast', 1.0, 8)], [('Equalize', 0.8, 8), ('Equalize', 0.6, 3)], ] pc = [[AugmentOp(*a, hparams=hparams) for a in sp] for sp in policy] return pc def auto_augment_policy_originalr(hparams): """auto_augment_policy_originalr""" # ImageNet policy from https://arxiv.org/abs/1805.09501 with research posterize variation policy = [ [('PosterizeIncreasing', 0.4, 8), ('Rotate', 0.6, 9)], [('Solarize', 0.6, 5), ('AutoContrast', 0.6, 5)], [('Equalize', 0.8, 8), ('Equalize', 0.6, 3)], [('PosterizeIncreasing', 0.6, 7), ('PosterizeIncreasing', 0.6, 6)], [('Equalize', 0.4, 7), ('Solarize', 0.2, 4)], [('Equalize', 0.4, 4), ('Rotate', 0.8, 8)], [('Solarize', 0.6, 3), ('Equalize', 0.6, 7)], [('PosterizeIncreasing', 0.8, 5), ('Equalize', 1.0, 2)], [('Rotate', 0.2, 3), ('Solarize', 0.6, 8)], [('Equalize', 0.6, 8), ('PosterizeIncreasing', 0.4, 6)], [('Rotate', 0.8, 8), ('Color', 0.4, 0)], [('Rotate', 0.4, 9), ('Equalize', 0.6, 2)], [('Equalize', 0.0, 7), ('Equalize', 0.8, 8)], [('Invert', 0.6, 4), ('Equalize', 1.0, 8)], [('Color', 0.6, 4), ('Contrast', 1.0, 8)], [('Rotate', 0.8, 8), ('Color', 1.0, 2)], [('Color', 0.8, 8), ('Solarize', 0.8, 7)], [('Sharpness', 0.4, 7), ('Invert', 0.6, 8)], [('ShearX', 0.6, 5), ('Equalize', 1.0, 9)], [('Color', 0.4, 0), ('Equalize', 0.6, 3)], [('Equalize', 0.4, 7), ('Solarize', 0.2, 4)], [('Solarize', 0.6, 5), ('AutoContrast', 0.6, 5)], [('Invert', 0.6, 4), ('Equalize', 1.0, 8)], [('Color', 0.6, 4), ('Contrast', 1.0, 8)], [('Equalize', 0.8, 8), ('Equalize', 0.6, 3)], ] pc = [[AugmentOp(*a, hparams=hparams) for a in sp] for sp in policy] return pc def auto_augment_policy(name='v0', hparams=None): """auto_augment_policy""" hparams = hparams or _HPARAMS_DEFAULT if name == 'original': func = auto_augment_policy_original(hparams) elif name == 'originalr': func = auto_augment_policy_originalr(hparams) elif name == 'v0': func = auto_augment_policy_v0(hparams) elif name == 'v0r': func = auto_augment_policy_v0r(hparams) else: raise ValueError(f'Unknown AA policy {name}') return func class AutoAugment: """AutoAugment""" def __init__(self, policy): self.policy = policy def __call__(self, img): """apply autoaugment""" sub_policy = random.choice(self.policy) for op in sub_policy: img = op(img) return img @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) def auto_augment_transform(config_str, hparams): """ Create a AutoAugment transform :param config_str: String defining configuration of auto augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the AutoAugment policy (one of 'v0', 'v0r', 'original', 'originalr'). The remaining sections, not order specific determine 'mstd' - float std deviation of magnitude noise applied Ex 'original-mstd0.5' results in AutoAugment with original policy, magnitude_std 0.5 :param hparams: Other hparams (kwargs) for the AutoAugmentation scheme :return: A MindSpore compatible Transform """ config = config_str.split('-') policy_name = config[0] config = config[1:] for c in config: cs = re.split(r'(\d.*)', c) if len(cs) >= 2: key, val = cs[:2] if key == 'mstd': # noise param injected via hparams for now hparams.setdefault('magnitude_std', float(val)) else: raise ValueError('Unknown AutoAugment config section') return AutoAugment(auto_augment_policy(policy_name, hparams=hparams)) _RAND_TRANSFORMS = [ 'AutoContrast', 'Equalize', 'Invert', 'Rotate', 'Posterize', 'Solarize', 'SolarizeAdd', 'Color', 'Contrast', 'Brightness', 'Sharpness', 'ShearX', 'ShearY', 'TranslateXRel', 'TranslateYRel', # 'Cutout' # NOTE I've implement this as random erasing separately ] _RAND_INCREASING_TRANSFORMS = [ 'AutoContrast', 'Equalize', 'Invert', 'Rotate', 'PosterizeIncreasing', 'SolarizeIncreasing', 'SolarizeAdd', 'ColorIncreasing', 'ContrastIncreasing', 'BrightnessIncreasing', 'SharpnessIncreasing', 'ShearX', 'ShearY', 'TranslateXRel', 'TranslateYRel', # 'Cutout' # NOTE I've implement this as random erasing separately ] # These experimental weights are based loosely on the relative improvements mentioned in paper. # They may not result in increased performance, but could likely be tuned to so. _RAND_CHOICE_WEIGHTS_0 = { 'Rotate': 0.3, 'ShearX': 0.2, 'ShearY': 0.2, 'TranslateXRel': 0.1, 'TranslateYRel': 0.1, 'Color': .025, 'Sharpness': 0.025, 'AutoContrast': 0.025, 'Solarize': .005, 'SolarizeAdd': .005, 'Contrast': .005, 'Brightness': .005, 'Equalize': .005, 'Posterize': 0, 'Invert': 0, } def _select_rand_weights(weight_idx=0, transforms=None): """_select_rand_weights""" transforms = transforms or _RAND_TRANSFORMS if weight_idx != 0: raise ValueError(f"only one set of weights currently, get weight_idx {weight_idx}") rand_weights = _RAND_CHOICE_WEIGHTS_0 probs = [rand_weights[k] for k in transforms] probs /= np.sum(probs) return probs def rand_augment_ops(magnitude=10, hparams=None, transforms=None): """rand_augment_ops""" hparams = hparams or _HPARAMS_DEFAULT transforms = transforms or _RAND_TRANSFORMS return [AugmentOp( name, prob=0.5, magnitude=magnitude, hparams=hparams) for name in transforms] class RandAugment(PyTensorOperation): """RandAugment""" def __init__(self, ops, num_layers=2, choice_weights=None): super(RandAugment, self).__init__() self.ops = ops self.num_layers = num_layers self.choice_weights = choice_weights def __call__(self, img): # no replacement when using weighted choice ops = np.random.choice( self.ops, self.num_layers, replace=self.choice_weights is None, p=self.choice_weights) for op in ops: img = op(img) return img @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) def rand_augment_transform(config_str, hparams): """ Create a RandAugment transform :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining sections, not order specific determine 'm' - integer magnitude of rand augment 'n' - integer num layers (number of transform ops selected per image) 'w' - integer probabiliy weight index (index of a set of weights to influence choice of op) 'mstd' - float std deviation of magnitude noise applied 'inc' - integer (bool), use augmentations that increase in severity with magnitude (default: 0) Ex 'rand-m9-n3-mstd0.5' results in RandAugment with magnitude 9, num_layers 3, magnitude_std 0.5 'rand-mstd1-w0' results in magnitude_std 1.0, weights 0, default magnitude of 10 and num_layers 2 :param hparams: Other hparams (kwargs) for the RandAugmentation scheme :return: A MindSpore compatible Transform """ magnitude = _MAX_LEVEL # default to _MAX_LEVEL for magnitude (currently 10) num_layers = 2 # default to 2 ops per image weight_idx = None # default to no probability weights for op choice transforms = _RAND_TRANSFORMS config = config_str.split('-') if config[0] != 'rand': raise ValueError(f"config[0] is not rand, get {config[0]}") # [rand, m9, mstd0.5, inc1] config = config[1:] for c in config: cs = re.split(r'(\d.*)', c) if len(cs) >= 2: key, val = cs[:2] if key == 'mstd': # noise param injected via hparams for now hparams.setdefault('magnitude_std', float(val)) elif key == 'inc': if bool(val): transforms = _RAND_INCREASING_TRANSFORMS elif key == 'm': magnitude = int(val) elif key == 'n': num_layers = int(val) elif key == 'w': weight_idx = int(val) else: raise ValueError('Unknown RandAugment config section') ra_ops = rand_augment_ops(magnitude=magnitude, hparams=hparams, transforms=transforms) choice_weights = None if weight_idx is None else _select_rand_weights(weight_idx) return RandAugment(ra_ops, num_layers, choice_weights=choice_weights) _AUGMIX_TRANSFORMS = [ 'AutoContrast', 'ColorIncreasing', # not in paper 'ContrastIncreasing', # not in paper 'BrightnessIncreasing', # not in paper 'SharpnessIncreasing', # not in paper 'Equalize', 'Rotate', 'PosterizeIncreasing', 'SolarizeIncreasing', 'ShearX', 'ShearY', 'TranslateXRel', 'TranslateYRel', ] def augmix_ops(magnitude=10, hparams=None, transforms=None): """augmix_ops""" hparams = hparams or _HPARAMS_DEFAULT transforms = transforms or _AUGMIX_TRANSFORMS return [AugmentOp( name, prob=1.0, magnitude=magnitude, hparams=hparams) for name in transforms] class AugMixAugment: """ AugMix Transform Adapted and improved from impl here From paper: 'AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty - https://arxiv.org/abs/1912.02781 """ def __init__(self, ops, alpha=1., width=3, depth=-1, blended=False): self.ops = ops self.alpha = alpha self.width = width self.depth = depth self.blended = blended # blended mode is faster but not well tested def _calc_blended_weights(self, ws, m): """_calc_blended_weights of AugMixAugment""" ws = ws * m cump = 1. rws = [] for w in ws[::-1]: alpha = w / cump cump *= (1 - alpha) rws.append(alpha) return np.array(rws[::-1], dtype=np.float32) def _apply_blended(self, img, mixing_weights, m): """_apply_blended of AugMixAugment""" # This is my first crack and implementing a slightly faster mixed augmentation. Instead # of accumulating the mix for each chain in a Numpy array and then blending with original, # it recomputes the blending coefficients and applies one PIL image blend per chain. # TODO the results appear in the right ballpark but they differ by more than rounding. img_orig = img.copy() ws = self._calc_blended_weights(mixing_weights, m) for w in ws: depth = self.depth if self.depth > 0 else np.random.randint(1, 4) ops = np.random.choice(self.ops, depth, replace=True) img_aug = img_orig # no ops are in-place, deep copy not necessary for op in ops: img_aug = op(img_aug) img = Image.blend(img, img_aug, w) return img def _apply_basic(self, img, mixing_weights, m): """_apply_basic of AugMixAugment""" # This is a literal adaptation of the paper/official implementation without normalizations and # PIL <-> Numpy conversions between every op. It is still quite CPU compute heavy compared to the # typical augmentation transforms, could use a GPU / Kornia implementation. img_shape = img.size[0], img.size[1], len(img.getbands()) mixed = np.zeros(img_shape, dtype=np.float32) for mw in mixing_weights: depth = self.depth if self.depth > 0 else np.random.randint(1, 4) ops = np.random.choice(self.ops, depth, replace=True) img_aug = img # no ops are in-place, deep copy not necessary for op in ops: img_aug = op(img_aug) mixed += mw * np.asarray(img_aug, dtype=np.float32) np.clip(mixed, 0, 255., out=mixed) mixed = Image.fromarray(mixed.astype(np.uint8)) return Image.blend(img, mixed, m) def __call__(self, img): """AugMixAugment apply""" mixing_weights = np.float32(np.random.dirichlet([self.alpha] * self.width)) m = np.float32(np.random.beta(self.alpha, self.alpha)) if self.blended: mixed = self._apply_blended(img, mixing_weights, m) else: mixed = self._apply_basic(img, mixing_weights, m) return mixed @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) def augment_and_mix_transform(config_str, hparams): """ Create AugMix MindSpore transform :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining sections, not order specific determine 'm' - integer magnitude (severity) of augmentation mix (default: 3) 'w' - integer width of augmentation chain (default: 3) 'd' - integer depth of augmentation chain (-1 is random [1, 3], default: -1) 'b' - integer (bool), blend each branch of chain into end result without a final blend, less CPU (default: 0) 'mstd' - float std deviation of magnitude noise applied (default: 0) Ex 'augmix-m5-w4-d2' results in AugMix with severity 5, chain width 4, chain depth 2 :param hparams: Other hparams (kwargs) for the Augmentation transforms :return: A MindSpore compatible Transform """ magnitude = 3 width = 3 depth = -1 alpha = 1. blended = False hparams['magnitude_std'] = float('inf') config = config_str.split('-') if config[0] != 'augmix': raise ValueError(f"config[0] is not augmix, get {config[0]}") config = config[1:] for c in config: cs = re.split(r'(\d.*)', c) if len(cs) >= 2: key, val = cs[:2] if key == 'mstd': # noise param injected via hparams for now hparams.setdefault('magnitude_std', float(val)) elif key == 'm': magnitude = int(val) elif key == 'w': width = int(val) elif key == 'd': depth = int(val) elif key == 'a': alpha = float(val) elif key == 'b': blended = bool(val) else: raise ValueError('Unknown AugMix config section') ops = augmix_ops(magnitude=magnitude, hparams=hparams) return AugMixAugment(ops, alpha=alpha, width=width, depth=depth, blended=blended) pengcheng.mind-test/mindformers/dataset/transforms/build_transforms.py000066400000000000000000000064441471400134000270200ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Transforms API.""" import inspect from mindspore.dataset import transforms as tf from mindspore.dataset import vision as vs from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_transforms( config: [dict, list] = None, default_args: dict = None, module_type: str = 'transforms', class_name: str = None, **kwargs): r"""Build transform For MindFormer. Instantiate the transform from MindFormerRegister's registry. Args: config (dict, list): The task transform's config. Default: None. default_args (dict): The default argument of transform API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'transform'. class_name (str): The class name of transform API. Default: None. Return: The function instance of transform API. Examples: >>> from mindformers import build_transforms >>> transform_config = [{'type': 'Decode'}, {'type': 'Resize', 'size': 256}] >>> # 1) use config dict to build transform >>> transform_from_config = build_transforms(transform_config) >>> # 2) use class name to build transform >>> transform_class_name = build_transforms(class_name='Decode') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) cfg_transforms = config if not isinstance(cfg_transforms, list): return MindFormerRegister.get_instance_from_cfg( cfg_transforms, MindFormerModuleType.TRANSFORMS, default_args=default_args) transforms = [] for transform in cfg_transforms: transform_op = MindFormerRegister.get_instance_from_cfg( transform, MindFormerModuleType.TRANSFORMS) transforms.append(transform_op) return transforms return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_transforms(): """register MindSpore builtin transforms class.""" for module_name in set(dir(vs.transforms) + dir(tf.transforms)): if module_name.startswith('__'): continue transforms = getattr(vs.transforms, module_name, None) \ if getattr(vs.transforms, module_name, None)\ else getattr(tf.transforms, module_name) if inspect.isclass(transforms): MindFormerRegister.register_cls(transforms, MindFormerModuleType.TRANSFORMS) register_ms_transforms() pengcheng.mind-test/mindformers/dataset/transforms/mixup.py000066400000000000000000000263651471400134000246110ustar00rootroot00000000000000# Mixup and Cutmix for MindSpore # # Papers: # mixup: Beyond Empirical Risk Minimization (https://arxiv.org/abs/1710.09412) # # CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features (https://arxiv.org/abs/1905.04899) # # Code Reference: # CutMix: https://github.com/clovaai/CutMix-PyTorch, Apache 2.0 # # Hacked together by / Copyright 2020 Ross Wightman # ============================================================================ """Mixup and Cutmix for MindSpore""" import numpy as np from mindspore import Tensor from mindspore import dtype as mstype from mindspore import ops as P from mindspore.dataset.vision.transforms import PyTensorOperation from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType def one_hot(x, num_classes, on_value=1., off_value=0.): """one hot to label""" x = x.reshape(-1) x = np.eye(num_classes)[x] x = np.clip(x, a_min=off_value, a_max=on_value, dtype=np.float32) return x def mixup_target(target, num_classes, lam=1., smoothing=0.0): """mixup_target""" off_value = smoothing / num_classes on_value = 1. - smoothing + off_value y1 = one_hot(target, num_classes, on_value=on_value, off_value=off_value) y2 = one_hot(np.flip(target, axis=0), num_classes, on_value=on_value, off_value=off_value) return y1 * lam + y2 * (1. - lam) def rand_bbox(img_shape, lam, margin=0., count=None): """ Standard CutMix bounding-box Generates a random square bbox based on lambda value. This impl includes support for enforcing a border margin as percent of bbox dimensions. Args: img_shape (tuple): Image shape as tuple lam (float): Cutmix lambda value margin (float): Percentage of bbox dimension to enforce as margin (reduce amount of box outside image) count (int): Number of bbox to generate """ ratio = np.sqrt(1 - lam) img_h, img_w = img_shape[-2:] cut_h, cut_w = int(img_h * ratio), int(img_w * ratio) margin_y, margin_x = int(margin * cut_h), int(margin * cut_w) cy = np.random.randint(0 + margin_y, img_h - margin_y, size=count) cx = np.random.randint(0 + margin_x, img_w - margin_x, size=count) yl = np.clip(cy - cut_h // 2, 0, img_h) yh = np.clip(cy + cut_h // 2, 0, img_h) xl = np.clip(cx - cut_w // 2, 0, img_w) xh = np.clip(cx + cut_w // 2, 0, img_w) return yl, yh, xl, xh def rand_bbox_minmax(img_shape, minmax, count=None): """ Min-Max CutMix bounding-box Inspired by Darknet cutmix impl, generates a random rectangular bbox based on min/max percent values applied to each dimension of the input image. Typical defaults for minmax are usually in the .2-.3 for min and .8-.9 range for max. Args: img_shape (tuple): Image shape as tuple minmax (tuple or list): Min and max bbox ratios (as percent of image size) count (int): Number of bbox to generate """ if len(minmax) != 2: raise ValueError(f"len(minmax) is not 2. ") img_h, img_w = img_shape[-2:] cut_h = np.random.randint(int(img_h * minmax[0]), int(img_h * minmax[1]), size=count) cut_w = np.random.randint(int(img_w * minmax[0]), int(img_w * minmax[1]), size=count) yl = np.random.randint(0, img_h - cut_h, size=count) xl = np.random.randint(0, img_w - cut_w, size=count) yu = yl + cut_h xu = xl + cut_w return yl, yu, xl, xu def cutmix_bbox_and_lam(img_shape, lam, ratio_minmax=None, correct_lam=True, count=None): """ Generate bbox and apply lambda correction. """ if ratio_minmax is not None: yl, yu, xl, xu = rand_bbox_minmax(img_shape, ratio_minmax, count=count) else: yl, yu, xl, xu = rand_bbox(img_shape, lam, count=count) if correct_lam or ratio_minmax is not None: bbox_area = (yu - yl) * (xu - xl) lam = 1. - bbox_area / float(img_shape[-2] * img_shape[-1]) return (yl, yu, xl, xu), lam @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class Mixup(PyTensorOperation): """ Mixup/Cutmix that applies different params to each element or whole batch Args: mixup_alpha (float): mixup alpha value, mixup is active if > 0. cutmix_alpha (float): cutmix alpha value, cutmix is active if > 0. cutmix_minmax (List[float]): cutmix min/max image ratio, cutmix is active and uses this vs alpha if not None. prob (float): probability of applying mixup or cutmix per batch or element switch_prob (float): probability of switching to cutmix instead of mixup when both are active mode (str): how to apply mixup/cutmix params (per 'batch', 'pair' (pair of elements), 'elem' (element) correct_lam (bool): apply lambda correction when cutmix bbox clipped by image borders label_smoothing (float): apply label smoothing to the mixed target tensor num_classes (int): number of classes for target """ def __init__(self, mixup_alpha=1., cutmix_alpha=0., cutmix_minmax=None, prob=1.0, switch_prob=0.5, mode='batch', correct_lam=True, label_smoothing=0.1, num_classes=1000): super(Mixup, self).__init__() self.mixup_alpha = mixup_alpha self.cutmix_alpha = cutmix_alpha self.cutmix_minmax = cutmix_minmax if self.cutmix_minmax is not None: if len(self.cutmix_minmax) != 2: raise ValueError(f"len(cutmix_minmax) is not 2.") # force cutmix alpha == 1.0 when minmax active to keep logic simple & safe self.cutmix_alpha = 1.0 self.mix_prob = prob self.switch_prob = switch_prob self.label_smoothing = label_smoothing self.num_classes = num_classes self.mode = mode self.correct_lam = correct_lam # correct lambda based on clipped area for cutmix self.mixup_enabled = True # set to false to disable mixing (intended tp be set by train loop) def _params_per_elem(self, batch_size): """_params_per_elem""" lam = np.ones(batch_size, dtype=np.float32) use_cutmix = np.zeros(batch_size, dtype=np.bool) if self.mixup_enabled: if self.mixup_alpha > 0. and self.cutmix_alpha > 0.: use_cutmix = np.random.rand(batch_size) < self.switch_prob lam_mix = np.where( use_cutmix, np.random.beta(self.cutmix_alpha, self.cutmix_alpha, size=batch_size), np.random.beta(self.mixup_alpha, self.mixup_alpha, size=batch_size)) elif self.mixup_alpha > 0.: lam_mix = np.random.beta(self.mixup_alpha, self.mixup_alpha, size=batch_size) elif self.cutmix_alpha > 0.: use_cutmix = np.ones(batch_size, dtype=np.bool) lam_mix = np.random.beta(self.cutmix_alpha, self.cutmix_alpha, size=batch_size) else: raise ValueError("One of mixup_alpha > 0., cutmix_alpha > 0., cutmix_minmax not None should be true.") lam = np.where(np.random.rand(batch_size) < self.mix_prob, lam_mix.astype(np.float32), lam) return lam, use_cutmix def _params_per_batch(self): """_params_per_batch""" lam = 1. use_cutmix = False if self.mixup_enabled and np.random.rand() < self.mix_prob: if self.mixup_alpha > 0. and self.cutmix_alpha > 0.: use_cutmix = np.random.rand() < self.switch_prob lam_mix = np.random.beta(self.cutmix_alpha, self.cutmix_alpha) if use_cutmix else \ np.random.beta(self.mixup_alpha, self.mixup_alpha) elif self.mixup_alpha > 0.: lam_mix = np.random.beta(self.mixup_alpha, self.mixup_alpha) elif self.cutmix_alpha > 0.: use_cutmix = True lam_mix = np.random.beta(self.cutmix_alpha, self.cutmix_alpha) else: raise ValueError("One of mixup_alpha > 0., cutmix_alpha > 0., cutmix_minmax not None should be true.") lam = float(lam_mix) return lam, use_cutmix def _mix_elem(self, x): """_mix_elem""" batch_size = len(x) lam_batch, use_cutmix = self._params_per_elem(batch_size) x_orig = x.clone() # need to keep an unmodified original for mixing source for i in range(batch_size): j = batch_size - i - 1 lam = lam_batch[i] if lam != 1.: if use_cutmix[i]: (yl, yh, xl, xh), lam = cutmix_bbox_and_lam( x[i].shape, lam, ratio_minmax=self.cutmix_minmax, correct_lam=self.correct_lam) x[i][:, yl:yh, xl:xh] = x_orig[j][:, yl:yh, xl:xh] lam_batch[i] = lam else: x[i] = x[i] * lam + x_orig[j] * (1 - lam) return P.ExpandDims()(Tensor(lam_batch, dtype=mstype.float32), 1) def _mix_pair(self, x): """_mix_pair""" batch_size = len(x) lam_batch, use_cutmix = self._params_per_elem(batch_size // 2) x_orig = x.clone() # need to keep an unmodified original for mixing source for i in range(batch_size // 2): j = batch_size - i - 1 lam = lam_batch[i] if lam != 1.: if use_cutmix[i]: (yl, yh, xl, xh), lam = cutmix_bbox_and_lam( x[i].shape, lam, ratio_minmax=self.cutmix_minmax, correct_lam=self.correct_lam) x[i][:, yl:yh, xl:xh] = x_orig[j][:, yl:yh, xl:xh] x[j][:, yl:yh, xl:xh] = x_orig[i][:, yl:yh, xl:xh] lam_batch[i] = lam else: x[i] = x[i] * lam + x_orig[j] * (1 - lam) x[j] = x[j] * lam + x_orig[i] * (1 - lam) lam_batch = np.concatenate((lam_batch, lam_batch[::-1])) return P.ExpandDims()(Tensor(lam_batch, dtype=mstype.float32), 1) def _mix_batch(self, x): """_mix_batch""" lam, use_cutmix = self._params_per_batch() if lam == 1.: return 1. if use_cutmix: (yl, yh, xl, xh), lam = cutmix_bbox_and_lam( x.shape, lam, ratio_minmax=self.cutmix_minmax, correct_lam=self.correct_lam) x[:, :, yl:yh, xl:xh] = np.flip(x, axis=0)[:, :, yl:yh, xl:xh] else: x_flipped = np.flip(x, axis=0) * (1. - lam) x *= lam x += x_flipped return lam def __call__(self, x, target): """Mixup apply""" # the same to image, label if len(x) % 2 != 0: if len(x) > 1: x = x[:-1] logger.warning('Batch size is odd. When using mixup, batch size should be even.' 'The last data in batch has been dropped to use mixip.' 'you can set "drop_remainder" true in dataset config manually.') else: logger.error('Batch size is 1.' 'If error occurs, please set "drop_remainder" true in dataset config.') if self.mode == 'elem': lam = self._mix_elem(x) elif self.mode == 'pair': lam = self._mix_pair(x) else: lam = self._mix_batch(x) target = mixup_target(target, self.num_classes, lam, self.label_smoothing) return x.astype(np.float32), target.astype(np.float32) pengcheng.mind-test/mindformers/dataset/transforms/random_erasing.py000066400000000000000000000113561471400134000264310ustar00rootroot00000000000000# Random Erasing (Cutout) for MindSpore # # Originally inspired by impl at https://github.com/zhunzhong07/Random-Erasing, Apache 2.0 # Copyright Zhun Zhong & Liang Zheng # # Hacked together by / Copyright 2020 Ross Wightman # # has been modified to adapt mindspore and mindformers # ============================================================================ """Random Erasing (Cutout) for MindSpore""" import math import random import numpy as np from mindspore.dataset.vision.transforms import PyTensorOperation from mindformers.tools.register import MindFormerRegister, MindFormerModuleType def _get_pixels(per_pixel, rand_color, patch_size, dtype=np.float32): """_get_pixels""" if per_pixel: func = np.random.normal(size=patch_size).astype(dtype) elif rand_color: func = np.random.normal(size=(patch_size[0], 1, 1)).astype(dtype) else: func = np.zeros((patch_size[0], 1, 1), dtype=dtype) return func @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomErasing(PyTensorOperation): """ Randomly selects a rectangle region in an image and erases its pixels. 'Random Erasing Data Augmentation' by Zhong et al. See https://arxiv.org/pdf/1708.04896.pdf This variant of RandomErasing is intended to be applied to either a batch or single image tensor after it has been normalized by dataset mean and std. Args: probability: Probability that the Random Erasing operation will be performed. min_area: Minimum percentage of erased area wrt input image area. max_area: Maximum percentage of erased area wrt input image area. min_aspect: Minimum aspect ratio of erased area. mode: pixel color mode, one of 'const', 'rand', or 'pixel' 'const' - erase block is constant color of 0 for all channels 'rand' - erase block is same per-channel random (normal) color 'pixel' - erase block is per-pixel random (normal) color max_count: maximum number of erasing blocks per image, area per box is scaled by count. per-image count is randomly chosen between 1 and this value. """ def __init__(self, probability=0.5, min_area=0.02, max_area=1 / 3, min_aspect=0.3, max_aspect=None, mode='const', min_count=1, max_count=None, num_splits=0): super(RandomErasing, self).__init__() self.probability = probability self.min_area = min_area self.max_area = max_area max_aspect = max_aspect or 1 / min_aspect self.log_aspect_ratio = (math.log(min_aspect), math.log(max_aspect)) self.min_count = min_count self.max_count = max_count or min_count self.num_splits = num_splits mode = mode.lower() self.rand_color = False self.per_pixel = False if mode == 'rand': self.rand_color = True # per block random normal elif mode == 'pixel': self.per_pixel = True # per pixel random normal else: if mode and mode == 'const': raise ValueError("mode is const") def _erase(self, img, chan, img_h, img_w, dtype): """_erase""" if random.random() > self.probability: pass else: area = img_h * img_w count = self.min_count if self.min_count == self.max_count else \ random.randint(self.min_count, self.max_count) for _ in range(count): for _ in range(10): target_area = random.uniform(self.min_area, self.max_area) * area / count aspect_ratio = math.exp(random.uniform(*self.log_aspect_ratio)) h = int(round(math.sqrt(target_area * aspect_ratio))) w = int(round(math.sqrt(target_area / aspect_ratio))) if w < img_w and h < img_h: top = random.randint(0, img_h - h) left = random.randint(0, img_w - w) img[:, top:top + h, left:left + w] = _get_pixels( self.per_pixel, self.rand_color, (chan, h, w), dtype=dtype) break return img def __call__(self, x): """RandomErasing apply""" if len(x.shape) == 3: output = self._erase(x, *x.shape, x.dtype) else: output = np.zeros_like(x) batch_size, chan, img_h, img_w = x.shape # skip first slice of batch if num_splits is set (for clean portion of samples) batch_start = batch_size // self.num_splits if self.num_splits > 1 else 0 for i in range(batch_start, batch_size): output[i] = self._erase(x[i], chan, img_h, img_w, x.dtype) return output pengcheng.mind-test/mindformers/dataset/transforms/text_transforms.py000066400000000000000000000141271471400134000267020ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Transforms.""" import re import numpy as np from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = [ 'RandomChoiceTokenizerForward', 'TokenizerForward', 'TokenizeWithLabel', 'LabelPadding', 'CaptionTransform' ] @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomChoiceTokenizerForward: """Random Choice Tokenizer Forward""" def __init__(self, tokenizer, max_length=77, padding="max_length", random_seed=2022): self.max_length = max_length self.padding = padding self.tokenizer = tokenizer self.random_seed = random_seed def __call__(self, text): text_list = text.tolist() np.random.seed(self.random_seed) index = np.random.choice(len(text_list)) token_id = self.tokenizer( text_list[index].decode("utf-8", "ignore") if isinstance(text_list[index], bytes) else text_list[index], max_length=self.max_length, padding=self.padding )["input_ids"] return token_id @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class TokenizerForward: """Tokenizer Forward""" def __init__(self, tokenizer, max_length=77, padding="max_length"): self.tokenizer = tokenizer self.max_length = max_length self.padding = padding def __call__(self, text): """call method""" text = text.tolist() if isinstance(text, np.ndarray) else text for i in range(len(text)): if isinstance(text[i], bytes): text[i] = text[i].decode("utf-8", "ignore") token_id = self.tokenizer( text, max_length=self.max_length, padding=self.padding)["input_ids"] return token_id @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class TokenizeWithLabel: """Tokenizer With Label""" def __init__(self, tokenizer, max_length=128, padding="max_length"): self.tokenizer = tokenizer self.max_length = max_length self.padding = padding def __call__(self, *inputs): """call method""" text = inputs[0] label_id = inputs[1] text = text.tolist() if isinstance(text, np.ndarray) else text if isinstance(text, bytes): text = text.decode("utf-8", "ignore") output = self.tokenizer(text, max_length=self.max_length, padding=self.padding) input_ids = np.array(output["input_ids"], dtype=np.int32) token_type_ids = np.array(output["token_type_ids"], dtype=np.int32) attention_mask = np.array(output["attention_mask"], dtype=np.int32) return input_ids, token_type_ids, attention_mask, label_id @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class LabelPadding: """Label Padding""" def __init__(self, max_length, padding_value=0): self.max_length = max_length self.padding_value = padding_value def __call__(self, label_id): """call method""" if len(label_id) > self.max_length - 1: label_id = label_id[:(self.max_length - 1)] pad_label_id = [] # For CLS token pad_label_id.append(self.padding_value) for i in range(len(label_id)): pad_label_id.append(label_id[i]) while len(pad_label_id) < self.max_length: pad_label_id.append(self.padding_value) pad_label_id = np.array(pad_label_id, dtype=np.int32) return pad_label_id @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class CaptionTransform: """ Caption Transform, preprocess captions and tokenize it, align with torch impl. """ def __init__(self, tokenizer, prompt="", max_words=50, max_length=32, padding="max_length", random_seed=2022, truncation=True, add_special_tokens=True): self.tokenizer = tokenizer self.prompt = prompt self.max_words = max_words self.max_length = max_length self.padding = padding self.random_seed = random_seed self.truncation = truncation self.add_special_tokens = add_special_tokens def __call__(self, caption): if caption.ndim == 1: caption_list = [] for i in range(caption.shape[0]): caption_list.append(self.pre_caption(caption[i])) return caption_list input_ids = self.pre_caption(caption) return input_ids def pre_caption(self, caption): """ Caption preprocessing removes any punctuationmarks except commas, tailing spaces and transform sentence into lower case. """ caption = str(caption) caption = self.prompt + caption caption = re.sub( r"([.!\"()*#:;~])", " ", caption.lower(), ) caption = re.sub( r"\s{2,}", " ", caption, ) caption = caption.rstrip("\n") caption = caption.strip(" ") # truncate caption caption_words = caption.split(" ") if len(caption_words) > self.max_words: caption = " ".join(caption_words[: self.max_words]) output = self.tokenizer(caption, max_length=self.max_length, padding=self.padding, truncation=self.truncation, add_special_tokens=self.add_special_tokens) input_ids = np.array(output["input_ids"], dtype=np.int32) return input_ids pengcheng.mind-test/mindformers/dataset/transforms/vision_transforms.py000066400000000000000000000277541471400134000272370ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ transform methods for vision models """ import numpy as np from PIL import Image from mindspore.dataset import vision import mindspore as ms from mindspore.dataset.vision.utils import Inter from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = [ 'BatchResize', 'BCHW2BHWC', 'BatchPILize', 'BatchNormalize', 'BatchCenterCrop', 'BatchToTensor', 'RandomCropDecodeResize', 'RandomResizedCrop', 'Resize', 'RandomHorizontalFlip' ] INTERPOLATION = {'nearest': Inter.NEAREST, 'antialias': Inter.ANTIALIAS, 'linear': Inter.LINEAR, 'cubic': Inter.PILCUBIC, 'bicubic': Inter.BICUBIC} class BCHW2BHWC: """ Transform a batch of image from CHW to HWC. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the channel should be (bz, c, h, w) or (c, h, w). for list, the item should be PIL.Image or numpy.array (c, h, w). Return: transformed image batch: for numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for PIL.Image input, it is returned directly. """ def __call__(self, image_batch): """the call function""" if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return image_batch.transpose(0, 2, 3, 1) if len(image_batch.shape) == 3: return image_batch.transpose(1, 2, 0) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") if isinstance(image_batch, Image.Image): return image_batch raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchResize: """ Resize a batch of image to the given shape. Args: image_resolution (int): the target size. interpolation (str): interpolate method, default is 'cubic'. """ def __init__(self, image_resolution, interpolation='cubic'): self.interpolation = INTERPOLATION.get(interpolation) self.resize = vision.Resize(image_resolution, self.interpolation) def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the shape should be (bz, h, w, c) or (h, w, c). for list, the item should be PIL.Image or numpy.array (h, w, c). Returns: resized image batch: for numpy or tensor input, return a numpy array; for PIL.Image input, it returns PIL.Image. """ if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self.resize(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return np.row_stack([self.resize(item)[np.newaxis, :] for item in image_batch]) if len(image_batch.shape) == 3: return self.resize(image_batch) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") if isinstance(image_batch, Image.Image): return self.resize(image_batch) raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchCenterCrop: """ CenterCrop a batch of image to the given shape. Args: image_resolution (int): the target size. """ def __init__(self, image_resolution): self.crop = vision.CenterCrop(image_resolution) def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the shape should be (bz, h, w, c) or (h, w, c). for list, the item should be PIL.Image or numpy.array (h, w, c). Returns: center cropped image batch: for numpy or tensor input, return a numpy array, the shape is (bz, image_resolution, image_resolution, c) or (image_resolution, image_resolution, c); for PIL.Image input, it is returned with shape (image_resolution, image_resolution). """ if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self.crop(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return np.row_stack([self.crop(item)[np.newaxis, :] for item in image_batch]) if len(image_batch.shape) == 3: return self.crop(image_batch) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") if isinstance(image_batch, Image.Image): return self.crop(image_batch) raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchToTensor: """Transform a batch of image to tensor and scale to (0, 1).""" def __init__(self): self.totensor = ms.dataset.vision.ToTensor() def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be PIL.Image or numpy.array. Returns: return a tensor or a list of tensor. """ if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self.totensor(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return np.row_stack([self.totensor(item)[np.newaxis, :] for item in image_batch]) if len(image_batch.shape) == 3: return self.totensor(image_batch) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") if isinstance(image_batch, Image.Image): return self.totensor(image_batch) raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchNormalize: """Normalize a batch of image.""" def __init__( self, mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711), is_hwc=False ): self.normalize = vision.Normalize(mean=mean, std=std, is_hwc=is_hwc) def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be numpy.array. Returns: return a tensor or a list of tensor. """ if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self.normalize(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 3: return self.normalize(image_batch) if len(image_batch.shape) == 4: return np.row_stack([self.normalize(item)[np.newaxis, :] for item in image_batch]) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchPILize: """transform a batch of image to PIL.Image list.""" def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be PIL.Image. Returns: return a tensor or a list of tensor. """ if isinstance(image_batch, Image.Image): return image_batch if isinstance(image_batch, list): for item in image_batch: if not isinstance(item, Image.Image): raise TypeError("unsupported type in list," " when the image_batch is a list," " the item in list should be PIL.Image.") return image_batch if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return [Image.fromarray(item.astype(np.uint8)) for item in image_batch] if len(image_batch.shape) == 3: return Image.fromarray(image_batch.astype(np.uint8)) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") raise ValueError("unsupported input type.") @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomCropDecodeResize(vision.transforms.RandomCropDecodeResize): """wrapper of RandomCropDecodeResize""" def __init__(self, size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolation='cubic', max_attempts=10): self.size = size self.scale = scale self.ratio = ratio self.interpolation = INTERPOLATION.get(interpolation) self.max_attempts = max_attempts super(RandomCropDecodeResize, self).__init__(size, scale, ratio, self.interpolation, max_attempts) @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomResizedCrop(vision.transforms.RandomResizedCrop): """wrapper of RandomCropDecodeResize""" def __init__(self, size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolation='cubic', max_attempts=10): self.size = size self.scale = scale self.ratio = ratio self.interpolation = INTERPOLATION.get(interpolation) self.max_attempts = max_attempts super(RandomResizedCrop, self).__init__(size, scale, ratio, self.interpolation, max_attempts) @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class Resize(vision.transforms.Resize): """wrapper of Resize""" def __init__(self, size, interpolation='cubic'): self.size = size self.interpolation = INTERPOLATION.get(interpolation) self.random = False super(Resize, self).__init__(size, self.interpolation) @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomHorizontalFlip(vision.transforms.RandomHorizontalFlip): """wrapper of RandomHorizontalFlip""" def __init__(self, prob=0.5): self.prob = prob super(RandomHorizontalFlip, self).__init__(prob) pengcheng.mind-test/mindformers/dataset/translation_dataset.py000066400000000000000000000162121471400134000253020ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """T5 Dataset.""" import os import copy import numpy as np import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset from ..auto_class import AutoTokenizer __all__ = ['TranslationDataset'] @MindFormerRegister.register(MindFormerModuleType.DATASET) class TranslationDataset(BaseDataset): """Translation dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TranslationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['translation']['t5_small'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/t5.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TranslationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TranslationDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Translation Dataset.") cls.init_dataset_config(dataset_config) if dataset_config.data_loader.type != 'MindDataset': dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, operations=type_cast_op, input_columns=input_arg) return dataset @classmethod def _tokenizer_map(cls, dataset, tokenizer_config): """Maps the tokenizer on the source and the output""" tokenizer = AutoTokenizer.from_pretrained(tokenizer_config.type) prefix = tokenizer_config.prefix src_max_length = tokenizer_config.src_max_length tgt_max_length = tokenizer_config.tgt_max_length logger.info("Start tokenize on the dataset using tokenizer: %s", tokenizer_config) def pad_max_function(src, tgt): src = src.tolist() if isinstance(src, bytes): src = src.decode() output = tokenizer(prefix + src, padding='max_length', max_length=src_max_length, truncation=True) tgt = tgt.tolist() if isinstance(tgt, bytes): tgt = tgt.decode() tgt_output = tokenizer(tgt, padding='max_length', max_length=tgt_max_length, truncation=True) input_ids = np.array(output['input_ids'], np.int32) attention_mask = np.array(output['attention_mask'], np.float32) labels = np.array(tgt_output['input_ids'], np.int32) return input_ids, attention_mask, labels dataset = get_dataset_map(dataset, pad_max_function, input_columns=['source', 'target'], output_columns=['input_ids', 'attention_mask', 'labels']) dataset = dataset.project(columns=['input_ids', 'attention_mask', 'labels']) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': device_num, 'shard_id': rank_id}) dataset = cls._tokenizer_map(dataset, dataset_config.tokenizer) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) dataset_files = [] if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if file.endswith(".mindrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if data_dir.endswith(".mindrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") logger.info("Using args %s to instance the dataset.", dataset_config.data_loader) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': device_num, 'shard_id': rank_id, 'columns_list': dataset_config.input_columns}) return dataset pengcheng.mind-test/mindformers/dataset/utils.py000066400000000000000000000036351471400134000224040ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Dataset Utils.""" def check_dataset_config(config): """Check dataset config.""" if config.train_dataset is not None: config.train_dataset.do_eval = False config.train_dataset.seed = config.seed config.train_dataset.auto_tune = config.auto_tune config.train_dataset.filepath_prefix = config.filepath_prefix config.train_dataset.autotune_per_step = config.autotune_per_step config.train_dataset.profile = config.profile config.train_dataset.batch_size = config.runner_config.batch_size if config.train_dataset.mixup_op: config.train_dataset.mixup_op.num_classes = config.runner_config.num_classes config.train_dataset_task.dataset_config = config.train_dataset if config.eval_dataset is not None: config.eval_dataset.do_eval = True config.eval_dataset.seed = config.seed config.eval_dataset.auto_tune = config.auto_tune config.eval_dataset.filepath_prefix = config.filepath_prefix config.eval_dataset.autotune_per_step = config.autotune_per_step config.eval_dataset.profile = config.profile config.eval_dataset.batch_size = config.runner_config.batch_size config.eval_dataset_task.dataset_config = config.eval_dataset pengcheng.mind-test/mindformers/dataset/zero_shot_image_classification_dataset.py000066400000000000000000000116601471400134000311770ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Zero Shot Image Classification Dataset.""" import os from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .transforms import build_transforms from .base_dataset import BaseDataset from ..tools import logger from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models.build_tokenizer import build_tokenizer @MindFormerRegister.register(MindFormerModuleType.DATASET) class ZeroShotImageClassificationDataset(BaseDataset): r""" Zero Shot Image Classification Dataset API. output image, text, and label columns Args: dataset_config (dict): Config for dataset. Returns: A dataset for ZeroShotImageClassificationTrainer. Examples: >>> import os >>> from mindformers import MindFormerBook, MindFormerConfig, build_dataset >>> project_path = MindFormerBook.get_project_path() >>> config_path = os.path.join(project_path, "configs", "clip", >>> "run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml") >>> config = MindFormerConfig(config_path) Note: Put cifar100 dataset to ./ The detailed data setting could refer to ./configs/clip/clip.md >>> config.eval_dataset_task.dataset_config.batch_size = 1 >>> dataset = build_dataset(config.eval_dataset_task) >>> for item in dataset: >>> print(item) >>> break [Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], [1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], [1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], ... [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000], [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000], [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000]]]]), Tensor(shape=[1, 100, 77], dtype=Int32, value= [[[49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], ... [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0]]]), Tensor(shape=[1], dtype=Int32, value= [49])] """ def __new__(cls, dataset_config: dict = None): """New method""" logger.info("Now Create Zero Shot Image Classification Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) tokenizer = build_tokenizer(dataset_config.tokenizer) text_transforms = build_transforms(dataset_config.text_transforms, default_args={"tokenizer": tokenizer}) if transforms is not None: dataset = get_dataset_map(dataset, input_columns="image", operations=transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if text_transforms is not None: dataset = get_dataset_map(dataset, input_columns="text", operations=text_transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-test/mindformers/generation/000077500000000000000000000000001471400134000213715ustar00rootroot00000000000000pengcheng.mind-test/mindformers/generation/__init__.py000066400000000000000000000017301471400134000235030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Mindformers generation.""" from .generation_config import * from .logits_process import * from .streamers import * from .text_generator import * __all__ = [] __all__.extend(generation_config.__all__) __all__.extend(logits_process.__all__) __all__.extend(streamers.__all__) __all__.extend(text_generator.__all__) pengcheng.mind-test/mindformers/generation/__pycache__/000077500000000000000000000000001471400134000236015ustar00rootroot00000000000000pengcheng.mind-test/mindformers/generation/__pycache__/__init__.cpython-39.pyc000066400000000000000000000006101471400134000277660ustar00rootroot00000000000000a ?cf@s\dZddlTddlTddlTddlTgZeejeejeejeejdS)z Mindformers generation.)*N)__doc__Zgeneration_configZlogits_processZ streamersZtext_generator__all__extendrrJ/home/ma-user/work/mindformers-baichuan/mindformers/generation/__init__.pys   pengcheng.mind-test/mindformers/generation/__pycache__/generation_config.cpython-39.pyc000066400000000000000000000164161471400134000317220ustar00rootroot00000000000000a ?cf @sLdZddlZddlmZmZddlmZddlmZdgZ GdddZ dS)zgeneration config.N)AnyDict) BaseConfig)loggerGenerationConfigc@sreZdZdZddZedddZeeee fddd Z ee dd d d Z d dZ eee fdddZdS)ra Class that holds a configuration for a generation task. Arg: > Parameters that control the length of the output max_length (`int`, *optional*, defaults to 20): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (`int`, *optional*): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. > Parameters that control the generation strategy used do_sample (`bool`, *optional*, defaults to `False`): Whether or not to use sampling ; use greedy decoding otherwise. use_past (`bool`, *optional*, defaults to `False`): Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. > Parameters for manipulation of the model output logits temperature (`float`, *optional*, defaults to 1.0): The value used to modulate the next token probabilities. top_k (`int`, *optional*, defaults to 50): The number of highest probability vocabulary tokens to keep for top-k-filtering. top_p (`float`, *optional*, defaults to 1.0): If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. encoder_repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for encoder_repetition_penalty. An exponential penalty on sequences that are not in the original input. 1.0 means no penalty. renormalize_logits (`bool`, *optional*, defaults to `False`): Whether to renormalize the logits after applying all the logits processors or warpers (including the custom ones). It's highly recommended to set this flag to `True` as the search algorithms suppose the score logits are normalized but some logit processors or warpers break the normalization. > Special tokens that can be used at generation time pad_token_id (`int`, *optional*): The id of the *padding* token. bos_token_id (`int`, *optional*): The id of the *beginning-of-sequence* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. > Wild card generation_kwargs: Additional generation kwargs will be forwarded to the `generate` function of the model. Kwargs that are not present in `generate`'s signature will be used in the model forward pass. c Ks>|dd|_|d|j|_|dd|_|dd|_|dd|_|dd |_|d d |_|d d |_|d d |_|dd |_ |dd|_ |dd|_ |dd|_ |dd|_ |dd|_|js:|D]T\}}zt|||Wqty6}z td||||WYd}~qd}~00qdS)NZmax_decode_length max_lengthmax_new_tokens do_sampleFuse_past temperatureg?top_k2top_prepetition_penaltyencoder_repetition_penaltyrenormalize_logits pad_token_id bos_token_id eos_token_id_from_model_configz!Can't set %s with value %s for %s)poprr r r r r rrrrrrrritemssetattrAttributeErrorrerror)selfkwargskeyvalueerrr!S/home/ma-user/work/mindformers-baichuan/mindformers/generation/generation_config.py__init__Qs,zGenerationConfig.__init__)returncCs t|jS)N)str__dict__)rr!r!r"__str__vszGenerationConfig.__str__) config_dictcKsN|dd}|fii||}|jfi|}td||rJ||fS|S)a Instantiates a [`GenerationConfig`] from a Python dictionary of parameters. Args: config_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the configuration object. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the configuration object. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. return_unused_kwargsFzGenerate config %s)rupdaterdebug)clsr(rr)config unused_kwargsr!r!r" from_dictys  zGenerationConfig.from_dict) model_configr$cCs(|}|dd|j|ddd}|S)a Instantiates a [`GenerationConfig`] from a [`PretrainedConfig`]. This function is useful to convert legacy [`BaseConfig`] objects, which may contain generation parameters, into a stand-alone [`GenerationConfig`]. Args: model_config (`PretrainedConfig`): The model config that will be used to instantiate the generation config. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. rNFT)r)r)to_dictrr/)r,r0r(r-r!r!r"from_model_configs z"GenerationConfig.from_model_configc sPg|D](\}}t||r t||||q fdd|D}|S)a Updates attributes of this class instance with attributes from `kwargs` if they match existing atributtes, returning all the unused kwargs. Args: kwargs (`Dict[str, Any]`): Dictionary of attributes to tentatively update this class. Returns: `Dict[str, Any]`: Dictionary containing all the key-value pairs that were not used to update the instance. csi|]\}}|vr||qSr!r!).0rr to_remover!r" z+GenerationConfig.update..)rhasattrrappend)rrrrr.r!r4r"r*s    zGenerationConfig.updatecCst|j}|S)zto dict convert function.)copydeepcopyr&)routputr!r!r"r1s zGenerationConfig.to_dictN)__name__ __module__ __qualname____doc__r#r%r' classmethodrrr/rr2r*r1r!r!r!r"rs7%) r@r:typingrrZmindformers.models.base_configrmindformers.toolsr__all__rr!r!r!r"s   pengcheng.mind-test/mindformers/generation/__pycache__/logits_process.cpython-39.pyc000066400000000000000000000201731471400134000312740ustar00rootroot00000000000000a ?cf"@sdZddlZddlZddlmZmZmZgdZGdddZ Gdd d Z Gd d d e Z Gd d d e Z Gddde ZGddde ZGddde ZGddde e ZdS)z Logits Processor for generation.N) log_softmaxsoftmaxtopk)LogitsProcessor LogitsWarperLogitsProcessorListc@seZdZdZddZdS)rzSAbstract base class for all logit processors that can be applied during generation.cCst|jddS)z#Torch method for processing logits.H is an abstract class. Only classes inheriting this class can be called.NNotImplementedError __class__self input_idsscoresrP/home/ma-user/work/mindformers-baichuan/mindformers/generation/logits_process.py__call__s zLogitsProcessor.__call__N__name__ __module__ __qualname____doc__rrrrrrsrc@seZdZdZddZdS)rznAbstract base class for all logit warpers that can be applied during generation with multinomial sampling.cCst|jddS)z Torch method for warping logits.r Nr r rrrr&s zLogitsWarper.__call__Nrrrrrr"src@seZdZdZddZdS)ra This class can be used to create a list of [`LogitsProcessor`] or [`LogitsWarper`] to subsequently process a `scores` input tensor. This class inherits from list and adds a specific *__call__* method to apply each [`LogitsProcessor`] or [`LogitsWarper`] to the inputs. c s|D]}t|jj}t|dkr|tfddt|ddDshtdt|d|j d|||fi}q|||}q|S)Nc3s|]}|vVqdSNr).0argkwargsrr 8z/LogitsProcessorList.__call__..z,Make sure that all the required parameters: z for z$ are passed to the logits processor.) inspect signaturer parameterslenalllistkeys ValueErrorr )rrrr processor function_argsrrrr4s & zLogitsProcessorList.__call__Nrrrrrr-src@s&eZdZdZedddZddZdS)TemperatureLogitsWarperz [`LogitsWarper`] for temperature (exponential scaling output probability distribution). Args: temperature (`float`): The value used to module the logits distribution.  temperaturecCs(t|}|dkrtd|||_dS)Nrz:`temperature` has to be a strictly positive float, but is )floatr(r-)rr-rrr__init__Ls z TemperatureLogitsWarper.__init__cCs||j}|Srr,r rrrrUs z TemperatureLogitsWarper.__call__Nrrrrr.r/rrrrrr+Cs r+c@s&eZdZdZedddZddZdS) RepetitionPenaltyLogitsProcessora# [`LogitsProcessor`] enforcing an exponential penalty on repeated sequences. Args: repetition_penalty (`float`): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. )penaltycCs(t|}|dkrtd|||_dS)Nrz6`penalty` has to be a strictly positive float, but is )r.r(r2)rr2rrrr/ds z)RepetitionPenaltyLogitsProcessor.__init__cCsXtj||dd}|dk}|}|||j||<|||j||<tj|||dd|S)Nraxisr)nptake_along_axisr2put_along_axis)rrrscoreZnegative_indexZpositive_indexrrrrmsz)RepetitionPenaltyLogitsProcessor.__call__Nr0rrrrr1Zs  r1c@s.eZdZdZd eeeedddZdd Zd S) TopPLogitsWarpera  [`LogitsWarper`] that performs top-p, i.e. restricting to top tokens summing to prob_cut_off <= prob_cut_off. Args: top_p (`float`): If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. filter_value (`float`, *optional*, defaults to `-50000`): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. candicate_token_num (`int`, *optional*, defaults to 200): Number of candicate tokens to calculate top_p. this can avoid sorting a huge seq, save time to speed up generation. 0 and < 1, but is z>`min_tokens_to_keep` has to be a non-negative integer, but is )r.r( isinstanceintr<r=r>r?)rr<r=r>r?rrrr/s zTopPLogitsWarper.__init__cCst||j\}}t|}tj|dd}||jk}tjtj|jddfd tj |dddfgdd}d|dd|j f<t | tj }tj |||dd|j||<|S)Nr3rr)shape.)rr?rr5cumsumr< concatenateonesrCastypebool_r> ones_liker7r=)rrrZcandicate_logitsZcandicate_indicesZcumulative_probsZsorted_indices_to_keepindices_to_removerrrrs ,  zTopPLogitsWarper.__call__N)r:rr;)rrrrr.rAr/rrrrrr9zs r9c@s4eZdZdZd eeedddZejddd Z d S) TopKLogitsWarpera [`LogitsWarper`] that performs top-k, i.e. restricting to the k highest probability elements. Args: top_k (`int`): The number of highest probability vocabulary tokens to keep for top-k-filtering. filter_value (`float`, *optional*, defaults to `-float("Inf")`): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. r:r)top_kr=r>cCs:t|tr|dkr td|t|||_t||_dS)Nrz6`top_k` has to be a strictly positive integer, but is )r@rAr(maxrLr.r=)rrLr=r>rrrr/s  zTopKLogitsWarper.__init__)rcCs6t|j|jd}|t||ddk}|j||<|S)NrBr).rBN)minrLrCrr=)rrrrLrJrrrrs zTopKLogitsWarper.__call__N)r:r) rrrrrAr.r/r5ndarrayrrrrrrKs  rKc@seZdZdZddZdS)LogitNormalizationa [`LogitsWarper`] and [`LogitsProcessor`] for normalizing the scores using log-softmax. It's important to normalize the scores during beam search, after applying the logits processors or warpers, since the search algorithm used in this library doesn't do it (it only does it before, but they may need re-normalization) but it still supposes that the scores are normalized when comparing the hypotheses. cCst|dd}|S)NrBr3)rr rrrrs zLogitNormalization.__call__NrrrrrrPsrP)rr!numpyr5utilsrrr__all__rrr&rr+r1r9rKrPrrrrs   8pengcheng.mind-test/mindformers/generation/__pycache__/streamers.cpython-39.pyc000066400000000000000000000212431471400134000302410ustar00rootroot00000000000000a ?cf,@sjdZddlmZddlmZddlZddlmZgdZ GdddZ Gd d d e Z Gd d d e Z dS) zStreamers for text generation.)Queue)OptionalN) BaseTokenizer) BaseStreamer TextStreamerTextIteratorStreamerc@s eZdZdZddZddZdS)rzG Base class from which `.generate()` streamers should inherit. cCs tdS)z;Function that is called by `.generate()` to push new tokensNNotImplementedErrorselfvaluer K/home/ma-user/work/mindformers-baichuan/mindformers/generation/streamers.pyputszBaseStreamer.putcCs tdS)zHFunction that is called by `.generate()` to signal the end of generationNrr r r rend#szBaseStreamer.endN)__name__ __module__ __qualname____doc__rrr r r rrsrc@sZeZdZdZdeeeedddZdd Zd d Z d d Z de edddZ ddZ dS)ra- Simple text streamer that prints the token(s) to stdout as soon as entire words are formed. The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`BaseTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextStreamer >>> tok = GPT2Tokenizer.from_pretrained("gpt2") >>> model = GPT2LMHeadModel.from_pretrained("gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) >>> streamer = TextStreamer(tok) >>> # Despite returning the usual output, the streamer will also print the generated text to stdout. >>> _ = model.generate(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) >>> 'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,' ``` NFT) tokenizer skip_promptskip_special_tokenscKs:||_||_||_||_d|_g|_d|_d|_d|_dS)NFrT) rrr decode_kwargs batch_stream token_cache text_cache print_lennext_tokens_are_prompt)r rrrrr r r__init__KszTextStreamer.__init__cCst|tr|j|t|tjr*|}t|trt|dkrbt|dtrb|j sd|_ d|_ qt|dkrt|dtr|d}|j rd|_ g|_d|_ |j |nt d|jr|jrd|_dS|j r|jj||jfi|j}n|jj|j|jfi|j}||}||dS)zm Receives tokens, decodes them, and prints them to stdout as soon as they form entire words. rTrFzJTextStreamer only supports int, or 1 ~ 2 dim numpy.ndarray/list as inputs.N) isinstanceintrappendnpndarraytolistlistlenrrrextend ValueErrorrrr batch_decoderrdecodeget_printable_texton_finalized_text)r r textprintable_textr r rr\s2      zTextStreamer.putcCs|jr |S|dr0||jd}g|_d|_nb|rh|t|drh||jd}|jt|7_n*||j|dd}|jt|7_|S)z/Get printable text when a new element comes in. Nr r!)rendswithrr_is_chinese_charordr)rfindr r0r1r r rr.s zTextStreamer.get_printable_textcCsf|js8|jr8|jj|j|jfi|j}||jd}nd}|j|ddd|_g|_d|_ d|_dS)z;Flushes any remaining cache and prints a newline to stdout.NrT) stream_endr) rrrr-rrrr/rrr9r r rrs zTextStreamer.endr0r:cCsj|jrP|js||_n|r0ddt|j|D|_td|jd|sFdnddnt|d|s^dndddS)zNPrints the new text to stdout. If the stream is ending, also prints a newline.cSsg|]\}}||qSr r ).0ijr r r z2TextStreamer.on_finalized_text.. TrN)flushr)rrzipprintr r0r:r r rr/s zTextStreamer.on_finalized_textcCsd|krdksnd|kr*dksnd|kr@dksnd|krVdksnd |krld ksnd |krd ksnd |krdksnd|krdkrnndSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr )r cpr r rr6sD  zTextStreamer._is_chinese_char)NFT)F)rrrrrrboolr rr.rstrr/r6r r r rr(s# ) rcsVeZdZdZdeeeeedfdd Zde eddd Z d d Z d d Z Z S)ra Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is useful for applications that benefit from accessing the generated text in a non-blocking way (e.g. in an interactive Gradio demo). The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`BaseTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. timeout (`float`, *optional*): The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextIteratorStreamer >>> from threading import Thread >>> tok = GPT2Tokenizer.from_pretrained("gpt2") >>> model = GPT2LMHeadModel.from_pretrained("gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) >>> streamer = TextIteratorStreamer(tok) >>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. >>> generation_kwargs = dict(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) >>> thread = Thread(target=model.generate, kwargs=generation_kwargs) >>> thread.start() >>> generated_text = "" >>> for new_text in streamer: ... generated_text += new_text >>> generated_text 'An increasing sequence: one, two, three, four, five, six, seven, eight,' ``` NF)rrtimeoutc s.tj||fi|t|_d|_||_dSN)superr r text_queue stop_signalrI)r rrrIr __class__r rr szTextIteratorStreamer.__init__r;cCs2|r|jj||jd|r.|jj|j|jddS)z\Put the new text in the queue. If the stream is ending, also put a stop signal in the queue.rIN)rLrrIrMrEr r rr/sz&TextIteratorStreamer.on_finalized_textcCs|SrJr rr r r__iter__szTextIteratorStreamer.__iter__cCs$|jj|jd}||jkr t|S)NrP)rLgetrIrM StopIterationr r r r__next__s zTextIteratorStreamer.__next__)NFN)F)rrrrrrrGfloatr rHr/rQrT __classcell__r r rNrrs/ r) rqueuertypingrnumpyr%!mindformers.models.base_tokenizerr__all__rrrr r r rs   (pengcheng.mind-test/mindformers/generation/__pycache__/text_generator.cpython-39.pyc000066400000000000000000000420311471400134000312640ustar00rootroot00000000000000a ?cfk@sdZddlZddlZddlmZmZmZddlZddl m Z ddl m Z ddlmmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZddlmZdd l m!Z!d d l"m#Z#dgZ$GdddZ%dS)z For text generation N)OptionalListUnion) functional) operations)Tensor) BaseStreamer)GenerationConfig)LogitNormalizationLogitsProcessorList RepetitionPenaltyLogitsProcessorTemperatureLogitsWarperTopKLogitsWarperTopPLogitsWarper)softmax)loggerGeneratorMixinc@seZdZdZddZddZedddZed d d Zd&ddZ e e e dddZ e e dddZe dddZddZd'ddZed ddZd(e e e e e ed d!d"Zd)e eeeeeefe e e e e ee ed#d$d%Zd S)*rzGenerator For the nlp modelscCsdS)N)selfrrP/home/ma-user/work/mindformers-baichuan/mindformers/generation/text_generator.py__init__-szGeneratorMixin.__init__cKs tddS)a prepare inputs for generation. A model class needs to define a `prepare_inputs_for_generation` method in order to use `.generate()` Raises: RuntimeError: Not implemented in model but call `.generate()` zeA model class needs to define a `prepare_inputs_for_generation` method in order to use `.generate()`.N) RuntimeError)r input_idskwargsrrrprepare_inputs_for_generation1s z,GeneratorMixin.prepare_inputs_for_generation) model_kwargscCsdS)z update model kwargs before generate. If your model needs to update model kwargs before generate, implement this method in your model, else do nothing. Nr)rrrrrr#update_model_kwargs_before_generate@sz2GeneratorMixin.update_model_kwargs_before_generate) model_inputscCs|d}t|tr|}g}t|D]8\}}t|||jd}|||||dq(tj |tj d}t|t j |d<dS)zr?Z processorsrrr_get_logits_processoras   z$GeneratorMixin._get_logits_processor) default_list custom_listcCsv|s|S|D]Z}|D]P}t|t|urd}td|dt|d|d|d|d|d|d qq |||S) z.merge custom processor list with default list.zlogits processorz A custom z of type z with values zS has been passed to `.generate()`, but it has already been created with the values z. z has been created by passing the corresponding arguments to generate or by the model's config default values. If you just want to change the default values of zO consider passing them as arguments to `.generate()` instead of using a custom .)type ValueErrorextend)rrFrGdefaultcustomZ object_typerrrrCss& z$GeneratorMixin._merge_processor_list)r>cCst}|jdur*|jdkr*|t|jd}|jdurV|jdkrV|t|j|d|jdur~|jdkr~|t|j|d|jdur|t |S)z This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsWarper`] instances used for multinomial sampling. Nr@r r)top_kmin_tokens_to_keep)top_prOT) r temperaturer'rrNrrPrrDr )rr>ZwarpersrOrrr_get_logits_warpers  z!GeneratorMixin._get_logits_warpercCsp|jd}t|tj}|t|tj|}t||jj f}t d|jt |}d|dddf<||||fS)z#generate the inputs for the decoderrz-Decoder: pad the origin inputs into shape: %sr N) r&rr+float32Zencoder_forwardr*r(zerosconfigmax_decode_lengthrdebug zeros_like)rr input_mask batch_size encoder_maskencoder_output target_maskrrr!_prepare_model_inputs_for_decoders    z0GeneratorMixin._prepare_model_inputs_for_decoderrcCsX|jj|jd}|dkr6td|jd|jjdtj|dd|ffdd|fd}|S) z#pad the input_ids to the max_lengthr0rzorigin_inputs size is z1, you shouldincrease the seq_length of the model rH)rrconstant)constant_values)rU seq_lengthr&rJr(pad)r origin_inputs pad_token_idZ pad_lengthrrrr_pad_inputs_using_max_lengths  z+GeneratorMixin._pad_inputs_using_max_lengthcCs|jr\|jddt|tj|d<tdgtj|d<t|gtj|d<|fi|}d|_nZ|jdd|||t|tj|d<tdgtj|d<t|gtj|d<|fi|}|S)z$model forward for incremental infer.T)is_first_iterationZinput_positionF init_resetbatch_valid_length)rfadd_flags_recursiverr+r*bool_r/)rrr,valid_length_each_exampleresrrr_incremental_infers$   z!GeneratorMixin._incremental_infer)r>r? logits_warperstreamerc% st}t}|jdur d|_|dur2|||jd} |jj} td|jgt| D](}  t t || |jkdq^t tdt } |jdur|j| |_|j|jjkrtd|j|jj|jj|_td|j| s"| |jkr"td| d |jd |j||jd } td |j| jt | }tjdD]} d|| d| f<q\d}d}| r|j|jjkr|jj|_td |j|| |\}}} }ddt| Ddg| }|| ||jrd|_d}t }t|}td|t || kr8t}| jdfddt| D}td| rt| tj}|d|||t|tjd}nH||d<|j| fi|}|jr|j}|j ||d}n|fi|}t|}t}|jj!st"|t#r |dn|}t"|tr$|$}t %|d|jdf}|r\|jdt&|kr\||}|| |}|| |}t 't (|jd| df}n0|\}}t"|tr|$}t"|tr|$}t|}t}ddt| D}t| D]} || rq|j)r,t*|| } t j+j,t&|| | d}!nt -|| }!|| |!}"|"| | | f<|durj|"g|| <| rt.d|| | <| t.d7<d|| | d<|| |!|j/ksЈ| |jkrd|| <qq|dur| dkr||dn ||t|}td||||||qg}#t| D],} |# | | dt.| f0t jqDtd|#|dur|1t |}$t|}t2d||$|$||#S)a. Text generation given the model and origin inputs Inputs: origin_inputs(list): The prompt for generation, should be a list of ids. generation_config(GenerationConfig): The controlling config for text generation. streamer: Streamer object that will be used to stream the generated sequences. model_kwargs: dict of model input kwargs, when be passed to model construct during the generation forward. Returns: outputs: the ids for the generated text NrzThe input shape is: %sr z%Get the valid for each example is: %szOmax_length %s can not exceeds model seq_length %s, set max_length = seq_length.zmax length is: %szthe input_ids length z exceeds the max length config zE.check your inputs and set max_length larger than your inputs length.)rcrdz,pad the origin inputs from %s into shape: %szmax decode length is: %scSsg|]}dqSr r.0_rrr 3z+GeneratorMixin._forward..FTzforward prepare time: %s scs g|]}|d|qSrpr)rrr.rarkrrrtGszvalidate length: %s)rattention_maskZencoder_outputsZdecoder_input_idsZdecoder_attention_maskr,)rr,rkr0cSsg|]}gqSrrrqrrrrtru)pzKforward time: %s s; sample time: %s s; update time: %s s; total count: %s szThe output is: %szJtotal time: %s s; generated tokens: %s tokens; generate speed: %s tokens/s)3timerdputr&rUis_encoder_decoderrrWranger'r(maxargwherer)max_new_tokens max_lengthrawarningrJrerXrVr^ruse_pastrfsumrr+r*rSrrmZis_sample_accelerationr"tupler#r1lentilearange do_samplerrandomchoiceargmaxr% eos_token_idastypeendinfo)%rrcr>r?rnror total_timeZ prepare_timerZr{r.Zinput_ids_lengthrrYr\r[r]Z is_finishedZneed_gather_logitsZ origin_lenZ forward_timer,inputsrlrZ sample_timer9probsZp_argsZ update_time target_listp_norm target_indextarget output_idsZ generate_lenrrvr_forwards6                                                zGeneratorMixin._forward)rr>r?roseedc Ks|j}|dt|dt|d}|dur8dn|}tj||durT|nt}|durnt |j }t |}|jfi|}|jsd|_d|_td||j||d}||} |jf|||| |d|} ||d k| S) u Generate the words according to the given the input ids. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, top_k=3, do_sample=True)`. Args: input_ids(List(str), List(List(str))): The token id list or a batch of token id list. When input a batch of token id list, the length of each token id list should be same. generation_config (`GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default config from the model configuration will be used. Please note that unspecified parameters will inherit [`GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. streamer: The streamer that generator uses. seed: Random seed used in sample. kwargs: Specific parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Supported `generate_config` keywords can be checked in [`GenerationConfig`]'s documentation. Mainly used Keywords are shown below: max_length(int): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (int): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. do_sample(bool): Whether to do sampling on the candidate ids. If set True it will be enabled, and set it to be False to disable the sampling, equivalent to topk 1. If set None, it follows the setting in the configureation in the model. top_k(int): Determine the topK numbers token id as candidate. This should be a positive number. If set None, it follows the setting in the configureation in the model. top_p(float): The accumulation probability of the candidate token ids below the top_p will be select as the condaite ids. The valid value of top_p is between (0, 1]. If the value is larger than 1, top_K algorithm will be enabled. If set None, it follows the setting in the configureation in the model. eos_token_id(int): The end of sentence token id. If set None, it follows the setting in the configureation in the model. pad_token_id(int): The pad token id. If set None, it follows the setting in the configureation in the model. repetition_penalty(float): The penalty factor of the frequency that generated words. The If set 1, the repetition_penalty will not be enabled. If set None, it follows the setting in the configureation in the model. Default None. Examples: >>> from mindformers import T5ForConditionalGeneration, T5Tokenizer >>> t5 = T5ForConditionalGeneration.from_pretrained("t5_small") >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> words = "translate the English to the Romanian: UN Chief Says There Is No Military Solution in Syria" >>> words = tokenizer(words, max_length=21, padding='max_length')['input_ids'] >>> output = t5.generate(words, do_sample=True) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) eful ONU declară că nu există o soluţie militară în Siria >>> # Enable the top p sampling >>> output = t5.generate(words, do_sample=True, top_p=0.4) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) eful ONU declară că nu există o soluţie militară în Siria >>> # Enable the top k sampling. >>> output = t5.generate(words, do_sample=True, top_k=10, top_p=1) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) Este comist de stat ale stateului membre nai uzusepa şi ONU Returns: A list of the generated token ids Fr0Nrr@zGeneration Config is: %sr=)rcr>r?rnrotrain)phase set_trainr(r)r1r&rrr r from_model_configrUcopydeepcopyupdaterrPrNrrWrErRr) rrr>r?rorr origin_phaserrnrrrrgenerates@S       zGeneratorMixin.generate)NF)r)NNN)NNNN)__name__ __module__ __qualname____doc__rrdictrr/r<r rr rErCrRr^rermrrrrr%rrrrrr*sJ     e)&rrrytypingrrrnumpyr( mindspore.opsrr6rr3mindspore.common.dtypecommonr r+mindspore.common.tensorrZ mindformers.generation.streamersrr>r logits_processr r r rrr streamersutilsrtoolsr__all__rrrrrs          pengcheng.mind-test/mindformers/generation/__pycache__/utils.cpython-39.pyc000066400000000000000000000030641471400134000273750ustar00rootroot00000000000000a ?cfn @s.dZddlZd ddZd ddZd d d ZdS)zutils for text generation.NcCstj||dd}|jdkr,d|t|<nt|s:d}||}t|}tjdd*tj||dd}t|}Wdn1s0Y||}|S)z}numpy implemented log softmax function. refers to https://github.com/scipy/scipy/blob/v1.11.1/scipy/special/_logsumexp.pyTaxiskeepdimsrignore)divideN)npamaxndimisfiniteexperrstatesumlog)xrx_maxtmpexp_tmpsoutrG/home/ma-user/work/mindformers-baichuan/mindformers/generation/utils.py log_softmaxs   (rcCs2tj||dd}t||}|tj||ddS)zynumpy implemented softmax function. refers to https://github.com/scipy/scipy/blob/v1.11.1/scipy/special/_logsumexp.pyTr)rrr r )rrr exp_x_shiftedrrrsoftmax*srTcCs|j||kr|j|d}|r4tj| ||d}ntj|||d}tj|t||d}tj|||d}|r|rtj| |dn tj||d}tj|||d}tj|||d}||fS)znumpy implemented topk sample.)r)shaper argpartitiontakearangetake_along_axisargsort)rtop_krlargestsortZ topk_indexZ topk_data sort_indexrrrtopk2s r&)N)N)rTT)__doc__numpyrrrr&rrrrs  pengcheng.mind-test/mindformers/generation/generation_config.py000066400000000000000000000203551471400134000254300ustar00rootroot00000000000000# Copyright 2022 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """generation config.""" import copy from typing import Any, Dict from mindformers.models.base_config import BaseConfig from mindformers.tools import logger __all__ = ["GenerationConfig"] class GenerationConfig: """Class that holds a configuration for a generation task. Arg: > Parameters that control the length of the output max_length (`int`, *optional*, defaults to 20): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (`int`, *optional*): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. > Parameters that control the generation strategy used do_sample (`bool`, *optional*, defaults to `False`): Whether or not to use sampling ; use greedy decoding otherwise. use_past (`bool`, *optional*, defaults to `False`): Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. > Parameters for manipulation of the model output logits temperature (`float`, *optional*, defaults to 1.0): The value used to modulate the next token probabilities. top_k (`int`, *optional*, defaults to 50): The number of highest probability vocabulary tokens to keep for top-k-filtering. top_p (`float`, *optional*, defaults to 1.0): If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. encoder_repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for encoder_repetition_penalty. An exponential penalty on sequences that are not in the original input. 1.0 means no penalty. renormalize_logits (`bool`, *optional*, defaults to `False`): Whether to renormalize the logits after applying all the logits processors or warpers (including the custom ones). It's highly recommended to set this flag to `True` as the search algorithms suppose the score logits are normalized but some logit processors or warpers break the normalization. > Special tokens that can be used at generation time pad_token_id (`int`, *optional*): The id of the *padding* token. bos_token_id (`int`, *optional*): The id of the *beginning-of-sequence* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. > Wild card generation_kwargs: Additional generation kwargs will be forwarded to the `generate` function of the model. Kwargs that are not present in `generate`'s signature will be used in the model forward pass. """ def __init__(self, **kwargs): # max generate length self.max_length = kwargs.pop("max_decode_length", 20) self.max_length = kwargs.pop("max_length", self.max_length) self.max_new_tokens = kwargs.pop("max_new_tokens", None) # do sample or not self.do_sample = kwargs.pop("do_sample", False) # incremental infer self.use_past = kwargs.pop("use_past", False) # logits processors self.temperature = kwargs.pop("temperature", 1.0) self.top_k = kwargs.pop("top_k", 50) self.top_p = kwargs.pop("top_p", 1.0) self.repetition_penalty = kwargs.pop("repetition_penalty", 1.0) self.encoder_repetition_penalty = kwargs.pop("encoder_repetition_penalty", 1.0) self.renormalize_logits = kwargs.pop("renormalize_logits", False) # Special tokens that can be used at generation time self.pad_token_id = kwargs.pop("pad_token_id", None) self.bos_token_id = kwargs.pop("bos_token_id", None) self.eos_token_id = kwargs.pop("eos_token_id", None) # interface. self._from_model_config = kwargs.pop("_from_model_config", False) # Additional attributes without default values if not self._from_model_config: # we don't want to copy values from the model config # if we're initializing a `GenerationConfig` from a # model's default configuration file for key, value in kwargs.items(): try: setattr(self, key, value) except AttributeError as err: logger.error("Can't set %s with value %s for %s", key, value, self) raise err def __str__(self) -> str: return str(self.__dict__) @classmethod def from_dict(cls, config_dict: Dict[str, Any], **kwargs): """ Instantiates a [`GenerationConfig`] from a Python dictionary of parameters. Args: config_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the configuration object. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the configuration object. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. """ return_unused_kwargs = kwargs.pop("return_unused_kwargs", False) config = cls(**{**config_dict, **kwargs}) unused_kwargs = config.update(**kwargs) logger.debug("Generate config %s", config) if return_unused_kwargs: return config, unused_kwargs return config @classmethod def from_model_config(cls, model_config: BaseConfig) -> "GenerationConfig": """ Instantiates a [`GenerationConfig`] from a [`PretrainedConfig`]. This function is useful to convert legacy [`BaseConfig`] objects, which may contain generation parameters, into a stand-alone [`GenerationConfig`]. Args: model_config (`PretrainedConfig`): The model config that will be used to instantiate the generation config. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. """ config_dict = model_config.to_dict() config_dict.pop("_from_model_config", None) config = cls.from_dict(config_dict, return_unused_kwargs=False, _from_model_config=True) return config def update(self, **kwargs): """ Updates attributes of this class instance with attributes from `kwargs` if they match existing atributtes, returning all the unused kwargs. Args: kwargs (`Dict[str, Any]`): Dictionary of attributes to tentatively update this class. Returns: `Dict[str, Any]`: Dictionary containing all the key-value pairs that were not used to update the instance. """ to_remove = [] for key, value in kwargs.items(): if hasattr(self, key): setattr(self, key, value) to_remove.append(key) unused_kwargs = {key: value for key, value in kwargs.items() if key not in to_remove} return unused_kwargs def to_dict(self) -> Dict[str, Any]: """to dict convert function.""" output = copy.deepcopy(self.__dict__) return output pengcheng.mind-test/mindformers/generation/logits_process.py000066400000000000000000000213011471400134000247770ustar00rootroot00000000000000# Copyright 2020 The HuggingFace Inc. team # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Logits Processor for generation.""" import inspect import numpy as np from .utils import log_softmax, softmax, topk __all__ = ["LogitsProcessor", "LogitsWarper", "LogitsProcessorList"] class LogitsProcessor: """Abstract base class for all logit processors that can be applied during generation.""" def __call__(self, input_ids, scores): """Torch method for processing logits.""" raise NotImplementedError( f"{self.__class__} is an abstract class. Only classes inheriting this class can be called." ) class LogitsWarper: """Abstract base class for all logit warpers that can be applied during generation with multinomial sampling.""" def __call__(self, input_ids, scores): """Torch method for warping logits.""" raise NotImplementedError( f"{self.__class__} is an abstract class. Only classes inheriting this class can be called." ) class LogitsProcessorList(list): """ This class can be used to create a list of [`LogitsProcessor`] or [`LogitsWarper`] to subsequently process a `scores` input tensor. This class inherits from list and adds a specific *__call__* method to apply each [`LogitsProcessor`] or [`LogitsWarper`] to the inputs. """ def __call__(self, input_ids, scores, **kwargs): for processor in self: function_args = inspect.signature(processor.__call__).parameters if len(function_args) > 2: if not all(arg in kwargs for arg in list(function_args.keys())[2:]): raise ValueError( f"Make sure that all the required parameters: {list(function_args.keys())} for " f"{processor.__class__} are passed to the logits processor." ) scores = processor(input_ids, scores, **kwargs) else: scores = processor(input_ids, scores) return scores class TemperatureLogitsWarper(LogitsWarper): r""" [`LogitsWarper`] for temperature (exponential scaling output probability distribution). Args: temperature (`float`): The value used to module the logits distribution. """ def __init__(self, temperature: float): temperature = float(temperature) if temperature <= 0: raise ValueError( f"`temperature` has to be a strictly positive float, but is {temperature}" ) self.temperature = temperature def __call__(self, input_ids, scores): scores = scores / self.temperature return scores class RepetitionPenaltyLogitsProcessor(LogitsProcessor): r""" [`LogitsProcessor`] enforcing an exponential penalty on repeated sequences. Args: repetition_penalty (`float`): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. """ def __init__(self, penalty: float): penalty = float(penalty) if penalty <= 0: raise ValueError( f"`penalty` has to be a strictly positive float, but is {penalty}" ) self.penalty = penalty def __call__(self, input_ids, scores): score = np.take_along_axis(scores, input_ids, axis=1) # if score < 0 then repetition penalty has to be multiplied to reduce the previous token probability negative_index = score < 0 positive_index = ~negative_index score[negative_index] = score[negative_index] * self.penalty score[positive_index] = score[positive_index] / self.penalty np.put_along_axis(scores, input_ids, score, axis=1) return scores class TopPLogitsWarper(LogitsWarper): """ [`LogitsWarper`] that performs top-p, i.e. restricting to top tokens summing to prob_cut_off <= prob_cut_off. Args: top_p (`float`): If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. filter_value (`float`, *optional*, defaults to `-50000`): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. candicate_token_num (`int`, *optional*, defaults to 200): Number of candicate tokens to calculate top_p. this can avoid sorting a huge seq, save time to speed up generation. """ def __init__(self, top_p: float, filter_value: float = -50000, min_tokens_to_keep: int = 1, candicate_token_num: int = 200): top_p = float(top_p) if top_p < 0 or top_p > 1.0: raise ValueError(f"`top_p` has to be a float > 0 and < 1, but is {top_p}") if not isinstance(min_tokens_to_keep, int) or (min_tokens_to_keep < 0): raise ValueError( f"`min_tokens_to_keep` has to be a non-negative integer, but is {min_tokens_to_keep}" ) self.top_p = top_p self.filter_value = float(filter_value) self.min_tokens_to_keep = min_tokens_to_keep self.candicate_token_num = candicate_token_num def __call__(self, input_ids, scores): candicate_logits, candicate_indices = topk(scores, self.candicate_token_num) cumulative_probs = softmax(candicate_logits) cumulative_probs = np.cumsum(cumulative_probs, axis=-1) # Remove tokens with cumulative top_p above the threshold (token with 0 are kept) sorted_indices_to_keep = cumulative_probs < self.top_p # add the last token that exceed top_p sorted_indices_to_keep = np.concatenate( [np.ones(shape=(scores.shape[0], 1)).astype(np.bool_), sorted_indices_to_keep[..., :-1]], axis=-1 ) # Keep at least min_tokens_to_keep sorted_indices_to_keep[..., :self.min_tokens_to_keep] = 1 # set remove indices, filter negative value indices_to_remove = np.ones_like(scores).astype(np.bool_) np.put_along_axis( indices_to_remove, candicate_indices, ~sorted_indices_to_keep, axis=-1 ) scores[indices_to_remove] = self.filter_value return scores class TopKLogitsWarper(LogitsWarper): r""" [`LogitsWarper`] that performs top-k, i.e. restricting to the k highest probability elements. Args: top_k (`int`): The number of highest probability vocabulary tokens to keep for top-k-filtering. filter_value (`float`, *optional*, defaults to `-float("Inf")`): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. """ def __init__(self, top_k: int, filter_value: float = -50000, min_tokens_to_keep: int = 1): if not isinstance(top_k, int) or top_k <= 0: raise ValueError( f"`top_k` has to be a strictly positive integer, but is {top_k}" ) self.top_k = max(top_k, min_tokens_to_keep) self.filter_value = float(filter_value) def __call__(self, input_ids, scores: np.ndarray): top_k = min(self.top_k, scores.shape[-1]) # Safety check # Remove all tokens with a probability less than the last token of the top-k indices_to_remove = scores < topk(scores, top_k)[0][..., -1, None] scores[indices_to_remove] = self.filter_value return scores class LogitNormalization(LogitsProcessor, LogitsWarper): r""" [`LogitsWarper`] and [`LogitsProcessor`] for normalizing the scores using log-softmax. It's important to normalize the scores during beam search, after applying the logits processors or warpers, since the search algorithm used in this library doesn't do it (it only does it before, but they may need re-normalization) but it still supposes that the scores are normalized when comparing the hypotheses. """ def __call__(self, input_ids, scores): scores = log_softmax(scores, axis=-1) return scores pengcheng.mind-test/mindformers/generation/streamers.py000066400000000000000000000262101471400134000237510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huggingface/transformers/blob/main/src/transformers/generation/streamers.py # ============================================================================ """Streamers for text generation.""" from queue import Queue from typing import Optional import numpy as np from mindformers.models.base_tokenizer import BaseTokenizer __all__ = ['BaseStreamer', 'TextStreamer', 'TextIteratorStreamer'] class BaseStreamer: """ Base class from which `.generate()` streamers should inherit. """ def put(self, value): """Function that is called by `.generate()` to push new tokens""" raise NotImplementedError() def end(self): """Function that is called by `.generate()` to signal the end of generation""" raise NotImplementedError() class TextStreamer(BaseStreamer): """ Simple text streamer that prints the token(s) to stdout as soon as entire words are formed. The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`BaseTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextStreamer >>> tok = GPT2Tokenizer.from_pretrained("gpt2") >>> model = GPT2LMHeadModel.from_pretrained("gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) >>> streamer = TextStreamer(tok) >>> # Despite returning the usual output, the streamer will also print the generated text to stdout. >>> _ = model.generate(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) >>> 'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,' ``` """ def __init__(self, tokenizer: Optional[BaseTokenizer] = None, skip_prompt: bool = False, skip_special_tokens: bool = True, **decode_kwargs): self.tokenizer = tokenizer self.skip_prompt = skip_prompt self.skip_special_tokens = skip_special_tokens self.decode_kwargs = decode_kwargs # variables used in the streaming process self.batch_stream = False self.token_cache = [] self.text_cache = "" self.print_len = 0 self.next_tokens_are_prompt = True def put(self, value): """ Receives tokens, decodes them, and prints them to stdout as soon as they form entire words. """ if isinstance(value, int): self.token_cache.append(value) if isinstance(value, np.ndarray): value = value.tolist() if isinstance(value, list): if len(value) > 1 and isinstance(value[0], list): # switch from single mode to batch mode if not self.batch_stream: self.batch_stream = True self.text_cache = "" else: # batch that equals 1 if len(value) == 1 and isinstance(value[0], list): value = value[0] # switch from batch mode to single mode if self.batch_stream: self.batch_stream = False self.token_cache = [] self.print_len = 0 self.token_cache.extend(value) else: raise ValueError("TextStreamer only supports int, or 1 ~ 2 dim numpy.ndarray/list as inputs.") if self.skip_prompt and self.next_tokens_are_prompt: self.next_tokens_are_prompt = False return # Add the new token to the cache and decodes the entire thing. if self.batch_stream: text = self.tokenizer.batch_decode(value, self.skip_special_tokens, **self.decode_kwargs) else: text = self.tokenizer.decode(self.token_cache, self.skip_special_tokens, **self.decode_kwargs) printable_text = self.get_printable_text(text) self.on_finalized_text(printable_text) def get_printable_text(self, text): """Get printable text when a new element comes in.""" # for batch streamer, we directly return the text if self.batch_stream: return text # After the symbol for a new line, we flush the cache. if text.endswith("\n"): printable_text = text[self.print_len :] self.token_cache = [] self.print_len = 0 # If the last token is a CJK character, we print the characters. elif text and self._is_chinese_char(ord(text[-1])): printable_text = text[self.print_len :] self.print_len += len(printable_text) # Otherwise, prints until the last space char (simple heuristic to avoid printing incomplete words, # which may change with the subsequent token -- there are probably smarter ways to do this!) else: printable_text = text[self.print_len : text.rfind(" ") + 1] self.print_len += len(printable_text) return printable_text def end(self): """Flushes any remaining cache and prints a newline to stdout.""" # Flush the cache, if it exists if not self.batch_stream and self.token_cache: text = self.tokenizer.decode(self.token_cache, self.skip_special_tokens, **self.decode_kwargs) printable_text = text[self.print_len :] else: printable_text = "" self.on_finalized_text(printable_text, stream_end=True) # always reset values when stream ends self.next_tokens_are_prompt = True self.token_cache = [] self.text_cache = "" self.print_len = 0 def on_finalized_text(self, text: str, stream_end: bool = False): """Prints the new text to stdout. If the stream is ending, also prints a newline.""" if self.batch_stream: if not self.text_cache: self.text_cache = text elif text: self.text_cache = [i + j for i, j in zip(self.text_cache, text)] print(f'\r{self.text_cache}', flush=True, end="" if not stream_end else None) else: print(text, flush=True, end="" if not stream_end else None) def _is_chinese_char(self, cp): """Checks whether CP is the codepoint of a CJK character.""" # This defines a "chinese character" as anything in the CJK Unicode block: # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block) # # Note that the CJK Unicode block is NOT all Japanese and Korean characters, # despite its name. The modern Korean Hangul alphabet is a different block, # as is Japanese Hiragana and Katakana. Those alphabets are used to write # space-separated words, so they are not treated specially and handled # like the all of the other languages. if ( (0x4E00 <= cp <= 0x9FFF) or (0x3400 <= cp <= 0x4DBF) or (0x20000 <= cp <= 0x2A6DF) or (0x2A700 <= cp <= 0x2B73F) or (0x2B740 <= cp <= 0x2B81F) or (0x2B820 <= cp <= 0x2CEAF) or (0xF900 <= cp <= 0xFAFF) or (0x2F800 <= cp <= 0x2FA1F) ): return True return False class TextIteratorStreamer(TextStreamer): """ Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is useful for applications that benefit from accessing the generated text in a non-blocking way (e.g. in an interactive Gradio demo). The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`BaseTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. timeout (`float`, *optional*): The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextIteratorStreamer >>> from threading import Thread >>> tok = GPT2Tokenizer.from_pretrained("gpt2") >>> model = GPT2LMHeadModel.from_pretrained("gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) >>> streamer = TextIteratorStreamer(tok) >>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. >>> generation_kwargs = dict(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) >>> thread = Thread(target=model.generate, kwargs=generation_kwargs) >>> thread.start() >>> generated_text = "" >>> for new_text in streamer: ... generated_text += new_text >>> generated_text 'An increasing sequence: one, two, three, four, five, six, seven, eight,' ``` """ def __init__(self, tokenizer: Optional[BaseTokenizer] = None, skip_prompt: bool = False, timeout: Optional[float] = None, **decode_kwargs): super().__init__(tokenizer, skip_prompt, **decode_kwargs) self.text_queue = Queue() self.stop_signal = None self.timeout = timeout def on_finalized_text(self, text: str, stream_end: bool = False): """Put the new text in the queue. If the stream is ending, also put a stop signal in the queue.""" if text: self.text_queue.put(text, timeout=self.timeout) if stream_end: self.text_queue.put(self.stop_signal, timeout=self.timeout) def __iter__(self): return self def __next__(self): value = self.text_queue.get(timeout=self.timeout) if value == self.stop_signal: raise StopIteration() return value pengcheng.mind-test/mindformers/generation/text_generator.py000066400000000000000000000656001471400134000250040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ For text generation """ import copy import time from typing import Optional, List, Union import numpy as np from mindspore.ops import functional as F from mindspore.ops import operations as P import mindspore.common.dtype as mstype from mindspore.common.tensor import Tensor from mindformers.generation.streamers import BaseStreamer from .generation_config import GenerationConfig from .logits_process import (LogitNormalization, LogitsProcessorList, RepetitionPenaltyLogitsProcessor, TemperatureLogitsWarper, TopKLogitsWarper, TopPLogitsWarper) from .streamers import BaseStreamer from .utils import softmax from ..tools import logger __all__ = ["GeneratorMixin"] class GeneratorMixin: """Generator For the nlp models""" def __init__(self): pass # pylint: disable=W0613 def prepare_inputs_for_generation(self, input_ids, **kwargs): """ prepare inputs for generation. A model class needs to define a `prepare_inputs_for_generation` method in order to use `.generate()` Raises: RuntimeError: Not implemented in model but call `.generate()` """ raise RuntimeError( "A model class needs to define a `prepare_inputs_for_generation`" " method in order to use `.generate()`." ) # pylint: disable=W0613 def update_model_kwargs_before_generate(self, input_ids, model_kwargs: dict): """ update model kwargs before generate. If your model needs to update model kwargs before generate, implement this method in your model, else do nothing. """ return def slice_incremental_inputs(self, model_inputs: dict, current_index): """used for non-first iterations, slice the inputs to length 1.""" input_ids = model_inputs.pop("input_ids") if isinstance(input_ids, Tensor): input_ids = input_ids.asnumpy() inputs_tmp = [] for i, index_value in enumerate(current_index): current_index_tmp = ( int(index_value) - i * input_ids.shape[1] ) # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp : current_index_tmp + 1]) inputs_tmp = np.array(inputs_tmp, dtype=np.int32) model_inputs["input_ids"] = Tensor(inputs_tmp, mstype.int32) def process_logits(self, logits, current_index=None, keep_all=False): """Process the logits""" logits = logits.reshape(-1, logits.shape[-1]) if not keep_all and current_index is not None: index = current_index.view(-1,) logits = P.Gather()(logits, index, 0) outputs = P.LogSoftmax(-1)(logits) outputs = F.tensor_pow(np.e, outputs) return outputs def _get_logits_processor(self, generation_config: GenerationConfig, logits_processor: Optional[LogitsProcessorList]): """ This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsProcessor`] instances used to modify the scores of the language model head. """ # instantiate processors list processors = LogitsProcessorList() if generation_config.repetition_penalty is not None and generation_config.repetition_penalty != 1.0: processors.append(RepetitionPenaltyLogitsProcessor(penalty=generation_config.repetition_penalty)) processors = self._merge_processor_list(processors, logits_processor) # `LogitNormalization` should always be the last logit processor, when present if generation_config.renormalize_logits is True: processors.append(LogitNormalization()) return processors def _merge_processor_list(self, default_list: LogitsProcessorList, custom_list: LogitsProcessorList): """merge custom processor list with default list.""" if not custom_list: return default_list for default in default_list: for custom in custom_list: if type(custom) is type(default): object_type = "logits processor" raise ValueError( f"A custom {object_type} of type {type(custom)} with values {custom} has been passed to" f" `.generate()`, but it has already been created with the values {default}." f" {default} has been created by passing the corresponding arguments to generate or" f" by the model's config default values. If you just want to change the default values" f" of {object_type} consider passing them as arguments to `.generate()`" f" instead of using a custom {object_type}." ) default_list.extend(custom_list) return default_list def _get_logits_warper(self, generation_config: GenerationConfig): """ This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsWarper`] instances used for multinomial sampling. """ # instantiate warpers list warpers = LogitsProcessorList() # all samplers can be found in `generation_utils_samplers.py` if generation_config.temperature is not None and generation_config.temperature != 1.0: warpers.append(TemperatureLogitsWarper(generation_config.temperature)) min_tokens_to_keep = 1 if generation_config.top_k is not None and generation_config.top_k != 0: warpers.append(TopKLogitsWarper(top_k=generation_config.top_k, min_tokens_to_keep=min_tokens_to_keep)) if generation_config.top_p is not None and generation_config.top_p < 1.0: warpers.append(TopPLogitsWarper(top_p=generation_config.top_p, min_tokens_to_keep=min_tokens_to_keep)) # `LogitNormalization` should always be the last logit processor, when present if generation_config.renormalize_logits is True: warpers.append(LogitNormalization()) return warpers def _prepare_model_inputs_for_decoder(self, input_ids, input_mask): """generate the inputs for the decoder""" batch_size = input_ids.shape[0] encoder_mask = Tensor(input_mask, mstype.float32) encoder_output = self.encoder_forward( Tensor(input_ids, mstype.int32), encoder_mask ) input_ids = np.zeros((batch_size, self.config.max_decode_length)) logger.debug("Decoder: pad the origin inputs into shape: %s", input_ids.shape) target_mask = np.zeros_like(input_ids) target_mask[:, 0] = 1 # As the decoder is generating from [START] token return encoder_output, encoder_mask, input_ids, target_mask def _pad_inputs_using_max_length(self, origin_inputs, pad_token_id=0): """pad the input_ids to the max_length""" pad_length = self.config.seq_length - origin_inputs.shape[-1] if pad_length < 0: raise ValueError( f"origin_inputs size is {origin_inputs.shape}, you should" f"increase the seq_length of the model {self.config.seq_length}." ) # Pad original inputs to model_origin_max_length input_ids = np.pad( origin_inputs, ((0, 0), (0, pad_length)), "constant", constant_values=(0, pad_token_id), ) return input_ids def _incremental_infer(self, model_inputs: dict, current_index, valid_length_each_example): """model forward for incremental infer.""" # Claim the first graph if self.is_first_iteration: self.add_flags_recursive(is_first_iteration=True) model_inputs["input_position"] = Tensor(current_index, mstype.int32) model_inputs["init_reset"] = Tensor([False], mstype.bool_) # init_reset (1,) bool False model_inputs["batch_valid_length"] = Tensor([valid_length_each_example], mstype.int32) # pylint: disable=E1102 res = self( **model_inputs, ) # first iter done, go to other iters self.is_first_iteration = False else: self.add_flags_recursive(is_first_iteration=False) # slice model inputs for incremental infer self.slice_incremental_inputs(model_inputs, current_index) model_inputs["input_position"] = Tensor(current_index, mstype.int32) model_inputs["init_reset"] = Tensor([True], mstype.bool_) # init_reset (1,) bool True model_inputs["batch_valid_length"] = Tensor([valid_length_each_example], mstype.int32) # pylint: disable=E1102 res = self( **model_inputs, ) return res def _forward(self, origin_inputs, generation_config: GenerationConfig, logits_processor: Optional[LogitsProcessorList] = None, logits_warper: Optional[LogitsProcessorList] = None, streamer: BaseStreamer = None, **model_kwargs): """ Text generation given the model and origin inputs Inputs: origin_inputs(list): The prompt for generation, should be a list of ids. generation_config(GenerationConfig): The controlling config for text generation. streamer: Streamer object that will be used to stream the generated sequences. model_kwargs: dict of model input kwargs, when be passed to model construct during the generation forward. Returns: outputs: the ids for the generated text """ total_time = time.time() prepare_time = time.time() if generation_config.pad_token_id is None: generation_config.pad_token_id = 0 if streamer is not None: streamer.put(origin_inputs) batch_size = origin_inputs.shape[0] is_encoder_decoder = self.config.is_encoder_decoder logger.debug("The input shape is: %s", origin_inputs.shape) valid_length_each_example = [] for i in range(batch_size): # As the nonzero returns the index and we need length valid_length_each_example.append( np.max(np.argwhere(origin_inputs[i] != generation_config.pad_token_id)) + 1 ) valid_length_each_example = np.array(valid_length_each_example) logger.debug("Get the valid for each example is: %s", valid_length_each_example) # Prepare `max_length` depending on other stopping criteria. input_ids_length = np.max(valid_length_each_example) if generation_config.max_new_tokens is not None: generation_config.max_length = generation_config.max_new_tokens + input_ids_length if generation_config.max_length > self.config.seq_length: logger.warning("max_length %s can not exceeds model seq_length %s, set max_length = seq_length.", generation_config.max_length, self.config.seq_length) generation_config.max_length = self.config.seq_length logger.debug("max length is: %s", generation_config.max_length) if not is_encoder_decoder and input_ids_length >= generation_config.max_length: raise ValueError( f"the input_ids length {input_ids_length} exceeds the max length config {generation_config.max_length}." f"check your inputs and set max_length larger than your inputs length." ) input_ids = self._pad_inputs_using_max_length( origin_inputs=origin_inputs, pad_token_id=generation_config.pad_token_id ) logger.debug( "pad the origin inputs from %s into shape: %s", origin_inputs.shape, input_ids.shape, ) input_mask = np.zeros_like(input_ids) for i in range(valid_length_each_example.shape[0]): input_mask[i, :valid_length_each_example[i]] = 1 encoder_output = None encoder_mask = None if is_encoder_decoder: if generation_config.max_length > self.config.max_decode_length: generation_config.max_length = self.config.max_decode_length logger.debug("max decode length is: %s", generation_config.max_length) # When do encoder and decoder prediction, the encoder can be cached # to speed up the inference ( encoder_output, encoder_mask, input_ids, target_mask, ) = self._prepare_model_inputs_for_decoder(input_ids, input_mask) valid_length_each_example = [1 for _ in range(batch_size)] # A single loop generates one token, loop until reaching target # model_origin_max_length or generating eod token is_finished = [False] * batch_size # update model kwargs once, before go into generate loop. self.update_model_kwargs_before_generate(input_ids, model_kwargs) # setup is_first_iteration flag for incremental infer if generation_config.use_past: self.is_first_iteration = True need_gather_logits = True origin_len = np.sum(valid_length_each_example) prepare_time = time.time() - prepare_time logger.debug("forward prepare time: %s s", prepare_time) while np.sum(is_finished) != batch_size: forward_time = time.time() seq_length = input_ids.shape[1] current_index = [ valid_length_each_example[i] - 1 + i * seq_length for i in range(batch_size) ] logger.debug("validate length: %s", valid_length_each_example) if is_encoder_decoder: inputs = Tensor(input_ids, mstype.int32) # pylint: disable=E1102 res = self( input_ids=None, attention_mask=encoder_mask, encoder_outputs=encoder_output, decoder_input_ids=inputs, decoder_attention_mask=Tensor(target_mask, mstype.float32), ) else: model_kwargs["current_index"] = current_index # model prepare input dict model_inputs = self.prepare_inputs_for_generation( # pylint: disable=E1111 input_ids, **model_kwargs ) # incremental generate if generation_config.use_past: # when first iteration, gather last logits; others keep all logits. need_gather_logits = self.is_first_iteration # incremental generate res = self._incremental_infer( model_inputs=model_inputs, current_index=current_index, valid_length_each_example=valid_length_each_example, ) # auto-aggressive generate else: res = self(**model_inputs) # pylint: disable=E1102 forward_time = time.time() - forward_time sample_time = time.time() # post process logits; skip this phase if post process is done in graph if not self.config.is_sample_acceleration: # convert to numpy for post process logits = res[0] if isinstance(res, tuple) else res if isinstance(logits, Tensor): logits = logits.asnumpy() logits = np.reshape(logits, (-1, logits.shape[-1])) # need gather last seq logits using current_index # compare length to determine if need gather; if not, gather should be done in model construct if need_gather_logits and logits.shape[0] > len(current_index): logits = logits[current_index] # post process logits, without changing logits shape and order probs = logits_processor(input_ids, logits) probs = logits_warper(input_ids, probs) p_args = np.tile(np.arange(logits.shape[-1]), (batch_size, 1)) else: probs, p_args = res if isinstance(probs, Tensor): probs = probs.asnumpy() if isinstance(p_args, Tensor): p_args = p_args.asnumpy() sample_time = time.time() - sample_time update_time = time.time() # Random select a token as final output for this round target_list = [[] for _ in range(batch_size)] for i in range(batch_size): if is_finished[i]: continue if generation_config.do_sample: # multinomial sample p_norm = softmax(probs[i]) target_index = np.random.choice(len(probs[i]), p=p_norm) else: # greedy target_index = np.argmax(probs[i]) # get target token id target = p_args[i][target_index] input_ids[i, valid_length_each_example[i]] = target if streamer is not None: # assign target element target_list[i] = [target] if is_encoder_decoder: target_mask[i][valid_length_each_example[i]] = int(1) valid_length_each_example[i] += int(1) input_mask[i][valid_length_each_example[i] - 1] = 1 # Stop judgment if p_args[i][target_index] == generation_config.eos_token_id \ or valid_length_each_example[i] == generation_config.max_length: is_finished[i] = True continue if streamer is not None: if batch_size == 1: streamer.put(target_list[0]) else: streamer.put(target_list) update_time = time.time() - update_time logger.debug("forward time: %s s; sample time: %s s; update time: %s s; total count: %s s", forward_time, sample_time, update_time, forward_time + sample_time + update_time) # Return valid outputs out of padded outputs output_ids = [] for i in range(batch_size): output_ids.append( input_ids[i, : int(valid_length_each_example[i])].astype(np.int32) ) logger.debug("The output is: %s", output_ids) if streamer is not None: streamer.end() generate_len = np.sum(valid_length_each_example) - origin_len total_time = time.time() - total_time logger.info("total time: %s s; generated tokens: %s tokens; generate speed: %s tokens/s", total_time, generate_len, generate_len / total_time) return output_ids def generate(self, input_ids: Optional[Union[List[int], List[List[int]]]], generation_config: Optional[GenerationConfig] = None, logits_processor: Optional[LogitsProcessorList] = None, streamer: Optional[BaseStreamer] = None, seed: Optional[int] = None, **kwargs): """ Generate the words according to the given the input ids. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, top_k=3, do_sample=True)`. Args: input_ids(List(str), List(List(str))): The token id list or a batch of token id list. When input a batch of token id list, the length of each token id list should be same. generation_config (`GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default config from the model configuration will be used. Please note that unspecified parameters will inherit [`GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. streamer: The streamer that generator uses. seed: Random seed used in sample. kwargs: Specific parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Supported `generate_config` keywords can be checked in [`GenerationConfig`]'s documentation. Mainly used Keywords are shown below: max_length(int): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (int): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. do_sample(bool): Whether to do sampling on the candidate ids. If set True it will be enabled, and set it to be False to disable the sampling, equivalent to topk 1. If set None, it follows the setting in the configureation in the model. top_k(int): Determine the topK numbers token id as candidate. This should be a positive number. If set None, it follows the setting in the configureation in the model. top_p(float): The accumulation probability of the candidate token ids below the top_p will be select as the condaite ids. The valid value of top_p is between (0, 1]. If the value is larger than 1, top_K algorithm will be enabled. If set None, it follows the setting in the configureation in the model. eos_token_id(int): The end of sentence token id. If set None, it follows the setting in the configureation in the model. pad_token_id(int): The pad token id. If set None, it follows the setting in the configureation in the model. repetition_penalty(float): The penalty factor of the frequency that generated words. The If set 1, the repetition_penalty will not be enabled. If set None, it follows the setting in the configureation in the model. Default None. Examples: >>> from mindformers import T5ForConditionalGeneration, T5Tokenizer >>> t5 = T5ForConditionalGeneration.from_pretrained("t5_small") >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> words = "translate the English to the Romanian: UN Chief Says There Is No Military Solution in Syria" >>> words = tokenizer(words, max_length=21, padding='max_length')['input_ids'] >>> output = t5.generate(words, do_sample=True) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) eful ONU declară că nu există o soluţie militară în Siria >>> # Enable the top p sampling >>> output = t5.generate(words, do_sample=True, top_p=0.4) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) eful ONU declară că nu există o soluţie militară în Siria >>> # Enable the top k sampling. >>> output = t5.generate(words, do_sample=True, top_k=10, top_p=1) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) Este comist de stat ale stateului membre nai uzusepa şi ONU Returns: A list of the generated token ids """ origin_phase = self.phase self.set_train(False) input_ids = np.array(input_ids).reshape(-1, np.shape(input_ids)[-1]) seed = 0 if seed is None else seed np.random.seed(seed) logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList() # Handle `generation_config` and kwargs that might update it # priority: `generation_config` argument > `model.generation_config` (default config) if generation_config is None: # legacy: users may modify the model configuration to control generation # model attribute accordingly, if it was created from the model config generation_config = GenerationConfig.from_model_config(self.config) generation_config = copy.deepcopy(generation_config) model_kwargs = generation_config.update( **kwargs ) # All unused kwargs must be model kwargs if not generation_config.do_sample: generation_config.top_p = 1.0 generation_config.top_k = 0 logger.debug("Generation Config is: %s", generation_config) logits_processor = self._get_logits_processor( generation_config=generation_config, logits_processor=logits_processor, ) logits_warper = self._get_logits_warper(generation_config) output_ids = self._forward( origin_inputs=input_ids, generation_config=generation_config, logits_processor=logits_processor, logits_warper=logits_warper, streamer=streamer, **model_kwargs, ) # set to original phase self.set_train(origin_phase == "train") return output_ids pengcheng.mind-test/mindformers/generation/utils.py000066400000000000000000000045561471400134000231150ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """utils for text generation.""" import numpy as np def log_softmax(x, axis=None): """numpy implemented log softmax function. refers to https://github.com/scipy/scipy/blob/v1.11.1/scipy/special/_logsumexp.py""" x_max = np.amax(x, axis=axis, keepdims=True) if x_max.ndim > 0: x_max[~np.isfinite(x_max)] = 0 elif not np.isfinite(x_max): x_max = 0 tmp = x - x_max exp_tmp = np.exp(tmp) # suppress warnings about log of zero with np.errstate(divide="ignore"): s = np.sum(exp_tmp, axis=axis, keepdims=True) out = np.log(s) out = tmp - out return out def softmax(x, axis=None): """numpy implemented softmax function. refers to https://github.com/scipy/scipy/blob/v1.11.1/scipy/special/_logsumexp.py""" x_max = np.amax(x, axis=axis, keepdims=True) exp_x_shifted = np.exp(x - x_max) return exp_x_shifted / np.sum(exp_x_shifted, axis=axis, keepdims=True) def topk(x, top_k, axis=-1, largest=True, sort=True): """numpy implemented topk sample.""" # safety check if x.shape[axis] < top_k: top_k = x.shape[axis] - 1 if largest: topk_index = np.argpartition(-x, top_k, axis=axis) else: topk_index = np.argpartition(x, top_k, axis=axis) topk_index = np.take(topk_index, np.arange(top_k), axis=axis) topk_data = np.take_along_axis(x, topk_index, axis=axis) if sort: sort_index = ( np.argsort(-topk_data, axis=axis) if largest else np.argsort(topk_data, axis=axis) ) topk_data = np.take_along_axis(topk_data, sort_index, axis=axis) topk_index = np.take_along_axis(topk_index, sort_index, axis=axis) return topk_data, topk_index pengcheng.mind-test/mindformers/inference/000077500000000000000000000000001471400134000211745ustar00rootroot00000000000000pengcheng.mind-test/mindformers/inference/__init__.py000066400000000000000000000015111471400134000233030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """infer module""" from .infer_config import InferConfig from .infer_task import InferTask from .pipeline import get_mslite_pipeline __all__ = ['InferConfig', 'InferTask'] pengcheng.mind-test/mindformers/inference/context.py000066400000000000000000000025301471400134000232320ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """context of mslite apis.""" from mindspore_lite.context import Context from mindformers.inference.infer_config import InferConfig def build_ascend_context(config: InferConfig): """build context for ascend.""" context = Context() context.ascend.rank_id = config.rank_id context.ascend.device_id = config.device_id context.ascend.provider = "ge" context.target = [config.target] return context def build_context(config: InferConfig): """build context for mslite.""" if config.target == "Ascend": return build_ascend_context(config) raise KeyError(f"Supported data type keywords include: " f"[Ascend], but get {config.target}") pengcheng.mind-test/mindformers/inference/infer_config.py000066400000000000000000000041251471400134000242000ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """lite config.""" from mindspore_lite import ModelType from mindformers.models import BaseConfig def convert_lite_model_type(model_type: str): """ convert str to ModelType :param model_type: str :return: ModelType """ if model_type == "mindir": return ModelType.MINDIR if model_type == "mindir_lite": return ModelType.MINDIR_LITE raise KeyError(f"Supported data type keywords include: " f"[mindir, mindir_lite], but get {model_type}") class InferConfig(BaseConfig): """ Inference config class. """ def __init__(self, prefill_model_path: str = "", increment_model_path: str = "", model_type: str = "mindir", model_name: str = "common", infer_seq_length: int = 1024, target: str = "Ascend", device_id: int = 0, rank_id: int = 0, ge_config_path: str = "", **kwargs): super(InferConfig, self).__init__(**kwargs) self.prefill_model_path = prefill_model_path self.increment_model_path = increment_model_path self.model_type = convert_lite_model_type(model_type) self.model_name = model_name self.seq_length = infer_seq_length self.target = target self.device_id = device_id self.rank_id = rank_id self.config_path = ge_config_path pengcheng.mind-test/mindformers/inference/infer_task.py000066400000000000000000000035111471400134000236730ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Infer task class""" from .infer_config import InferConfig from .infers.base_infer import BaseInfer from .infers.text_generator_infer import TextGeneratorInfer class InferTask: """ Infer Task Factory. """ task_mapping = { "text_generation": TextGeneratorInfer, } @classmethod def get_infer_task(cls, task_type: str, task_config: InferConfig, **kwargs) -> BaseInfer: """ Get a infer task obj. by task_type. Args: task_type (str): task name task_config (InferConfig): config of infer task. Returns: BaseInfer. infer task obj. """ return InferTask.task_mapping[task_type](task_config, **kwargs) @classmethod def check_task_valid(cls, task_type: str) -> bool: """ check whether task type is valid or not. Args: task_type: str. task name Returns: bool """ return task_type in InferTask.support_list() @classmethod def support_list(cls): """ get infer task support list. Returns: List[str] """ return InferTask.task_mapping.keys() pengcheng.mind-test/mindformers/inference/infers/000077500000000000000000000000001471400134000224625ustar00rootroot00000000000000pengcheng.mind-test/mindformers/inference/infers/__init__.py000066400000000000000000000013001471400134000245650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """inference tasks.""" __all__ = [] pengcheng.mind-test/mindformers/inference/infers/base_infer.py000066400000000000000000000063731471400134000251420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """base infer class""" import abc from typing import Optional from mindspore_lite import Model, ModelGroup, ModelGroupFlag from mindformers.models import BaseTokenizer, BaseImageProcessor from mindformers.inference.infer_config import InferConfig from mindformers.inference.context import build_context class BaseInfer(metaclass=abc.ABCMeta): """ BaseInfer. """ def __init__(self, config: InferConfig = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None): if config is None: config = InferConfig() self.config = config self.model_type = config.model_type self.model_name = config.model_name self.seq_length = config.seq_length self.config_path = config.config_path self.context = build_context(config=self.config) self.tokenizer = tokenizer self.image_processor = image_processor self.full_model = None self.cache_model = None if config.prefill_model_path and config.increment_model_path: self.full_model, self.cache_model = self._load_increment_models( config.prefill_model_path, config.increment_model_path ) else: self.full_model = self._load_model(config.prefill_model_path) def _load_model(self, model_path): """ load single model from model path. """ model = Model() model.build_from_file(model_path, model_type=self.model_type, context=self.context, config_path=self.config_path) return model def _load_increment_models(self, full_model_path, cache_model_path): """ load kv cache models. """ full_model = Model() cache_model = Model() model_group = ModelGroup(ModelGroupFlag.SHARE_WEIGHT) model_group.add_model([full_model, cache_model]) full_model.build_from_file(full_model_path, self.model_type, self.context, self.config_path) cache_model.build_from_file(cache_model_path, self.model_type, self.context, self.config_path) return full_model, cache_model def __call__(self, inputs, **kwargs): """call infer process.""" return self.infer(inputs, **kwargs) @abc.abstractmethod def infer(self, inputs, **kwargs): """infer interface.""" @abc.abstractmethod def preprocess(self, input_data, **kwargs): """preprocess interface.""" @abc.abstractmethod def postprocess(self, predict_data, **kwargs): """postprocess interface.""" pengcheng.mind-test/mindformers/inference/infers/text_generator_infer.py000066400000000000000000000536301471400134000272600ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Generator Infer.""" import abc import time from typing import Union, List, Optional import numpy as np from mindspore_lite import Model from mindformers.tools.logger import logger from mindformers.models import BaseTokenizer from mindformers.generation import GenerationConfig, LogitsProcessorList from mindformers.generation.logits_process import RepetitionPenaltyLogitsProcessor, LogitNormalization, \ TemperatureLogitsWarper, TopKLogitsWarper, TopPLogitsWarper from mindformers.generation.streamers import BaseStreamer from mindformers.generation.utils import softmax from .base_infer import BaseInfer class BaseInputsOfInfer: """ BaseInputsOfInfer interface. """ @abc.abstractmethod def get_inputs(self, model: Model, **kwargs): pass def get_lite_tensor_list(self, inputs, model): input_list = [] for item in inputs: if item is None: continue input_list.append(item) lite_inputs = model.get_inputs() for input_np, tensor in zip(input_list, lite_inputs): tensor.set_data_from_numpy(input_np) return lite_inputs class CommonInputsOfInfer(BaseInputsOfInfer): """ common infer inputs of llm models. """ # pylint: disable=W0221 def get_inputs(self, model: Model, input_ids=None, current_index=None, valid_length=None, init_reset=None, is_first_iteration=True, **kwargs): if not is_first_iteration: inputs_tmp = [] for i in range(len(current_index)): current_index_tmp = int(current_index[i]) - i * input_ids.shape[1] # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp:current_index_tmp + 1]) input_ids = np.array(inputs_tmp, dtype=np.int32) inputs = [input_ids, current_index, init_reset, valid_length] lite_inputs = self.get_lite_tensor_list(inputs, model) return lite_inputs class GLMInputsOfInfer(BaseInputsOfInfer): """ glm infer inputs. """ def get_masks_np(self, input_ids, tokenizer: BaseTokenizer): batch_size, seq_length = input_ids.shape context_lengths = [list(seq).index(tokenizer.bos_token_id) for seq in input_ids] attention_mask = np.tril(np.ones((batch_size, seq_length, seq_length))) for i, context_length in enumerate(context_lengths): attention_mask[i, :, :context_length] = 1 attention_mask = np.expand_dims(attention_mask, axis=1) attention_mask = np.array(attention_mask < 0.5, np.bool_) return attention_mask def get_position_ids_np(self, input_ids, mask_positions, tokenizer: BaseTokenizer, use_gmasks=None, position_encoding_2d=True): """Get position ids from input_ids and mask_positions with numpy""" batch_size, seq_length = input_ids.shape if use_gmasks is None: use_gmasks = [False] * batch_size context_lengths = [list(seq).index(tokenizer.bos_token_id) for seq in input_ids] if position_encoding_2d: position_ids = np.repeat(np.expand_dims(np.arange(seq_length), 0), batch_size, axis=0) for i, context_length in enumerate(context_lengths): position_ids[i, context_length:] = mask_positions[i] block_position_ids = [np.concatenate(( np.zeros(context_length, np.int32), np.arange(seq_length - context_length, dtype=np.int32) + 1 )) for context_length in context_lengths] block_position_ids = np.stack(block_position_ids, axis=0) position_ids = np.stack((position_ids, block_position_ids), axis=1) else: position_ids = np.repeat(np.expand_dims(np.arange(seq_length), 0), batch_size, axis=0) for i, context_length in enumerate(context_lengths): if not use_gmasks[i]: position_ids[context_length:] = mask_positions[i] return position_ids def create_position_ids_np(self, input_ids, tokenizer, position_encoding_2d=True): """Get position ids from input_ids with numpy""" mask, gmask = tokenizer.mask_token_id, tokenizer.gmask_token_id seqs = list(input_ids) mask_positions, use_gmasks = [], [] for seq in seqs: mask_token = gmask if gmask in seq else mask use_gmask = mask_token == gmask mask_positions.append(list(seq).index(mask_token)) use_gmasks.append(use_gmask) position_ids = self.get_position_ids_np(input_ids, mask_positions, tokenizer, use_gmasks=None, position_encoding_2d=position_encoding_2d) return position_ids # pylint: disable=W0221 def get_inputs(self, model: Model, input_ids=None, current_index=None, valid_length=None, init_reset=None, tokenizer=None, position_encoding_2d=True, is_first_iteration=True, **kwargs): attention_mask_ = self.get_masks_np(input_ids, tokenizer).astype(np.int32) position_ids_ = self.create_position_ids_np(input_ids, tokenizer, position_encoding_2d).astype(np.int32) if not is_first_iteration: attention_mask = [] position_ids = [] inputs_tmp = [] for i in range(len(current_index)): current_index_tmp = int(current_index[i]) - i * input_ids.shape[1] # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp:current_index_tmp + 1]) position_ids.append(position_ids_[i][:, current_index_tmp:current_index_tmp + 1]) attention_mask.append(attention_mask_[i][:, current_index_tmp:current_index_tmp + 1, :]) input_ids = np.array(inputs_tmp, np.int32) position_ids = np.array(position_ids, np.int32) attention_mask = np.array(attention_mask, np.int32) else: attention_mask = attention_mask_ position_ids = position_ids_ inputs = [input_ids, position_ids, attention_mask, current_index, init_reset, valid_length] lite_inputs = self.get_lite_tensor_list(inputs, model) return lite_inputs class InputOfInfer: """ Input of llm model. """ MAPPING = { "bloom": CommonInputsOfInfer, "llama": CommonInputsOfInfer, "glm2": CommonInputsOfInfer, "glm": GLMInputsOfInfer, "common": CommonInputsOfInfer } @classmethod def get_inputs(cls, model_name: str, model, **kwargs): """ Get input tensor list of mslite. Args: model_name: str, model name. model: mindspore_lite.Model. mode obj of mslite. Returns: tensor list of mslite. """ name = "" if model_name not in InputOfInfer.MAPPING: for k in InputOfInfer.MAPPING: if model_name.startswith(k): name = k break if not name: logger.warning("Model name not in support maps.Common input format will be used to do inference.") name = "common" else: name = model_name return InputOfInfer.MAPPING[name]().get_inputs(model, **kwargs) class TextGeneratorInfer(BaseInfer): """ Text generator infer implement class. """ # pylint: disable=W0221 def infer(self, inputs: Union[str, List[str]], do_sample: bool = False, top_k: int = 1, top_p: float = 1.0, temperature: float = 1.0, repetition_penalty: float = 1.0, eos_token_id: int = 2, pad_token_id: int = 0, max_length: int = 256, is_sample_acceleration: bool = False, add_special_tokens: bool = False, streamer: Optional[BaseStreamer] = None, **kwargs): """ text generator inference api Args: inputs(List(str), List(List(str))): The token id list or a list of token id list. do_sample(bool): Whether to do sampling on the candidate ids. If set True it will be enabled, and set it to be False to disable the sampling, equivalent to topk 1. If set None, it follows the setting in the configureation in the model. Default None. top_k(int): Determine the topK numbers token id as candidate. This should be a positive number. If set None, it follows the setting in the configureation in the model. Default 1. top_p(float): The accumulation probability of the candidate token ids below the top_p will be select as the condaite ids. The valid value of top_p is between (0, 1]. If the value is larger than 1, top_K algorithm will be enabled. If set None, it follows the setting in the configureation in the model. Default 1. temperature (`float`, *optional*, defaults to 1.0): The value used to modulate the next token probabilities. eos_token_id(int): The end of sentence token id. If set None, it follows the setting in the configureation in the model. Default 2. pad_token_id(int): The padding of sentence token id. If set None, it follows the setting in the configureation in the model. Default 0. repetition_penalty(float): The penalty factor of the frequency that generated words. The If set 1, the repetition_penalty will not be enabled. If set None, it follows the setting in the configureation in the model. Default 1. max_length: The maximum length of the generated words. If set None, it follows the setting in the configureation in the model. Default 256. is_sample_acceleration: The postprocess are processing in model. Default False. add_special_tokens: Add special tokens for preprocess. streamer: The streamer that generator uses. Returns: outputs of model infer """ input_ids = self.preprocess(inputs, add_special_tokens) output_ids = self.generate(input_ids, do_sample, top_k, top_p, temperature, repetition_penalty, eos_token_id, pad_token_id, max_length, is_sample_acceleration, streamer, **kwargs) outputs = self.postprocess(output_ids) return outputs # pylint: disable=W0613 def preprocess(self, input_data, add_special_tokens=False, **kwargs): """preprocess.""" tokens = self.tokenizer(input_data, add_special_tokens=add_special_tokens) input_ids = tokens["input_ids"] input_list = [] if isinstance(input_data, str): input_list.append(input_ids) else: input_list = input_ids return input_list # pylint: disable=W0613 def postprocess(self, predict_data, **kwargs): """postprocess.""" outputs = self.tokenizer.decode(predict_data, skip_special_tokens=True) return outputs def _get_logits_processor(self, generation_config: GenerationConfig, logits_processor: Optional[LogitsProcessorList]): """ This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsProcessor`] instances used to modify the scores of the language model head. """ # instantiate processors list processors = LogitsProcessorList() if generation_config.repetition_penalty is not None and generation_config.repetition_penalty != 1.0: processors.append(RepetitionPenaltyLogitsProcessor(penalty=generation_config.repetition_penalty)) processors = self._merge_processor_list(processors, logits_processor) # `LogitNormalization` should always be the last logit processor, when present if generation_config.renormalize_logits is True: processors.append(LogitNormalization()) return processors def _merge_processor_list(self, default_list: LogitsProcessorList, custom_list: LogitsProcessorList): """merge custom processor list with default list.""" if not custom_list: return default_list for default in default_list: for custom in custom_list: if type(custom) is type(default): object_type = "logits processor" raise ValueError( f"A custom {object_type} of type {type(custom)} with values {custom} has been passed to" f" `.generate()`, but it has already been created with the values {default}." f" {default} has been created by passing the corresponding arguments to generate or" f" by the model's config default values. If you just want to change the default values" f" of {object_type} consider passing them as arguments to `.generate()`" f" instead of using a custom {object_type}." ) default_list.extend(custom_list) return default_list def _get_logits_warper(self, generation_config: GenerationConfig): """ This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsWarper`] instances used for multinomial sampling. """ # instantiate warpers list warpers = LogitsProcessorList() # all samplers can be found in `generation_utils_samplers.py` if generation_config.temperature is not None and generation_config.temperature != 1.0: warpers.append(TemperatureLogitsWarper(generation_config.temperature)) min_tokens_to_keep = 1 if generation_config.top_k is not None and generation_config.top_k != 0: warpers.append(TopKLogitsWarper(top_k=generation_config.top_k, min_tokens_to_keep=min_tokens_to_keep)) if generation_config.top_p is not None and generation_config.top_p < 1.0: warpers.append(TopPLogitsWarper(top_p=generation_config.top_p, min_tokens_to_keep=min_tokens_to_keep)) # `LogitNormalization` should always be the last logit processor, when present if generation_config.renormalize_logits is True: warpers.append(LogitNormalization()) return warpers def generate(self, input_ids, do_sample, top_k, top_p, temperature, repetition_penalty, eos_token_id, pad_token_id, max_length, is_sample_acceleration, streamer, **kwargs): """token generator.""" sampler_dict = {"do_sample": do_sample, "top_k": top_k, "top_p": top_p, "temperature": temperature, "repetition_penalty": repetition_penalty, "max_length": max_length, **kwargs} generation_config = GenerationConfig(**sampler_dict) if not generation_config.do_sample: generation_config.top_p = 1.0 generation_config.top_k = 0 logits_processor = LogitsProcessorList() logits_processor = self._get_logits_processor( generation_config=generation_config, logits_processor=logits_processor, ) logits_warper = self._get_logits_warper(generation_config) if streamer: streamer.put(np.array(input_ids[0])) batch_size = len(input_ids) valid_length = [] for i in range(batch_size): # As the nonzero returns the index and we need length valid_length.append(np.max(np.argwhere(np.array(input_ids[i]) != pad_token_id)) + 1) valid_length = np.array(valid_length, np.int32) target_length = self.seq_length if max_length > self.seq_length else max_length # pad original input ids to seq_length pad_length = self.seq_length - valid_length pad_input_ids = np.array([ np.pad(input_ids[i], (0, pad_length[i]), 'constant', constant_values=(pad_token_id)) for i in range(len(input_ids)) ], np.int32) input_ids = pad_input_ids # setup is_first_iteration flag for incremental infer is_first_iteration = True is_finished = [False] * batch_size use_past = self.full_model and self.cache_model while np.sum(is_finished) != batch_size: start_time = time.time() seq_length = input_ids.shape[1] current_index = [valid_length[i] - 1 + i * seq_length for i in range(batch_size)] current_index = np.array(current_index, np.int32) logger.debug("validate length: %s", valid_length) if use_past: outputs = self._inc_infer(input_ids, current_index, valid_length, is_first_iteration, **kwargs) else: outputs = self._full_infer(input_ids, current_index, is_sample_acceleration, **kwargs) if not is_sample_acceleration: logits = outputs[0].get_data_to_numpy() vocab_size = logits.shape[-1] if len(logits.shape) < 3: logits = logits.reshape(batch_size, -1, vocab_size) # gather logits if is_first_iteration: logits = np.array([logits[i][current_index[i] - i * seq_length, :] for i in range(batch_size)]) else: logits = np.array([logits[i][0, :] for i in range(batch_size)]) logits = logits.reshape(-1, vocab_size) log_probs = logits_processor(input_ids, logits) p = logits_warper(input_ids, log_probs) p_args = np.tile(np.arange(logits.shape[-1]), (batch_size, 1)) else: p = outputs[0].get_data_to_numpy().astype(np.int32) p_args = outputs[1].get_data_to_numpy() # Random select a token as final output for this round for i in range(batch_size): if is_finished[i]: continue # target_index = np.random.choice(len(p[i]), p=p[i]) if generation_config.do_sample: # multinomial sample p_norm = softmax(p[i]) target_index = np.random.choice(len(p[i]), p=p_norm) else: # greedy target_index = np.argmax(p[i]) # Stop judgment when length exceeds target_length. if valid_length[i] >= target_length: is_finished[i] = True continue # add next token to input_ids. target = p_args[i][target_index] input_ids[i, valid_length[i]] = target if streamer: streamer.put(np.asarray([target])) valid_length[i] += int(1) # Stop judgment when output is EOS token, with the output # is appended to input_ids and streamer. if target == eos_token_id: is_finished[i] = True continue is_first_iteration = not use_past print(f"one token takes {time.time() - start_time} s") # Return valid outputs out of padded outputs output_ids = [] for i in range(batch_size): output_ids.append(input_ids[i, : int(valid_length[i])].astype(np.int32)) logger.debug("The output is: %s", output_ids) if streamer: streamer.end() return output_ids def _inc_infer(self, input_ids, current_index, valid_length, is_first_iteration, **kwargs): """kvcache infer""" if is_first_iteration: init_reset = np.array([False]) lite_inputs = self.get_predict_inputs(self.full_model, input_ids, current_index, valid_length, init_reset, is_first_iteration, **kwargs) outputs = self.full_model.predict(lite_inputs) else: init_reset = np.array([True]) lite_inputs = self.get_predict_inputs(self.cache_model, input_ids, current_index, valid_length, init_reset, is_first_iteration, **kwargs) outputs = self.cache_model.predict(lite_inputs) return outputs def _full_infer(self, input_ids, current_index, is_npu_acceleration, **kwargs): """infer""" # get inputs if is_npu_acceleration: lite_inputs = self.get_predict_inputs(self.full_model, input_ids, current_index, **kwargs) else: lite_inputs = self.get_predict_inputs(self.full_model, input_ids, **kwargs) # do infer outputs = self.full_model.predict(lite_inputs) return outputs def get_predict_inputs(self, mode: Model, input_ids, current_index=None, valid_length=None, init_reset=None, is_first_iteration=True, **kwargs): """Get inputs of llm model for mslite.""" return InputOfInfer.get_inputs(self.model_name, mode, input_ids=input_ids, current_index=current_index, valid_length=valid_length, init_reset=init_reset, tokenizer=self.tokenizer, is_first_iteration=is_first_iteration, **kwargs) pengcheng.mind-test/mindformers/inference/pipeline.py000066400000000000000000000236151471400134000233620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """lite infer pipeline.""" import glob import os.path from typing import Optional, Union, Tuple from mindformers.models import BaseTokenizer, BaseImageProcessor, BaseAudioProcessor, \ build_tokenizer, build_processor from mindformers.mindformer_book import MindFormerBook from mindformers.tools.register import MindFormerConfig from .infer_task import InferTask from .infers.base_infer import BaseInfer from .infer_config import InferConfig SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() def is_model_name(model): if not isinstance(model, str): return False if os.path.exists(model): return False return True def is_tuple(model): return isinstance(model, tuple) def is_path(model): return isinstance(model, str) and os.path.isfile(model) def is_dir(model): return isinstance(model, str) and os.path.isdir(model) # align with pipeline() # pylint: disable=W0613 def get_infer_pipeline_from_dir(task, model, tokenizer, image_processor, audio_processor, ge_config_path=None, **kwargs) -> BaseInfer: """ Support user project contain certain files. (*.yaml/config.ini/tokenizer.model/graph1.mindir graph2.mindir) Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (str): The model dir. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. ge_config_path: ge config file path Returns: BaseInfer. """ if not os.path.exists(model): raise FileNotFoundError(f"{model} is not found.") proj_dir = os.path.realpath(model) # check model run config run_config = glob.glob(proj_dir + "/*.yaml") if len(run_config) > 1: raise ValueError(f"{model} contains multi config files: {run_config}.") config = MindFormerConfig(run_config[0]) if config.infer is None: raise ValueError(f"There is no infer config in {run_config[0]}.") if tokenizer is None: tokenizer = build_tokenizer(config.processor.tokenizer) if image_processor is None and hasattr(config.processor, 'image_processor'): image_processor = build_processor(config.processor.image_processor) # check ge config.ini if ge_config_path is None: ge_config_path = os.path.join(proj_dir, "config.ini") if not os.path.exists(ge_config_path): raise ValueError(f"There is ge config.ini file in {model}.") config.infer.config_path = ge_config_path # graph.mindir graphs = glob.glob(proj_dir + "/*.mindir") if len(graphs) > 2: raise ValueError(f"MindIR files in {model} must equal to 1 or 2. But get {len(graphs)} files.") if len(graphs) == 2: if "prefill" in graphs[0]: config.infer.prefill_model_path = graphs[0] config.infer.increment_model_path = graphs[1] else: config.infer.prefill_model_path = graphs[1] config.infer.increment_model_path = graphs[0] else: config.infer.prefill_model_path = graphs[0] # set device id and rank id if "device_id" in kwargs: config.infer.device_id = kwargs["device_id"] if "rank_id" in kwargs: config.infer.rank_id = kwargs["rank_id"] # add model name config.infer.model_name = config.trainer.model_name infer_config = InferConfig( **config.infer ) task_pipeline = InferTask.get_infer_task(task, infer_config, tokenizer=tokenizer, image_processor=image_processor) return task_pipeline # align with pipeline() # pylint: disable=W0613 def get_infer_pipeline_from_model_name(task, model, tokenizer, image_processor, audio_processor, ge_config_path, **kwargs) -> BaseInfer: """ Support get infer pipeline from model name. Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (str): The model name. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. ge_config_path: ge config file path Returns: BaseInfer. """ if model not in SUPPORT_MODEL_NAMES: raise KeyError( f"model must be in {SUPPORT_MODEL_NAMES} when model's type is string, but get {model}.") pipeline_config = MindFormerConfig(SUPPORT_PIPELINES.get(task).get(model)) if pipeline_config.infer is None: raise ValueError(f"There is no infer config in {model} config.") if image_processor is None and hasattr(pipeline_config.processor, 'image_processor'): image_processor = build_processor(pipeline_config.processor.image_processor) if audio_processor is None and hasattr(pipeline_config.processor, 'audio_processor'): audio_processor = build_processor(pipeline_config.processor.audio_processor) if tokenizer is None: tokenizer = build_tokenizer(pipeline_config.processor.tokenizer) pipeline_config.infer.model_name = model # set device id and rank id if "device_id" in kwargs: pipeline_config.infer.device_id = kwargs["device_id"] if "rank_id" in kwargs: pipeline_config.infer.rank_id = kwargs["rank_id"] infer_config = InferConfig( **pipeline_config.infer ) task_pipeline = InferTask.get_infer_task(task, infer_config, tokenizer=tokenizer, image_processor=image_processor) return task_pipeline # align with pipeline() # pylint: disable=W0613 def get_infer_pipeline_from_model_files(task, model, tokenizer, image_processor, audio_processor, ge_config_path, **kwargs) -> BaseInfer: """ Support get infer pipeline from model files. Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (Union[str, Tuple[str, str]]): The model paths of tuple(model_path, model_path). tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. ge_config_path: ge config file path Returns: BaseInfer. """ if isinstance(model, str): full_model_path = model cache_model_path = "" else: full_model_path, cache_model_path = model full_model_path = os.path.realpath(full_model_path) if not os.path.exists(full_model_path): raise FileNotFoundError(f"{full_model_path} is not found.") if cache_model_path: cache_model_path = os.path.realpath(cache_model_path) if not os.path.exists(cache_model_path): raise FileNotFoundError(f"{cache_model_path} is not found.") config = InferConfig( prefill_model_path=full_model_path, increment_model_path=cache_model_path, ge_config_path=ge_config_path, **kwargs ) task_pipeline = InferTask.get_infer_task(task, config, tokenizer=tokenizer, image_processor=image_processor) return task_pipeline STRATEGY_MAP = { is_model_name: get_infer_pipeline_from_model_name, is_tuple: get_infer_pipeline_from_model_files, is_path: get_infer_pipeline_from_model_files, is_dir: get_infer_pipeline_from_dir, } def get_mslite_pipeline(task: str = None, model: Optional[Union[str, BaseInfer, Tuple[str, str]]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, ge_config_path: str = None, **kwargs) -> BaseInfer: """ Get mslite infer pipeline. Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (Union[str, Tuple[str, str]]): The model name. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. ge_config_path: ge config file path Returns: BaseInfer. """ if not InferTask.check_task_valid(task): raise KeyError(f"{task} is not supported by pipeline. please select" f" a task from {InferTask.support_list()}.") # init infer_pipeline = None for k, v in STRATEGY_MAP.items(): if k(model): infer_pipeline = v(task, model, tokenizer, image_processor, audio_processor, ge_config_path, **kwargs) if not infer_pipeline: raise ValueError(f"model must be a dir or a mindir file path or a tuple of file path, but get {model}.") return infer_pipeline pengcheng.mind-test/mindformers/mindformer_book.py000066400000000000000000001724601471400134000227760ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ MindFormerBook class, which contains the lists of models, pipelines, tasks, and default settings in MindFormer repository. """ import os import copy from collections import OrderedDict from mindformers.tools import logger def print_dict(input_dict): """ Print dict function for show support method of MindFormerBook or other BaseClasses. Args: input_dict (dict): the dict to be printed. """ if isinstance(input_dict, dict): for key, val in input_dict.items(): if isinstance(val, dict): logger.info("%s:", key) print_dict(val) else: logger.info(" %s : %s", key, val) else: raise TypeError(f"{type(input_dict)} is unspoorted by print_dict") def print_path_or_list(input_path_or_list): """ Print path or list function for show support method of MindFormerBook or other BaseClasses. Args: input_path_or_list (str, list): the path or list to be printed. """ if isinstance(input_path_or_list, (str, list)): logger.info(" %s", input_path_or_list) logger.info("-------------------------------------") else: raise TypeError(f"{type(input_path_or_list)} is unsupported by print_path_or_list") class MindFormerBook: """ MindFormerBook class, which contains the lists of models, pipelines, tasks, and default settings in MindFormer repository When adding a new Model or assemble in this project, the following constants list and dict need adding. Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> >>> # 1) Fill the following constant list and dict in this class >>> # 2) Overwrite the support_list when define a new Model or Pipeline. >>> @MindFormerRegister.register(MindFormerModuleType.MODELS) ... class ViTForImageClassification(BaseModel): ... _support_list = MindFormerBook.get_model_support_list()['vit'] >>> # 3) Then you can use auto class and from pretrain to init an instance. >>> vit_model = AutoModel.from_pretrained('vit_base_p16') """ _PROJECT_PATH = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) _DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER = os.getenv("CHECKPOINT_DOWNLOAD_FOLDER", os.path.join('.', 'checkpoint_download')) _DEFAULT_CHECKPOINT_SAVE_FOLDER = os.getenv("CHECKPOINT_SAVE_FOLDER", os.path.join('.', 'checkpoint_save')) _TRAINER_SUPPORT_TASKS_LIST = OrderedDict([ ("general", OrderedDict([ ("common", os.path.join( _PROJECT_PATH, "configs/general/run_general_task.yaml"))]) ), ("masked_image_modeling", OrderedDict([ ("mae_vit_base_p16", os.path.join( _PROJECT_PATH, "configs/mae/run_mae_vit_base_p16_224_800ep.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/mae/run_mae_vit_base_p16_224_800ep.yaml"))]) ), ("image_classification", OrderedDict([ ("vit_base_p16", os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")), ("swin_base_p4w7", os.path.join( _PROJECT_PATH, "configs/swin/run_swin_base_p4w7_224_100ep.yaml")), ("mindspore/vit_base_p16", os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")), ("mindspore/swin_base_p4w7", os.path.join( _PROJECT_PATH, "configs/swin/run_swin_base_p4w7_224_100ep.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml"))]) ), ("fill_mask", OrderedDict([ ("bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_base_uncased.yaml")), ("bert_tiny_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_tiny_uncased.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_tiny_uncased.yaml"))]) ), ("contrastive_language_image_pretrain", OrderedDict([ ("clip_vit_b_32", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml")), ("blip2_stage1_vit_g", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_qformer_pretrain.yaml")), ("blip2_stage2_vit_g_baichuan_7b", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_baichuan_7b.yaml")), ("blip2_stage2_vit_g_llama_7b", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_llama_7b.yaml")), ("mindspore/clip_vit_b_32", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml")), ("clip_vit_b_16", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_16_pretrain_flickr8k.yaml")), ("clip_vit_l_14", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14_pretrain_flickr8k.yaml")), ("clip_vit_l_14@336", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14@336_pretrain_flickr8k.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml"))]) ), ("image_to_text_retrieval", OrderedDict([ ("blip2_stage1_vit_g", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yaml"))]) ), ("zero_shot_image_classification", OrderedDict([ ("clip_vit_b_32", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ("mindspore/clip_vit_b_32", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ("clip_vit_b_16", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_16_zero_shot_image_classification_cifar100.yaml")), ("clip_vit_l_14", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14_zero_shot_image_classification_cifar100.yaml")), ("clip_vit_l_14@336", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14@336_zero_shot_image_classification_cifar100.yaml")), ('blip2_stage1_classification', os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml"))]) ), ("image_to_text_generation", OrderedDict([ ("itt_blip2_stage2_vit_g_baichuan_7b", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yaml")), ("itt_blip2_stage2_vit_g_llama_7b", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/clip/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml"))]) ), ("translation", OrderedDict([ ("t5_small", os.path.join( _PROJECT_PATH, "configs/t5/run_t5_small_on_wmt16.yaml")), ("t5_tiny", os.path.join( _PROJECT_PATH, "configs/t5/run_t5_tiny_on_wmt16.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/t5/run_t5_small_on_wmt16.yaml"))]) ), ("text_classification", OrderedDict([ ("txtcls_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased.yaml")), ("txtcls_bert_base_uncased_mnli", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml")), ("mindspore/txtcls_bert_base_uncased_mnli", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml")), ("gpt2_txtcls", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_txtcls.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased.yaml"))]) ), ("token_classification", OrderedDict([ ("tokcls_bert_base_chinese", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese.yaml")), ("tokcls_bert_base_chinese_cluener", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese_cluener.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese.yaml"))]) ), ("question_answering", OrderedDict([ ("qa_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")), ("qa_bert_base_uncased_squad", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")), ("mindspore/qa_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml"))]) ), ("text_generation", OrderedDict([ ("gpt2", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2.yaml")), ("gpt2_lora", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_lora.yaml")), ("gpt2_13b", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_13b.yaml")), ("gpt2_52b", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_52b.yaml")), ("gpt2_xl", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_xl.yaml")), ("gpt2_xl_lora", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_xl_lora.yaml")), ("llama_7b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_7b.yaml")), ("llama_13b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_13b.yaml")), ("llama_65b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_65b.yaml")), ("llama2_7b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_7b.yaml")), ("llama2_13b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_13b.yaml")), ("llama2_70b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_70b.yaml")), ("llama_7b_lora", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_7b_lora.yaml")), ("pangualpha_2_6b", os.path.join( _PROJECT_PATH, "configs/pangualpha/run_pangualpha_2_6b.yaml")), ("pangualpha_13b", os.path.join( _PROJECT_PATH, "configs/pangualpha/run_pangualpha_13b.yaml")), ("glm_6b", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_finetune.yaml")), ("glm_6b_chat", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_infer.yaml")), ("glm_6b_lora", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_lora.yaml")), ("glm_6b_lora_chat", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_lora_infer.yaml")), ("glm2_6b", os.path.join( _PROJECT_PATH, "configs/glm2/run_glm2_6b.yaml")), ("glm2_6b_lora", os.path.join( _PROJECT_PATH, "configs/glm2/run_glm2_6b_lora.yaml")), ("bloom_560m", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_560m.yaml")), ("bloom_7.1b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_7.1b.yaml")), ("bloom_65b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_65b.yaml")), ("bloom_176b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_176b.yaml")), ("baichuan_7b", os.path.join( _PROJECT_PATH, "research/baichuan/run_baichuan_7b.yaml")), ("baichuan2_7b", os.path.join( _PROJECT_PATH, "research/baichuan2/run_baichuan2_7b.yaml")), ("baichuan2_13b", os.path.join( _PROJECT_PATH, "research/baichuan2/run_baichuan2_13b.yaml")), ("ziya_13b", os.path.join( _PROJECT_PATH, "research/ziya/run_ziya_13b.yaml")), ("internlm_7b", os.path.join( _PROJECT_PATH, "research/internlm/run_internlm_7b.yaml")), ("internlm_7b_lora", os.path.join( _PROJECT_PATH, "research/internlm/run_internlm_7b_lora.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2.yaml"))]) ), ("segment_anything", OrderedDict([ ("sam_vit_b", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-b.yaml")), ("sam_vit_l", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-l.yaml")), ("sam_vit_h", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-h.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-h.yaml"))]) ) ]) _PIPELINE_SUPPORT_TASK_LIST = OrderedDict([ ('zero_shot_image_classification', OrderedDict([ ('clip_vit_b_16', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_16_zero_shot_image_classification_cifar100.yaml")), ('mindspore/clip_vit_b_32', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ('clip_vit_b_32', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ('clip_vit_l_14', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14_zero_shot_image_classification_cifar100.yaml")), ('clip_vit_l_14@336', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14@336_zero_shot_image_classification_cifar100.yaml")), ('common', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ('blip2_stage1_classification', os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml")) ])), ('image_to_text_generation', OrderedDict([ ('itt_blip2_stage2_vit_g_baichuan_7b', os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yaml")), ('itt_blip2_stage2_vit_g_llama_7b', os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml")) ])), ('masked_image_modeling', OrderedDict([ ('mae_vit_base_p16', os.path.join( _PROJECT_PATH, "configs/mae/run_mae_vit_base_p16_224_800ep.yaml")), ('common', os.path.join( _PROJECT_PATH, "configs/mae/run_mae_vit_base_p16_224_800ep.yaml")) ])), ('image_classification', OrderedDict([ ('vit_base_p16', os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")), ("swin_base_p4w7", os.path.join( _PROJECT_PATH, "configs/swin/run_swin_base_p4w7_224_100ep.yaml")), ('mindspore/vit_base_p16', os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")), ("mindspore/swin_base_p4w7", os.path.join( _PROJECT_PATH, "configs/swin/run_swin_base_p4w7_224_100ep.yaml")), ('common', os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")) ])), ('translation', OrderedDict([ ('t5_small', os.path.join( _PROJECT_PATH, "configs/t5/run_t5_small_on_wmt16.yaml")), ('common', os.path.join( _PROJECT_PATH, "configs/t5/run_t5_small_on_wmt16.yaml")) ])), ("fill_mask", OrderedDict([ ("bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_base_uncased.yaml")), ("bert_tiny_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_tiny_uncased.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_tiny_uncased.yaml")) ])), ("text_classification", OrderedDict([ ("txtcls_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased.yaml")), ("txtcls_bert_base_uncased_mnli", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml")), ("mindspore/txtcls_bert_base_uncased_mnli", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml")), ("gpt2_txtcls", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_txtcls.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased.yaml")) ])), ("token_classification", OrderedDict([ ("tokcls_bert_base_chinese", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese.yaml")), ("tokcls_bert_base_chinese_cluener", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese_cluener.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese.yaml")) ])), ("question_answering", OrderedDict([ ("qa_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")) ])), ("text_generation", OrderedDict([ ("gpt2", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2.yaml")), ("gpt2_lora", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_lora.yaml")), ("gpt2_xl", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_xl.yaml")), ("gpt2_xl_lora", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_xl_lora.yaml")), ("gpt2_13b", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_13b.yaml")), ("llama_7b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_7b.yaml")), ("llama_13b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_13b.yaml")), ("llama_65b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_65b.yaml")), ("llama2_7b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_7b.yaml")), ("llama2_13b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_13b.yaml")), ("llama2_70b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_70b.yaml")), ("llama_7b_lora", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_7b_lora.yaml")), ("pangualpha_2_6b", os.path.join( _PROJECT_PATH, "configs/pangualpha/run_pangualpha_2_6b.yaml")), ("pangualpha_13b", os.path.join( _PROJECT_PATH, "configs/pangualpha/run_pangualpha_13b.yaml")), ("glm_6b", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_finetune.yaml")), ("glm_6b_chat", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_infer.yaml")), ("glm_6b_lora", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_lora.yaml")), ("glm_6b_lora_chat", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_lora_infer.yaml")), ("glm2_6b", os.path.join( _PROJECT_PATH, "configs/glm2/run_glm2_6b.yaml")), ("glm2_6b_lora", os.path.join( _PROJECT_PATH, "configs/glm2/run_glm2_6b_lora.yaml")), ("bloom_560m", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_560m.yaml")), ("bloom_7.1b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_7.1b.yaml")), ("bloom_65b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_65b.yaml")), ("bloom_176b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_176b.yaml")), ("baichuan_7b", os.path.join( _PROJECT_PATH, "research/baichuan/run_baichuan_7b.yaml")), ("baichuan2_7b", os.path.join( _PROJECT_PATH, "research/baichuan2/run_baichuan2_7b.yaml")), ("baichuan2_13b", os.path.join( _PROJECT_PATH, "research/baichuan2/run_baichuan2_13b.yaml")), ("ziya_13b", os.path.join( _PROJECT_PATH, "research/ziya/run_ziya_13b.yaml")), ("internlm_7b", os.path.join( _PROJECT_PATH, "research/internlm/run_internlm_7b.yaml")), ("internlm_7b_lora", os.path.join( _PROJECT_PATH, "research/internlm/run_internlm_7b_lora.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2.yaml")) ])), ("image_to_text_retrieval", OrderedDict([ ("blip2_stage1_vig_g", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yaml")) ])), ("segment_anything", OrderedDict([ ("sam_vit_b", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-b.yaml")), ("sam_vit_l", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-l.yaml")), ("sam_vit_h", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-h.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-h.yaml")) ])) ]) _CONFIG_SUPPORT_LIST = OrderedDict([ ('clip', [ 'clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336', 'mindspore/clip_vit_b_32' ]), ('blip2', [ 'blip2_stage1_vig_g', 'blip2_stage1_classification', 'blip2_stage2_vit_g_baichuan_7b', 'blip2_stage2_vit_g_llama_7b', 'itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b', ]), ('itt', OrderedDict([ ('blip2', ['itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b']) ])), ('mae', [ 'mae_vit_base_p16', ]), ('vit', [ 'vit_base_p16', 'mindspore/vit_base_p16', 'vit_g_p16' ]), ('swin', [ 'swin_base_p4w7', 'mindspore/swin_base_p4w7' ]), ('bert', [ 'bert_base_uncased', 'bert_tiny_uncased', ]), ('tokcls', OrderedDict([ ('bert', ['tokcls_bert_base_chinese', 'tokcls_bert_base_chinese_cluener', 'mindspore/tokcls_bert_base_chinese_cluener']) ])), ('txtcls', OrderedDict([ ('bert', ['txtcls_bert_base_uncased', 'txtcls_bert_base_uncased_mnli', 'mindspore/txtcls_bert_base_uncased_mnli']), ('gpt2', ['gpt2_txtcls']) ])), ('qa', OrderedDict([ ('bert', ['qa_bert_base_uncased', 'qa_bert_base_uncased_squad']) ])), ('t5', [ 't5_small', ]), ('filip', [ 'filip_vit_l_14', ]), ('gpt2', [ 'gpt2', 'gpt2_lora', 'gpt2_xl', 'gpt2_xl_lora', 'gpt2_13b' ]), ('llama', [ 'llama_7b', 'llama_13b', 'llama_65b', 'llama2_7b', 'llama2_13b', 'llama2_70b', 'llama_7b_lora', 'baichuan_7b', 'baichuan2_7b', 'baichuan2_13b', 'ziya_13b', 'internlm_7b', 'internlm_7b_lora', ]), ('llama2', [ 'llama2_7b', 'llama2_13b', 'llama2_70b', ]), ('pangualpha', [ 'pangualpha_2_6b', 'pangualpha_13b' ]), ('bloom', [ 'bloom_560m', 'bloom_7.1b', 'bloom_65b', 'bloom_176b', ]), ('glm', [ 'glm_6b', 'glm_6b_chat', 'glm_6b_lora', 'glm_6b_lora_chat' ]), ('glm2', [ 'glm2_6b', 'glm2_6b_lora', ]), ('sam', [ 'sam_vit_b', 'sam_vit_l', 'sam_vit_h' ]) ]) _MODEL_SUPPORT_LIST = OrderedDict([ ('clip', [ 'clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336', 'mindspore/clip_vit_b_32' ]), ('blip2', OrderedDict([ ('1-stg', ['blip2_stage1_vit_g', 'blip2_stage1_classification']), ('2-stg', ['blip2_stage1_pretrained', 'blip2_stage2_vit_g_baichuan_7b', 'blip2_stage2_vit_g_llama_7b']) ])), ('itt', OrderedDict([ ('blip2', ['itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b']) ])), ('mae', [ 'mae_vit_base_p16', ]), ('vit', [ 'vit_base_p16', 'mindspore/vit_base_p16', 'mae_vit_base_p16', 'vit_g_p16' ]), ('swin', [ 'swin_base_p4w7', 'mindspore/swin_base_p4w7' ]), ('bert', [ 'bert_base_uncased', 'bert_tiny_uncased', ]), ('tokcls', OrderedDict([ ('bert', ['tokcls_bert_base_chinese', 'tokcls_bert_base_chinese_cluener', 'mindspore/tokcls_bert_base_chinese_cluener']) ])), ('txtcls', OrderedDict([ ('bert', ['txtcls_bert_base_uncased', 'txtcls_bert_base_uncased_mnli', 'mindspore/txtcls_bert_base_uncased_mnli']), ('gpt2', ['gpt2_txtcls']) ])), ('qa', OrderedDict([ ('bert', ['qa_bert_base_uncased', 'qa_bert_base_uncased_squad']) ])), ('t5', [ 't5_small', ]), ('mt5', [ 't5_pegasus_base', ]), ('filip', [ 'filip_vit_l_14', ]), ('gpt2', [ 'gpt2', 'gpt2_lora', 'gpt2_xl', 'gpt2_xl_lora', 'gpt2_13b' ]), ('llama', [ 'llama_7b', 'llama_13b', 'llama_65b', 'llama2_7b', 'llama2_13b', 'llama2_70b', 'llama_7b_lora' ]), ('llama2', [ 'llama2_7b', 'llama2_13b', 'llama2_70b', ]), ('pangualpha', [ 'pangualpha_2_6b', 'pangualpha_13b' ]), ('bloom', [ 'bloom_560m', 'bloom_7.1b', 'bloom_65b', 'bloom_176b', ]), ('glm', [ 'glm_6b', 'glm_6b_chat', 'glm_6b_lora', 'glm_6b_lora_chat' ]), ('glm2', [ 'glm2_6b', 'glm2_6b_lora' ]), ('internlm', [ 'internlm_7b', 'internlm_7b_lora', ]), ('sam', [ 'sam_vit_b', 'sam_vit_l', 'sam_vit_h' ]) ]) _PROCESSOR_SUPPORT_LIST = OrderedDict([ ('clip', [ 'clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336', 'mindspore/clip_vit_b_32' ]), ('blip2', [ 'blip2_stage1_vit_g', 'blip2_stage1_classification', 'blip2_stage2_vit_g_baichuan_7b', 'blip2_stage2_vit_g_llama_7b' ]), ('itt', OrderedDict([ ('blip2', ['itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b']) ])), ('mae', [ 'mae_vit_base_p16', ]), ('vit', [ 'vit_base_p16', 'mindspore/vit_base_p16', 'vit_g_p16' ]), ('swin', [ 'swin_base_p4w7', 'mindspore/swin_base_p4w7', ]), ('bert', [ 'bert_base_uncased', 'bert_tiny_uncased', ]), ('tokcls', OrderedDict([ ('bert', ['tokcls_bert_base_chinese', 'tokcls_bert_base_chinese_cluener', 'mindspore/tokcls_bert_base_chinese_cluener']) ])), ('txtcls', OrderedDict([ ('bert', ['txtcls_bert_base_uncased', 'txtcls_bert_base_uncased_mnli', 'mindspore/txtcls_bert_base_uncased_mnli']), ('gpt2', ['gpt2_txtcls']) ])), ('t5', [ 't5_small', ]), ('gpt2', [ 'gpt2', 'gpt2_lora', 'gpt2_xl', 'gpt2_xl_lora', 'gpt2_13b' ]), ('llama', [ 'llama', ]), ('pangualpha', [ 'pangualpha_2_6b', 'pangualpha_13b' ]), ('glm', [ 'glm', ]), ('bloom', [ 'bloom_560m', 'bloom_7.1b', 'bloom_65b', 'bloom_176b', ]), ('sam', [ 'sam_vit_b', 'sam_vit_l', 'sam_vit_h' ]) ]) _TOKENIZER_SUPPORT_LIST = OrderedDict([ ('clip', [ 'clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336', 'mindspore/clip_vit_b_32' ]), ('blip2', [ 'blip2_stage1_vit_g', 'blip2_stage1_classification' ]), ('bert', [ 'bert_base_uncased', 'bert_tiny_uncased', ]), ('tokcls', OrderedDict([ ('bert', ['tokcls_bert_base_chinese', 'tokcls_bert_base_chinese_cluener', 'mindspore/tokcls_bert_base_chinese_cluener']) ])), ('txtcls', OrderedDict([ ('bert', ['txtcls_bert_base_uncased', 'txtcls_bert_base_uncased_mnli', 'mindspore/txtcls_bert_base_uncased_mnli']), ('gpt2', ['gpt2_txtcls']) ])), ('qa', OrderedDict([ ('bert', ['qa_bert_base_uncased', 'qa_bert_base_uncased_squad']) ])), ('t5', [ 't5_small', ]), ('gpt2', [ 'gpt2', ]), ('llama', [ 'llama_7b', 'llama_13b', 'llama_65b', 'llama2_7b', 'llama2_13b', 'llama2_70b', 'llama_7b_lora', 'baichuan_7b', 'ziya_13b' ]), ('llama2', [ 'llama2_7b', 'llama2_13b', 'llama2_70b', ]), ('pangualpha', [ 'pangualpha_2_6b', 'pangualpha_13b' ]), ('glm', [ 'glm_6b', ]), ('glm2', [ 'glm2_6b' ]), ('bloom', [ 'bloom_560m', 'bloom_7.1b', 'bloom_65b', 'bloom_176b', ]), ('internlm', [ 'internlm_7b', 'internlm_7b_lora', ]), ('baichuan2', [ 'baichuan2_7b', 'baichuan2_13b', ]), ]) _MODEL_CONFIG_TO_NAME = OrderedDict([ ]) _MODEL_CKPT_URL_LIST = OrderedDict([ ('clip_vit_b_32', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/clip/clip_vit_b_32.ckpt' ]), ('mindspore/clip_vit_b_32', ['https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/clip_vit_b_32/file/clip_vit_b_32.ckpt']), ('clip_vit_b_16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_b_16.ckpt' ]), ('clip_vit_l_14', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_l_14.ckpt' ]), ('clip_vit_l_14@336', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_l_14%40336.ckpt' ]), ('mae_vit_base_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/mae/mae_vit_base_p16.ckpt' ]), ('vit_base_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/vit/vit_base_p16.ckpt' ]), ('vit_g_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/vit_g_p16.ckpt' ]), ('blip2_stage1_classification', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('blip2_stage1_pretrained', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('blip2_stage2_vit_g_baichuan_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('itt_blip2_stage2_vit_g_baichuan_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('blip2_stage2_vit_g_llama_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('itt_blip2_stage2_vit_g_llama_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('mindspore/vit_base_p16', ['https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/vit_base_p16/file/vit_base_p16.ckpt']), ('swin_base_p4w7', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/swin/swin_base_p4w7.ckpt' ]), ('mindspore/swin_base_p4w7', ['https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/swin_base_p4w7/file/swin_base_p4w7.ckpt']), ('t5_small', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/t5/mindspore_model.ckpt' ]), ('bert_tiny_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/bert_tiny/bert_tiny_uncased.ckpt' ]), ('bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/bert_base/bert_base_uncased.ckpt' ]), ('tokcls_bert_base_chinese', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/' 'token_classification/tokcls_bert_base_chinese.ckpt' ]), ('tokcls_bert_base_chinese_cluener', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/token_classification/' 'tokcls_bert_base_chinese_cluener.ckpt' ]), ('mindspore/tokcls_bert_base_chinese_cluener', ['https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/tokcls_bert_base_chineses_cluener/' 'file/tokcls_bert_base_chinese_cluener.ckpt']), ('txtcls_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased.ckpt' ]), ('txtcls_bert_base_uncased_mnli', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/txtcls/' 'txtcls_bert_base_uncased_mnli.ckpt' ]), ('mindspore/txtcls_bert_base_uncased_mnli', ['https://xihe.mindspore.cn/' 'api/v1/repo/model/MindSpore/txtcls_bert_base_uncased_mnli/file/txtcls_bert_base_uncased_mnli.ckpt' ]), ('qa_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased.ckpt' ]), ('qa_bert_base_uncased_squad', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/qa/' 'qa_bert_base_uncased_squad.ckpt' ]), ('gpt2', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2.ckpt' ]), ('gpt2_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2_lora.ckpt' ]), ('gpt2_xl', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2_xl.ckpt' ]), ('gpt2_xl_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2_xl_lora.ckpt' ]), ('gpt2_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2_13b.ckpt' ]), ('pangualpha_2_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/pangualpha/pangualpha_2_6b.ckpt' ]), ('pangualpha_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/pangualpha/pangualpha_13b.ckpt' ]), ('glm_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/glm_6b.ckpt' ]), ('glm_6b_chat', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/glm_6b.ckpt' ]), ('glm_6b_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/glm_6b_lora.ckpt' ]), ('glm_6b_lora_chat', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/glm_6b_lora.ckpt' ]), ('glm2_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm2/glm2_6b.ckpt' ]), ('glm2_6b_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm2/glm2_6b_lora.ckpt' ]), ('llama_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/open_llama_7b.ckpt' ]), ('llama_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/open_llama_13b.ckpt' ]), ('llama2_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/llama2/llama2_7b.ckpt' ]), ('llama2_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/llama2/llama2-13b-fp16.ckpt' ]), ('llama_7b_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/open_llama_7b_lora.ckpt' ]), ('bloom_560m', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/bloom_560m.ckpt' ]), ('bloom_7.1b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/bloom_7.1b.ckpt' ]), ('sam_vit_b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/sam/sam_vit_b_01ec64.ckpt' ]), ('sam_vit_l', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/sam/sam_vit_l_0b3195.ckpt' ]), ('sam_vit_h', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/sam/sam_vit_h_4b8939.ckpt' ]) ]) _MODEL_CONFIG_URL_LIST = OrderedDict([ ('clip_vit_b_32', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/clip/clip_vit_b_32.yaml' ]), ('clip_vit_b_16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_b_16.yaml' ]), ('clip_vit_l_14', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_l_14.yaml' ]), ('clip_vit_l_14@336', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_l_14%40336.yaml' ]), ('blip2_stage1_vit_g', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_stage1_vit_g.yaml' ]), ('blip2_stage1_classification', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_stage1_classification.yaml' ]), ('mae_vit_base_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/mae/mae_vit_base_p16.yaml' ]), ('vit_base_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/vit/vit_base_p16.yaml' ]), ('vit_g_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/vit_g_p16.yaml' ]), ('swin_base_p4w7', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/swin/swin_base_p4w7.yaml' ]), ('t5_small', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/t5/mindspore_model.yaml' ]), ('bert_tiny_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/bert_tiny/bert_tiny_uncased.yaml' ]), ('bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/bert_base/bert_base_uncased.yaml' ]), ('tokcls_bert_base_chinese', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese.yaml' ]), ('tokcls_bert_base_chinese_cluener', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/token_classification/' 'tokcls_bert_base_chinese_cluener.yaml' ]), ('txtcls_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased.yaml' ]), ('txtcls_bert_base_uncased_mnli', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/txtcls/' 'txtcls_bert_base_uncased_mnli.yaml' ]), ('qa_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased.yaml' ]), ('qa_bert_base_uncased_squad', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/qa/' 'qa_bert_base_uncased_squad.yaml' ]), ('gpt2', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2.yaml' ]) ]) _PIPELINE_INPUT_DATA_LIST = OrderedDict([ ('zero_shot_image_classification', "https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") ]) _TOKENIZER_SUPPORT_URL_LIST = OrderedDict([ ('clip_vit_b_32', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" ]), ('mindspore/clip_vit_b_32', [ "https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/clip_vit_b_32/file/bpe_simple_vocab_16e6.txt.gz" ]), ('clip_vit_b_16', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" ]), ('clip_vit_l_14', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" ]), ('clip_vit_l_14@336', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" ]), ('t5_small', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/t5/spiece.model" ]), ('bert_base_uncased', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/vocab.txt" ]), ('bert_tiny_uncased', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/vocab.txt" ]), ('tokcls_bert_base_chinese', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/bert_base_chinese/vocab.txt" ]), ('tokcls_bert_base_chinese_cluener', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/bert_base_chinese/vocab.txt" ]), ('mindspore/tokcls_bert_base_chinese_cluener', [ "https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/tokcls_bert_base_chineses_cluener/file/vocab.txt" ]), ('txtcls_bert_base_uncased', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/bert_base_english/vocab.txt" ]), ('txtcls_bert_base_uncased_mnli', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/bert_base_english/vocab.txt" ]), ('mindspore/txtcls_bert_base_uncased_mnli', [ "https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/txtcls_bert_base_uncased_mnli/file/vocab.txt" ]), ('qa_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/' 'XFormer_for_mindspore/bert/bert_base_english/vocab.txt' ]), ('qa_bert_base_uncased_squad', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/' 'XFormer_for_mindspore/bert/bert_base_english/vocab.txt' ]), ('gpt2', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('gpt2_xl', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('gpt2_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('gpt2_xl_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('gpt2_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('pangualpha_2_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/pangualpha/vocab.model' ]), ('pangualpha_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/pangualpha/vocab.model' ]), ('glm_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/ice_text.model' ]), ('glm2_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm2/tokenizer.model' ]), ('llama_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/tokenizer.model' ]), ('llama_7b_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/tokenizer.model' ]), ('llama2_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/llama2/tokenizer.model' ]), ('bloom_560m', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/tokenizer.json' ]), ('bloom_7.1b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/tokenizer.json' ]), ('bloom_65b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/tokenizer.json' ]), ('bloom_176b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/tokenizer.json' ]), ]) _TOKENIZER_NAME_TO_PROCESSOR = OrderedDict([ ('ChatGLMTokenizer', 'GLMProcessor'), ('ChatGLM2Tokenizer', 'GLMProcessor'), ('CLIPTokenizer', 'CLIPProcessor'), ('BertTokenizer', 'BertProcessor'), ('T5Tokenizer', 'T5Processor'), ('LlamaTokenizer', 'LlamaProcessor'), ('GPT2Tokenizer', 'GPT2Processor'), ('PanguAlphaTokenizer', 'PanguAlphaProcessor'), ('BloomTokenizer', 'BloomProcessor'), ('InternLMTokenizer', 'LlamaProcessor'), ('Baichuan2Tokenizer', 'LlamaProcessor') ]) @classmethod def show_trainer_support_model_list(cls, task=None): """show_trainer_support_model_list""" all_list = copy.deepcopy(cls._TRAINER_SUPPORT_TASKS_LIST) all_list.pop("general") for key, val in all_list.items(): val.pop("common") temp_list = [] for model_name, _ in val.items(): temp_list.append(model_name) all_list[key] = temp_list if task: if task in all_list.keys(): all_list = all_list[task] logger.info("Trainer support model list for %s task is: ", str(task)) print_path_or_list(all_list) else: raise KeyError("unsupported task") else: logger.info("Trainer support model list of MindFormer is: ") print_dict(all_list) @classmethod def show_pipeline_support_model_list(cls, task=None): """show_pipeline_support_model_list""" all_list = copy.deepcopy(cls._PIPELINE_SUPPORT_TASK_LIST) for key, val in all_list.items(): val.pop("common") temp_list = [] for model_name, _ in val.items(): temp_list.append(model_name) all_list[key] = temp_list if task: if task in all_list.keys(): all_list = all_list[task] logger.info("Pipeline support model list for %s task is: ", str(task)) print_path_or_list(all_list) else: raise KeyError("unsupported task") else: logger.info("Pipeline support model list of MindFormer is: ") print_dict(all_list) @classmethod def show_tokenizer_name_to_processor(cls): """show_tokenizer_name_to_processor function""" logger.info("_TRAINER_SUPPORT_TASKS_LIST of MindFormer is: ") print_dict(cls._TOKENIZER_NAME_TO_PROCESSOR) @classmethod def get_tokenizer_name_to_processor(cls): """get_tokenizer_name_to_processor function""" return cls._TOKENIZER_NAME_TO_PROCESSOR @classmethod def show_trainer_support_task_list(cls): """show_trainer_support_task_list function""" logger.info("_TRAINER_SUPPORT_TASKS_LIST of MindFormer is: ") print_dict(cls._TRAINER_SUPPORT_TASKS_LIST) @classmethod def show_pipeline_support_input_data_list(cls): """show_pipeline_support_input_data_list function""" logger.info("_PIPELINE_INPUT_DATA_LIST of MindFormer is: ") print_dict(cls._PIPELINE_INPUT_DATA_LIST) @classmethod def get_pipeline_support_input_data_list(cls): """get_pipeline_support_input_data_list function""" return cls._PIPELINE_INPUT_DATA_LIST @classmethod def get_trainer_support_task_list(cls): """get_trainer_support_task_list function""" return cls._TRAINER_SUPPORT_TASKS_LIST @classmethod def show_pipeline_support_task_list(cls): """show_pipeline_support_task_list function""" logger.info("_PIPELINE_SUPPORT_TASK_LIST of MindFormer is: ") print_dict(cls._PIPELINE_SUPPORT_TASK_LIST) @classmethod def get_pipeline_support_task_list(cls): """get_pipeline_support_task_list function""" return cls._PIPELINE_SUPPORT_TASK_LIST @classmethod def show_model_config_to_name(cls): """show_model_config_to_name function""" print_dict(cls._MODEL_CONFIG_TO_NAME) @classmethod def get_model_config_to_name(cls): """show_model_config_to_name function""" return cls._MODEL_CONFIG_TO_NAME @classmethod def set_model_config_to_name(cls, model_config, model_name): """ set_model_config_to_name function Args: model_config (str): the name of model config model_name (str): the name of model """ cls._MODEL_CONFIG_TO_NAME.update({ model_config: model_name }) @classmethod def show_model_config_url_list(cls): """show_model_config_url_list function""" logger.info("_MODEL_CONFIG_URL_LIST of MindFormer is: ") print_dict(cls._MODEL_CONFIG_URL_LIST) @classmethod def get_model_config_url_list(cls): """get_model_config_url_list function""" return cls._MODEL_CONFIG_URL_LIST @classmethod def show_project_path(cls): """show_project_path function""" logger.info("_PROJECT_PATH of MindFormer is: ") print_path_or_list(cls._PROJECT_PATH) @classmethod def get_project_path(cls): """get_project_path function""" return cls._PROJECT_PATH @classmethod def get_xihe_checkpoint_download_folder(cls): """get xihe's mindspore checkpoint download folder.""" return os.path.join(cls._DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER, 'mindspore') @classmethod def show_default_checkpoint_download_folder(cls): """show_default_checkpoint_download_folder function""" logger.info("_DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER of MindFormer is: ") print_path_or_list(cls._DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER) @classmethod def get_default_checkpoint_download_folder(cls): """get_default_checkpoint_download_folder function""" return cls._DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER @classmethod def set_default_checkpoint_download_folder(cls, download_folder): """ set_default_checkpoint_download_folder function Args: download_folder (str): the path of default checkpoint download folder """ if not os.path.isdir(download_folder): raise TypeError(f"{download_folder} should be a directory.") cls._DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER = download_folder @classmethod def get_default_checkpoint_save_folder(cls): """get_default_checkpoint_save_folder function""" return cls._DEFAULT_CHECKPOINT_SAVE_FOLDER @classmethod def set_default_checkpoint_save_folder(cls, save_folder): """ set_default_checkpoint_save_folder function Args: save_folder (str): the path of default checkpoint save folder """ if not os.path.isdir(save_folder): raise TypeError(f"{save_folder} should be a directory.") cls._DEFAULT_CHECKPOINT_SAVE_FOLDER = save_folder @classmethod def show_default_checkpoint_save_folder(cls): """show_default_checkpoint_save_folder function""" logger.info("_DEFAULT_CHECKPOINT_SAVE_FOLDER of MindFormer is: ") print_path_or_list(cls._DEFAULT_CHECKPOINT_SAVE_FOLDER) @classmethod def show_config_support_list(cls): """show_config_support_list function""" logger.info("CONFIG_SUPPORT_LIST of MindFormer is: ") print_dict(cls._CONFIG_SUPPORT_LIST) @classmethod def get_config_support_list(cls): """get_config_support_list function""" return cls._CONFIG_SUPPORT_LIST @classmethod def show_model_support_list(cls): """show_model_support_list function""" logger.info("MODEL_SUPPORT_LIST of MindFormer is: ") print_dict(cls._MODEL_SUPPORT_LIST) @classmethod def get_model_support_list(cls): """get_model_support_list function""" return cls._MODEL_SUPPORT_LIST @classmethod def show_processor_support_list(cls): """show_processor_support_list function""" logger.info("PROCESSOR_SUPPORT_LIST of MindFormer is: ") print_dict(cls._PROCESSOR_SUPPORT_LIST) @classmethod def get_processor_support_list(cls): """get_processor_support_list function""" return cls._PROCESSOR_SUPPORT_LIST @classmethod def get_tokenizer_support_list(cls): """get_tokenizer_support_list function""" return cls._TOKENIZER_SUPPORT_LIST @classmethod def show_tokenizer_support_list(cls): """show_tokenizer_support_list function""" print_dict(cls._TOKENIZER_SUPPORT_LIST) @classmethod def get_tokenizer_url_support_list(cls): """get_tokenizer_url_support_list function""" return cls._TOKENIZER_SUPPORT_URL_LIST @classmethod def show_tokenizer_url_support_list(cls): """show_tokenizer_url_support_list function""" print_dict(cls._TOKENIZER_SUPPORT_URL_LIST) @classmethod def get_model_name_support_list(cls): """get_model_name_support_list""" support_model_name = [] for task_name in cls._TRAINER_SUPPORT_TASKS_LIST.keys(): support_model_name.extend(cls._TRAINER_SUPPORT_TASKS_LIST.get(task_name).keys()) return set(support_model_name) @classmethod def show_model_ckpt_url_list(cls): """show_model_ckpt_url_list function""" logger.info("MODEL_CKPT_URL_LIST of MindFormer is: ") print_dict(cls._MODEL_CKPT_URL_LIST) @classmethod def get_model_ckpt_url_list(cls): """get_model_ckpt_url_list function""" return cls._MODEL_CKPT_URL_LIST pengcheng.mind-test/mindformers/models/000077500000000000000000000000001471400134000205215ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/.ipynb_checkpoints/000077500000000000000000000000001471400134000243125ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000036231471400134000305540ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """models init""" from .bert import * #from .mae import * #from .vit import * #from .swin import * #from .blip2 import * #from .clip import * #from .t5 import * #from .gpt2 import * #from .glm import * #from .glm2 import * from .llama import * #from .pangualpha import * #from .filip import * #from .bloom import * #from .sam import * from .base_tokenizer import * from .base_config import BaseConfig from .base_model import BaseModel from .base_processor import BaseProcessor, BaseImageProcessor, BaseAudioProcessor from .build_tokenizer import build_tokenizer from .build_processor import build_processor from .build_model import build_model_config, build_head, \ build_model, build_encoder __all__ = ['BaseConfig', 'BaseModel', 'BaseProcessor', 'BaseImageProcessor', 'BaseAudioProcessor'] #__all__.extend(blip2.__all__) #__all__.extend(mae.__all__) #__all__.extend(vit.__all__) #__all__.extend(swin.__all__) #__all__.extend(clip.__all__) #__all__.extend(t5.__all__) #__all__.extend(gpt2.__all__) #__all__.extend(glm.__all__) #__all__.extend(glm2.__all__) #__all__.extend(pangualpha.__all__) #__all__.extend(filip.__all__) #__all__.extend(bloom.__all__) __all__.extend(bert.__all__) __all__.extend(llama.__all__) __all__.extend(base_tokenizer.__all__) pengcheng.mind-test/mindformers/models/.ipynb_checkpoints/base_config-checkpoint.py000066400000000000000000000243531471400134000312570ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BaseConfig class, which is all model configs' base class """ import os import shutil import yaml from ..mindformer_book import MindFormerBook from ..mindformer_book import print_path_or_list from ..tools import logger from ..tools.register.config import MindFormerConfig from ..models.build_config import build_model_config class BaseConfig(dict): """ Base Config for all models' config Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_config import BaseConfig >>> class MyConfig(BaseConfig): ... _support_list = MindFormerBook.get_config_support_list()['my_model'] ... ... def __init__(self, ... data_size: int = 32, ... net_size: list = [1, 2, 3], ... loss_type: str = "MyLoss", ... checkpoint_name_or_path: str = '', ... **kwargs): ... self.data_size = data_size ... self.net_size = net_size ... loss_type = loss_type ... self.checkpoint_name_or_path = checkpoint_name_or_path ... super(MyConfig, self).__init__(**kwargs) ... >>> mynet = MyModel(MyConfig) >>> output = mynet(input) """ _support_list = [] _model_type = 0 _model_name = 1 def __init__(self, **kwargs): super(BaseConfig, self).__init__() self.update(kwargs) def __getattr__(self, key): if key not in self: return None return self[key] def __setattr__(self, key, value): self[key] = value def __delattr__(self, key): del self[key] def to_dict(self): """ for yaml dump, transform from Config to a strict dict class """ return_dict = {} for key, val in self.items(): if isinstance(val, BaseConfig): val = val.to_dict() return_dict[key] = val return return_dict @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a config by yaml name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}.") if os.path.exists(yaml_name_or_path): if not yaml_name_or_path.endswith(".yaml"): raise ValueError(f"{yaml_name_or_path} should be a .yaml file for model" " config.") config_args = MindFormerConfig(yaml_name_or_path) logger.info("the content in %s is used for" " config building.", yaml_name_or_path) elif yaml_name_or_path not in cls._support_list: raise ValueError(f"{yaml_name_or_path} is not a supported" f" model type or a valid path to model config." f" supported model could be selected from {cls._support_list}.") else: yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) config = build_model_config(config_args.model.model_config) MindFormerBook.set_model_config_to_name(id(config), config_args.model.arch.type) return config def save_pretrained(self, save_directory=None, save_name="mindspore_model"): """ Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. """ if save_directory is None: save_directory = MindFormerBook.get_default_checkpoint_save_folder() if not isinstance(save_directory, str) or not isinstance(save_name, str): raise TypeError(f"save_directory and save_name should be a str," f" but got {type(save_directory)} and {type(save_name)}.") if not os.path.exists(save_directory): os.makedirs(save_directory, exist_ok=True) save_path = os.path.join(save_directory, save_name + ".yaml") parsed_config, removed_list = self._inverse_parse_config() wraped_config = self._wrap_config(parsed_config) for key, val in removed_list: self[key] = val self.remove_type() meraged_dict = {} if os.path.exists(save_path): with open(save_path, 'r') as file_reader: meraged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) file_reader.close() meraged_dict.update(wraped_config) with open(save_path, 'w') as file_pointer: file_pointer.write(yaml.dump(meraged_dict)) file_pointer.close() logger.info("config saved successfully!") def remove_type(self): """remove type caused by save’""" if isinstance(self, BaseConfig): self.pop("type") for key, val in self.items(): if isinstance(val, BaseConfig): val.pop("type") self.update({key: val}) def inverse_parse_config(self): """inverse_parse_config""" val, _ = self._inverse_parse_config() return val def _inverse_parse_config(self): """ Inverse parse config method, which builds yaml file content for model config. Returns: A model config, which follows the yaml content. """ self.update({"type": self.__class__.__name__}) removed_list = [] for key, val in self.items(): if isinstance(val, BaseConfig): val = val.inverse_parse_config() elif not isinstance(val, (str, int, float, bool)): removed_list.append((key, val)) continue self.update({key: val}) for key, _ in removed_list: self.pop(key) return self, removed_list def _wrap_config(self, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. """ model_name = self.pop("model_name", None) if model_name is None: model_name = MindFormerBook.get_model_config_to_name().get(id(config), None) return {"model": {"model_config": config.to_dict(), "arch": {"type": model_name}}} @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list pengcheng.mind-test/mindformers/models/__init__.py000066400000000000000000000036231471400134000226360ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """models init""" from .bert import * #from .mae import * #from .vit import * #from .swin import * #from .blip2 import * #from .clip import * #from .t5 import * #from .gpt2 import * #from .glm import * #from .glm2 import * from .llama import * #from .pangualpha import * #from .filip import * #from .bloom import * #from .sam import * from .base_tokenizer import * from .base_config import BaseConfig from .base_model import BaseModel from .base_processor import BaseProcessor, BaseImageProcessor, BaseAudioProcessor from .build_tokenizer import build_tokenizer from .build_processor import build_processor from .build_model import build_model_config, build_head, \ build_model, build_encoder __all__ = ['BaseConfig', 'BaseModel', 'BaseProcessor', 'BaseImageProcessor', 'BaseAudioProcessor'] #__all__.extend(blip2.__all__) #__all__.extend(mae.__all__) #__all__.extend(vit.__all__) #__all__.extend(swin.__all__) #__all__.extend(clip.__all__) #__all__.extend(t5.__all__) #__all__.extend(gpt2.__all__) #__all__.extend(glm.__all__) #__all__.extend(glm2.__all__) #__all__.extend(pangualpha.__all__) #__all__.extend(filip.__all__) #__all__.extend(bloom.__all__) __all__.extend(bert.__all__) __all__.extend(llama.__all__) __all__.extend(base_tokenizer.__all__) pengcheng.mind-test/mindformers/models/__pycache__/000077500000000000000000000000001471400134000227315ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/__pycache__/__init__.cpython-39.pyc000066400000000000000000000013351471400134000271230ustar00rootroot00000000000000a ?cf@sdZddlTddlTddlTddlmZddlmZddlm Z m Z m Z ddl m Z ddl m Z ddlmZmZmZmZgd Zeejeejeejd S) z models init)*) BaseConfig) BaseModel) BaseProcessorBaseImageProcessorBaseAudioProcessor)build_tokenizer)build_processor)build_model_config build_head build_model build_encoder)rrrrrN)__doc__ZbertllamaZbase_tokenizer base_configr base_modelrZbase_processorrrrrr r r r r __all__extendrrF/home/ma-user/work/mindformers-baichuan/mindformers/models/__init__.pys       pengcheng.mind-test/mindformers/models/__pycache__/base_config.cpython-39.pyc000066400000000000000000000203131471400134000276200ustar00rootroot00000000000000a ?cf(@sldZddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z Gd d d e ZdS) z: BaseConfig class, which is all model configs' base class N)MindFormerBook)print_path_or_list)logger)MindFormerConfig)build_model_configcseZdZdZgZdZdZfddZddZdd Z d d Z d d Z e ddZ d ddZddZddZddZddZe ddZe ddZZS)! BaseConfiga Base Config for all models' config Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_config import BaseConfig >>> class MyConfig(BaseConfig): ... _support_list = MindFormerBook.get_config_support_list()['my_model'] ... ... def __init__(self, ... data_size: int = 32, ... net_size: list = [1, 2, 3], ... loss_type: str = "MyLoss", ... checkpoint_name_or_path: str = '', ... **kwargs): ... self.data_size = data_size ... self.net_size = net_size ... loss_type = loss_type ... self.checkpoint_name_or_path = checkpoint_name_or_path ... super(MyConfig, self).__init__(**kwargs) ... >>> mynet = MyModel(MyConfig) >>> output = mynet(input) rc stt|||dSN)superr__init__update)selfkwargs __class__I/home/ma-user/work/mindformers-baichuan/mindformers/models/base_config.pyr =szBaseConfig.__init__cCs||vr dS||Sr rrkeyrrr __getattr__AszBaseConfig.__getattr__cCs |||<dSr r)rrvaluerrr __setattr__FszBaseConfig.__setattr__cCs ||=dSr rrrrr __delattr__IszBaseConfig.__delattr__cCs4i}|D]"\}}t|tr&|}|||<q |S)zU for yaml dump, transform from Config to a strict dict class )items isinstancerto_dict)r return_dictrvalrrrrLs   zBaseConfig.to_dictc Ks|dd}|dur|}t|ts6tdt|dtj|rr|dsZt |dt |}t d|n||j vrt |d|j dn|}|d r|d |j}tjt|d |j}ntjt|d |j}tj|s tj|d d tj||d}dd}tj|s||} tj| rrtj| rrt| |t d|ntd| t |}t|jj} tt| |jj j| S)a From pretrain method, which instantiates a config by yaml name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. pretrained_model_name_or_pathNz+yaml_name_or_path should be a str, but got ..yamlz) should be a .yaml file for model config.z.the content in %s is used for config building.zg is not a supported model type or a valid path to model config. supported model could be selected from mindspore/_Texist_okcSs0d}tD]}||vr||}q,q|S)N)rget_trainer_support_task_listvaluesget) model_namedefault_yaml_file model_dictrrrget_default_yaml_files  z9BaseConfig.from_pretrained..get_default_yaml_filez"default yaml config in %s is used.z0default yaml file path must be correct, but get )!poprstr TypeErrortypeospathexistsendswith ValueErrorrrinfo _support_list startswithsplit _model_namejoinr#get_xihe_checkpoint_download_folder _model_type&get_default_checkpoint_download_foldermakedirsrealpathshutilcopyFileNotFoundErrorrmodel model_configset_model_config_to_nameidarch) clsZyaml_name_or_pathrr config_argsZ yaml_nameZcheckpoint_path yaml_filer.r,configrrrfrom_pretrainedXsR            zBaseConfig.from_pretrainedNmindspore_modelc CsT|durt}t|tr$t|tsBtdt|dt|dtj|s\tj |ddtj ||d}| \}}| |}|D]\}}|||<q| i} tj|rt|d$} tj| tjd } Wdn1s0Y| | |t|d  } | t| Wdn1s40Y| td dS) z Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. Nz6save_directory and save_name should be a str, but got z and r Tr%r!r)Loaderwzconfig saved successfully!)r"get_default_checkpoint_save_folderrr0r1r2r3r4r5rAr=_inverse_parse_config _wrap_config remove_typeopenyamlloadreadrRcloser writedumprr8) rsave_directory save_name save_pathZ parsed_config removed_listZ wraped_configrrZ meraged_dict file_reader file_pointerrrrsave_pretraineds4         2  0zBaseConfig.save_pretrainedcCsLt|tr|d|D]*\}}t|tr|d|||iqdS)uremove type caused by save’r2N)rrr/rr )rrrrrrrWs     zBaseConfig.remove_typecCs|\}}|S)inverse_parse_config)rU)rrr$rrrrfs zBaseConfig.inverse_parse_configcCs|d|jjig}|D]L\}}t|tr:|}n"t|ttt t fs\| ||fq|||iq|D]\}}| |qp||fS)z Inverse parse config method, which builds yaml file content for model config. Returns: A model config, which follows the yaml content. r2) r r__name__rrrrfr0intfloatboolappendr/)rrbrrr$rrrrUs    z BaseConfig._inverse_parse_configcCs>|dd}|dur(tt|d}d|d|idiS)a Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. r+NrFr2)rGrJ)r/rget_model_config_to_namer*rIr)rrNr+rrrrVs zBaseConfig._wrap_configcCstd|jt|jdS)zshow support list methodzsupport list of %s is:N)rr8rgrr9rKrrrshow_support_listszBaseConfig.show_support_listcCs|jS)zget support list method)r9rmrrrget_support_list szBaseConfig.get_support_list)NrP)rg __module__ __qualname____doc__r9r?r<r rrrr classmethodrOrerWrfrUrVrnro __classcell__rrrrr s(   N '  r)rrr3rCrYmindformer_bookrrtoolsrZtools.register.configrZmodels.build_configrdictrrrrrs     pengcheng.mind-test/mindformers/models/__pycache__/base_model.cpython-39.pyc000066400000000000000000000263001471400134000274550ustar00rootroot00000000000000a ?cfA@sdZddlmZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z ddlmZmZd d lmZd d lmZdd lmZdd lmZddlmZddlmZGdddeje ZdS)z BaseModel )OptionalN)nn)load_checkpointload_param_into_net)GeneratorMixin)MindFormerBookprint_path_or_list)build_model_config) BaseConfig)MindFormerConfig)download_with_progress_bar)logger) try_sync_filecseZdZdZgZdZdZedfdd ZddZ de e e d d d Z ddZ ddZddZee dddZeddZeddZZS) BaseModela[ The base model that contains the class method `from_pretained` and `save_pretrained`, any new model that should inherit the class. Note: GeneratorMixin provides the method `generate` that enable the generation for nlp models. Args: config(BaseConfig): The model configuration that inherits the `BaseConfig`. rr )configc s&tt|jfi|||_d|_dS)N)superr__init__r default_checkpoint_download_path)selfrkwargs __class__H/home/ma-user/work/mindformers-baichuan/mindformers/models/base_model.pyr5szBaseModel.__init__c Cs|j}|rt|ts(tdt|tj|r~t|}|}zt ||t d|Wnt yxt d|Yn0q||jvrt|d|jdq|}|dr|d|j}tjt|d|j}ntjt|d|j}tj|stj|d d tj||d }tj|sjt||j}t||}|sjt d d St|||_t dtj|t|}zt ||t d|Wn t yt d|Yn0n t dd S)a1 load checkpoint for models. Args: config (ModelConfig): a model config instance, which could have attribute "checkpoint_name_or_path (str)". set checkpoint_name_or_path to a supported model name or a path to checkpoint, to load model weights. z1checkpoint_name_or_path should be a str, but got zweights in %s are loadedzJthe given config and weights in %s are mismatched, and weights load failedzT is not a supported default model or a valid path to checkpoint, please select from . mindspore/_Texist_ok.ckptz@checkpoint download failed, and pretrained weights are unloaded.Nzstart to read the ckpt file: %szmodel built, but weights is unloaded, since the config has no checkpoint_name_or_path attribute or checkpoint_name_or_path is None.)checkpoint_name_or_path isinstancestr TypeErrortypeospathexistsrrrinfo RuntimeErrorerror _support_list ValueError startswithsplit _model_namejoinr#get_xihe_checkpoint_download_folder _model_type&get_default_checkpoint_download_foldermakedirsget_model_ckpt_url_listrrrgetsize) rrr#param ckpt_fileZcheckpoint_nameZ"default_checkpoint_download_folderurlsucceedrrrr:sh           zBaseModel.load_checkpointNmindspore_model)save_directory save_namec Cs|durt}t|tr$t|tsBtdt|dt|dtj|s\tj |ddtj ||d}tj ||d}t |||j durtd ||j \}}||}|D]\}} | |j |<q||j i} tj|r6t|d $} tj| tjd } Wdn1s$0Y| | |t|d  } | t| Wdn1sr0Y| td dS)a  Save the model weight and configuration file. (only supports standalone mode, and distribute mode waits for developing) Args: save_directory(str): a directory to save the model weight and configuration. If None, the directory will be `./checkpoint_save`, which can be obtained by the `MindFormerBook.get_default_checkpoint_save_folder()`. If set, the directory will be what is set. save_name(str): the name of saved files, including model weight and configuration file. Default mindspore_model. Examples: >>> import os >>> from mindformers import T5ForConditionalGeneration, MindFormerBook >>> net = T5ForConditionalGeneration.from_pretrained('t5_small') >>> net.save_pretrained() >>> output_path = MindFormerBook.get_default_checkpoint_save_folder() >>> print(os.listdir(output_path)) ['mindspore_model.yaml', 'mindspore_model.ckpt'] Nz6save_directory and save_name should be a str, but got z and rTr r".yamlz"the model has no config attribute.r)Loaderwzmodel saved successfully!)r"get_default_checkpoint_save_folderr$r%r&r'r(r)r*r7r3mssave_checkpointrAttributeError_inverse_parse_config _wrap_config remove_typeopenyamlloadreadrCcloseupdatewritedumprr+) rr?r@checkpoint_pathZ config_path parsed_configZ remove_list wraped_configkeyval meraged_dict file_reader file_pointerrrrsave_pretraineds<         4  0zBaseModel.save_pretrainedcCsLt|tr|d|D]*\}}t|tr|d|||iqdS)uremove type caused by save’r'N)r$r popitemsrQ)rrrWrXrrrrKs     zBaseModel.remove_typecCsg}t|ts||fS|jj}|d|i|D]H\}}||\}}t|ttt t tfsn| ||fq4|||iq4|D]\}}| |q||fS)a Inverse parse config method, which builds yaml file content for model config. Args: config (BaseConfig): a model config inherited from BaseConfig. Returns: A model config, which follows the yaml content. r') r$r r__name__rQr^rIr%intfloatboolappendr])rr removed_list class_namerWrXnew_valrrrrrIs    zBaseModel._inverse_parse_configcCs|jj}d|d|idiS)a Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. modelr') model_configarch)rr_to_dict)rr model_namerrrrJs zBaseModel._wrap_config)pretrained_model_name_or_dircKs|dd}|dur|}t|ts4tdt|tj|}tj|}|sx||j vrxt |d|j d|j d|r|st |d|rLdd t |D}d d t |D}|r|st d |dtj|||j}tj|||j} td || t|} | jjjfi|t| jj} | d | i|| } n4|} |dr| d|j} tjt| d|j}ntjt| d|j}tj|stj|ddtj|| d}dd}tj|s:|| }tj|r,tj|r,t ||td|nt d|t!|t|} | jjjfi|t| jj} | d |i|| } td| S)a. Instantiates a model by the pretrained_model_name_or_dir. It download the model weights if the user pass a model name, or load the weight from the given directory if given the path. (only support standalone mode, and distribute mode waits for developing!) Args: pretrained_model_name_or_dir (str): It supports the following two input types. If `pretrained_model_name_or_dir` is a supported model name, for example, `vit_base_p16` and `t5_small`, it will download the necessary files from the cloud. User can pass one from the support list by call `MindFormerBook.get_model_support_list()`. If `pretrained_model_name_or_dir` is a path to the local directory where there should have model weights ended with `.ckpt` and configuration file ended with `yaml`. pretrained_model_name_or_path (Optional[str]): Equal to "pretrained_model_name_or_dir", if "pretrained_model_name_or_path" is set, "pretrained_model_name_or_dir" is useless. Examples: >>> from mindformers import T5ForConditionalGeneration >>> net = T5ForConditionalGeneration.from_pretrained('t5_small') Returns: A model, which inherited from BaseModel. pretrained_model_name_or_pathNz6pretrained_model_name_or_dir should be a str, but got z+ does not exist, or it is not supported by z. please select from rz is not a directory.cSsg|]}|dr|qS)rAendswith.0filerrr s z-BaseModel.from_pretrained..cSsg|]}|dr|qS)r"rnrprrrrs s zIthere is no yaml file for model config or ckpt file for model weights in z;config in %s and weights in %s are used for model building.r#rrrTr rAcSs0d}tD]}||vr||}q,q|S)N)rget_trainer_support_task_listvaluesget)rkdefault_yaml_file model_dictrrrget_default_yaml_fileDs  z8BaseModel.from_pretrained..get_default_yaml_filez"default yaml config in %s is used.z0default yaml file path must be correct, but get zmodel built successfully!)"r]r$r%r&r'r(r)r*isdirr.r/r_listdirFileNotFoundErrorr3r5rr+r rgrhrQr r0r1r2rr4r6r7realpathshutilcopyr)clsrlrrmis_existis_dir yaml_list ckpt_list yaml_filer; config_argsrrgZpretrained_model_namerTrzrxrrrfrom_pretrainedsz              zBaseModel.from_pretrainedcCstd|jt|jdS)zshow_support_list methodzsupport list of %s is:N)rr+r_r r.rrrrshow_support_list\szBaseModel.show_support_listcCs|jS)zget_support_list method)r.rrrrget_support_listbszBaseModel.get_support_list)Nr>)r_ __module__ __qualname____doc__r.r5r2r rrrr%r\rKrIrJ classmethodrrr __classcell__rrrrr&s* I <  i r)rtypingrr(rrMrrFrmindspore.train.serializationrr generationrmindformer_bookrr build_configr base_configr Ztools.registerr Ztools.download_toolsrZ tools.loggerrZ tools.utilsrCellrrrrrs          pengcheng.mind-test/mindformers/models/__pycache__/base_processor.cpython-39.pyc000066400000000000000000000252011471400134000303730ustar00rootroot00000000000000a ?cfO4@sdZddlZddlZddlZddlmZmZddlmZddlm Z ddl m Z dd l m Z Gd d d ZGd d d ZGdddZdS)z BaseProcessor N)print_path_or_listMindFormerBook)build_processor) BaseTokenizer)logger)MindFormerConfigc@s(eZdZdZddZddZddZdS) BaseImageProcessora BaseImageProcessor for all image preprocess. Examples: >>> from mindspore.dataset.vision.transforms import CenterCrop >>> from mindformers.models.base_processor import BaseImageProcessor >>> image_resolution = 224 >>> class MyImageProcessor(BaseImageProcessor): ... def __init__(self, image_resolution): ... super(MyImageProcessor, self).__init__(image_resolution=image_resolution) ... self.center_crop = CenterCrop(image_resolution) ... ... def preprocess(self, images, **kwargs): ... res = [] ... for image in images: ... image = self.center_crop(image) ... res.append(image) ... return res ... >>> my_image_processor = MyImageProcessor(image_resolution) >>> output = my_image_processor(image) cKsi|_|j|dSNconfigupdateselfkwargsrL/home/ma-user/work/mindformers-baichuan/mindformers/models/base_processor.py__init__6szBaseImageProcessor.__init__cKs|j|fi|Szforward process preprocess)rZ image_datarrrr__call__:szBaseImageProcessor.__call__cKs tddS)preprocess methodz=Each image processor must implement its own preprocess methodNNotImplementedError)rimagesrrrrr>szBaseImageProcessor.preprocessN__name__ __module__ __qualname____doc__rrrrrrrr sr c@s(eZdZdZddZddZddZdS) BaseAudioProcessora BaseAudioProcessor for all audio preprocess. Examples: >>> from mindspore.dataset.audio import AllpassBiquad >>> from mindformers.models.base_processor import BaseAudioProcessor >>> sample_rate = 44100 >>> central_freq = 200.0 >>> class MyAudioProcessor(BaseAudioProcessor): ... def __init__(self, audio_property): ... super(MyAudioProcessor, self).__init__(sample_rate=sample_rate, central_freq=central_freq ) ... self.all_pass_biquad = AllpassBiquad(44100, 200.0) ... ... def preprocess(self, audio_data, **kwargs): ... res = [] ... for audio in audio_data: ... audio = self.all_pass_biquad(audio) ... res.append(audio) ... return res ... >>> my_audio_processor = MyAudioProcessor(sample_rate, central_freq) >>> output = my_audio_processor(audio) cKsi|_|j|dSr r rrrrr[szBaseAudioProcessor.__init__cKs|j|fi|SrrrZ audio_datarrrrr_szBaseAudioProcessor.__call__cKs tddS)rz=Each audio processor must implement its own preprocess methodNrr#rrrrcszBaseAudioProcessor.preprocessNrrrrrr"Csr"c@sleZdZdZgZdZdZddZdddZdd d Z d d Z ddZ e ddZ e ddZe ddZdS) BaseProcessoraT Base processor Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_processor import BaseProcessor >>> class MyProcessor(BaseProcessor): ... _support_list = MindFormerBook.get_processor_support_list()['my_model'] ... ... def __init__(self, image_processor=None, audio_processor=None, tokenizer=None, return_tensors='ms'): ... super(MyProcessor, self).__init__( ... image_processor=image_processor, ... audio_processor=audio_processor, ... tokenizer=tokenizer, ... return_tensors=return_tensors) ... >>> myprocessor = MyProcessor(image_processor, audio_processor, tokenizer) >>> output = mynet(image, audio, text) rrcKsji|_|j||dd|_|dd|_|dd|_|dd|_|dd|_|dd|_dS)Nimage_processoraudio_processor tokenizer max_lengthpaddingFreturn_tensors) r rpopr%r&r'r(r)r*rrrrrs zBaseProcessor.__init__NcCsi}|durF|jrFt|jts4tdt|jd||}||d<|dur|jrt|jtsvtdt|jdt|tr|g}|j||j|j |j dd}||d<|S) z call functionNzFfeature_extractor should inherit from the BaseImageProcessor, but got .imagez;tokenizer should inherited from the BaseTokenizer, but got )r*r(r) input_idstext) r% isinstancer TypeErrortyper'rstrr*r(r))r image_input text_inputoutputZ image_output text_outputrrrrs.       zBaseProcessor.__call__mindspore_modelc Cs6|durt}t|tr$t|tsBtdt|dt|dtj|s\tj |dd| |j }| |}tj ||d}i}tj|rt|d$}tj|tjd }Wdn1s0Y|||t|d  }|t|Wdn1s0Y|td dS) z Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. Nz6save_directory and save_name should be a str, but got z and r,Texist_ok.yamlr)Loaderwzprocessor saved successfully!)r"get_default_checkpoint_save_folderr0r3r1r2ospathexistsmakedirs_inverse_parse_configr _wrap_configjoinopenyamlloadreadr=closerwritedumprinfo) rsave_directory save_name parsed_config wraped_config config_path meraged_dict file_reader file_pointerrrrsave_pretraineds.      2  0zBaseProcessor.save_pretrainedcCsd|jji}|D]~\}}t|trNd|jji}||j|||iqt|ttfrd|jji}||j |||iq|||iq|S)a Inverse parse config method, which builds yaml file content for feature extractor config. Args: Config (dict): a dict, which contains input parameters of feature extractor. Returns: A dict, which follows the yaml content. r2) __class__ritemsr0rr init_kwargsr r"r )rr rQkeyvalZparsed_sub_configrrrrDs      z#BaseProcessor._inverse_parse_configcCsd|iS)z Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (dict): a dict processed by _inverse_parse_config function. Returns: A dict for yaml.dump. processorr)rr rrrrEs zBaseProcessor._wrap_configc Ks|dd}|dur|}t|ts4tdt|tj|}|sl||jvrlt |d|j d|jd|rt d|t |}n|}|dr|d |j}tjt|d |j}ntjt|d |j}tj|stj|d d tj||d }dd} tj|sp| |} tj| rbtj| rbt| |t d|ntd| t |}t|j} t d| S)a From pretrain method, which instantiates a processor by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A processor which inherited from BaseProcessor. pretrained_model_name_or_pathNz+yaml_name_or_path should be a str, but got z+ does not exist, or it is not supported by z. please select from r,z,config in %s is used for processor building. mindspore/_Tr9r;cSs0d}tD]}||vr||}q,q|S)N)rget_trainer_support_task_listvaluesget) model_namedefault_yaml_file model_dictrrrget_default_yaml_file"s  z.get_default_yaml_filez"default yaml config in %s is used.z0default yaml file path must be correct, but get zprocessor built successfully!)r+r0r3r1r2r@rArB _support_list ValueErrorrrrNr startswithsplit _model_namerFr#get_xihe_checkpoint_download_folder _model_type&get_default_checkpoint_download_folderrCrealpathshutilcopyFileNotFoundErrorrr]) clsyaml_name_or_pathrr^is_exist config_args yaml_namecheckpoint_path yaml_filerirgr]rrrfrom_pretrainedsT            zBaseProcessor.from_pretrainedcCstd|jt|jdS)show_support_listzsupport list of %s is:N)rrNrrrjrvrrrr~8szBaseProcessor.show_support_listcCs|jS)zget_support_list method)rjrrrrget_support_list>szBaseProcessor.get_support_list)NN)Nr8)rrr r!rjrprnrrrWrDrE classmethodr}r~rrrrrr$hs  "  L r$)r!r@rsrHmindformer_bookrrrbase_tokenizerrtoolsrtools.registerr r r"r$rrrrs    $%pengcheng.mind-test/mindformers/models/__pycache__/base_tokenizer.cpython-39.pyc000066400000000000000000004426771471400134000304120ustar00rootroot00000000000000a ?cf8@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z m Z ddl mZddlmZmZmZmZmZmZmZmZddlmZmZddlmZmZddlmZddl Z ddl!Z"ddl#Z$dd l%m&Z&dd l'm(Z(m)Z)m*Z*d d l+m+Z+d dl,m-Z-d dl.m/Z/d dl0m1Z1gdZ2dZ3dZ4e5dZ6e5dZ7e8Z9ee8Z:ee5Z;ee8e8fZdZ?dZ@dZAdZBeCdZDGddde8eZEGdddeEZFe dddGdd d ZGe Gd!d"d"ZHGd#d$d$eEZIGd%d&d&eZJGd'd(d(eZKGd)d*d*eZLGd+d,d,eEZMGd-d.d.ZNd/ZOd0ZPd1ZQd2d3ZRd4d5ZSd6d7ZTd8d9ZUd:d;ZVdd?ZXee8e8d@dAdBZYdCdDZZGdEdFdFZ[eReQGdGdHdHe[Z\eReQe(]e)j^GdIdJdJe\Z_dS)Kz+Base Tokenizer for the pretrained tokenizerN) dataclassfield)Enum)AnyDictList NamedTupleOptionalSequenceTupleUnion) OrderedDictUserDict)MappingSized)contextmanager)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_tokenizer)download_with_progress_bar) try_sync_file)MindFormerBook) BaseTokenizer TokenizerSpecialTokensMixinzspecial_tokens_map.jsontokenizer_config.jsongꌠ9Y>)Fg@xDzadded_tokens.jsonztokenizer.jsonztokenizer\.(.*)\.jsonc@seZdZdZeddZdS) ExplicitEnumzC Enum with more explicit error message for missing values. cCs(t|d|jdt|jdS)Nz is not a valid z, please select one of ) ValueError__name__list_value2member_map_keys)clsvaluer(L/home/ma-user/work/mindformers-baichuan/mindformers/models/base_tokenizer.py _missing_MszExplicitEnum._missing_N)r" __module__ __qualname____doc__ classmethodr*r(r(r(r)r Hsr c@seZdZdZdZdZdS) TensorTypez Possible values for the `return_tensors` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. npmsN)r"r+r,r-NUMPY MINDSPOREr(r(r(r)r/Tsr/T)frozeneqc@sdeZdZUdZeedZeed<dZe ed<dZ e ed<dZ e ed<dZ e ed <d d Z d d ZdS) AddedTokenz AddedToken represents a token to be added to a Tokenizer An AddedToken can have special options defining the way it should behave. )default_factorycontentF single_wordlstriprstripT normalizedcCs|jSN)__dict__selfr(r(r) __getstate__jszAddedToken.__getstate__cCs|jSr=)r8r?r(r(r)__str__mszAddedToken.__str__N)r"r+r,r-rstrr8__annotations__r9boolr:r;r<rArBr(r(r(r)r6]s     r6c@seZdZdZdS) EncodingFastz_This is dummy class because without the `tokenizers` library we don't have these objects anywayN)r"r+r,r-r(r(r(r)rFqsrFc@s eZdZdZdZdZdZdZdS)TruncationStrategyz Possible values for the `truncation` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. Z only_firstZ only_second longest_firstdo_not_truncateN)r"r+r,r- ONLY_FIRST ONLY_SECOND LONGEST_FIRSTDO_NOT_TRUNCATEr(r(r(r)rGvs rGc@s"eZdZUdZeed<eed<dS)CharSpanz Character span in the original string. Args: start (`int`): Index of the first character in the original string. end (`int`): Index of the character following the last character in the original string. startendNr"r+r,r-intrDr(r(r(r)rNs rNc@s"eZdZUdZeed<eed<dS) TokenSpanz Token span in an encoded string (list of tokens). Args: start (`int`): Index of the first token in the span. end (`int`): Index of the token following the last token in the span. rOrPNrQr(r(r(r)rSs rScs(eZdZdZde eee dd"d#Zd?e eee dd$d%Zd@e eee dd&d'ZdAe ee e d(d)d*ZdBe ee e d(d+d,Z dCe ee e ee!d-d.d/Z"dDe ee e#d(d0d1Z$dEe ee e e d2d3d4Z%dFe ee e e#d-d5d6Z&dGe ee e e d2d7d8Z'dHeeee fe d9d:d;Z(Z)S)I BatchEncodinga& Holds the output of the [`~tokenization_utils_base.PreTrainedTokenizerBase.__call__`], [`~tokenization_utils_base.PreTrainedTokenizerBase.encode_plus`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.batch_encode_plus`] methods (tokens, attention_masks, etc). This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes utility methods to map from word/character space to token space. Args: data (`dict`): Dictionary of lists/arrays/tensors returned by the `__call__`/`encode_plus`/`batch_encode_plus` methods ('input_ids', 'attention_mask', etc.). encoding (`tokenizers.Encoding` or `Sequence[tokenizers.Encoding]`, *optional*): If the tokenizer is a fast tokenizer which outputs additional information like mapping from word/character space to token space the `tokenizers.Encoding` instance or list of instance (for batches) hold this information. tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in Mindspore/Numpy Tensors at initialization. prepend_batch_axis (`bool`, *optional*, defaults to `False`): Whether or not to add a batch axis when converting to tensors (see `tensor_type` above). n_sequences (`Optional[int]`, *optional*): You can give a tensor_type here to convert the lists of integers in Mindspore/Numpy Tensors at initialization. NF)dataencoding tensor_typeprepend_batch_axis n_sequencescsXt|t|tr|g}||_|dur@|dur@|r@|dj}||_|j||ddS)NrrWrX)super__init__ isinstancerF _encodingsrY _n_sequencesconvert_to_tensors)r@rUrVrWrXrY __class__r(r)r\s   zBatchEncoding.__init__returncCs|jS)z `Optional[int]`: The number of sequences used to generate each sample from the batch encoded in this [`BatchEncoding`]. Currently can be one of `None` (unknown), `1` (a single sentence) or `2` (a pair of sentences) )r_r?r(r(r)rYszBatchEncoding.n_sequencescCs |jduS)z `bool`: Indicate whether this [`BatchEncoding`] was generated from the result of a [`PreTrainedTokenizerFast`] or not. Nr^r?r(r(r)is_fastszBatchEncoding.is_fast)itemrdcsXttrjSjdur(jSttrLfddjDStddS)a If the key is a string, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.). If the key is an integer, get the `tokenizers.Encoding` for batch item with index `key`. If the key is a slice, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.) with the constraint of slice. Ncsi|]}|j|qSr(rU).0keyrgr@r(r) z-BatchEncoding.__getitem__..zInvalid key. Only three types of key are available: (1) string, (2) integers for backend Encoding, and (3) slices for data subsetting.)r]rCrUr^slicer%KeyErrorr@rgr(rkr) __getitem__s     zBatchEncoding.__getitem__)rgcCs(z |j|WSty"tYn0dSr=)rUroAttributeErrorrpr(r(r) __getattr__s  zBatchEncoding.__getattr__cCs|j|jdS)N)rU encodingsrUr^r?r(r(r)rAszBatchEncoding.__getstate__cCs(d|vr|d|_d|vr$|d|_dS)NrUrtru)r@stater(r(r) __setstate__s zBatchEncoding.__setstate__cCs |jSr=)rUr%r?r(r(r)r%szBatchEncoding.keyscCs |jSr=)rUvaluesr?r(r(r)rxszBatchEncoding.valuescCs |jSr=)rUitemsr?r(r(r)ry szBatchEncoding.itemscCs|jS)z `Optional[List[tokenizers.Encoding]]`: The list all encodings from the tokenization process. Returns `None` if the input was tokenized through Python (i.e., not a fast) tokenizer. rer?r(r(r)rtszBatchEncoding.encodingsr) batch_indexrdcCs|jstd|j|jS)a Return the list of tokens (sub-parts of the input strings after word/subword splitting and before conversion to integer indices) at a given batch index (only works for the output of a fast tokenizer). Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[str]`: The list of tokens at that index. zgtokens() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)r^r!tokensr@rzr(r(r)r{s zBatchEncoding.tokenscCs|jstd|j|jS)a Return a list mapping the tokens to the id of their original sentences: - `None` for special tokens added around or between sequences, - `0` for tokens corresponding to words in the first sequence, - `1` for tokens corresponding to words in the second sequence when a pair of sequences was jointly encoded. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the sequence id corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding sequence. zmsequence_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)r^r! sequence_idsr|r(r(r)r}*s zBatchEncoding.sequence_idscCs$|jstdtdt||S)> Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word). zfwords() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).z`BatchEncoding.words()` property is deprecated and should be replaced with the identical, but more self-explanatory `BatchEncoding.word_ids()` property.)r^r!warningswarn FutureWarningword_idsr|r(r(r)wordsBs zBatchEncoding.wordscCs|jstd|j|jS)r~ziword_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)r^r!rr|r(r(r)rZs zBatchEncoding.word_ids)batch_or_token_index token_indexrdcCsX|jstd|dur|}nd}|}|dkr6|j|}|dkrH|j|}|j||S)a Get the index of the sequence represented by the given token. In the general use case, this method returns `0` for a single sequence or the first sequence of a pair, and `1` for the second sequence of a pair Can be called as: - `self.token_to_sequence(token_index)` if batch size is 1 - `self.token_to_sequence(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. zGtoken_to_sequence() is not available when using Python based tokenizersNr)r^r! _batch_size_seq_lentoken_to_sequencer@rrrzr(r(r)rms  zBatchEncoding.token_to_sequencecCsX|jstd|dur|}nd}|}|dkr6|j|}|dkrH|j|}|j||S)a  Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch. Can be called as: - `self.token_to_word(token_index)` if batch size is 1 - `self.token_to_word(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. zCtoken_to_word() is not available when using Python based tokenizersNr)r^r!rr token_to_wordrr(r(r)rs  zBatchEncoding.token_to_word)batch_or_word_index word_indexsequence_indexrdcCsn|jstd|dur|}nd}|}|dkr6|j|}|dkrH|j|}|j|||}|durjt|SdS)a- Get the encoded token span corresponding to a word in a sequence of the batch. Token spans are returned as a [`~tokenization_utils_base.TokenSpan`] with: - **start** -- Index of the first token. - **end** -- Index of the token following the last token. Can be called as: - `self.word_to_tokens(word_index, sequence_index: int = 0)` if batch size is 1 - `self.word_to_tokens(batch_index, word_index, sequence_index: int = 0)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the word in the sequence. word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: ([`~tokenization_utils_base.TokenSpan`], *optional*): Span of tokens in the encoded sequence. Returns `None` if no tokens correspond to the word. This can happen especially when the token is a special token that has been used to format the tokenization. For example when we add a class token at the very beginning of the tokenization. zDword_to_tokens() is not available when using Python based tokenizersNr)r^r!rrword_to_tokensrS)r@rrrrzspanr(r(r)rs'  zBatchEncoding.word_to_tokenscCsH|jstd|dur|}nd}|}|j||}|durDt|SdS)a Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a [`~tokenization_utils_base.CharSpan`] with: - **start** -- Index of the first character in the original string associated to the token. - **end** -- Index of the character following the last character in the original string associated to the token. Can be called as: - `self.token_to_chars(token_index)` if batch size is 1 - `self.token_to_chars(batch_index, token_index)` if batch size is greater or equal to 1 Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token or tokens in the sequence. Returns: [`~tokenization_utils_base.CharSpan`]: Span of characters in the original string, or None, if the token (e.g. , ) doesn't correspond to any chars in the origin string. zDtoken_to_chars() is not available when using Python based tokenizersNr)r^r!token_to_charsrN)r@rrrzZ span_indicesr(r(r)rszBatchEncoding.token_to_chars)batch_or_char_index char_indexrrdcCs6|jstd|dur|}nd}|}|j|||S)a Get the index of the token in the encoded output comprising a character in the original string for a sequence of the batch. Can be called as: - `self.char_to_token(char_index)` if batch size is 1 - `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int`: Index of the token. zCchar_to_token() is not available when using Python based tokenizersNr)r^r! char_to_tokenr@rrrrzr(r(r)rs zBatchEncoding.char_to_tokencCs:|jstd|dur|}nd}|}t|j|||S)a/ Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned as a CharSpan NamedTuple with: - start: index of the first character in the original string - end: index of the character following the last character in the original string Can be called as: - `self.word_to_chars(word_index)` if batch size is 1 - `self.word_to_chars(batch_index, word_index)` if batch size is greater or equal to 1 Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: `CharSpan` or `List[CharSpan]`: Span(s) of the associated character or characters in the string. CharSpan are NamedTuple with: - start: index of the first character associated to the token in the original string - end: index of the character following the last character associated to the token in the original string zCword_to_chars() is not available when using Python based tokenizersNr)r^r!rN word_to_chars)r@rrrrzr(r(r)r?s$zBatchEncoding.word_to_charscCs6|jstd|dur|}nd}|}|j|||S)a@ Get the word in the original string corresponding to a character in the original string of a sequence of the batch. Can be called as: - `self.char_to_word(char_index)` if batch size is 1 - `self.char_to_word(batch_index, char_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the character in the original string. char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the character in the original string. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int` or `List[int]`: Index or indices of the associated encoded token(s). zBchar_to_word() is not available when using Python based tokenizersNr)r^r! char_to_wordrr(r(r)rlszBatchEncoding.char_to_wordrZc s|dur |St|tst|}|tjkrBtj}tjdd}|}n d fdd dd}tj}|}|D]v\}}z$|rx||s||d} | ||<Wqjty} z0|d krt d | t d |d | WYd} ~ qjd} ~ 00qj|S)a Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. prepend_batch_axis (`int`, *optional*, defaults to `False`): Whether or not to add the batch dimension during the conversion. NcSs t|tjSr=)r]r1Tensorxr(r(r) is_ms_tensorsz6BatchEncoding.convert_to_tensors..is_ms_tensorcsnt|ttfr`t|dtttjfr`dd|D}tt|dkr`|dur`dd|Dtd}tj||dS)NrcSsg|] }t|qSr(lenrivalr(r(r) rmzGBatchEncoding.convert_to_tensors..as_tensor..rcSsg|]}t|qSr()r0asarrayrr(r(r)rrmdtype) r]r#tupler0ndarrayrsetobjectr)r'rZ value_lens as_tensorr(r)rs $z3BatchEncoding.convert_to_tensors..as_tensorcSs t|tjSr=)r]r0rrr(r(r)is_numpy_arraysz8BatchEncoding.convert_to_tensors..is_numpy_arrayroverflowing_tokenszUnable to create tensor returning overflowing tokens of different lengths. Please see if a fast version of this tokenizer is available to have this feature available.zUnable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`zY` in this case) have excessive nesting (inputs type `list` where type `int` is expected).)N) r]r/r3r1int32rr0ry Exceptionr!) r@rWrX tensor_dtyper is_tensorrrjr'tensorer(rr)r`sD    z BatchEncoding.convert_to_tensors)NNNFN)r)r)r)r)N)N)Nr)N)Nr)Nr)Nr)NF)*r"r+r,r-r rrCrr rFr r/rErRr\propertyrYrfrqrsrArwr%rxryrrtr{r}rrrrrSrrNrrrrr` __classcell__r(r(rar)rTsj  ''  5(  *  -(rTc@seZdZdZdZdZdZdS)PaddingStrategyz Possible values for the `padding` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. longest max_lengthZ do_not_padN)r"r+r,r-LONGEST MAX_LENGTH DO_NOT_PADr(r(r(r)rsrc@s`eZdZdZddZedddZeeeddd Zdd d Z dd dZ dddZ ddZ d S)Triez Trie in Python. Creates a Trie out of a list of words. The trie is used to split on `added_tokens` in one pass Loose reference https://en.wikipedia.org/wiki/Trie cCs i|_dSr=rhr?r(r(r)r\sz Trie.__init__)wordcCsD|sdS|j}|D]$}||vr&||ni||<||}qd|d<dS)uy Passes over every char (utf-8 char) on word and recursively adds it to the internal `data` trie representation. The special key `""` is used to represent termination. This function is idempotent, adding twice the same word will leave the trie unchanged Example: ```python >>> trie = Trie() >>> trie.add("Hello 友達") >>> trie.data {"H": {"e": {"l": {"l": {"o": {" ": {"友": {"達": {"": 1}}}}}}}}} >>> trie.add("Hello") >>> trie.data {"H": {"e": {"l": {"l": {"o": {"": 1, " ": {"友": {"達": {"": 1}}}}}}}}} ``` Nrrh)r@rrefcharr(r(r)adds zTrie.addtextrdc Cst}dg}d}t|D]\}}|r.||kr.qt}d}|D]n\} } d| vr|j|||| |d\}} } }|| || d}qq@|| vr| |} | || <q@|| q@|j|||d}||kr||jvr|j|||<q|j |||d}| ||S)aY Will look for the words added to the trie within `text`. Output is the original string split along the boundaries of the words found. This trie will match the longest possible word first ! Example: ```python >>> trie = Trie() >>> trie.split("[CLS] This is a extra_id_100") ["[CLS] This is a extra_id_100"] >>> trie.add("[CLS]") >>> trie.add("extra_id_1") >>> trie.add("extra_id_100") >>> trie.split("[CLS] This is a extra_id_100") ["[CLS]", " This is a ", "extra_id_100"] ``` rFr)statescurrentrrOskipT)reset to_remover)rroffsets) r enumeraterry split_atom_3appendr split_atom_2rU split_atom_1cut_text) r@rrrrrZ current_charrrrO trie_pointerrPr(r(r)splits2!       z Trie.splitNcCs>|D]0\}}d|vrt|}||||q:q|S)atomic act of splitr)ryrr)r@rrrrOrrPr(r(r)r|s  zTrie.split_atom_1cCs|r i}n|D] }||=q|S)rr()r@rrrrOr(r(r)rs zTrie.split_atom_2c Cs|D]\}}||krqn"||kr8|d}|d} n|}|} |t|krT||nd} d|vrl|}|} |}| |vr|| }|d7}d|vr|}|} |}|t|krq||} qlq||| |fS)rrNr)ryr) r@rrrrOrZ lookstartZlooktrie_pointerZlookahead_indexrPZ next_charr(r(r)rs0   zTrie.split_atom_3cCsZ|t|g}d}|D]:}||kr4tdqn ||kr>q|||||}q|S)z cut thr text.rzbThere was a bug in Trie algorithm in tokenization. Attempting to recover. Please report it anyway.)rrrerror)r@rrr{rOrPr(r(r)rsz Trie.cut_text)NNN)NNN)NNNNN) r"r+r,r-r\rCrrrrrrrr(r(r(r)rsk  %ra add_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to encode the sequences with the special tokens relative to their model. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. stride (`int`, *optional*, defaults to 0): If set to a number along with `max_length`, the overflowing tokens returned when `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflowing sequences. The value of this argument defines the number of overlapping tokens. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. Requires `padding` to be activated. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return Numpy `ms.Tensor` objects. a return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are token type IDs?](../glossary#token-type-ids) return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_overflowing_tokens (`bool`, *optional*, defaults to `False`): Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead of returning overflowing tokens. return_special_tokens_mask (`bool`, *optional*, defaults to `False`): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*, defaults to `False`): Whether or not to return `(char_start, char_end)` for each token. This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using Python's tokenizer, this method will raise `NotImplementedError`. return_length (`bool`, *optional*, defaults to `False`): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. **kwargs: passed to the `self.tokenize()` method Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or if *"token_type_ids"* is in `self.model_input_names`). [What are token type IDs?](../glossary#token-type-ids) - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and `return_overflowing_tokens=True`). - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and `return_overflowing_tokens=True`). - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`). - **length** -- The length of the inputs (when `return_length=True`) a Class attributes (overridden by derived classes) - **vocab_files_names** (`Dict[str, str]`) -- A dictionary with, as keys, the `__init__` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - **pretrained_vocab_files_map** (`Dict[str, Dict[str, str]]`) -- A dictionary of dictionaries, with the high-level keys being the `__init__` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` of the pretrained models with, as associated values, the `url` to the associated pretrained vocabulary file. - **max_model_input_sizes** (`Dict[str, Optional[int]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or `None` if the model has no maximum input size. - **pretrained_init_configuration** (`Dict[str, Dict[str, Any]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, a dictionary of specific arguments to pass to the `__init__` method of the tokenizer class for this pretrained model when loading the tokenizer with the [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`] method. - **model_input_names** (`List[str]`) -- A list of inputs expected in the forward pass of the model. - **padding_side** (`str`) -- The default value for the side on which the model should have padding applied. Should be `'right'` or `'left'`. - **truncation_side** (`str`) -- The default value for the side on which the model should have truncation applied. Should be `'right'` or `'left'`. Args: model_max_length (`int`, *optional*): The maximum length (in number of tokens) for the inputs to the transformer model. When the tokenizer is loaded with [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`], this will be set to the value stored for the associated model in `max_model_input_sizes` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. truncation_side (`str`, *optional*): The side on which the model should have truncation applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. model_input_names (`List[string]`, *optional*): The list of inputs accepted by the forward pass of the model (like `"token_type_ids"` or `"attention_mask"`). Default value is picked from the class attribute of the same name. bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. Will be associated to `self.bos_token` and `self.bos_token_id`. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. Will be associated to `self.eos_token` and `self.eos_token_id`. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. Will be associated to `self.unk_token` and `self.unk_token_id`. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). Will be associated to `self.sep_token` and `self.sep_token_id`. pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. Will be associated to `self.pad_token` and `self.pad_token_id`. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). Will be associated to `self.cls_token` and `self.cls_token_id`. mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). Will be associated to `self.mask_token` and `self.mask_token_id`. additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. Add them here to ensure they won't be split by the tokenization process. Will be associated to `self.additional_special_tokens` and `self.additional_special_tokens_ids`. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. csfdd}|S)Ncs$|jdur|jndd|_|SNr)r-join)fndocstrr(r)docstring_decorators z/add_end_docstrings..docstring_decoratorr()rrr(rr)add_end_docstringss rcCsrt|ttfr dd|DSt|ttfr.cSsg|] }t|qSr(r)rior(r(r)rrmzto_py_obj..) r]dictrryr#rr1rasnumpytolistr0rnumber)objr(r(r)rs  rcCs&|dvr dSt|}|dkr"dSdS)z0Checks whether `char` is a whitespace character.)    TZsF) unicodedatacategoryrcatr(r(r)_is_whitespaces  rcCs(|dvr dSt|}|dr$dSdS)z-Checks whether `char` is a control character.)rrrFCT)rr startswithrr(r(r) _is_controls   rcCst|}d|krdksbnd|kr2dksbnd|krHdksbnd|kr^dkrfnnd St|}|d r~d Sd S) z1Checks whether `char` is a punctuation character.!/:@[`{~TPF)ordrrr)rcprr(r(r)_is_punctuationsZ  rcCs$|d}tt|t|Bt|BS)zcChecks whether the last character in text is one of a punctuation, control or whitespace character.rErrr)rZ last_charr(r(r)_is_end_of_wordsrcCs$|d}tt|t|Bt|BS)zdChecks whether the first character in text is one of a punctuation, control or whitespace character.rr)r first_charr(r(r)_is_start_of_wordsr) token_list new_tokencCs4t||}|t|kr$|||ks0|||dS)zm Inserts one token to an ordered list if it does not already exist. Note: token_list must be sorted. N)bisect bisect_leftrinsert)rrZ insertion_idxr(r(r)!_insert_one_token_to_ordered_lists rcCs<t|ttfr&td|tdntt|ddS)z Print path or list function for show support method of MindFormerBook or other BaseClasses. Args: input_path_or_list (str, list): the path or list to be printed. z %sz%-------------------------------------z% is unsupported by print_path_or_listN)r]rCr#rinfo TypeErrortype)Zinput_path_or_listr(r(r)print_path_or_lists  r c @sveZdZdZgdZdOddZedddZdPee e e e ffed d d Z dQe e e e e e e ffeed ddZdRe e e e e feed ddZee dddZee dddZee dddZee dddZee dddZee dddZee dddZee e dd d!Zejd"dZejd#dZejd$dZejd%dZejd&dZejd'dZejd(dZejd)d!Zeeedd*d+Zeeedd,d-Zeeedd.d/Zeeedd0d1Zeeedd2d3Z eedd4d5Z!eeedd6d7Z"eeedd8d9Z#ee edd:d;Z$ejdd/Zejd?d1Ze jd@d3Z e"jdAd7Z"e#jdBd9Z#e$jdCd;Z$eee e e e e ffddDdEZ%eee e e e e e e e fffddFdGZ&ee e ddHdIZ'ee e e e fddJdKZ(ee eddLdMZ)dNS)Sra A mixin derived by [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`] to handle specific behaviors related to special tokens. In particular, this class hold the attributes which can be used to directly access these special tokens in a model-independent manner and allow to set and update the special tokens. Args: bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. ) bos_token eos_token unk_token sep_token pad_token cls_token mask_tokenadditional_special_tokensTcKsd|_d|_d|_d|_d|_d|_d|_d|_g|_||_ | D]\}}|durVqD||j vrD|dkrt |t tfsJd|dtdd|DsJdt|||qDt |ttfrt|||qDtd|d t|qDdS) NrrzValue z is not a list or tuplecss|]}t|ttfVqdSr=r]rCr6ritr(r(r) sz.SpecialTokensMixin.__init__..z2One of the tokens is not a string or an AddedTokenzspecial token z- has to be either str or AddedToken but got: ) _bos_token _eos_token _unk_token _sep_token _pad_token _cls_token _mask_token_pad_token_type_id_additional_special_tokensverboserySPECIAL_TOKENS_ATTRIBUTESr]r#rallsetattrrCr6rr )r@r kwargsrjr'r(r(r)r\ s2  zSpecialTokensMixin.__init__rccCs|j|jddS)aJ Make sure that all the special tokens attributes of the tokenizer (`tokenizer.mask_token`, `tokenizer.cls_token`, etc.) are in the vocabulary. Add the missing ones to the vocabulary if needed. Return: `int`: The number of tokens added in the vocabulary during the operation. Tspecial_tokens) add_tokensall_special_tokens_extendedr?r(r(r)sanitize_special_tokens(s z*SpecialTokensMixin.sanitize_special_tokens)special_tokens_dictrdc CsP|sdSd}|D]4\}}||jvs8Jd|d|jrLtd|||dkrt|ttfrvtdd|DsJd|d |d |rt |||nRt ||}t |}g}|D]&} t | |vrt | |vr| | q||||_||j|d d 7}qt|t tfs*Jd |d |dt |||||j|gd d 7}q|S)aD Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary). Note,None When adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Using `add_special_tokens` will ensure your special tokens can be used in several ways: - Special tokens are carefully handled by the tokenizer (they are never split). - You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This makes it easy to develop model-agnostic training and fine-tuning scripts. When possible, special tokens are already registered for provided pretrained models (for instance [`BertTokenizer`] `cls_token` is already registered to be :obj*'[CLS]'* and XLM's one is also registered to be `''`). Args: special_tokens_dict (dictionary *str* to *str* or `tokenizers.AddedToken`): Keys should be in the list of predefined special attributes: [`bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`]. Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). replace_additional_special_tokens (`bool`, *optional*,, defaults to `True`): If `True`, the existing list of additional special tokens will be replaced by the one specified in `special_tokens_dict`. Otherwise, `self._additional_special_tokens` is updated. In the former case, the tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged as non-special tokens. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to add a new classification token to GPT-2 tokenizer = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2Model.from_pretrained("gpt2") special_tokens_dict = {"cls_token": ""} num_added_toks = tokenizer.add_special_tokens(special_tokens_dict) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) assert tokenizer.cls_token == "" ```rzKey z is not a special tokenz+Assigning %s to the %s key of the tokenizerrcss|]}t|ttfVqdSr=rrr(r(r)rvsz8SpecialTokensMixin.add_special_tokens..zTokens z for key z* should all be str or AddedToken instancesTr%Token z* should be a str or an AddedToken instance)ryr!r rrr]r#rr"r#getattrrrCrextendrr'r6) r@r*Z!replace_additional_special_tokens added_tokensrjr'rZadditional_special_tokens_setto_addtokenr(r(r)add_special_tokens4s@7      z%SpecialTokensMixin.add_special_tokensF new_tokensr&rdcCs*|sdSt|ttfs|g}|j||dS)a Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary and and will be isolated before the tokenization algorithm is applied. Added tokens and tokens from the vocabulary of the tokenization algorithm are therefore not treated in the same way. Note, when adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Args: new_tokens (`str`, `tokenizers.AddedToken` or a list of *str* or `tokenizers.AddedToken`): Tokens are only added if they are not already in the vocabulary. `tokenizers.AddedToken` wraps a string token to let you personalize its behavior: whether this token should only match against a single word, whether this token should strip all potential whitespaces on the left side, whether this token should strip all potential whitespaces on the right side, etc. special_tokens (`bool`, *optional*, defaults to `False`): Can be used to specify if the token is a special token. This mostly change the normalization behavior (special tokens like CLS or [MASK] are usually not lower-cased for instance). See details for `tokenizers.AddedToken` in HuggingFace tokenizers library. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```rr%)r]r#r _add_tokensr@r3r&r(r(r)r's )zSpecialTokensMixin.add_tokenscCstdSr=NotImplementedErrorr5r(r(r)r4szSpecialTokensMixin._add_tokenscCs(|jdur|jrtddSt|jS)ze `str`: Beginning of sentence token. Log an error if used while not having been set. Nz'Using bos_token, but it is not set yet.)rr rrrCr?r(r(r)r s   zSpecialTokensMixin.bos_tokencCs(|jdur|jrtddSt|jS)z_ `str`: End of sentence token. Log an error if used while not having been set. Nz'Using eos_token, but it is not set yet.)rr rrrCr?r(r(r)r s   zSpecialTokensMixin.eos_tokencCs(|jdur|jrtddSt|jS)zW `str`: Unknown token. Log an error if used while not having been set. Nz'Using unk_token, but it is not set yet.)rr rrrCr?r(r(r)r s   zSpecialTokensMixin.unk_tokencCs(|jdur|jrtddSt|jS)z `str`: Separation token, to separate context and query in an input sequence. Log an error if used while not having been set. Nz'Using sep_token, but it is not set yet.)rr rrrCr?r(r(r)rs   zSpecialTokensMixin.sep_tokencCs(|jdur|jrtddSt|jS)zW `str`: Padding token. Log an error if used while not having been set. Nz'Using pad_token, but it is not set yet.)rr rrrCr?r(r(r)rs   zSpecialTokensMixin.pad_tokencCs(|jdur|jrtddSt|jS)z `str`: Classification token, to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set. Nz'Using cls_token, but it is not set yet.)rr rrrCr?r(r(r)rs   zSpecialTokensMixin.cls_tokencCs(|jdur|jrtddSt|jS)z `str`: Mask token, to use when training a model with masked-language modeling. Log an error if used while not having been set. Nz(Using mask_token, but it is not set yet.)rr rrrCr?r(r(r)r s   zSpecialTokensMixin.mask_tokencCs.|jdur|jrtddSdd|jDS)z `List[str]`: All the additional special tokens you may want to use. Log an error if used while not having been set. Nz7Using additional_special_tokens, but it is not set yet.cSsg|] }t|qSr(rCritokr(r(r)r rmz@SpecialTokensMixin.additional_special_tokens..)rr rrr?r(r(r)rs   z,SpecialTokensMixin.additional_special_tokenscCs ||_dSr=)rr@r'r(r(r)r "scCs ||_dSr=)rr;r(r(r)r &scCs ||_dSr=)rr;r(r(r)r *scCs ||_dSr=)rr;r(r(r)r.scCs ||_dSr=)rr;r(r(r)r2scCs ||_dSr=)rr;r(r(r)r6scCs ||_dSr=)rr;r(r(r)r:scCs ||_dSr=rr;r(r(r)r>scCs|jdurdS||jS)z `Optional[int]`: Id of the beginning of sentence token in the vocabulary. Returns `None` if the token has not been set. N)rconvert_tokens_to_idsr r?r(r(r) bos_token_idBs zSpecialTokensMixin.bos_token_idcCs|jdurdS||jS)z `Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been set. N)rr=r r?r(r(r) eos_token_idLs zSpecialTokensMixin.eos_token_idcCs|jdurdS||jS)z{ `Optional[int]`: Id of the unknown token in the vocabulary. Returns `None` if the token has not been set. N)rr=r r?r(r(r) unk_token_idVs zSpecialTokensMixin.unk_token_idcCs|jdurdS||jS)z `Optional[int]`: Id of the separation token in the vocabulary, to separate context and query in an input sequence. Returns `None` if the token has not been set. N)rr=rr?r(r(r) sep_token_id_s zSpecialTokensMixin.sep_token_idcCs|jdurdS||jS)z{ `Optional[int]`: Id of the padding token in the vocabulary. Returns `None` if the token has not been set. N)rr=rr?r(r(r) pad_token_idis zSpecialTokensMixin.pad_token_idcCs|jS)zH `int`: Id of the padding token type in the vocabulary. )rr?r(r(r)pad_token_type_idrsz$SpecialTokensMixin.pad_token_type_idcCs|jdurdS||jS)z `Optional[int]`: Id of the classification token in the vocabulary, to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Returns `None` if the token has not been set. N)rr=rr?r(r(r) cls_token_idys zSpecialTokensMixin.cls_token_idcCs|jdurdS||jS)z `Optional[int]`: Id of the mask token in the vocabulary, used when training a model with masked-language modeling. Returns `None` if the token has not been set. N)rr=rr?r(r(r) mask_token_ids z SpecialTokensMixin.mask_token_idcCs ||jS)z `List[int]`: Ids of all the additional special tokens in the vocabulary. Log an error if used while not having been set. )r=rr?r(r(r)additional_special_tokens_idssz0SpecialTokensMixin.additional_special_tokens_idscCs|dur||nd|_dSr=)convert_ids_to_tokensrr;r(r(r)r>scCs|dur||nd|_dSr=)rGrr;r(r(r)r?scCs|dur||nd|_dSr=)rGrr;r(r(r)r@scCs|dur||nd|_dSr=)rGrr;r(r(r)rAscCs|dur||nd|_dSr=)rGrr;r(r(r)rBscCs|dur||nd|_dSr=)rGrr;r(r(r)rDscCs|dur||nd|_dSr=)rGrr;r(r(r)rEscsfdd|D_dS)Ncsg|]}|qSr()rG)rir'r?r(r)rrmzDSpecialTokensMixin.additional_special_tokens_ids..r<)r@rxr(r?r)rFscCsVi}|jD]F}t|d|}|r t|ttfrDt|dd|Dnt|||<q |S)a  `Dict[str, Union[str, List[str]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Convert potential tokens of `tokenizers.AddedToken` type to string. _css|]}t|VqdSr=r8)riZattr_value_subr(r(r)rrmz8SpecialTokensMixin.special_tokens_map..)r!r,r]r#rr rCr@set_attrattr attr_valuer(r(r)special_tokens_maps  z%SpecialTokensMixin.special_tokens_mapcCs.i}|jD]}t|d|}|r |||<q |S)a `Dict[str, Union[str, tokenizers.AddedToken, List[Union[str, tokenizers.AddedToken]]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. rH)r!r,rIr(r(r)special_tokens_map_extendeds   z.SpecialTokensMixin.special_tokens_map_extendedcCsdd|jD}|S)z `List[str]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Convert tokens of `tokenizers.AddedToken` type to string. cSsg|] }t|qSr(r8)risr(r(r)rrmz9SpecialTokensMixin.all_special_tokens..)r()r@all_toksr(r(r)all_special_tokenssz%SpecialTokensMixin.all_special_tokenscCsJg}|j}|D]$}|t|ttfr.t|n|g}qtt|}|S)a1 `List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. )rNrxr]r#rr fromkeys)r@rPrJrLr(r(r)r(s  "z.SpecialTokensMixin.all_special_tokens_extendedcCs|j}||}|S)zy `List[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes. )rQr=)r@rPall_idsr(r(r)all_special_idss z"SpecialTokensMixin.all_special_idsN)T)T)F)F)*r"r+r,r-r!r\rRr)rrCr r6r1rrEr'r4rr r r rrrrrsetterr r>r?r@rArBrCrDrErFrMrNrQr(rTr(r(r(r)rs   _ 1$                             $. rcseZdZUdZiZeeefed<iZeeeeeffed<iZ eeeee ffed<iZ eee e fed<dZe eed<gZeeed<gd Zeeed <d Zeed <d Zeed <dZdZdZfddZee dddZee dddZeje dddZeje dddZedddZedddZe dddZeee fdd d!Ze ed"d#d$Z!e d%d&Z"e dd'd(Z#e d)d*Z$e d+d,Z%de eeed/d0d1Z&dee ee'ed2d3d4Z(dee ee)eed6d7d8Z*e+e,d9d:de-e.e/e0fe e-e.e/e0fe)e-e)ee1fe-e)ee2fe e e e e-ee3fee d< d=d>Z4de)e d?d@dAZ5ddBdCZ6e+e,e7de-e.e/ee.ee/fe e-e.e/ee.ee/fe-e.e/ee.ee/fe e-e.e/ee.ee/fe)e-e)ee1fe-e)ee2fe e e e)e e e e-ee3fe e)e e)e)e)e)e)e)e8dDdEdFZ9de-e.e/ee.ee/fe e-e.e/ee.ee/fe)e-e)ee1fe-e)ee2fe e e e)e e e e-ee3fe e)e e)e)e)e)e)e)e8dGdHdIZ:e+e,e7de-e.e/e0fe e-e.e/e0fe)e-e)ee1fe-e)ee2fe e e e)e e e e-ee3fe e)e e)e)e)e)e)e)e8dGdJdKZ;dd;e1je+e,e7de-ee.ee?ee/ee@ee0eeAfe)e-e)ee1fe-e)ee2fe e e e)e e e e-ee3fe e)e e)e)e)e)e)e)e8dOdPdQZBd;e1jsz%BaseTokenizer.max_len_single_sentencecCs|j|jddS)zi `int`: The maximum combined length of a pair of sentences that can be fed to the model. Trqrsr?r(r(r)max_len_sentences_pairEsz$BaseTokenizer.max_len_sentences_paircCsL||j|jddkr@|jr@|jdds4tdd|jd<ntddS)NFrqruzXSetting 'max_len_single_sentence' is now deprecated. This value is automatically set up.Trertr rogetrwarningr!r;r(r(r)ruLs cCsL||j|jddkr@|jr@|jdds4tdd|jd<ntddS)NTrqrvFzWSetting 'max_len_sentences_pair' is now deprecated. This value is automatically set up.rwr;r(r(r)rvZs )rdcCs ||_dS)z%Sets processor class as an attribute.N)rm)r@rdr(r(r)_set_processor_classfsz"BaseTokenizer._set_processor_classcCsN|jjd|jd|jd|jd|jd|jd|jd|jd|j d S) Nz(name_or_path='z', vocab_size=z, model_max_length=z , is_fast=z, padding_side='z', truncation_side='z', special_tokens=z, clean_up_tokenization_spaces=)) rbr"rc vocab_sizererfrarbrNrhr?r(r(r)__repr__js zBaseTokenizer.__repr__cCs tdSr=r6r?r(r(r)__len__sszBaseTokenizer.__len__cCs tdS)a Returns the vocabulary as a dictionary of token to index. `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the vocab. Returns: `Dict[str, int]`: The vocabulary. Nr6r?r(r(r) get_vocabvs zBaseTokenizer.get_vocab)rccKs|dd}|dur|}tj|}tj|}|s\||jvr\t|d|jd|jd|rr|srt|dt}d}i}|t vr| |\}} | |\}}| }d} tj|rddt |D} t| d krtd || | r$tj|| d } td | t| }| |\}}||\} } d | vrV| d dd}| d }n td| |||| |s|j}td||tfd|i|}|}|rtd|S)ax Instantiates a tokenizer by the name_or_path. User can get the name using `get_support_list` of any tokenizer, it will download the necessary files from the cloud. or pass a directory where contains the vocabulary file and tokenizers yaml configuration file. Args: name_or_path (str): It supports the following two input types: If the name_or_path is a supported tokenizer name, for example, `clip_vit_b_32` and `t5_small`, it will download the necessary files from the cloud. User can select one from the support list by call `MindFormerBook.show_tokenizer_support_list()`. If name_or_path is a path to the local directory where there should have vocaburary files and configuration file ended with `yaml`. The vocaburary file needed by the tokenizer is determined by `.vocab_files_names`. pretrained_model_name_or_path (Optional[str]): Equal to "name_or_path", if "pretrained_model_name_or_path" is set, "name_or_path" is useless. Examples: >>> from mindformers import T5Tokenizer >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> res = tokenizer.encode("hello world!") >>> print(res) [21820, 296, 55, 1] Returns: A instanced tokenizer. pretrained_model_name_or_pathNz+ does not exist, or it is not supported by z. please select from .z is not a directory.cSsg|]}|dr|qS).yaml)endswith)rifiler(r(r)rrmz1BaseTokenizer.from_pretrained..rzWThere should be only one yaml file under the directory %s, but followings are found: %srz;config in the yaml file %s are used for tokenizer building.rtokenizer_classzWCan't find the tokenizer_config.json in the file_dict. The content of file_dict is : %sz0build tokenizer class name is: %s using args %s. class_namezuSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.)rlospathexistsisdir _support_listr!r"rrget_tokenizer_url_support_list_download_using_name$_get_class_name_and_args_form_configlistdirrrryrrr*read_files_according_specific_by_tokenizerupdaterr))r&rcr$ris_existis_dirrZ loaded_kwargsconfig cache_pathZ yaml_list yaml_file vocab_dict file_dict tokenizerr.r(r(r)from_pretrainedsf            zBaseTokenizer.from_pretrainedc CsT|}|dr>|d|j}tjt|d|j}ntjt |d|j}tj |svtj |ddtj||d}dd}tj |s||}tj |rtj |rt ||td |ntd |t|}|D]N}|dd } tj|| } tj | s:td || t|| t| qt|} | |fS) z9Given the supported model name, download it from the urls mindspore/rHTexist_okrcSs0d}tD]}||vr||}q,q|Sr)rZget_trainer_support_task_listrxrx) model_namedefault_yaml_fileZ model_dictr(r(r)get_default_yaml_files  zABaseTokenizer._download_using_name..get_default_yaml_filez"default yaml config in %s is used.z0default yaml file path must be correct, but get rz)Download the vocab from the url %s to %s.)rr _model_namerrrrZ#get_xihe_checkpoint_download_folder _model_type&get_default_checkpoint_download_folderrmakedirsrealpathshutilrirrFileNotFoundErrorrrrr) r&rcZtokenizer_namerrrrtokenizer_need_filesurl_filelocal_file_name file_pathrr(r(r)rs:         z"BaseTokenizer._download_using_namec Cs|s>> from mindformers import T5Tokenizer, MindFormerBook >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> tokenizer.save_pretrained() >>> output_path = MindFormerBook.get_default_checkpoint_save_folder() >>> print(os.listdir(output_path)) ['mindspore_model.yaml', 'spiece.model'] NTrr)rrz2format should be one of [`yaml`, `json`], but got rrr rr)Loaderrrwrr)indentz3file_format should be one of [json, yaml], but got )rZ"get_default_checkpoint_save_folderrrr!rirjrkrbr"rrrrrrrreadrZget_tokenizer_name_to_processorr]ryr6r8dumpTOKENIZER_CONFIG_NAMErsave_vocabulary)r@rrrZdefault_directoryr$ merged_dictr file_readerZprocessor_nameZ token_keyZ token_valueZtokenizer_config_pathfpr(r(r)save_pretrainedBsD     $     .   2zBaseTokenizer.save_pretrained)rfilename_prefixrdcCstdS)aO Save only the vocabulary of the tokenizer (vocabulary + added tokens). This method won't save the configuration and special token mappings of the tokenizer. Use [`~PreTrainedTokenizerFast._save_pretrained`] to save the whole state of the tokenizer. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. Nr6)r@rrr(r(r)rszBaseTokenizer.save_vocabularyFrrrr1rdcKstdS)a Converts a string in a sequence of tokens, replacing unknown tokens with the `unk_token`. Args: text (`str`): The sequence to be encoded. pair (`str`, *optional*): A second sequence to be encoded with the first. add_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add the special tokens associated with the corresponding model. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific encode method. See details in [`~PreTrainedTokenizerBase.__call__`] Returns: `List[str]`: The list of tokens. Nr6)r@rrrr1r$r(r(r)tokenizeszBaseTokenizer.tokenizezI **kwargs: Passed along to the `.tokenize()` method. zp Returns: `List[int]`, `ms.Tensor`, or `np.ndarray`: The tokenized ids of the text. T) r text_pairr1padding truncationrstridereturn_tensorsrdc Ks*|j|f|||||||d| } | dS)aC Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing `self.convert_tokens_to_ids(self.tokenize(text))`. Args: text (`str`, `List[str]` or `List[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). )rr1rrrrrr\) encode_plus) r@rrr1rrrrrr$encoded_inputsr(r(r)encodes%  zBaseTokenizer.encoderrrdcCstdSr=r6)r@rrr(r(r)rtsz'BaseTokenizer.num_special_tokens_to_addc Ks|dd}|dd}|durZ|durZ|durZ|rV|jddsLtdd|jd<d }|dur|r|rvtd t|durtj } ntj } n|dur|dur|r|dur|dus|dus|dkrtd |durtd tj } n$t |tst|} nt |tr|} ntj } |durD|dkrD|r:td tt |} nT|dur|dur|durjt j} n&t |t st |} nt |t r|} nt j} |durB| tj kr|jtkr|r|jddstdd|jd<tj } n|j}| t jkrB|jtkr<|r4|jdds*tdd|jd<t j} n|j}| tj krj|jrb|jdkrjtd| t jkr| tj kr|dur|dur||dkrtd|d|d| | ||fS)z Find the correct padding/truncation strategy with backward compatibility for old arguments (truncation_strategy and pad_to_max_length) and behaviors. truncation_strategyrIZpad_to_max_lengthFNz#Truncation-not-explicitly-activatedatTruncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.TrHaThe `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).z`max_length` is ignored when `padding`=`True` and there is no truncation strategy. To pad to max length, use `padding='max_length'`.zLThough `pad_to_max_length` = `True`, it is ignored because `padding`=`True`.aThe `truncation_strategy` argument is deprecated and will be removed in a future version, use `truncation=True` to truncate examples to a max length. You can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input size of the model (e.g. 512 for Bert). If you have pairs of inputs, you can give a specific truncation strategy selected among `truncation='only_first'` (will only truncate the first sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence in the pairs).zAsking-to-pad-to-max_lengthzAsking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.z Asking-to-truncate-to-max_lengthzAsking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.rzAsking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.zATruncation and padding are both activated but truncation length (z+) is not a multiple of pad_to_multiple_of (z).)rlrorxrryrrrrrrr]rrGrLrMre LARGE_INTEGERrrBr!) r@rrrpad_to_multiple_ofr r$Zold_truncation_strategyZold_pad_to_max_lengthpadding_strategyrr(r(r)"_get_padding_truncation_strategiess                       z0BaseTokenizer._get_padding_truncation_strategies)rr text_targettext_pair_targetr1rrrris_split_into_wordsrrreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthr rdcKs||||| | | | | ||||||d}|||durF|durFtd|durr|js\||jf||d|}|dur||jf||d|}||dur|S|dur|S|d|d<|S)a  Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences. Args: text (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). )r1rrrrrrrrrrrrrr Nz3You need to specify either `text` or `text_target`.)rrr\labels)rr!rp_switch_to_input_mode _call_one_switch_to_target_mode)r@rrrrr1rrrrrrrrrrrrrr r$Z all_kwargsrtZtarget_encodingsr(r(r)__call__h sB2  zBaseTokenizer.__call__)rrr1rrrrrrrrrrrrrr rdcKs6dd}||std|dur0||s0td|rZt|ttfoV|oVt|dttf}nt|ttf}|rt|trtd|durt|t|krtdt|dt|d |durtt||n|}|jf|||||||| | | | | ||||d |S|j f||||||||| | | | | ||||d |S) zused by self.__call__cSslt|trdSt|ttfrh|s$dSt|dtr6dSt|dttfrd|d pbt|ddtSdSdS)NTrF)r]rCr#r)rr(r(r)_is_valid_text_input s z5BaseTokenizer._call_one.._is_valid_text_inputztext input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).Nrzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: r)batch_text_or_text_pairsr1rrrrrrrrrrrrrr )rrr1rrrrrrrrrrrrrr ) r!r]r#rrCrrzipbatch_encode_plusr)r@rrr1rrrrrrrrrrrrrr r$r is_batchedrr(r(r)r s&  zBaseTokenizer._call_onecKsX|jf|||| |d|\}}}}|jf||||||||| | | | | ||||d|S)a Tokenize and prepare for the model a sequence or a pair of sequences. This method is deprecated, `__call__` should be used instead. Args: text (`str`, `List[str]` or `List[int]` (the latter only for not-fast tokenizers)): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). rrrrr )rrr1rrrrrrrrrrrrrr )r _encode_plus)r@rrr1rrrrrrrrrrrrrr r$rrr(r(r)r. s<+ zBaseTokenizer.encode_plusrrr1rrrrrrrrrrrrrr rdcKstdS)zused by self.encode_plusNr6)r@rrr1rrrrrrrrrrrrrr r$r(r(r)rw szBaseTokenizer._encode_plus)rr1rrrrrrrrrrrrrr rdcKsV|jf|||||d|\}}}}|jf||||||||| | | | | |||d|S)a Tokenize and prepare for the model a list of sequences or a list of pairs of sequences. This method is deprecated, `__call__` should be used instead. Args: batch_text_or_text_pairs ( `List[str]`, `List[Tuple[str, str]]`, `List[List[str]]`, `List[Tuple[List[str], List[str]]]`, and for not-fast tokenizers, also `List[List[int]]`, `List[Tuple[List[int], List[int]]]`): Batch of sequences or pair of sequences to be encoded. This can be a list of string/string-sequences/int-sequences or a list of pair of string/string-sequences/int-sequence (see details in `encode_plus`). r)rr1rrrrrrrrrrrrrr )r_batch_encode_plus)r@rr1rrrrrrrrrrrrrr r$rrr(r(r)r s:/ zBaseTokenizer.batch_encode_plusrr1rrrrrrrrrrrrrr rdcKstdS)zused by self.batch_encode_plusNr6)r@rr1rrrrrrrrrrrrrr r$r(r(r)r sz BaseTokenizer._batch_encode_plus)rrrrrrr rdcs|jjdr<|jdds|d ur8dn|}n If the `encoded_inputs` passed are dictionary of numpy arrays, or Mindspore tensors, the result will use the same type unless you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the specific device of your tensors however. Args: encoded_inputs ( [`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`): Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str, List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader collate function. Instead of `List[int]` you can have tensors (numpy arrays or Mindspore tensors), see the note above for the return type. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return Numpy `ms.Tensor` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. ZFastzAsking-to-pad-a-fast-tokenizerFzYou're using a z tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.Trcs i|]fddDqS)csg|] }|qSr(r()riexamplerjr(r)rP rmz0BaseTokenizer.pad...r()ri)rrr)rlP rmz%BaseTokenizer.pad..zRYou should supply an encoding or a list of encodings to this method that includes z, but you provided Nr^r1r0ztype of z unknown: z7. Should be one of a python, mindspore or numpy object.)rrr )rrrrrWc3s|]}t|kVqdSr=r)rir) batch_sizer(r)r sz$BaseTokenizer.pad..zLSome items in the output dictionary have a different batch size than others.css|]}t|VqdSr=r)riinputsr(r(r)r rmcsi|]\}}||qSr(r(r)ir(r)rl rm)#rbr"rrorxrZwarning_advicer]r#rrr%r_r!rrRr1rr0rr ryrr_padrTrr"rxrrmaxrranger)r@rrrrrrr required_input first_elementrgrjr'rrH batch_outputsroutputsr()rrrr)pad sK        zBaseTokenizer.pad) token_ids_0 token_ids_1rdcCs2|durt|dgSdgt|dgt|S)a Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids) Should be overridden in a subclass if the model has a special way of building those. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The token type ids. Nrrrr@rrr(r(r)$create_token_type_ids_from_sequences sz2BaseTokenizer.create_token_type_ids_from_sequencescCs|dur |S||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. This implementation does not add special tokens and this method should be overridden in a subclass. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The model input with special tokens. Nr(rr(r(r) build_inputs_with_special_tokens sz.BaseTokenizer.build_inputs_with_special_tokens)idspair_idsr1rrrrrrrrrrrr rXrdcKs |jf|||||d|\}}}}t|du}t|}|rDt|nd}| rX|sXtd| rv|tjkrv|durvtd| durd|jv} | durd|jv} i}|||r|j|dnd}g}|tjkr|r||kr|j ||||||d \}}}| r ||d <|||d <|r,| ||}| ||}n8|r:||n|}dgt||r^dgt|ng}||d <| rz||d<| r|r| |||d <ndgt||d <| |d |||tjks| r|j|||j|| d}|rt|d |d<t|| |d}|S)a Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens. Please Note, for *pair_ids* different than `None` and *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a combination of arguments will raise an error. Args: ids (`List[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. rNrzAsking to return token_type_ids while setting add_special_tokens to False results in an undefined behavior. Please set add_special_tokens to True or set return_token_type_ids to None.zNot possible to return overflowing tokens for pair of sequences with the `longest_first`. Please select another truncation strategy than `longest_first`, for instance `only_second` or `only_first`.r]r^rq)rnum_tokens_to_removerrrZnum_truncated_tokensr\special_tokens_mask)rrrrlengthrZ)rrErr!rGrLr_rtrMtruncate_sequencesrrget_special_tokens_mask&_eventual_warn_about_too_long_sequencerrrr'rT)r@rrr1rrrrrrrrrrrr rXr$rrrrZlen_idsZ len_pair_idsr total_lenrsequencer]rr(r(r)prepare_for_model s&      &zBaseTokenizer.prepare_for_modelrH)rrrrrrdc Cs|dkr||gfSt|ts$t|}g}|tjksH|tjkr|durt||krtt|||}|jdkr|d|}||d}n:|jdkr|| d}|d| }ntd|jdn:d|dt|d }|tjkr|d |d }t |n|tjkrt d tjj t |D]} |dusHt|t|kr|jdkrb|dd }n,|jdkr||dd}ntdt |jnF|jdkr|dd }n,|jdkr|dd}ntdt |jq(n|tjkr|durt||krvtt|||}|jdkr<|| d}|d| }n8|jdkrb|d|}||d}ntdt |jnt d|t|||||fS)a Truncates a sequence pair in-place following the strategy. Args: ids (`List[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. num_tokens_to_remove (`int`, *optional*, defaults to 0): Number of tokens to remove using the truncation strategy. truncation_strategy (`str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): The strategy to follow for truncation. Can be: - `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). stride (`int`, *optional*, defaults to 0): If set to a positive number, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. Returns: `Tuple[List[int], List[int], List[int]]`: The truncated `ids`, the truncated `pair_ids` and the list of overflowing tokens. Note: The *longest_first* strategy returns empty list of overflowing tokens if a pair of sequences (or a batch of pairs) is provided. rNrgr`zinvalid truncation strategy: z, use 'left' or 'right'.zWe need to remove z; to truncate the input but the first sequence has a length z. z/Please select another truncation strategy than z0, for instance 'longest_first' or 'only_second'.zBe aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the %s truncation strategy. So the returned list will always be empty even if some tokens have been removed.rrzinvalid truncation strategy:zWe need to remove %s to truncate the input but the second sequence has a length %s. Please select another truncation strategy than %s, for instance 'longest_first' or 'only_first'.)r]rGrJrLrminrbr!rrryr'rrCrK) r@rrrrrr window_len error_msgrHr(r(r)r T sz-                  z BaseTokenizer.truncate_sequences)rrrrrrdc Cs|durd|jv}||jd}|tjkr2t|}|dur^|dur^||dkr^||d|}|tjkort||k}|rd|vrdgt||d<|r|t|}|jdkr&|r|ddg||d<d|vr|d|jg||d<d|vr |ddg||d<||jg|||jd<n|jdkr|rNdg||d|d<d|vrp|jg||d|d<d|vrdg||d|d<|jg||||jd<ntd t |j|S) a? Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) Nr^rrr`r]rrgzInvalid padding strategy:) r_rrrrrarCrBr!rC) r@rrrrrrZneeds_to_be_padded differencer(r(r)r sB            zBaseTokenizer._padr{rdcCstdS)aT Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove sub-word tokenization artifacts at the same time. Args: tokens (`List[str]`): The token to join in a string. Returns: `str`: The joined tokens. Nr6r@r{r(r(r)convert_tokens_to_string s z&BaseTokenizer.convert_tokens_to_stringz np.ndarrayz ms.Tensor) sequencesskip_special_tokensrhrdc sfdd|DS)a\ Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, ms.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. cs,g|]$}|r$j|fdndqS))rrhr)decode)riseqrhr$r@rr(r)r= sz.BaseTokenizer.batch_decode..r()r@rrrhr$r(rr) batch_decode% szBaseTokenizer.batch_decode token_idsrrhrdcKsdt|}t|dtrHg}|D]&}|jf|||d|}||qn|jf|||d|}|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, ms.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. r)rrrh)rr]r#_decoder)r@rrrhr$outputrgZnew_strsr(r(r)rG s(zBaseTokenizer.decodecKstdSr=r6)r@rrrhr$r(r(r)rv szBaseTokenizer._decoderralready_has_special_tokensrdcs0|r |dusJd|jfdd|D}|S) Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. Na You cannot use ``already_has_special_tokens=False`` with this tokenizer. Please use a slow (full python) tokenizer to activate this argument. Or set `return_special_tokens_mask=True` when calling the encoding method to get the special tokens mask in any tokenizer. csg|]}|vrdndqS)rrr()rir0rTr(r)r rmz9BaseTokenizer.get_special_tokens_mask..r#)r@rrr!rr(r#r)r  s z%BaseTokenizer.get_special_tokens_mask) out_stringrdcCsX|ddddddddd d d d d ddddddd}|S)a Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms. Args: out_string (`str`): The text to clean up. Returns: `str`: The cleaned-up string. z .rz ??z !!z ,,z ' 'z n'tzn'tz 'mz'mz 'sz'sz 'vez'vez 'rez're)replace)r$r(r(r)clean_up_tokenization s*   z#BaseTokenizer.clean_up_tokenization)rrr cCsJ|durFt||jkrF|rF|jdds %s). Running this sequence through the model will result in indexing errorsT)rrerorxrry)r@rrr r(r(r)r  s  z4BaseTokenizer._eventual_warn_about_too_long_sequencecCsdS)zs Private method to put the tokenizer in input mode (when it has different modes for input/outputs) Nr(r?r(r(r)r sz#BaseTokenizer._switch_to_input_modecCsdS)zt Private method to put the tokenizer in target mode (when it has different modes for input/outputs) Nr(r?r(r(r)r sz$BaseTokenizer._switch_to_target_modeccs0td|d|_dVd|_|dS)z Temporarily sets the tokenizer for encoding the targets. Useful for tokenizer associated to sequence-to-sequence models that need a slightly different processing for the labels. a`as_target_tokenizer` is deprecated and will be removed in v5 of Transformers. You can tokenize your labels by using the argument `text_target` of the regular `__call__` method (either in the same call as your input texts if you use the same keyword arguments, or in a separate call.TNF)rrrrprr?r(r(r)as_target_tokenizer sz!BaseTokenizer.as_target_tokenizerr) src_texts tgt_textsrmax_target_lengthrrrrdc Ksd} t| t|dd|dd|dur6|j}||fd||||d|} |dur^| S|durj|}|,||fd||||d|} Wdn1s0Y| d| d <| S) aa Prepare model inputs for translation. For best performance, translate one sentence at a time. Arguments: src_texts (`List[str]`): List of documents to summarize or source language texts. tgt_texts (`list`, *optional*): List of summaries or target language texts. max_length (`int`, *optional*): Controls the maximum length for encoder inputs (documents to summarize or source language texts) If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. max_target_length (`int`, *optional*): Controls the maximum length of decoder inputs (target language texts or summaries) If left unset or set to `None`, this will use the max_length value. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return PyTorch `ms.Tensor` objects. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `True`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). **kwargs: Additional keyword arguments passed along to `self.__call__`. Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to the encoder. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model. - **labels** -- List of token ids for tgt_texts. The full set of keys `[input_ids, attention_mask, labels]`, will only be returned if tgt_texts is passed. Otherwise, input_ids, attention_mask will be the only keys. a `prepare_seq2seq_batch` is deprecated and will be removed in version 5 of HuggingFace Transformers. Use the regular `__call__` method to prepare your inputs and targets. Here is a short example: model_inputs = tokenizer(src_texts, text_target=tgt_texts, ...) If you either need to use different keyword arguments for the source and target texts, you should do two calls like this: model_inputs = tokenizer(src_texts, ...) labels = tokenizer(text_target=tgt_texts, ...) model_inputs["labels"] = labels["input_ids"] See the documentation of your specific tokenizer for more details on the specific arguments to the tokenizer of choice. For a more complete example, see the implementation of `prepare_seq2seq_batch`. Zsrc_langNZtgt_langT)r1rrrr)r1rrrrr\r)rrrrlrer+) r@r,r-rr.rrrr$Zformatted_warning model_inputsrr(r(r)prepare_seq2seq_batch sFG     $ z#BaseTokenizer.prepare_seq2seq_batch)N)Nrr)N)NF)NTFNNrN)F)FNNNT)NNNNTFNNrFNNNNFFFFT)NTFNNrFNNNNFFFFT)NTFNNrFNNNNFFFFT)TFNNrFNNNNFFFFT)TNNNNT)N)N)NTFNNrNNNNFFFTF)NrrHr)FN)FN)FN)NF)NNNrNT)Yr"r+r,r-rVrrCrDrWrXrrYr rRrZr[rr_rarbZslow_tokenizer_classrrr\rrurvrUrzr}r~rr.rrrrrrr rrErrENCODE_KWARGS_DOCSTRINGr TextInputPreTokenizedInput EncodedInputrrGr/rrtr'ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRINGrTrrrrrMr TextInputPairPreTokenizedInputPairEncodedInputPairrrrrrrr rrrrrrr  staticmethodr*r rrrr+r0rr(r(rar)rs    &    R 3    @    )    Z   n   K     U '   .         z M % 2   "  rcsfeZdZdZgZfddZeedddZee dddZ e e e fdd d Z d d ZdCeee eefee dddZddZdDee dddZdEeee eee dddZddZee ee fee ee fdddZd d!Zd"d#Zddejejdd$d ddddd d d d dfeee e!feeee e!feeeee e eee eee e"feeeeeeeeee#d%d&d'Z$dejejdd$d ddddd d d d dfeeeee%ee ee&ee!ee'feeeee e eee eee e"feeeeeeeeee#d(d)d*Z(e)e*e+dejejdd$ddddd d d df eee&e,ee dffeeeee e ee ee eeeeeeeee#d+d,d-Z-e e,e e e e.ffd.d/d0Z/dFeeeeee d1fd2d3 Z0dGee ee feee ee fd4d5d6Z1e e d7d8d9Z2ee e dd:d;Z3dHee eee d<d=d>Z4e5d?d@Z6e5dAdBZ7Z8S)Ira Base class for all slow tokenizers. Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary. This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). c s6tjfi|i|_i|_g|_t|_d|_dSNF)r[r\added_tokens_encoderadded_tokens_decoderunique_no_split_tokensr tokens_trie_decode_use_source_tokenizer)r@r$rar(r)r\ts zTokenizer.__init__rccCsdSr:r(r?r(r(r)rfszTokenizer.is_fastcCstdS)zP `int`: Size of the base vocabulary (without the added tokens). Nr6r?r(r(r)r|szTokenizer.vocab_sizecCs|jS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. )r;r?r(r(r)get_added_vocabszTokenizer.get_added_vocabcCs|jt|jS)zD Size of the full vocabulary with the added tokens. )r|rr;r?r(r(r)r~szTokenizer.__len__Fr2cspdd|D}g}|D]}t|ts>td|dt|d|sZtdrZjrZ|}|jkr|jkr||vr| |j rt d|qfdd t |D}d d |D}j|j||r"t|d krtj|d nttjt|_n:t|d krBtj|d nttjt|_jt|S) a Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Args: new_tokens (`List[str]`or `List[tokenizers.AddedToken]`): Token(s) to add in vocabulary. A token is only added if it's not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the tokens should be added as special tokens. Returns: `int`: The number of tokens actually added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```cSsg|] }t|qSr(r8r9r(r(r)rrmz)Tokenizer._add_tokens..r+z is not a string but a r do_lower_casezAdding %s to the vocabularycsi|]\}}|t|qSr(r)rirr:r?r(r)rlrmz)Tokenizer._add_tokens..cSsi|]\}}||qSr(r(rr(r(r)rlrmrr)r]rCrr hasattrrAlowerr r=rr rrrryr;rr<rrr=sortedrunion _create_trie)r@r3r&Z tokens_to_addr0Zadded_tok_encoderZadded_tok_decoderr(r?r)r4s:     zTokenizer._add_tokenscCsNt}|D]8}t|dr8|jr8||jvr8||q ||q ||_dS)NrA)rrBrArQrrCr>)r@r=trier0r(r(r)rFs  zTokenizer._create_triercCs g}g}t|||r|ndS)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. N)rr)r@rrrrr(r(r)rtsz#Tokenizer.num_special_tokens_to_addNTrcKsdd|jD}|j|fi|\}}|r6td|t|dr|jrdd|j|jD}dd|d d }t |d d |}t |j}|j |} t| D]\} } | |vr|| d } | dkr| | dnd } | t| dkr| | dnd }t| tr@| jr |r || | d<| jrj| rj| | | d<q|rV|| | d<| r| | | d<qg}| D]6} | sqt| |vr|| n||| qt|S)a" Converts a string in a sequence of tokens, using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Takes care of added tokens. Args: text (`str`): The sequence to be encoded. **kwargs (additional keyword arguments): Passed along to the model-specific `prepare_for_tokenization` preprocessing method. Returns: `List[str]`: The list of tokens. cSs i|]}t|trt||qSr()r]r6rCrr(r(r)rlsz&Tokenizer.tokenize..z$Keyword arguments %s not recognized.rAcSsg|]}t|qSr()reescape)riZs_tokr(r(r)rsz&Tokenizer.tokenize..(|z)|z(.+?)cSs|dp|dS)Nrr)groupsrC)mr(r(r)rmz$Tokenizer.tokenize..Nrr)r(prepare_for_tokenizationrryrBrAr=rQrrHsubrr>rrrxrr]r6r;r:rr- _tokenize)r@rrrr1r$r(Zescaped_special_tokspatternZno_split_tokenr{rr0Z tok_extendedrgr`Ztokenized_textr(r(r)rsH         zTokenizer.tokenizecKstdS)a Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Do NOT take care of added tokens. Nr6r@rr$r(r(r)rQCszTokenizer._tokenizercCsB|dur dSt|tr ||Sg}|D]}|||q(|S)aT Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. N)r]rC#_convert_token_to_id_with_added_vocr)r@r{rr0r(r(r)r=Ls   zTokenizer.convert_tokens_to_idscCs*|dur dS||jvr |j|S||Sr=)r;_convert_token_to_idr@r0r(r(r)rTbs   z-Tokenizer._convert_token_to_id_with_added_voccCstdSr=r6rVr(r(r)rUjszTokenizer._convert_token_to_idrrc sffdd}|rtd||}|dur4||nd}j||||j|j||| | d| | | |||dS)Ncst|tr&j|fi}|St|ttfr||r|t|dtr|rrttjfdd|D}|S|St|ttfr|rt|dtr|Srt d|dt d|ddS)Nrc3s$|]}j|fddiVqdSrTNrrr$r@r(r)rrmz@Tokenizer._encode_plus..get_input_ids..zInput z] is not valid. Should be a string or a list/tuple of strings when `is_split_into_words=True`.zW is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers. r]rCrr=r#r itertoolschainrRr!rr{rr$r@r(r) get_input_idss&        z-Tokenizer._encode_plus..get_input_idsa return_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast. More information on available tokenizers at https://github.com/huggingface/transformers/pull/2674T)rr1rrrrrrrXrrrrrr )r7rr')r@rrr1rrrrrrrrrrrrrr r$r_ first_ids second_idsr(r^r)rms2zTokenizer._encode_plusrc sfdd}|rtdg}|D]r}t|ttfsB|d}}n*rdt|dttfsd|d}}n|\}}||}|dur||nd}|||fq$j|||||||| | | | || |d}t|S)Ncst|tr&j|fi}|St|ttfr||r|t|dtr|rrttjfdd|D}|S|St|ttfr|rt|dtr|St ddS)Nrc3s$|]}j|fddiVqdSrWrXrrYr(r)rrmzFTokenizer._batch_encode_plus..get_input_ids..z\Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.rZr]r^r(r)r_s      z3Tokenizer._batch_encode_plus..get_input_idszreturn_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast.r) r1rrrrrrrrrrrr )r7r]r#rr_batch_prepare_for_modelrT)r@rr1rrrrrrrrrrrrrr r$r_r\Zids_or_pair_idsrrr`rarr(r^r)rs@  zTokenizer._batch_encode_plus)batch_ids_pairsr1rrrrrrrrrrrr rdcCsi}|D]h\}}|j|||tjj|j||dd| | | | dd|d}|D]&\}}||vr`g||<|||qHq|j||j||| d}t||d}|S)a Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens Args: batch_ids_pairs: list of tokenized input ids or input ids pairs NF)r1rrrrrrrrrrrrXr )rrrrr)rrrr'ryrrrT)r@rcr1rrrrrrrrrrrr rr`rarrjr'r(r(r)rb sB  z"Tokenizer._batch_prepare_for_modelrcKs||fS)a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. kwargs: Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. r(rSr(r(r)rOLsz"Tokenizer.prepare_for_tokenizationr csD|r&|durtdtj||ddSdg|r6t|ndt|S)r"NzYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.T)rrr!r)r!r[r r)r@rrr!rar(r)r dsz!Tokenizer.get_special_tokens_mask)rrrdcCst|trB||jvr|j|S||jkr8td|d||Sg}|D]b}t|}|rf||jvrfqJ||jvr||j|qJ||jkrtd|d|||qJ|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). z The token id zb is out of the size of vocabulary, please check your tokenizer and corresponding vocabulary files.)r]rRr<r| IndexError_convert_id_to_tokenrTr)r@rrr{indexr(r(r)rGs&        zTokenizer.convert_ids_to_tokens)rfrdcCstdSr=r6)r@rfr(r(r)reszTokenizer._convert_id_to_tokencCs d|S)Nr)rrr(r(r)rsz"Tokenizer.convert_tokens_to_stringrc Ks|dd|_|dd}|j||d}g}g}|D]L} |rH| |jvrHq4| |jvrv|rj|||g}|| q4|| q4|r||||rd|} n d|} |dur|n|j}|r| | } | S| S)NZuse_source_tokenizerFspaces_between_special_tokens)rrr) rlr?rGrTr;rrrrhr*) r@rrrhr$rgZfiltered_tokensZ sub_textsZcurrent_sub_textr0r clean_textr(r(r)rs6       zTokenizer._decodecCstd|jt|jdS)zshow_support_list methodzsupport list of %s is:N)rrr"r rr&r(r(r)show_support_listszTokenizer.show_support_listcCs|jS)zget_support_list method)rrir(r(r)get_support_listszTokenizer.get_support_list)F)F)NT)NF)F)FN)9r"r+r,r-rr\rrErfrRr|rrCr@r~r rr6r4rFrtr2r rrQr=rTrUrrrGrMr3r4r/rTrr6r7r8rrr1r5r rbrrOr rGrerrr.rjrkrr(r(rar)rds:   $A   H (  U SA   & - r)`r-rirrHrrrrrr[ dataclassesrrenumrtypingrrrrr r r r collectionsr rcollections.abcrr contextlibrrnumpyr0rr1mindformers.toolsrmindformers.tools.registerrrrrZtools.download_toolsrZ tools.utilsrmindformer_bookr__all__ZSPECIAL_TOKEN_FILE_NAMErrRrnrrCr2r3r4r6r7r8ZSPECIAL_TOKENS_MAP_FILEZADDED_TOKENS_FILEZTOKENIZER_CONFIG_FILEZFULL_TOKENIZER_FILEcompileZ_re_tokenizer_filer r/r6rFrGrNrSrTrrr1r5ZINIT_TOKENIZER_DOCSTRINGrrrrrrrrr rrregister TOKENIZERrr(r(r(r)s (             H h56E   s pengcheng.mind-test/mindformers/models/__pycache__/build_config.cpython-39.pyc000066400000000000000000000033461471400134000300140ustar00rootroot00000000000000a ?cfF @s2dZddlmZmZmZdeeeedddZdS) z build model config modules )MindFormerRegisterMindFormerModuleTypeMindFormerConfigNconfig)r default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|jdurTt|j|_|jdurjt|j|_|jdurt|j|_tj|t j |dStj ||fi|S)afBuild model config For MindFormer. Instantiate the model config from MindFormerRegister's registry. Args: config (dict): The task model config's config. Default: None. default_args (dict): The default argument of model config API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'config'. class_name (str): The class name of model config API. Default: None. Return: The function instance of model config API. Examples: >>> from mindformers import build_model_config >>> model_config = {'type': 'ViTConfig'} >>> # 1) use config dict to build model >>> model_config_from_config = build_model_config(model_config) >>> # 2) use class name to build model >>> model_config_class_name = build_model_config(class_name='ViTConfig') N)r) isinstancedictrZ text_configbuild_model_configZ vision_configZ head_configrget_instance_from_cfgrCONFIG get_instance)rrrrkwargsrJ/home/ma-user/work/mindformers-baichuan/mindformers/models/build_config.pyr s      r )NNrN)__doc__mindformers.tools.registerrrrr strr rrrrspengcheng.mind-test/mindformers/models/__pycache__/build_model.cpython-39.pyc000066400000000000000000000050251471400134000276430ustar00rootroot00000000000000a ?cf@sjdZddlmZmZmZddlmZdeeeeddd Z deeeedd d Z deeeedddZ dS)zBuild Model API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_model_configNmodels)config default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|durJi}t|j|d}|dur|dur|D]\}}|||qpi}|d|tj |j t j |dSdStj ||fi|S)aBuild model For MindFormer. Instantiate the model from MindFormerRegister's registry. Args: config (dict): The task model's config. Default: None. default_args (dict): The default argument of model API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'model'. class_name (str): The class name of model API. Default: None. Return: The function instance of model API. Examples: >>> from mindformers import build_model >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> model_from_config = build_model(config.model) Nr r) isinstancedictrr model_configitems __setattr__ setdefaultrget_instance_from_cfgarchrMODELS get_instance)rr r r kwargsrkeyvaluerI/home/ma-user/work/mindformers-baichuan/mindformers/models/build_model.py build_models&  rencoderc Ksz|dur|durdS|durf|}t|tss(  - pengcheng.mind-test/mindformers/models/__pycache__/build_processor.cpython-39.pyc000066400000000000000000000032341471400134000305620ustar00rootroot00000000000000a ?cf @s>dZddlmZmZmZddlmZd eeeeddd ZdS) zBuild Processor API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_tokenizerN processor)config default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|jdurTt|j|_|jdurrt|jfi||_tj|t j |dStj ||fi|S)a Build processor For MindFormer. Instantiate the processor from MindFormerRegister's registry. Args: config (dict): The task processor's config. Default: None. default_args (dict): The default argument of processor API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'processor'. class_name (str): The class name of processor API. Default: None. Return: The function instance of processor API. Examples: >>> from mindformers import build_processor >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> processor_from_config = build_processor(config.processor) N)r ) isinstancedictrimage_processorbuild_processor tokenizerrrget_instance_from_cfgr PROCESSOR get_instance)rr r r kwargsrM/home/ma-user/work/mindformers-baichuan/mindformers/models/build_processor.pyrs   r)NNrN) __doc__mindformers.tools.registerrrrrr strrrrrrs pengcheng.mind-test/mindformers/models/__pycache__/build_tokenizer.cpython-39.pyc000066400000000000000000000046631471400134000305640ustar00rootroot00000000000000a ?cf@sFdZddlmZmZmZddlmZddZd eee e d d d Z dS) zBuild Tokenizer API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)MindFormerBookc Ksd|vr dS|d}tjd|d}|js,dS|}|dd}t}||vrz|rz|j|d\}}||||n>| d} | | vr| } n | d } |j | d} || dS) zNAnd the vocab file path to the config if there is not vocab file in the config vocab_fileNtype tokenizer) module_type class_namelib_path) name_or_pathcheckpoint_name_or_pathr) rget_clsvocab_files_namespoprget_tokenizer_url_support_listkeys*read_files_according_specific_by_tokenizerupdategetget_support_listcache_vocab_files) configkwargsr Z dynamic_classr pathZremote_tokenizer_support_listread_vocab_file_dictread_tokenizer_file_dictrZ support_namerrM/home/ma-user/work/mindformers-baichuan/mindformers/models/build_tokenizer.pycheck_and_add_vocab_file_paths(          r Nr )r default_argsr r cKst|dur|durdS|dur`t|tr>t|ts>tfi|}t|fi|tj|tj|dStj||fi|S)aBuild tokenizer For MindFormer. Instantiate the tokenizer from MindFormerRegister's registry. Args: config (dict): The task tokenizer's config. Default: None. default_args (dict): The default argument of tokenizer API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'tokenizer'. class_name (str): The class name of tokenizer API. Default: None. Return: The function instance of tokenizer API. Examples: >>> from mindformers import build_tokenizer >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> tokenizer_from_config = build_tokenizer(config.processor.tokenizer) N)r!) isinstancedictrr rget_instance_from_cfgr TOKENIZER get_instance)rr!r r rrrrbuild_tokenizer/sr')NNr N) __doc__mindformers.tools.registerrrrmindformer_bookrr r#strr'rrrrs pengcheng.mind-test/mindformers/models/__pycache__/utils.cpython-39.pyc000066400000000000000000000014671471400134000265320ustar00rootroot00000000000000a ?cf@s>dZddlmmZddlmZd edddZd d Z dS) zCheck Model Input Config.Nget_cell_reusefloat16ms_typecCs>t|tjr|S|dkrtjS|dkr,tjStd|dS)z*Convert the string type to MindSpore type.rfloat32zBSupported data type keywords include: [float16, float32], but get N) isinstancemstypeFloatrrKeyErrorrr C/home/ma-user/work/mindformers-baichuan/mindformers/models/utils.pyconvert_mstypes rcCstS)Nrr r r r cell_reusesr)r) __doc__Zmindspore.common.dtypecommondtyper version_controlrstrrrr r r rs  pengcheng.mind-test/mindformers/models/base_config.py000066400000000000000000000243531471400134000233410ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BaseConfig class, which is all model configs' base class """ import os import shutil import yaml from ..mindformer_book import MindFormerBook from ..mindformer_book import print_path_or_list from ..tools import logger from ..tools.register.config import MindFormerConfig from ..models.build_config import build_model_config class BaseConfig(dict): """ Base Config for all models' config Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_config import BaseConfig >>> class MyConfig(BaseConfig): ... _support_list = MindFormerBook.get_config_support_list()['my_model'] ... ... def __init__(self, ... data_size: int = 32, ... net_size: list = [1, 2, 3], ... loss_type: str = "MyLoss", ... checkpoint_name_or_path: str = '', ... **kwargs): ... self.data_size = data_size ... self.net_size = net_size ... loss_type = loss_type ... self.checkpoint_name_or_path = checkpoint_name_or_path ... super(MyConfig, self).__init__(**kwargs) ... >>> mynet = MyModel(MyConfig) >>> output = mynet(input) """ _support_list = [] _model_type = 0 _model_name = 1 def __init__(self, **kwargs): super(BaseConfig, self).__init__() self.update(kwargs) def __getattr__(self, key): if key not in self: return None return self[key] def __setattr__(self, key, value): self[key] = value def __delattr__(self, key): del self[key] def to_dict(self): """ for yaml dump, transform from Config to a strict dict class """ return_dict = {} for key, val in self.items(): if isinstance(val, BaseConfig): val = val.to_dict() return_dict[key] = val return return_dict @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a config by yaml name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}.") if os.path.exists(yaml_name_or_path): if not yaml_name_or_path.endswith(".yaml"): raise ValueError(f"{yaml_name_or_path} should be a .yaml file for model" " config.") config_args = MindFormerConfig(yaml_name_or_path) logger.info("the content in %s is used for" " config building.", yaml_name_or_path) elif yaml_name_or_path not in cls._support_list: raise ValueError(f"{yaml_name_or_path} is not a supported" f" model type or a valid path to model config." f" supported model could be selected from {cls._support_list}.") else: yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) config = build_model_config(config_args.model.model_config) MindFormerBook.set_model_config_to_name(id(config), config_args.model.arch.type) return config def save_pretrained(self, save_directory=None, save_name="mindspore_model"): """ Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. """ if save_directory is None: save_directory = MindFormerBook.get_default_checkpoint_save_folder() if not isinstance(save_directory, str) or not isinstance(save_name, str): raise TypeError(f"save_directory and save_name should be a str," f" but got {type(save_directory)} and {type(save_name)}.") if not os.path.exists(save_directory): os.makedirs(save_directory, exist_ok=True) save_path = os.path.join(save_directory, save_name + ".yaml") parsed_config, removed_list = self._inverse_parse_config() wraped_config = self._wrap_config(parsed_config) for key, val in removed_list: self[key] = val self.remove_type() meraged_dict = {} if os.path.exists(save_path): with open(save_path, 'r') as file_reader: meraged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) file_reader.close() meraged_dict.update(wraped_config) with open(save_path, 'w') as file_pointer: file_pointer.write(yaml.dump(meraged_dict)) file_pointer.close() logger.info("config saved successfully!") def remove_type(self): """remove type caused by save’""" if isinstance(self, BaseConfig): self.pop("type") for key, val in self.items(): if isinstance(val, BaseConfig): val.pop("type") self.update({key: val}) def inverse_parse_config(self): """inverse_parse_config""" val, _ = self._inverse_parse_config() return val def _inverse_parse_config(self): """ Inverse parse config method, which builds yaml file content for model config. Returns: A model config, which follows the yaml content. """ self.update({"type": self.__class__.__name__}) removed_list = [] for key, val in self.items(): if isinstance(val, BaseConfig): val = val.inverse_parse_config() elif not isinstance(val, (str, int, float, bool)): removed_list.append((key, val)) continue self.update({key: val}) for key, _ in removed_list: self.pop(key) return self, removed_list def _wrap_config(self, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. """ model_name = self.pop("model_name", None) if model_name is None: model_name = MindFormerBook.get_model_config_to_name().get(id(config), None) return {"model": {"model_config": config.to_dict(), "arch": {"type": model_name}}} @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list pengcheng.mind-test/mindformers/models/base_model.py000066400000000000000000000407301471400134000231710ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BaseModel """ from typing import Optional import os import shutil import yaml import mindspore as ms from mindspore import nn from mindspore.train.serialization import load_checkpoint, load_param_into_net from ..generation import GeneratorMixin from ..mindformer_book import MindFormerBook, print_path_or_list from .build_config import build_model_config from .base_config import BaseConfig from ..tools.register import MindFormerConfig from ..tools.download_tools import download_with_progress_bar from ..tools.logger import logger from ..tools.utils import try_sync_file class BaseModel(nn.Cell, GeneratorMixin): """ The base model that contains the class method `from_pretained` and `save_pretrained`, any new model that should inherit the class. Note: GeneratorMixin provides the method `generate` that enable the generation for nlp models. Args: config(BaseConfig): The model configuration that inherits the `BaseConfig`. """ _support_list = [] _model_type = 0 _model_name = 1 def __init__(self, config: BaseConfig, **kwargs): super(BaseModel, self).__init__(**kwargs) self.config = config self.default_checkpoint_download_path = None def load_checkpoint(self, config): """ load checkpoint for models. Args: config (ModelConfig): a model config instance, which could have attribute "checkpoint_name_or_path (str)". set checkpoint_name_or_path to a supported model name or a path to checkpoint, to load model weights. """ checkpoint_name_or_path = config.checkpoint_name_or_path if checkpoint_name_or_path: if not isinstance(checkpoint_name_or_path, str): raise TypeError(f"checkpoint_name_or_path should be a str," f" but got {type(checkpoint_name_or_path)}") if os.path.exists(checkpoint_name_or_path): param = load_checkpoint(checkpoint_name_or_path) ckpt_file = checkpoint_name_or_path try: load_param_into_net(self, param) logger.info("weights in %s are loaded", ckpt_file) except RuntimeError: logger.error("the given config and weights in %s are" " mismatched, and weights load failed", ckpt_file) elif checkpoint_name_or_path not in self._support_list: raise ValueError(f"{checkpoint_name_or_path} is not a supported default model" f" or a valid path to checkpoint," f" please select from {self._support_list}.") else: checkpoint_name = checkpoint_name_or_path if checkpoint_name_or_path.startswith('mindspore'): # Adaptation the name of checkpoint at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" checkpoint_name = checkpoint_name_or_path.split('/')[self._model_name] default_checkpoint_download_folder = os.path.join( MindFormerBook.get_xihe_checkpoint_download_folder(), checkpoint_name.split('_')[self._model_type]) else: # Default the name of checkpoint, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" default_checkpoint_download_folder = os.path.join( MindFormerBook.get_default_checkpoint_download_folder(), checkpoint_name_or_path.split("_")[self._model_type]) if not os.path.exists(default_checkpoint_download_folder): os.makedirs(default_checkpoint_download_folder, exist_ok=True) ckpt_file = os.path.join(default_checkpoint_download_folder, checkpoint_name + ".ckpt") if not os.path.exists(ckpt_file): url = MindFormerBook.get_model_ckpt_url_list()[checkpoint_name_or_path][self._model_type] succeed = download_with_progress_bar(url, ckpt_file) if not succeed: logger.info("checkpoint download failed, and pretrained weights are unloaded.") return try_sync_file(ckpt_file) self.default_checkpoint_download_path = ckpt_file logger.info("start to read the ckpt file: %s", os.path.getsize(ckpt_file)) param = load_checkpoint(ckpt_file) try: load_param_into_net(self, param) logger.info("weights in %s are loaded", ckpt_file) except RuntimeError: logger.error("the given config and weights in %s are" " mismatched, and weights load failed", ckpt_file) else: logger.info("model built, but weights is unloaded, since the config has no" " checkpoint_name_or_path attribute or" " checkpoint_name_or_path is None.") def save_pretrained(self, save_directory: Optional[str] = None, save_name: str = "mindspore_model"): """ Save the model weight and configuration file. (only supports standalone mode, and distribute mode waits for developing) Args: save_directory(str): a directory to save the model weight and configuration. If None, the directory will be `./checkpoint_save`, which can be obtained by the `MindFormerBook.get_default_checkpoint_save_folder()`. If set, the directory will be what is set. save_name(str): the name of saved files, including model weight and configuration file. Default mindspore_model. Examples: >>> import os >>> from mindformers import T5ForConditionalGeneration, MindFormerBook >>> net = T5ForConditionalGeneration.from_pretrained('t5_small') >>> net.save_pretrained() >>> output_path = MindFormerBook.get_default_checkpoint_save_folder() >>> print(os.listdir(output_path)) ['mindspore_model.yaml', 'mindspore_model.ckpt'] """ if save_directory is None: save_directory = MindFormerBook.get_default_checkpoint_save_folder() if not isinstance(save_directory, str) or not isinstance(save_name, str): raise TypeError(f"save_directory and save_name should be a str," f" but got {type(save_directory)} and {type(save_name)}.") if not os.path.exists(save_directory): os.makedirs(save_directory, exist_ok=True) checkpoint_path = os.path.join(save_directory, save_name + '.ckpt') config_path = os.path.join(save_directory, save_name + '.yaml') ms.save_checkpoint(self, checkpoint_path) if self.config is None: # A model should have "config" attribute for model save. raise AttributeError("the model has no config attribute.") parsed_config, remove_list = self._inverse_parse_config(self.config) wraped_config = self._wrap_config(parsed_config) for key, val in remove_list: self.config[key] = val self.remove_type(self.config) meraged_dict = {} if os.path.exists(config_path): with open(config_path, 'r') as file_reader: meraged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) file_reader.close() meraged_dict.update(wraped_config) with open(config_path, 'w') as file_pointer: file_pointer.write(yaml.dump(meraged_dict)) file_pointer.close() logger.info("model saved successfully!") def remove_type(self, config): """remove type caused by save’""" if isinstance(config, BaseConfig): config.pop("type") for key, val in config.items(): if isinstance(val, BaseConfig): val.pop("type") config.update({key: val}) def _inverse_parse_config(self, config): """ Inverse parse config method, which builds yaml file content for model config. Args: config (BaseConfig): a model config inherited from BaseConfig. Returns: A model config, which follows the yaml content. """ removed_list = [] if not isinstance(config, BaseConfig): return config, removed_list class_name = config.__class__.__name__ config.update({"type": class_name}) for key, val in config.items(): new_val, _ = self._inverse_parse_config(val) if not isinstance(new_val, (str, int, float, bool, BaseConfig)): removed_list.append((key, new_val)) continue config.update({key: new_val}) for key, _ in removed_list: config.pop(key) return config, removed_list def _wrap_config(self, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. """ model_name = self.__class__.__name__ return {"model": {"model_config": config.to_dict(), "arch": {"type": model_name}}} @classmethod def from_pretrained(cls, pretrained_model_name_or_dir: str, **kwargs): """ Instantiates a model by the pretrained_model_name_or_dir. It download the model weights if the user pass a model name, or load the weight from the given directory if given the path. (only support standalone mode, and distribute mode waits for developing!) Args: pretrained_model_name_or_dir (str): It supports the following two input types. If `pretrained_model_name_or_dir` is a supported model name, for example, `vit_base_p16` and `t5_small`, it will download the necessary files from the cloud. User can pass one from the support list by call `MindFormerBook.get_model_support_list()`. If `pretrained_model_name_or_dir` is a path to the local directory where there should have model weights ended with `.ckpt` and configuration file ended with `yaml`. pretrained_model_name_or_path (Optional[str]): Equal to "pretrained_model_name_or_dir", if "pretrained_model_name_or_path" is set, "pretrained_model_name_or_dir" is useless. Examples: >>> from mindformers import T5ForConditionalGeneration >>> net = T5ForConditionalGeneration.from_pretrained('t5_small') Returns: A model, which inherited from BaseModel. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: pretrained_model_name_or_dir = pretrained_model_name_or_path if not isinstance(pretrained_model_name_or_dir, str): raise TypeError(f"pretrained_model_name_or_dir should be a str," f" but got {type(pretrained_model_name_or_dir)}") is_exist = os.path.exists(pretrained_model_name_or_dir) is_dir = os.path.isdir(pretrained_model_name_or_dir) if not is_exist and (pretrained_model_name_or_dir not in cls._support_list): raise ValueError(f'{pretrained_model_name_or_dir} does not exist,' f' or it is not supported by {cls.__name__}. ' f'please select from {cls._support_list}.') if is_exist and not is_dir: raise ValueError(f"{pretrained_model_name_or_dir} is not a directory.") if is_dir: yaml_list = [file for file in os.listdir(pretrained_model_name_or_dir) if file.endswith(".yaml")] ckpt_list = [file for file in os.listdir(pretrained_model_name_or_dir) if file.endswith(".ckpt")] if not yaml_list or not ckpt_list: raise FileNotFoundError(f"there is no yaml file for model config or ckpt file " f"for model weights in {pretrained_model_name_or_dir}.") yaml_file = os.path.join(pretrained_model_name_or_dir, yaml_list[cls._model_type]) ckpt_file = os.path.join(pretrained_model_name_or_dir, ckpt_list[cls._model_type]) logger.info("config in %s and weights in %s are used for " "model building.", yaml_file, ckpt_file) config_args = MindFormerConfig(yaml_file) config_args.model.model_config.update(**kwargs) config = build_model_config(config_args.model.model_config) config.update({"checkpoint_name_or_path": ckpt_file}) model = cls(config) else: pretrained_model_name = pretrained_model_name_or_dir if pretrained_model_name_or_dir.startswith('mindspore'): # Adaptation the name of pretrained model at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" pretrained_model_name = pretrained_model_name.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), pretrained_model_name.split('_')[cls._model_type]) else: # Default the name of pretrained model, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), pretrained_model_name.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, pretrained_model_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(pretrained_model_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') try_sync_file(yaml_file) config_args = MindFormerConfig(yaml_file) config_args.model.model_config.update(**kwargs) config = build_model_config(config_args.model.model_config) config.update({"checkpoint_name_or_path": pretrained_model_name_or_dir}) model = cls(config) logger.info("model built successfully!") return model @classmethod def show_support_list(cls): """show_support_list method""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get_support_list method""" return cls._support_list pengcheng.mind-test/mindformers/models/base_processor.py000066400000000000000000000321171471400134000241100ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BaseProcessor """ import os import shutil import yaml from ..mindformer_book import print_path_or_list, MindFormerBook from .build_processor import build_processor from .base_tokenizer import BaseTokenizer from ..tools import logger from ..tools.register import MindFormerConfig class BaseImageProcessor: """ BaseImageProcessor for all image preprocess. Examples: >>> from mindspore.dataset.vision.transforms import CenterCrop >>> from mindformers.models.base_processor import BaseImageProcessor >>> image_resolution = 224 >>> class MyImageProcessor(BaseImageProcessor): ... def __init__(self, image_resolution): ... super(MyImageProcessor, self).__init__(image_resolution=image_resolution) ... self.center_crop = CenterCrop(image_resolution) ... ... def preprocess(self, images, **kwargs): ... res = [] ... for image in images: ... image = self.center_crop(image) ... res.append(image) ... return res ... >>> my_image_processor = MyImageProcessor(image_resolution) >>> output = my_image_processor(image) """ def __init__(self, **kwargs): self.config = {} self.config.update(kwargs) def __call__(self, image_data, **kwargs): """forward process""" return self.preprocess(image_data, **kwargs) def preprocess(self, images, **kwargs): """preprocess method""" raise NotImplementedError("Each image processor must implement its own preprocess method") class BaseAudioProcessor: """ BaseAudioProcessor for all audio preprocess. Examples: >>> from mindspore.dataset.audio import AllpassBiquad >>> from mindformers.models.base_processor import BaseAudioProcessor >>> sample_rate = 44100 >>> central_freq = 200.0 >>> class MyAudioProcessor(BaseAudioProcessor): ... def __init__(self, audio_property): ... super(MyAudioProcessor, self).__init__(sample_rate=sample_rate, central_freq=central_freq ) ... self.all_pass_biquad = AllpassBiquad(44100, 200.0) ... ... def preprocess(self, audio_data, **kwargs): ... res = [] ... for audio in audio_data: ... audio = self.all_pass_biquad(audio) ... res.append(audio) ... return res ... >>> my_audio_processor = MyAudioProcessor(sample_rate, central_freq) >>> output = my_audio_processor(audio) """ def __init__(self, **kwargs): self.config = {} self.config.update(kwargs) def __call__(self, audio_data, **kwargs): """forward process""" return self.preprocess(audio_data, **kwargs) def preprocess(self, audio_data, **kwargs): """preprocess method""" raise NotImplementedError("Each audio processor must implement its own preprocess method") class BaseProcessor: """ Base processor Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_processor import BaseProcessor >>> class MyProcessor(BaseProcessor): ... _support_list = MindFormerBook.get_processor_support_list()['my_model'] ... ... def __init__(self, image_processor=None, audio_processor=None, tokenizer=None, return_tensors='ms'): ... super(MyProcessor, self).__init__( ... image_processor=image_processor, ... audio_processor=audio_processor, ... tokenizer=tokenizer, ... return_tensors=return_tensors) ... >>> myprocessor = MyProcessor(image_processor, audio_processor, tokenizer) >>> output = mynet(image, audio, text) """ _support_list = [] _model_type = 0 _model_name = 1 def __init__(self, **kwargs): self.config = {} self.config.update(kwargs) self.image_processor = kwargs.pop("image_processor", None) self.audio_processor = kwargs.pop("audio_processor", None) self.tokenizer = kwargs.pop("tokenizer", None) self.max_length = kwargs.pop("max_length", None) self.padding = kwargs.pop("padding", False) self.return_tensors = kwargs.pop("return_tensors", None) def __call__(self, image_input=None, text_input=None): """call function""" output = {} if image_input is not None and self.image_processor: if not isinstance(self.image_processor, BaseImageProcessor): raise TypeError(f"feature_extractor should inherit from the BaseImageProcessor," f" but got {type(self.image_processor)}.") image_output = self.image_processor(image_input) output['image'] = image_output if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output def save_pretrained(self, save_directory=None, save_name="mindspore_model"): """ Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. """ if save_directory is None: save_directory = MindFormerBook.get_default_checkpoint_save_folder() if not isinstance(save_directory, str) or not isinstance(save_name, str): raise TypeError(f"save_directory and save_name should be a str," f" but got {type(save_directory)} and {type(save_name)}.") if not os.path.exists(save_directory): os.makedirs(save_directory, exist_ok=True) parsed_config = self._inverse_parse_config(self.config) wraped_config = self._wrap_config(parsed_config) config_path = os.path.join(save_directory, save_name + '.yaml') meraged_dict = {} if os.path.exists(config_path): with open(config_path, 'r') as file_reader: meraged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) file_reader.close() meraged_dict.update(wraped_config) with open(config_path, 'w') as file_pointer: file_pointer.write(yaml.dump(meraged_dict)) file_pointer.close() logger.info("processor saved successfully!") def _inverse_parse_config(self, config): """ Inverse parse config method, which builds yaml file content for feature extractor config. Args: Config (dict): a dict, which contains input parameters of feature extractor. Returns: A dict, which follows the yaml content. """ parsed_config = {"type": self.__class__.__name__} for key, val in config.items(): if isinstance(val, BaseTokenizer): parsed_sub_config = {"type": val.__class__.__name__} parsed_sub_config.update(val.init_kwargs) parsed_config.update({key: parsed_sub_config}) elif isinstance(val, (BaseImageProcessor, BaseAudioProcessor)): parsed_sub_config = {"type": val.__class__.__name__} parsed_sub_config.update(val.config) parsed_config.update({key: parsed_sub_config}) else: parsed_config.update({key: val}) return parsed_config def _wrap_config(self, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (dict): a dict processed by _inverse_parse_config function. Returns: A dict for yaml.dump. """ return {"processor": config} @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a processor by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A processor which inherited from BaseProcessor. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}") is_exist = os.path.exists(yaml_name_or_path) if not is_exist and yaml_name_or_path not in cls._support_list: raise ValueError(f'{yaml_name_or_path} does not exist,' f' or it is not supported by {cls.__name__}. ' f'please select from {cls._support_list}.') if is_exist: logger.info("config in %s is used for processor" " building.", yaml_name_or_path) config_args = MindFormerConfig(yaml_name_or_path) else: yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) processor = build_processor(config_args.processor) logger.info("processor built successfully!") return processor @classmethod def show_support_list(cls): """show_support_list""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get_support_list method""" return cls._support_list pengcheng.mind-test/mindformers/models/base_tokenizer.py000066400000000000000000006074701471400134000241150ustar00rootroot00000000000000# coding=utf-8 # Copyright 2020 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Base Tokenizer for the pretrained tokenizer""" import copy import os import re import json import shutil import warnings import unicodedata import bisect import itertools from dataclasses import dataclass, field from enum import Enum from typing import Any, Dict, List, NamedTuple, Optional, Sequence, Tuple, Union from collections import OrderedDict, UserDict from collections.abc import Mapping, Sized from contextlib import contextmanager import yaml import numpy as np import mindspore as ms from mindformers.tools import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .build_tokenizer import build_tokenizer from ..tools.download_tools import download_with_progress_bar from ..tools.utils import try_sync_file from ..mindformer_book import MindFormerBook __all__ = ['BaseTokenizer', 'Tokenizer', 'SpecialTokensMixin'] SPECIAL_TOKEN_FILE_NAME = 'special_tokens_map.json' TOKENIZER_CONFIG_NAME = 'tokenizer_config.json' VERY_LARGE_INTEGER = int(1e30) # This is used to set the max input length for a model with infinite size input LARGE_INTEGER = int(1e20) # This is used when we need something big but slightly smaller than VERY_LARGE_INTEGER # Define type aliases and NamedTuples TextInput = str PreTokenizedInput = List[str] EncodedInput = List[int] TextInputPair = Tuple[str, str] PreTokenizedInputPair = Tuple[List[str], List[str]] EncodedInputPair = Tuple[List[int], List[int]] # Slow tokenizers used to be saved in three separated files SPECIAL_TOKENS_MAP_FILE = "special_tokens_map.json" ADDED_TOKENS_FILE = "added_tokens.json" TOKENIZER_CONFIG_FILE = "tokenizer_config.json" # Fast tokenizers (provided by HuggingFace tokenizer's library) can be saved in a single file FULL_TOKENIZER_FILE = "tokenizer.json" _re_tokenizer_file = re.compile(r"tokenizer\.(.*)\.json") class ExplicitEnum(str, Enum): """ Enum with more explicit error message for missing values. """ @classmethod def _missing_(cls, value): raise ValueError( f"{value} is not a valid {cls.__name__}, please select one of {list(cls._value2member_map_.keys())}" ) class TensorType(ExplicitEnum): """ Possible values for the `return_tensors` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. """ NUMPY = "np" MINDSPORE = "ms" @dataclass(frozen=True, eq=True) class AddedToken: """ AddedToken represents a token to be added to a Tokenizer An AddedToken can have special options defining the way it should behave. """ content: str = field(default_factory=str) single_word: bool = False lstrip: bool = False rstrip: bool = False normalized: bool = True def __getstate__(self): return self.__dict__ def __str__(self): return self.content @dataclass class EncodingFast: """This is dummy class because without the `tokenizers` library we don't have these objects anyway""" class TruncationStrategy(ExplicitEnum): """ Possible values for the `truncation` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. """ ONLY_FIRST = "only_first" ONLY_SECOND = "only_second" LONGEST_FIRST = "longest_first" DO_NOT_TRUNCATE = "do_not_truncate" class CharSpan(NamedTuple): """ Character span in the original string. Args: start (`int`): Index of the first character in the original string. end (`int`): Index of the character following the last character in the original string. """ start: int end: int class TokenSpan(NamedTuple): """ Token span in an encoded string (list of tokens). Args: start (`int`): Index of the first token in the span. end (`int`): Index of the token following the last token in the span. """ start: int end: int class BatchEncoding(UserDict): """ Holds the output of the [`~tokenization_utils_base.PreTrainedTokenizerBase.__call__`], [`~tokenization_utils_base.PreTrainedTokenizerBase.encode_plus`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.batch_encode_plus`] methods (tokens, attention_masks, etc). This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes utility methods to map from word/character space to token space. Args: data (`dict`): Dictionary of lists/arrays/tensors returned by the `__call__`/`encode_plus`/`batch_encode_plus` methods ('input_ids', 'attention_mask', etc.). encoding (`tokenizers.Encoding` or `Sequence[tokenizers.Encoding]`, *optional*): If the tokenizer is a fast tokenizer which outputs additional information like mapping from word/character space to token space the `tokenizers.Encoding` instance or list of instance (for batches) hold this information. tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in Mindspore/Numpy Tensors at initialization. prepend_batch_axis (`bool`, *optional*, defaults to `False`): Whether or not to add a batch axis when converting to tensors (see `tensor_type` above). n_sequences (`Optional[int]`, *optional*): You can give a tensor_type here to convert the lists of integers in Mindspore/Numpy Tensors at initialization. """ def __init__( self, data: Optional[Dict[str, Any]] = None, encoding: Optional[Union[EncodingFast, Sequence[EncodingFast]]] = None, tensor_type: Union[None, str, TensorType] = None, prepend_batch_axis: bool = False, n_sequences: Optional[int] = None, ): super().__init__(data) if isinstance(encoding, EncodingFast): encoding = [encoding] self._encodings = encoding if n_sequences is None and encoding is not None and encoding: n_sequences = encoding[0].n_sequences self._n_sequences = n_sequences self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis) @property def n_sequences(self) -> Optional[int]: """ `Optional[int]`: The number of sequences used to generate each sample from the batch encoded in this [`BatchEncoding`]. Currently can be one of `None` (unknown), `1` (a single sentence) or `2` (a pair of sentences) """ return self._n_sequences @property def is_fast(self) -> bool: """ `bool`: Indicate whether this [`BatchEncoding`] was generated from the result of a [`PreTrainedTokenizerFast`] or not. """ return self._encodings is not None def __getitem__(self, item: Union[int, str]) -> Union[Any, EncodingFast]: """ If the key is a string, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.). If the key is an integer, get the `tokenizers.Encoding` for batch item with index `key`. If the key is a slice, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.) with the constraint of slice. """ if isinstance(item, str): return self.data[item] if self._encodings is not None: return self._encodings[item] if isinstance(item, slice): return {key: self.data[key][item] for key in self.data.keys()} raise KeyError( "Invalid key. Only three types of key are available: " "(1) string, (2) integers for backend Encoding, and (3) slices for data subsetting." ) def __getattr__(self, item: str): try: return self.data[item] except KeyError: raise AttributeError def __getstate__(self): return {"data": self.data, "encodings": self._encodings} def __setstate__(self, state): if "data" in state: self.data = state["data"] if "encodings" in state: self._encodings = state["encodings"] def keys(self): return self.data.keys() def values(self): return self.data.values() def items(self): return self.data.items() # After this point: # Extended properties and methods only available for fast (Rust-based) tokenizers # provided by HuggingFace tokenizers library. @property def encodings(self) -> Optional[List[EncodingFast]]: """ `Optional[List[tokenizers.Encoding]]`: The list all encodings from the tokenization process. Returns `None` if the input was tokenized through Python (i.e., not a fast) tokenizer. """ return self._encodings def tokens(self, batch_index: int = 0) -> List[str]: """ Return the list of tokens (sub-parts of the input strings after word/subword splitting and before conversion to integer indices) at a given batch index (only works for the output of a fast tokenizer). Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[str]`: The list of tokens at that index. """ if not self._encodings: raise ValueError( "tokens() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`" " class)." ) return self._encodings[batch_index].tokens def sequence_ids(self, batch_index: int = 0) -> List[Optional[int]]: """ Return a list mapping the tokens to the id of their original sentences: - `None` for special tokens added around or between sequences, - `0` for tokens corresponding to words in the first sequence, - `1` for tokens corresponding to words in the second sequence when a pair of sequences was jointly encoded. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the sequence id corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding sequence. """ if not self._encodings: raise ValueError( "sequence_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`" " class)." ) return self._encodings[batch_index].sequence_ids def words(self, batch_index: int = 0) -> List[Optional[int]]: """ Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word). """ if not self._encodings: raise ValueError( "words() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`" " class)." ) warnings.warn( "`BatchEncoding.words()` property is deprecated and should be replaced with the identical, " "but more self-explanatory `BatchEncoding.word_ids()` property.", FutureWarning, ) return self.word_ids(batch_index) def word_ids(self, batch_index: int = 0) -> List[Optional[int]]: """ Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word). """ if not self._encodings: raise ValueError( "word_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`" " class)." ) return self._encodings[batch_index].word_ids def token_to_sequence(self, batch_or_token_index: int, token_index: Optional[int] = None) -> int: """ Get the index of the sequence represented by the given token. In the general use case, this method returns `0` for a single sequence or the first sequence of a pair, and `1` for the second sequence of a pair Can be called as: - `self.token_to_sequence(token_index)` if batch size is 1 - `self.token_to_sequence(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. """ if not self._encodings: raise ValueError("token_to_sequence() is not available when using Python based tokenizers") if token_index is not None: batch_index = batch_or_token_index else: batch_index = 0 token_index = batch_or_token_index if batch_index < 0: batch_index = self._batch_size + batch_index if token_index < 0: token_index = self._seq_len + token_index return self._encodings[batch_index].token_to_sequence(token_index) def token_to_word(self, batch_or_token_index: int, token_index: Optional[int] = None) -> int: """ Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch. Can be called as: - `self.token_to_word(token_index)` if batch size is 1 - `self.token_to_word(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. """ if not self._encodings: raise ValueError("token_to_word() is not available when using Python based tokenizers") if token_index is not None: batch_index = batch_or_token_index else: batch_index = 0 token_index = batch_or_token_index if batch_index < 0: batch_index = self._batch_size + batch_index if token_index < 0: token_index = self._seq_len + token_index return self._encodings[batch_index].token_to_word(token_index) def word_to_tokens( self, batch_or_word_index: int, word_index: Optional[int] = None, sequence_index: int = 0 ) -> Optional[TokenSpan]: """ Get the encoded token span corresponding to a word in a sequence of the batch. Token spans are returned as a [`~tokenization_utils_base.TokenSpan`] with: - **start** -- Index of the first token. - **end** -- Index of the token following the last token. Can be called as: - `self.word_to_tokens(word_index, sequence_index: int = 0)` if batch size is 1 - `self.word_to_tokens(batch_index, word_index, sequence_index: int = 0)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the word in the sequence. word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: ([`~tokenization_utils_base.TokenSpan`], *optional*): Span of tokens in the encoded sequence. Returns `None` if no tokens correspond to the word. This can happen especially when the token is a special token that has been used to format the tokenization. For example when we add a class token at the very beginning of the tokenization. """ if not self._encodings: raise ValueError("word_to_tokens() is not available when using Python based tokenizers") if word_index is not None: batch_index = batch_or_word_index else: batch_index = 0 word_index = batch_or_word_index if batch_index < 0: batch_index = self._batch_size + batch_index if word_index < 0: word_index = self._seq_len + word_index span = self._encodings[batch_index].word_to_tokens(word_index, sequence_index) return TokenSpan(*span) if span is not None else None def token_to_chars(self, batch_or_token_index: int, token_index: Optional[int] = None) -> CharSpan: """ Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a [`~tokenization_utils_base.CharSpan`] with: - **start** -- Index of the first character in the original string associated to the token. - **end** -- Index of the character following the last character in the original string associated to the token. Can be called as: - `self.token_to_chars(token_index)` if batch size is 1 - `self.token_to_chars(batch_index, token_index)` if batch size is greater or equal to 1 Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token or tokens in the sequence. Returns: [`~tokenization_utils_base.CharSpan`]: Span of characters in the original string, or None, if the token (e.g. , ) doesn't correspond to any chars in the origin string. """ if not self._encodings: raise ValueError("token_to_chars() is not available when using Python based tokenizers") if token_index is not None: batch_index = batch_or_token_index else: batch_index = 0 token_index = batch_or_token_index span_indices = self._encodings[batch_index].token_to_chars(token_index) return CharSpan(*span_indices) if span_indices is not None else None def char_to_token( self, batch_or_char_index: int, char_index: Optional[int] = None, sequence_index: int = 0 ) -> int: """ Get the index of the token in the encoded output comprising a character in the original string for a sequence of the batch. Can be called as: - `self.char_to_token(char_index)` if batch size is 1 - `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int`: Index of the token. """ if not self._encodings: raise ValueError("char_to_token() is not available when using Python based tokenizers") if char_index is not None: batch_index = batch_or_char_index else: batch_index = 0 char_index = batch_or_char_index return self._encodings[batch_index].char_to_token(char_index, sequence_index) def word_to_chars( self, batch_or_word_index: int, word_index: Optional[int] = None, sequence_index: int = 0 ) -> CharSpan: """ Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned as a CharSpan NamedTuple with: - start: index of the first character in the original string - end: index of the character following the last character in the original string Can be called as: - `self.word_to_chars(word_index)` if batch size is 1 - `self.word_to_chars(batch_index, word_index)` if batch size is greater or equal to 1 Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: `CharSpan` or `List[CharSpan]`: Span(s) of the associated character or characters in the string. CharSpan are NamedTuple with: - start: index of the first character associated to the token in the original string - end: index of the character following the last character associated to the token in the original string """ if not self._encodings: raise ValueError("word_to_chars() is not available when using Python based tokenizers") if word_index is not None: batch_index = batch_or_word_index else: batch_index = 0 word_index = batch_or_word_index return CharSpan(*(self._encodings[batch_index].word_to_chars(word_index, sequence_index))) def char_to_word(self, batch_or_char_index: int, char_index: Optional[int] = None, sequence_index: int = 0) -> int: """ Get the word in the original string corresponding to a character in the original string of a sequence of the batch. Can be called as: - `self.char_to_word(char_index)` if batch size is 1 - `self.char_to_word(batch_index, char_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the character in the original string. char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the character in the original string. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int` or `List[int]`: Index or indices of the associated encoded token(s). """ if not self._encodings: raise ValueError("char_to_word() is not available when using Python based tokenizers") if char_index is not None: batch_index = batch_or_char_index else: batch_index = 0 char_index = batch_or_char_index return self._encodings[batch_index].char_to_word(char_index, sequence_index) def convert_to_tensors( self, tensor_type: Optional[Union[str, TensorType]] = None, prepend_batch_axis: bool = False ): """ Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. prepend_batch_axis (`int`, *optional*, defaults to `False`): Whether or not to add the batch dimension during the conversion. """ if tensor_type is None: return self # Convert to TensorType if not isinstance(tensor_type, TensorType): tensor_type = TensorType(tensor_type) # Get a function reference for the correct framework if tensor_type == TensorType.MINDSPORE: tensor_dtype = ms.int32 as_tensor = ms.Tensor def is_ms_tensor(x): return isinstance(x, ms.Tensor) is_tensor = is_ms_tensor else: def as_tensor(value, dtype=None): if isinstance(value, (list, tuple)) and isinstance(value[0], (list, tuple, np.ndarray)): value_lens = [len(val) for val in value] if len(set(value_lens)) > 1 and dtype is None: # we have a ragged list so handle explicitly value = as_tensor([np.asarray(val) for val in value], dtype=object) return np.asarray(value, dtype=dtype) def is_numpy_array(x): return isinstance(x, np.ndarray) tensor_dtype = np.int32 is_tensor = is_numpy_array # Do the tensor conversion in batch for key, value in self.items(): try: if prepend_batch_axis: # value = [value] pass if not is_tensor(value): tensor = as_tensor(value, dtype=tensor_dtype) # Removing this for now in favor of controlling the shape with `prepend_batch_axis` # # at-least2d # if tensor.ndim > 2: # tensor = tensor.squeeze(0) # elif tensor.ndim < 2: # tensor = tensor[None, :] self[key] = tensor except Exception as e: if key == "overflowing_tokens": raise ValueError( "Unable to create tensor returning overflowing tokens of different lengths. " "Please see if a fast version of this tokenizer is available to have this feature available." ) from e raise ValueError( "Unable to create tensor, you should probably activate truncation and/or padding with" " 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your" f" features (`{key}` in this case) have excessive nesting (inputs type `list` where type `int` is" " expected)." ) from e return self class PaddingStrategy(ExplicitEnum): """ Possible values for the `padding` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. """ LONGEST = "longest" MAX_LENGTH = "max_length" DO_NOT_PAD = "do_not_pad" class Trie: """ Trie in Python. Creates a Trie out of a list of words. The trie is used to split on `added_tokens` in one pass Loose reference https://en.wikipedia.org/wiki/Trie """ def __init__(self): self.data = {} def add(self, word: str): """ Passes over every char (utf-8 char) on word and recursively adds it to the internal `data` trie representation. The special key `""` is used to represent termination. This function is idempotent, adding twice the same word will leave the trie unchanged Example: ```python >>> trie = Trie() >>> trie.add("Hello 友達") >>> trie.data {"H": {"e": {"l": {"l": {"o": {" ": {"友": {"達": {"": 1}}}}}}}}} >>> trie.add("Hello") >>> trie.data {"H": {"e": {"l": {"l": {"o": {"": 1, " ": {"友": {"達": {"": 1}}}}}}}}} ``` """ if not word: # Prevent empty string return ref = self.data for char in word: ref[char] = ref[char] if char in ref else {} ref = ref[char] ref[""] = 1 def split(self, text: str) -> List[str]: """ Will look for the words added to the trie within `text`. Output is the original string split along the boundaries of the words found. This trie will match the longest possible word first ! Example: ```python >>> trie = Trie() >>> trie.split("[CLS] This is a extra_id_100") ["[CLS] This is a extra_id_100"] >>> trie.add("[CLS]") >>> trie.add("extra_id_1") >>> trie.add("extra_id_100") >>> trie.split("[CLS] This is a extra_id_100") ["[CLS]", " This is a ", "extra_id_100"] ``` """ # indexes are counted left of the chars index. # "hello", index 0, is left of h, index 1 is between h and e. # index 5 is right of the "o". # States are going to capture every possible start (indexes as above) # as keys, and have as values, a pointer to the position in the trie # where we're at. This is a partial match for now. # This enables to keep track of multiple matches while we're iterating # the string # If the trie contains, "blowing", and "lower" and we encounter the # string "blower", we need to split into ["b", "lower"]. # This is where we need to keep track of multiple possible starts. states = OrderedDict() # This will contain every indices where we need # to cut. # We force to cut at offset 0 and len(text) (added later) offsets = [0] # This is used by the lookahead which needs to skip over # some text where the full match exceeded the place in the initial # for loop skip = 0 # Main loop, Giving this algorithm O(n) complexity for current, current_char in enumerate(text): if skip and current < skip: # Prevents the lookahead for matching twice # like extra_id_100 and id_100 continue # This will track every state # that stop matching, we need to stop tracking them. # If we look at "lowball", we're going to match "l" (add it to states), "o", "w", then # fail on "b", we need to remove 0 from the valid states. to_remove = set() # Whenever we found a match, we need to drop everything # this is a greedy algorithm, it will match on the first found token reset = False # In this case, we already have partial matches (But unfinished) for start, trie_pointer in states.items(): if "" in trie_pointer: # This is a final match, we need to reset and # store the results in `offsets`. # Lookahead to match longest first # Important in case of extra_id_1 vs extra_id_100 # Here we are also actively looking for other earlier partial # matches # "[CLS]", "L", we need to match CLS even if L is special states, start, end, skip = \ self.split_atom_3(states=states, current=current, text=text, start=start, skip=skip) offsets.append(start) offsets.append(end) reset = True break elif current_char in trie_pointer: # The current character being looked at has a match within the trie # update the pointer (it will be stored back into states later). trie_pointer = trie_pointer[current_char] # Storing back the new pointer into the states. # Partial matches got longer by one. states[start] = trie_pointer else: # The new character has not match in the trie, we need # to stop keeping track of this partial match. # We can't do it directly within the loop because of how # python iteration works to_remove.add(start) # Either clearing the full start (we found a real match) # Or clearing only the partial matches that didn't work. states = self.split_atom_2(reset=reset, to_remove=to_remove, states=states) # If this character is a starting character within the trie # start keeping track of this partial match. if current >= skip and current_char in self.data: states[current] = self.data[current_char] # We have a cut at the end with states. offsets = self.split_atom_1(states=states, text=text, offsets=offsets) return self.cut_text(text, offsets) def split_atom_1(self, states=None, text=None, offsets=None): """atomic act of split""" for start, trie_pointer in states.items(): if "" in trie_pointer: # This is a final match, we need to reset and # store the results in `offsets`. end = len(text) offsets.append(start) offsets.append(end) # Longest cut is always the one with lower start so the first # item so we need to break. break return offsets def split_atom_2(self, reset=None, to_remove=None, states=None): """atomic act of split""" if reset: states = {} else: for start in to_remove: del states[start] return states def split_atom_3(self, states=None, current=None, text=None, start=None, skip=None): """atomic act of split""" for lookstart, looktrie_pointer in states.items(): if lookstart > start: # This partial match is later, we can stop looking break elif lookstart < start: # This partial match is earlier, the trie pointer # was already updated, so index is + 1 lookahead_index = current + 1 end = current + 1 else: # Here lookstart == start and # looktrie_pointer == trie_pointer # It wasn't updated yet so indices are current ones lookahead_index = current end = current next_char = text[lookahead_index] if lookahead_index < len(text) else None if "" in looktrie_pointer: start = lookstart end = lookahead_index skip = lookahead_index while next_char in looktrie_pointer: looktrie_pointer = looktrie_pointer[next_char] lookahead_index += 1 if "" in looktrie_pointer: start = lookstart end = lookahead_index skip = lookahead_index if lookahead_index == len(text): # End of string break next_char = text[lookahead_index] return states, start, end, skip def cut_text(self, text, offsets): """cut thr text.""" # We have all the offsets now, we just need to do the actual splitting. # We need to eventually add the first part of the string and the eventual # last part. offsets.append(len(text)) tokens = [] start = 0 for end in offsets: if start > end: logger.error( "There was a bug in Trie algorithm in tokenization. Attempting to recover. Please report it" " anyway." ) continue elif start == end: # This might happen if there's a match at index 0 # we're also preventing zero-width cuts in case of two # consecutive matches continue tokens.append(text[start:end]) start = end return tokens ENCODE_KWARGS_DOCSTRING = r""" add_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to encode the sequences with the special tokens relative to their model. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. stride (`int`, *optional*, defaults to 0): If set to a number along with `max_length`, the overflowing tokens returned when `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflowing sequences. The value of this argument defines the number of overlapping tokens. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. Requires `padding` to be activated. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return Numpy `ms.Tensor` objects. """ ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING = r""" return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are token type IDs?](../glossary#token-type-ids) return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_overflowing_tokens (`bool`, *optional*, defaults to `False`): Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead of returning overflowing tokens. return_special_tokens_mask (`bool`, *optional*, defaults to `False`): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*, defaults to `False`): Whether or not to return `(char_start, char_end)` for each token. This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using Python's tokenizer, this method will raise `NotImplementedError`. return_length (`bool`, *optional*, defaults to `False`): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. **kwargs: passed to the `self.tokenize()` method Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or if *"token_type_ids"* is in `self.model_input_names`). [What are token type IDs?](../glossary#token-type-ids) - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and `return_overflowing_tokens=True`). - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and `return_overflowing_tokens=True`). - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`). - **length** -- The length of the inputs (when `return_length=True`) """ INIT_TOKENIZER_DOCSTRING = r""" Class attributes (overridden by derived classes) - **vocab_files_names** (`Dict[str, str]`) -- A dictionary with, as keys, the `__init__` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - **pretrained_vocab_files_map** (`Dict[str, Dict[str, str]]`) -- A dictionary of dictionaries, with the high-level keys being the `__init__` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` of the pretrained models with, as associated values, the `url` to the associated pretrained vocabulary file. - **max_model_input_sizes** (`Dict[str, Optional[int]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or `None` if the model has no maximum input size. - **pretrained_init_configuration** (`Dict[str, Dict[str, Any]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, a dictionary of specific arguments to pass to the `__init__` method of the tokenizer class for this pretrained model when loading the tokenizer with the [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`] method. - **model_input_names** (`List[str]`) -- A list of inputs expected in the forward pass of the model. - **padding_side** (`str`) -- The default value for the side on which the model should have padding applied. Should be `'right'` or `'left'`. - **truncation_side** (`str`) -- The default value for the side on which the model should have truncation applied. Should be `'right'` or `'left'`. Args: model_max_length (`int`, *optional*): The maximum length (in number of tokens) for the inputs to the transformer model. When the tokenizer is loaded with [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`], this will be set to the value stored for the associated model in `max_model_input_sizes` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. truncation_side (`str`, *optional*): The side on which the model should have truncation applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. model_input_names (`List[string]`, *optional*): The list of inputs accepted by the forward pass of the model (like `"token_type_ids"` or `"attention_mask"`). Default value is picked from the class attribute of the same name. bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. Will be associated to `self.bos_token` and `self.bos_token_id`. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. Will be associated to `self.eos_token` and `self.eos_token_id`. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. Will be associated to `self.unk_token` and `self.unk_token_id`. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). Will be associated to `self.sep_token` and `self.sep_token_id`. pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. Will be associated to `self.pad_token` and `self.pad_token_id`. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). Will be associated to `self.cls_token` and `self.cls_token_id`. mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). Will be associated to `self.mask_token` and `self.mask_token_id`. additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. Add them here to ensure they won't be split by the tokenization process. Will be associated to `self.additional_special_tokens` and `self.additional_special_tokens_ids`. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. """ def add_end_docstrings(*docstr): def docstring_decorator(fn): fn.__doc__ = (fn.__doc__ if fn.__doc__ is not None else "") + "".join(docstr) return fn return docstring_decorator def to_py_obj(obj): """ Convert a Mindspore tensor, Numpy array or python list to a python list. """ if isinstance(obj, (dict, UserDict)): return {k: to_py_obj(v) for k, v in obj.items()} if isinstance(obj, (list, tuple)): return [to_py_obj(o) for o in obj] if isinstance(obj, ms.Tensor): return obj.asnumpy().tolist() if isinstance(obj, (np.ndarray, np.number)): # tolist also works on 0d np arrays return obj.tolist() return obj def _is_whitespace(char): """Checks whether `char` is a whitespace character.""" # \t, \n, and \r are technically control characters but we treat them # as whitespace since they are generally considered as such. if char in (' ', '\t', '\n', '\r'): return True cat = unicodedata.category(char) if cat == "Zs": return True return False def _is_control(char): """Checks whether `char` is a control character.""" # These are technically control characters but we count them as whitespace # characters. if char in ('\t', '\n', '\r'): return False cat = unicodedata.category(char) if cat.startswith("C"): return True return False def _is_punctuation(char): """Checks whether `char` is a punctuation character.""" cp = ord(char) # We treat all non-letter/number ASCII as punctuation. # Characters such as "^", "$", and "`" are not in the Unicode # Punctuation class but we treat them as punctuation anyways, for # consistency. if (33 <= cp <= 47) or (58 <= cp <= 64) or (91 <= cp <= 96) or (123 <= cp <= 126): return True cat = unicodedata.category(char) if cat.startswith("P"): return True return False def _is_end_of_word(text): """Checks whether the last character in text is one of a punctuation, control or whitespace character.""" last_char = text[-1] return bool(_is_control(last_char) | _is_punctuation(last_char) | _is_whitespace(last_char)) def _is_start_of_word(text): """Checks whether the first character in text is one of a punctuation, control or whitespace character.""" first_char = text[0] return bool(_is_control(first_char) | _is_punctuation(first_char) | _is_whitespace(first_char)) def _insert_one_token_to_ordered_list(token_list: List[str], new_token: str): """ Inserts one token to an ordered list if it does not already exist. Note: token_list must be sorted. """ insertion_idx = bisect.bisect_left(token_list, new_token) # Checks if new_token is already in the ordered token_list if not (insertion_idx < len(token_list) and token_list[insertion_idx] == new_token): # new_token is in token_list, don't add token_list.insert(insertion_idx, new_token) def print_path_or_list(input_path_or_list): """ Print path or list function for show support method of MindFormerBook or other BaseClasses. Args: input_path_or_list (str, list): the path or list to be printed. """ if isinstance(input_path_or_list, (str, list)): logger.info(" %s", input_path_or_list) logger.info("-------------------------------------") else: raise TypeError(f"{type(input_path_or_list)} is unsupported by print_path_or_list") class SpecialTokensMixin: """ A mixin derived by [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`] to handle specific behaviors related to special tokens. In particular, this class hold the attributes which can be used to directly access these special tokens in a model-independent manner and allow to set and update the special tokens. Args: bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. """ SPECIAL_TOKENS_ATTRIBUTES = [ "bos_token", "eos_token", "unk_token", "sep_token", "pad_token", "cls_token", "mask_token", "additional_special_tokens", ] def __init__(self, verbose=True, **kwargs): self._bos_token = None self._eos_token = None self._unk_token = None self._sep_token = None self._pad_token = None self._cls_token = None self._mask_token = None self._pad_token_type_id = 0 self._additional_special_tokens = [] self.verbose = verbose # We directly set the hidden value to allow initialization with special tokens # which are not yet in the vocabulary. Necessary for serialization/de-serialization # TODO clean this up at some point (probably by switching to fast tokenizers) for key, value in kwargs.items(): if value is None: continue if key in self.SPECIAL_TOKENS_ATTRIBUTES: if key == "additional_special_tokens": assert isinstance(value, (list, tuple)), f"Value {value} is not a list or tuple" assert all( isinstance(t, (str, AddedToken)) for t in value ), "One of the tokens is not a string or an AddedToken" setattr(self, key, value) elif isinstance(value, (str, AddedToken)): setattr(self, key, value) else: raise TypeError(f"special token {key} has to be either str or AddedToken but got: {type(value)}") def sanitize_special_tokens(self) -> int: """ Make sure that all the special tokens attributes of the tokenizer (`tokenizer.mask_token`, `tokenizer.cls_token`, etc.) are in the vocabulary. Add the missing ones to the vocabulary if needed. Return: `int`: The number of tokens added in the vocabulary during the operation. """ return self.add_tokens(self.all_special_tokens_extended, special_tokens=True) def add_special_tokens( self, special_tokens_dict: Dict[str, Union[str, AddedToken]], replace_additional_special_tokens=True ) -> int: """ Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary). Note,None When adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Using `add_special_tokens` will ensure your special tokens can be used in several ways: - Special tokens are carefully handled by the tokenizer (they are never split). - You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This makes it easy to develop model-agnostic training and fine-tuning scripts. When possible, special tokens are already registered for provided pretrained models (for instance [`BertTokenizer`] `cls_token` is already registered to be :obj*'[CLS]'* and XLM's one is also registered to be `''`). Args: special_tokens_dict (dictionary *str* to *str* or `tokenizers.AddedToken`): Keys should be in the list of predefined special attributes: [`bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`]. Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). replace_additional_special_tokens (`bool`, *optional*,, defaults to `True`): If `True`, the existing list of additional special tokens will be replaced by the one specified in `special_tokens_dict`. Otherwise, `self._additional_special_tokens` is updated. In the former case, the tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged as non-special tokens. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to add a new classification token to GPT-2 tokenizer = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2Model.from_pretrained("gpt2") special_tokens_dict = {"cls_token": ""} num_added_toks = tokenizer.add_special_tokens(special_tokens_dict) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) assert tokenizer.cls_token == "" ```""" if not special_tokens_dict: return 0 added_tokens = 0 for key, value in special_tokens_dict.items(): assert key in self.SPECIAL_TOKENS_ATTRIBUTES, f"Key {key} is not a special token" if self.verbose: logger.info("Assigning %s to the %s key of the tokenizer", value, key) if key == "additional_special_tokens": assert isinstance(value, (list, tuple)) and all( isinstance(t, (str, AddedToken)) for t in value ), f"Tokens {value} for key {key} should all be str or AddedToken instances" if replace_additional_special_tokens: setattr(self, key, value) else: # This is a copy of `self._additional_special_tokens` additional_special_tokens = getattr(self, key) additional_special_tokens_set = set(additional_special_tokens) to_add = [] for token in value: if str(token) not in additional_special_tokens_set and str(token) not in to_add: to_add.append(token) # update the property additional_special_tokens.extend(to_add) self.additional_special_tokens = additional_special_tokens added_tokens += self.add_tokens(value, special_tokens=True) else: assert isinstance( value, (str, AddedToken) ), f"Token {value} for key {key} should be a str or an AddedToken instance" setattr(self, key, value) added_tokens += self.add_tokens([value], special_tokens=True) return added_tokens def add_tokens( self, new_tokens: Union[str, AddedToken, List[Union[str, AddedToken]]], special_tokens: bool = False ) -> int: """ Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary and and will be isolated before the tokenization algorithm is applied. Added tokens and tokens from the vocabulary of the tokenization algorithm are therefore not treated in the same way. Note, when adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Args: new_tokens (`str`, `tokenizers.AddedToken` or a list of *str* or `tokenizers.AddedToken`): Tokens are only added if they are not already in the vocabulary. `tokenizers.AddedToken` wraps a string token to let you personalize its behavior: whether this token should only match against a single word, whether this token should strip all potential whitespaces on the left side, whether this token should strip all potential whitespaces on the right side, etc. special_tokens (`bool`, *optional*, defaults to `False`): Can be used to specify if the token is a special token. This mostly change the normalization behavior (special tokens like CLS or [MASK] are usually not lower-cased for instance). See details for `tokenizers.AddedToken` in HuggingFace tokenizers library. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```""" if not new_tokens: return 0 if not isinstance(new_tokens, (list, tuple)): new_tokens = [new_tokens] return self._add_tokens(new_tokens, special_tokens=special_tokens) def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int: raise NotImplementedError @property def bos_token(self) -> str: """ `str`: Beginning of sentence token. Log an error if used while not having been set. """ if self._bos_token is None: if self.verbose: logger.error("Using bos_token, but it is not set yet.") return None return str(self._bos_token) @property def eos_token(self) -> str: """ `str`: End of sentence token. Log an error if used while not having been set. """ if self._eos_token is None: if self.verbose: logger.error("Using eos_token, but it is not set yet.") return None return str(self._eos_token) @property def unk_token(self) -> str: """ `str`: Unknown token. Log an error if used while not having been set. """ if self._unk_token is None: if self.verbose: logger.error("Using unk_token, but it is not set yet.") return None return str(self._unk_token) @property def sep_token(self) -> str: """ `str`: Separation token, to separate context and query in an input sequence. Log an error if used while not having been set. """ if self._sep_token is None: if self.verbose: logger.error("Using sep_token, but it is not set yet.") return None return str(self._sep_token) @property def pad_token(self) -> str: """ `str`: Padding token. Log an error if used while not having been set. """ if self._pad_token is None: if self.verbose: logger.error("Using pad_token, but it is not set yet.") return None return str(self._pad_token) @property def cls_token(self) -> str: """ `str`: Classification token, to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set. """ if self._cls_token is None: if self.verbose: logger.error("Using cls_token, but it is not set yet.") return None return str(self._cls_token) @property def mask_token(self) -> str: """ `str`: Mask token, to use when training a model with masked-language modeling. Log an error if used while not having been set. """ if self._mask_token is None: if self.verbose: logger.error("Using mask_token, but it is not set yet.") return None return str(self._mask_token) @property def additional_special_tokens(self) -> List[str]: """ `List[str]`: All the additional special tokens you may want to use. Log an error if used while not having been set. """ if self._additional_special_tokens is None: if self.verbose: logger.error("Using additional_special_tokens, but it is not set yet.") return None return [str(tok) for tok in self._additional_special_tokens] @bos_token.setter def bos_token(self, value): self._bos_token = value @eos_token.setter def eos_token(self, value): self._eos_token = value @unk_token.setter def unk_token(self, value): self._unk_token = value @sep_token.setter def sep_token(self, value): self._sep_token = value @pad_token.setter def pad_token(self, value): self._pad_token = value @cls_token.setter def cls_token(self, value): self._cls_token = value @mask_token.setter def mask_token(self, value): self._mask_token = value @additional_special_tokens.setter def additional_special_tokens(self, value): self._additional_special_tokens = value @property def bos_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the beginning of sentence token in the vocabulary. Returns `None` if the token has not been set. """ if self._bos_token is None: return None return self.convert_tokens_to_ids(self.bos_token) @property def eos_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been set. """ if self._eos_token is None: return None return self.convert_tokens_to_ids(self.eos_token) @property def unk_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the unknown token in the vocabulary. Returns `None` if the token has not been set. """ if self._unk_token is None: return None return self.convert_tokens_to_ids(self.unk_token) @property def sep_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the separation token in the vocabulary, to separate context and query in an input sequence. Returns `None` if the token has not been set. """ if self._sep_token is None: return None return self.convert_tokens_to_ids(self.sep_token) @property def pad_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the padding token in the vocabulary. Returns `None` if the token has not been set. """ if self._pad_token is None: return None return self.convert_tokens_to_ids(self.pad_token) @property def pad_token_type_id(self) -> int: """ `int`: Id of the padding token type in the vocabulary. """ return self._pad_token_type_id @property def cls_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the classification token in the vocabulary, to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Returns `None` if the token has not been set. """ if self._cls_token is None: return None return self.convert_tokens_to_ids(self.cls_token) @property def mask_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the mask token in the vocabulary, used when training a model with masked-language modeling. Returns `None` if the token has not been set. """ if self._mask_token is None: return None return self.convert_tokens_to_ids(self.mask_token) @property def additional_special_tokens_ids(self) -> List[int]: """ `List[int]`: Ids of all the additional special tokens in the vocabulary. Log an error if used while not having been set. """ return self.convert_tokens_to_ids(self.additional_special_tokens) @bos_token_id.setter def bos_token_id(self, value): self._bos_token = self.convert_ids_to_tokens(value) if value is not None else None @eos_token_id.setter def eos_token_id(self, value): self._eos_token = self.convert_ids_to_tokens(value) if value is not None else None @unk_token_id.setter def unk_token_id(self, value): self._unk_token = self.convert_ids_to_tokens(value) if value is not None else None @sep_token_id.setter def sep_token_id(self, value): self._sep_token = self.convert_ids_to_tokens(value) if value is not None else None @pad_token_id.setter def pad_token_id(self, value): self._pad_token = self.convert_ids_to_tokens(value) if value is not None else None @cls_token_id.setter def cls_token_id(self, value): self._cls_token = self.convert_ids_to_tokens(value) if value is not None else None @mask_token_id.setter def mask_token_id(self, value): self._mask_token = self.convert_ids_to_tokens(value) if value is not None else None @additional_special_tokens_ids.setter def additional_special_tokens_ids(self, values): self._additional_special_tokens = [self.convert_ids_to_tokens(value) for value in values] @property def special_tokens_map(self) -> Dict[str, Union[str, List[str]]]: """ `Dict[str, Union[str, List[str]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Convert potential tokens of `tokenizers.AddedToken` type to string. """ set_attr = {} for attr in self.SPECIAL_TOKENS_ATTRIBUTES: attr_value = getattr(self, "_" + attr) if attr_value: set_attr[attr] = ( type(attr_value)(str(attr_value_sub) for attr_value_sub in attr_value) if isinstance(attr_value, (list, tuple)) else str(attr_value) ) return set_attr @property def special_tokens_map_extended(self) -> Dict[str, Union[str, AddedToken, List[Union[str, AddedToken]]]]: """ `Dict[str, Union[str, tokenizers.AddedToken, List[Union[str, tokenizers.AddedToken]]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. """ set_attr = {} for attr in self.SPECIAL_TOKENS_ATTRIBUTES: attr_value = getattr(self, "_" + attr) if attr_value: set_attr[attr] = attr_value return set_attr @property def all_special_tokens(self) -> List[str]: """ `List[str]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Convert tokens of `tokenizers.AddedToken` type to string. """ all_toks = [str(s) for s in self.all_special_tokens_extended] return all_toks @property def all_special_tokens_extended(self) -> List[Union[str, AddedToken]]: """ `List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. """ all_toks = [] set_attr = self.special_tokens_map_extended for attr_value in set_attr.values(): all_toks = all_toks + (list(attr_value) if isinstance(attr_value, (list, tuple)) else [attr_value]) all_toks = list(OrderedDict.fromkeys(all_toks)) return all_toks @property def all_special_ids(self) -> List[int]: """ `List[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes. """ all_toks = self.all_special_tokens all_ids = self.convert_tokens_to_ids(all_toks) return all_ids @add_end_docstrings(INIT_TOKENIZER_DOCSTRING) class BaseTokenizer(SpecialTokensMixin): """ Base class for [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`]. Handles shared (mostly boiler plate) methods for those two classes. """ vocab_files_names: Dict[str, str] = {} pretrained_vocab_files_map: Dict[str, Dict[str, str]] = {} pretrained_init_configuration: Dict[str, Dict[str, Any]] = {} max_model_input_sizes: Dict[str, Optional[int]] = {} _auto_class: Optional[str] = None FILE_LIST: List[str] = [] # first name has to correspond to main model input name # to make sure `tokenizer.pad(...)` works correctly model_input_names: List[str] = ["input_ids", "token_type_ids", "attention_mask"] padding_side: str = "right" truncation_side: str = "right" slow_tokenizer_class = None _model_type = 0 _model_name = 1 def __init__(self, **kwargs): super().__init__(self) # inputs and kwargs for saving and re-loading (see ``from_pretrained`` and ``save_pretrained``) self.init_inputs = () self.init_kwargs = copy.deepcopy(kwargs) self.name_or_path = kwargs.pop("name_or_path", "") self._processor_class = kwargs.pop("processor_class", None) # For backward compatibility we fallback to set model_max_length from max_len if provided model_max_length = kwargs.pop("model_max_length", kwargs.pop("max_len", None)) self.model_max_length = model_max_length if model_max_length is not None else VERY_LARGE_INTEGER # Padding and truncation side are right by default and overridden in subclasses. If specified in the kwargs, it # is changed. self.padding_side = kwargs.pop("padding_side", self.padding_side) if self.padding_side not in ["right", "left"]: raise ValueError( f"Padding side should be selected between 'right' and 'left', current value: {self.padding_side}" ) self.truncation_side = kwargs.pop("truncation_side", self.truncation_side) if self.truncation_side not in ["right", "left"]: raise ValueError( f"Padding side should be selected between 'right' and 'left', current value: {self.truncation_side}" ) self.model_input_names = kwargs.pop("model_input_names", self.model_input_names) # By default, cleaning tokenization spaces for both fast and slow tokenizers self.clean_up_tokenization_spaces = kwargs.pop("clean_up_tokenization_spaces", True) self.deprecation_warnings = ( {} ) # Use to store when we have already noticed a deprecation warning (avoid overlogging). self._in_target_context_manager = False super().__init__(**kwargs) @property def max_len_single_sentence(self) -> int: """ `int`: The maximum length of a sentence that can be fed to the model. """ return self.model_max_length - self.num_special_tokens_to_add(pair=False) @property def max_len_sentences_pair(self) -> int: """ `int`: The maximum combined length of a pair of sentences that can be fed to the model. """ return self.model_max_length - self.num_special_tokens_to_add(pair=True) @max_len_single_sentence.setter def max_len_single_sentence(self, value) -> int: # For backward compatibility, allow to try to setup 'max_len_single_sentence'. if value == self.model_max_length - self.num_special_tokens_to_add(pair=False) and self.verbose: if not self.deprecation_warnings.get("max_len_single_sentence", False): logger.warning( "Setting 'max_len_single_sentence' is now deprecated. This value is automatically set up." ) self.deprecation_warnings["max_len_single_sentence"] = True else: raise ValueError( "Setting 'max_len_single_sentence' is now deprecated. This value is automatically set up." ) @max_len_sentences_pair.setter def max_len_sentences_pair(self, value) -> int: # For backward compatibility, allow to try to setup 'max_len_sentences_pair'. if value == self.model_max_length - self.num_special_tokens_to_add(pair=True) and self.verbose: if not self.deprecation_warnings.get("max_len_sentences_pair", False): logger.warning( "Setting 'max_len_sentences_pair' is now deprecated. This value is automatically set up." ) self.deprecation_warnings["max_len_sentences_pair"] = True else: raise ValueError("Setting 'max_len_sentences_pair' is now deprecated. This value is automatically set up.") def _set_processor_class(self, processor_class: str): """Sets processor class as an attribute.""" self._processor_class = processor_class def __repr__(self) -> str: return ( f"{self.__class__.__name__}(name_or_path='{self.name_or_path}'," f" vocab_size={self.vocab_size}, model_max_length={self.model_max_length}, is_fast={self.is_fast}," f" padding_side='{self.padding_side}', truncation_side='{self.truncation_side}'," f" special_tokens={self.special_tokens_map_extended}, " f"clean_up_tokenization_spaces={self.clean_up_tokenization_spaces})" ) def __len__(self) -> int: raise NotImplementedError() def get_vocab(self) -> Dict[str, int]: """ Returns the vocabulary as a dictionary of token to index. `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the vocab. Returns: `Dict[str, int]`: The vocabulary. """ raise NotImplementedError() @classmethod def from_pretrained(cls, name_or_path: str, **kwargs): """ Instantiates a tokenizer by the name_or_path. User can get the name using `get_support_list` of any tokenizer, it will download the necessary files from the cloud. or pass a directory where contains the vocabulary file and tokenizers yaml configuration file. Args: name_or_path (str): It supports the following two input types: If the name_or_path is a supported tokenizer name, for example, `clip_vit_b_32` and `t5_small`, it will download the necessary files from the cloud. User can select one from the support list by call `MindFormerBook.show_tokenizer_support_list()`. If name_or_path is a path to the local directory where there should have vocaburary files and configuration file ended with `yaml`. The vocaburary file needed by the tokenizer is determined by `.vocab_files_names`. pretrained_model_name_or_path (Optional[str]): Equal to "name_or_path", if "pretrained_model_name_or_path" is set, "name_or_path" is useless. Examples: >>> from mindformers import T5Tokenizer >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> res = tokenizer.encode("hello world!") >>> print(res) [21820, 296, 55, 1] Returns: A instanced tokenizer. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: name_or_path = pretrained_model_name_or_path is_exist = os.path.exists(name_or_path) is_dir = os.path.isdir(name_or_path) if not is_exist and (name_or_path not in cls._support_list): raise ValueError(f'{name_or_path} does not exist,' f' or it is not supported by {cls.__name__}. ' f'please select from {cls._support_list}.') if is_exist and not is_dir: raise ValueError(f"{name_or_path} is not a directory.") kwargs = dict() class_name = None loaded_kwargs = {} if name_or_path in MindFormerBook.get_tokenizer_url_support_list(): config, cache_path = cls._download_using_name(name_or_path) class_name, loaded_kwargs = cls._get_class_name_and_args_form_config(config) name_or_path = cache_path yaml_list = None if os.path.isdir(name_or_path): yaml_list = [file for file in os.listdir(name_or_path) if file.endswith(".yaml")] if len(yaml_list) > 1: logger.warning("There should be only one yaml file under the directory %s, " "but followings are found: %s", name_or_path, yaml_list) if yaml_list: yaml_file = os.path.join(name_or_path, yaml_list[0]) logger.info("config in the yaml file %s are used for tokenizer building.", yaml_file) config = MindFormerConfig(yaml_file) class_name, loaded_kwargs = cls._get_class_name_and_args_form_config(config) vocab_dict, file_dict = cls.read_files_according_specific_by_tokenizer(name_or_path) if 'tokenizer_config.json' in file_dict: class_name = file_dict['tokenizer_config.json'].pop('tokenizer_class', None) loaded_kwargs = file_dict['tokenizer_config.json'] else: logger.warning("Can't find the tokenizer_config.json in the file_dict. " "The content of file_dict is : %s", file_dict) kwargs.update(loaded_kwargs) kwargs.update(vocab_dict) if not class_name: class_name = cls.__name__ logger.info("build tokenizer class name is: %s using args %s.", class_name, kwargs) tokenizer = build_tokenizer(class_name=class_name, **kwargs) # Check all our special tokens are registered as "no split" token (we don't cut them) and are in the vocab added_tokens = tokenizer.sanitize_special_tokens() if added_tokens: logger.warning( "Special tokens have been added in the vocabulary, make sure the associated word embeddings are" " fine-tuned or trained." ) return tokenizer @classmethod def _download_using_name(cls, name_or_path): """Given the supported model name, download it from the urls""" tokenizer_name = name_or_path if name_or_path.startswith('mindspore'): # Adaptation the name of tokenizer at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/clip_vit_b_32" tokenizer_name = name_or_path.split('/')[cls._model_name] cache_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), tokenizer_name.split('_')[cls._model_type]) else: # Default the name of tokenizer, # the relevant file will be downloaded from the Obs platform. # such as "clip_vit_b_32" cache_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), name_or_path.split('_')[cls._model_type]) if not os.path.exists(cache_path): os.makedirs(cache_path, exist_ok=True) yaml_file = os.path.join(cache_path, tokenizer_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(tokenizer_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') # some tokenizers rely on more than one file, e.g gpt2 tokenizer_need_files = MindFormerBook.get_tokenizer_url_support_list()[name_or_path] for url_file in tokenizer_need_files: local_file_name = url_file.split('/')[-1] file_path = os.path.join(cache_path, local_file_name) if not os.path.exists(file_path): logger.info("Download the vocab from the url %s to %s.", url_file, file_path) download_with_progress_bar(url_file, file_path) try_sync_file(file_path) config = MindFormerConfig(yaml_file) return config, cache_path @classmethod def cache_vocab_files(cls, name_or_path, cache_path=None): """Cache the vocab files to the default dir""" if not cache_path: cache_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), name_or_path.split("_")[cls._model_type]) if not os.path.exists(cache_path): os.makedirs(cache_path, exist_ok=True) # some tokenizers rely on more than one file, e.g gpt2 tokenizer_need_files = MindFormerBook.get_tokenizer_url_support_list()[name_or_path] for url_file in tokenizer_need_files: local_file_name = url_file.split('/')[-1] file_path = os.path.join(cache_path, local_file_name) if not os.path.exists(file_path): logger.info("Download the yaml from the url %s to %s.", url_file, file_path) download_with_progress_bar(url_file, file_path) try_sync_file(file_path) read_vocab_file_dict, _ = cls.read_files_according_specific_by_tokenizer(cache_path) return read_vocab_file_dict @classmethod def _get_class_name_and_args_form_config(cls, config): """Lookup the yaml files under the name_or_path""" class_name = None tokenizer_args = {} if config and 'processor' in config and 'tokenizer' in config['processor'] \ and config.processor.tokenizer and 'type' in config.processor.tokenizer: tokenizer_args = config['processor']['tokenizer'] class_name = tokenizer_args.pop('type', None) else: logger.info("There is no matched format config['processor']['tokenizer'] in config %s", config) return class_name, tokenizer_args @classmethod def read_files_according_specific_by_tokenizer(cls, name_or_path): """Read the file path specific by the class variable in the tokenizer""" read_vocab_file_dict = {} read_tokenizer_file_dict = {} for k, name in cls.vocab_files_names.items(): if isinstance(name, str): path = os.path.join(name_or_path, name) if os.path.isfile(path): read_vocab_file_dict[k] = path # To support tokenizer like clip that has two types for vocab files. elif isinstance(name, list): for sub_name in name: path = os.path.join(name_or_path, sub_name) if os.path.isfile(path): read_vocab_file_dict[k] = path for item in cls.FILE_LIST: path = os.path.join(name_or_path, item) if os.path.isfile(path): read_tokenizer_file_dict[item] = json.load(open(path, 'r')) return read_vocab_file_dict, read_tokenizer_file_dict def save_pretrained(self, save_directory: Optional[str] = None, save_name: str = "mindspore_model", file_format: str = 'yaml'): """ Save the tokenizer by writing the `save_name`.yaml and vocaburary files those are determinied by `.vocab_files_names` to the disk. The kwargs passed to initialize the tokenizer will be saved. Args: save_directory(str): The output file directory. If None, the directory will be `./checkpoint_save`, which can be obtained by the `MindFormerBook.get_default_checkpoint_save_folder()`. Default None. save_name(str): The file name of the saved files. Default mindspore_model. file_format(str): Support json or yaml. Default yaml. Examples: >>> from mindformers import T5Tokenizer, MindFormerBook >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> tokenizer.save_pretrained() >>> output_path = MindFormerBook.get_default_checkpoint_save_folder() >>> print(os.listdir(output_path)) ['mindspore_model.yaml', 'spiece.model'] """ default_directory = MindFormerBook.get_default_checkpoint_save_folder() if save_directory is None: save_directory = default_directory os.makedirs(save_directory, exist_ok=True) if save_name is None: save_name = "mindspore_model" if file_format not in ('yaml', 'json'): raise ValueError(f"format should be one of [`yaml`, `json`], but got {file_format}.") kwargs = copy.deepcopy(self.init_kwargs) # Start to save the kwargs for the tokenizer if file_format == 'yaml': kwargs['type'] = self.__class__.__name__ merged_dict = dict() yaml_file = os.path.join(save_directory, save_name + '.yaml') if os.path.exists(yaml_file): with open(yaml_file, 'r') as file_reader: merged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) if merged_dict is None: merged_dict = dict() processor_name = MindFormerBook.get_tokenizer_name_to_processor()[kwargs['type']] merged_dict['processor'] = {"type": processor_name} if isinstance(kwargs, dict): for token_key, token_value in kwargs.items(): if isinstance(token_value, AddedToken): kwargs[token_key] = token_value.content merged_dict['processor']['tokenizer'] = kwargs with open(yaml_file, 'w') as file_reader: yaml.dump(merged_dict, file_reader) elif file_format == 'json': kwargs["tokenizer_class"] = self.__class__.__name__ tokenizer_config_path = os.path.join(save_directory, TOKENIZER_CONFIG_NAME) with open(tokenizer_config_path, 'w') as fp: json.dump(kwargs, fp, indent=4) else: raise ValueError(f"file_format should be one of [json, yaml], but got {file_format}.") self.save_vocabulary(save_directory) def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]: """ Save only the vocabulary of the tokenizer (vocabulary + added tokens). This method won't save the configuration and special token mappings of the tokenizer. Use [`~PreTrainedTokenizerFast._save_pretrained`] to save the whole state of the tokenizer. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. """ raise NotImplementedError def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]: """ Converts a string in a sequence of tokens, replacing unknown tokens with the `unk_token`. Args: text (`str`): The sequence to be encoded. pair (`str`, *optional*): A second sequence to be encoded with the first. add_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add the special tokens associated with the corresponding model. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific encode method. See details in [`~PreTrainedTokenizerBase.__call__`] Returns: `List[str]`: The list of tokens. """ raise NotImplementedError @add_end_docstrings( ENCODE_KWARGS_DOCSTRING, """ **kwargs: Passed along to the `.tokenize()` method. """, """ Returns: `List[int]`, `ms.Tensor`, or `np.ndarray`: The tokenized ids of the text. """, ) def encode( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, return_tensors: Optional[Union[str, TensorType]] = None, **kwargs, ) -> List[int]: """ Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing `self.convert_tokens_to_ids(self.tokenize(text))`. Args: text (`str`, `List[str]` or `List[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). """ encoded_inputs = self.encode_plus( text, text_pair=text_pair, add_special_tokens=add_special_tokens, padding=padding, truncation=truncation, max_length=max_length, stride=stride, return_tensors=return_tensors, **kwargs, ) return encoded_inputs["input_ids"] def num_special_tokens_to_add(self, pair: bool = False) -> int: raise NotImplementedError def _get_padding_truncation_strategies( self, padding=False, truncation=None, max_length=None, pad_to_multiple_of=None, verbose=True, **kwargs ): """ Find the correct padding/truncation strategy with backward compatibility for old arguments (truncation_strategy and pad_to_max_length) and behaviors. """ old_truncation_strategy = kwargs.pop("truncation_strategy", "do_not_truncate") old_pad_to_max_length = kwargs.pop("pad_to_max_length", False) # Backward compatibility for previous behavior, maybe we should deprecate it: # If you only set max_length, it activates truncation for max_length if max_length is not None and padding is False and truncation is None: if verbose: if not self.deprecation_warnings.get("Truncation-not-explicitly-activated", False): logger.warning( "Truncation was not explicitly activated but `max_length` is provided a specific value, please" " use `truncation=True` to explicitly truncate examples to max length. Defaulting to" " 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the" " tokenizer you can select this strategy more precisely by providing a specific strategy to" " `truncation`." ) self.deprecation_warnings["Truncation-not-explicitly-activated"] = True truncation = "longest_first" # Get padding strategy if padding is False and old_pad_to_max_length: if verbose: warnings.warn( "The `pad_to_max_length` argument is deprecated and will be removed in a future version, " "use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or " "use `padding='max_length'` to pad to a max length. In this case, you can give a specific " "length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the " "maximal input size of the model (e.g. 512 for Bert).", FutureWarning, ) if max_length is None: padding_strategy = PaddingStrategy.LONGEST else: padding_strategy = PaddingStrategy.MAX_LENGTH elif padding is not False: if padding is True: if verbose: if max_length is not None and ( truncation is None or truncation is False or truncation == "do_not_truncate" ): warnings.warn( "`max_length` is ignored when `padding`=`True` and there is no truncation strategy. " "To pad to max length, use `padding='max_length'`." ) if old_pad_to_max_length is not False: warnings.warn("Though `pad_to_max_length` = `True`, it is ignored because `padding`=`True`.") padding_strategy = PaddingStrategy.LONGEST # Default to pad to the longest sequence in the batch elif not isinstance(padding, PaddingStrategy): padding_strategy = PaddingStrategy(padding) elif isinstance(padding, PaddingStrategy): padding_strategy = padding else: padding_strategy = PaddingStrategy.DO_NOT_PAD # Get truncation strategy if truncation is None and old_truncation_strategy != "do_not_truncate": if verbose: warnings.warn( "The `truncation_strategy` argument is deprecated and will be removed in a future version, use" " `truncation=True` to truncate examples to a max length. You can give a specific length with" " `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input" " size of the model (e.g. 512 for Bert). If you have pairs of inputs, you can give a specific" " truncation strategy selected among `truncation='only_first'` (will only truncate the first" " sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the" " pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence" " in the pairs).", FutureWarning, ) truncation_strategy = TruncationStrategy(old_truncation_strategy) elif truncation is not False and truncation is not None: if truncation is True: truncation_strategy = ( TruncationStrategy.LONGEST_FIRST ) # Default to truncate the longest sequences in pairs of inputs elif not isinstance(truncation, TruncationStrategy): truncation_strategy = TruncationStrategy(truncation) elif isinstance(truncation, TruncationStrategy): truncation_strategy = truncation else: truncation_strategy = TruncationStrategy.DO_NOT_TRUNCATE # Set max length if needed if max_length is None: if padding_strategy == PaddingStrategy.MAX_LENGTH: if self.model_max_length > LARGE_INTEGER: if verbose: if not self.deprecation_warnings.get("Asking-to-pad-to-max_length", False): logger.warning( "Asking to pad to max_length but no maximum length is provided and the model has no" " predefined maximum length. Default to no padding." ) self.deprecation_warnings["Asking-to-pad-to-max_length"] = True padding_strategy = PaddingStrategy.DO_NOT_PAD else: max_length = self.model_max_length if truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE: if self.model_max_length > LARGE_INTEGER: if verbose: if not self.deprecation_warnings.get("Asking-to-truncate-to-max_length", False): logger.warning( "Asking to truncate to max_length but no maximum length is provided and the model has" " no predefined maximum length. Default to no truncation." ) self.deprecation_warnings["Asking-to-truncate-to-max_length"] = True truncation_strategy = TruncationStrategy.DO_NOT_TRUNCATE else: max_length = self.model_max_length # Test if we have a padding token if padding_strategy != PaddingStrategy.DO_NOT_PAD and (not self.pad_token or self.pad_token_id < 0): raise ValueError( "Asking to pad but the tokenizer does not have a padding token. " "Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` " "or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`." ) # Check that we will truncate to a multiple of pad_to_multiple_of if both are provided if ( truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE and padding_strategy != PaddingStrategy.DO_NOT_PAD and pad_to_multiple_of is not None and max_length is not None and (max_length % pad_to_multiple_of != 0) ): raise ValueError( "Truncation and padding are both activated but " f"truncation length ({max_length}) is not a multiple of pad_to_multiple_of ({pad_to_multiple_of})." ) return padding_strategy, truncation_strategy, max_length, kwargs @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def __call__( self, text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None, text_pair: Optional[Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]]] = None, text_target: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None, text_pair_target: Optional[ Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] ] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """ Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences. Args: text (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). """ # To avoid duplicating all_kwargs = { "add_special_tokens": add_special_tokens, "padding": padding, "truncation": truncation, "max_length": max_length, "stride": stride, "is_split_into_words": is_split_into_words, "pad_to_multiple_of": pad_to_multiple_of, "return_tensors": return_tensors, "return_token_type_ids": return_token_type_ids, "return_attention_mask": return_attention_mask, "return_overflowing_tokens": return_overflowing_tokens, "return_special_tokens_mask": return_special_tokens_mask, "return_offsets_mapping": return_offsets_mapping, "return_length": return_length, "verbose": verbose, } all_kwargs.update(kwargs) if text is None and text_target is None: raise ValueError("You need to specify either `text` or `text_target`.") if text is not None: # The context manager will send the inputs as normal texts and not text_target, but we shouldn't change the # input mode in this case. if not self._in_target_context_manager: self._switch_to_input_mode() encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) if text_target is not None: self._switch_to_target_mode() target_encodings = self._call_one(text=text_target, text_pair=text_pair_target, **all_kwargs) # Leave back tokenizer in input mode self._switch_to_input_mode() if text_target is None: return encodings if text is None: return target_encodings encodings["labels"] = target_encodings["input_ids"] return encodings def _call_one( self, text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]], text_pair: Optional[Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]]] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """used by self.__call__""" # Input type checking for clearer error def _is_valid_text_input(t): if isinstance(t, str): # Strings are fine return True if isinstance(t, (list, tuple)): # List are fine as long as they are... if not t: # ... empty return True if isinstance(t[0], str): # ... list of strings return True if isinstance(t[0], (list, tuple)): # ... list with an empty list or with a list of strings return not t[0] or isinstance(t[0][0], str) return False return False if not _is_valid_text_input(text): raise ValueError( "text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) " "or `List[List[str]]` (batch of pretokenized examples)." ) if text_pair is not None and not _is_valid_text_input(text_pair): raise ValueError( "text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) " "or `List[List[str]]` (batch of pretokenized examples)." ) if is_split_into_words: is_batched = isinstance(text, (list, tuple)) and text and isinstance(text[0], (list, tuple)) else: is_batched = isinstance(text, (list, tuple)) if is_batched: if isinstance(text_pair, str): raise TypeError( "when tokenizing batches of text, `text_pair` must be a list or tuple with the same length as" " `text`." ) if text_pair is not None and len(text) != len(text_pair): raise ValueError( f"batch length of `text`: {len(text)} does not match batch length of `text_pair`:" f" {len(text_pair)}." ) batch_text_or_text_pairs = list(zip(text, text_pair)) if text_pair is not None else text return self.batch_encode_plus( batch_text_or_text_pairs=batch_text_or_text_pairs, add_special_tokens=add_special_tokens, padding=padding, truncation=truncation, max_length=max_length, stride=stride, is_split_into_words=is_split_into_words, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, return_token_type_ids=return_token_type_ids, return_attention_mask=return_attention_mask, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_offsets_mapping=return_offsets_mapping, return_length=return_length, verbose=verbose, **kwargs, ) return self.encode_plus( text=text, text_pair=text_pair, add_special_tokens=add_special_tokens, padding=padding, truncation=truncation, max_length=max_length, stride=stride, is_split_into_words=is_split_into_words, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, return_token_type_ids=return_token_type_ids, return_attention_mask=return_attention_mask, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_offsets_mapping=return_offsets_mapping, return_length=return_length, verbose=verbose, **kwargs, ) @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def encode_plus( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """ Tokenize and prepare for the model a sequence or a pair of sequences. This method is deprecated, `__call__` should be used instead. Args: text (`str`, `List[str]` or `List[int]` (the latter only for not-fast tokenizers)): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). """ # Backward compatibility for 'truncation_strategy', 'pad_to_max_length' padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( padding=padding, truncation=truncation, max_length=max_length, pad_to_multiple_of=pad_to_multiple_of, verbose=verbose, **kwargs, ) return self._encode_plus( text=text, text_pair=text_pair, add_special_tokens=add_special_tokens, padding_strategy=padding_strategy, truncation_strategy=truncation_strategy, max_length=max_length, stride=stride, is_split_into_words=is_split_into_words, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, return_token_type_ids=return_token_type_ids, return_attention_mask=return_attention_mask, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_offsets_mapping=return_offsets_mapping, return_length=return_length, verbose=verbose, **kwargs, ) def _encode_plus( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None, add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """used by self.encode_plus""" raise NotImplementedError @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def batch_encode_plus( self, batch_text_or_text_pairs: Union[ List[TextInput], List[TextInputPair], List[PreTokenizedInput], List[PreTokenizedInputPair], List[EncodedInput], List[EncodedInputPair], ], add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """ Tokenize and prepare for the model a list of sequences or a list of pairs of sequences. This method is deprecated, `__call__` should be used instead. Args: batch_text_or_text_pairs ( `List[str]`, `List[Tuple[str, str]]`, `List[List[str]]`, `List[Tuple[List[str], List[str]]]`, and for not-fast tokenizers, also `List[List[int]]`, `List[Tuple[List[int], List[int]]]`): Batch of sequences or pair of sequences to be encoded. This can be a list of string/string-sequences/int-sequences or a list of pair of string/string-sequences/int-sequence (see details in `encode_plus`). """ # Backward compatibility for 'truncation_strategy', 'pad_to_max_length' padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( padding=padding, truncation=truncation, max_length=max_length, pad_to_multiple_of=pad_to_multiple_of, verbose=verbose, **kwargs, ) return self._batch_encode_plus( batch_text_or_text_pairs=batch_text_or_text_pairs, add_special_tokens=add_special_tokens, padding_strategy=padding_strategy, truncation_strategy=truncation_strategy, max_length=max_length, stride=stride, is_split_into_words=is_split_into_words, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, return_token_type_ids=return_token_type_ids, return_attention_mask=return_attention_mask, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_offsets_mapping=return_offsets_mapping, return_length=return_length, verbose=verbose, **kwargs, ) def _batch_encode_plus( self, batch_text_or_text_pairs: Union[ List[TextInput], List[TextInputPair], List[PreTokenizedInput], List[PreTokenizedInputPair], List[EncodedInput], List[EncodedInputPair], ], add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """used by self.batch_encode_plus""" raise NotImplementedError def pad( self, encoded_inputs: Union[ BatchEncoding, List[BatchEncoding], Dict[str, EncodedInput], Dict[str, List[EncodedInput]], List[Dict[str, EncodedInput]], ], padding: Union[bool, str, PaddingStrategy] = True, max_length: Optional[int] = None, pad_to_multiple_of: Optional[int] = None, return_attention_mask: Optional[bool] = None, return_tensors: Optional[Union[str, TensorType]] = None, verbose: bool = True, ) -> BatchEncoding: """ Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch. Padding side (left/right) padding token ids are defined at the tokenizer level (with `self.padding_side`, `self.pad_token_id` and `self.pad_token_type_id`). Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding. If the `encoded_inputs` passed are dictionary of numpy arrays, or Mindspore tensors, the result will use the same type unless you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the specific device of your tensors however. Args: encoded_inputs ( [`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`): Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str, List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader collate function. Instead of `List[int]` you can have tensors (numpy arrays or Mindspore tensors), see the note above for the return type. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return Numpy `ms.Tensor` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. """ if self.__class__.__name__.endswith("Fast"): if not self.deprecation_warnings.get("Asking-to-pad-a-fast-tokenizer", False): logger.warning_advice( f"You're using a {self.__class__.__name__} tokenizer. Please note that with a fast tokenizer," " using the `__call__` method is faster than using a method to encode the text followed by a call" " to the `pad` method to get a padded encoding." ) self.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = True # If we have a list of dicts, let's convert it in a dict of lists # We do this to allow using this method as a collate_fn function in PyTorch Dataloader if isinstance(encoded_inputs, (list, tuple)) and isinstance(encoded_inputs[0], Mapping): encoded_inputs = {key: [example[key] for example in encoded_inputs] for key in encoded_inputs[0].keys()} # The model's main input name, usually `input_ids`, has be passed for padding if self.model_input_names[0] not in encoded_inputs: raise ValueError( "You should supply an encoding or a list of encodings to this method " f"that includes {self.model_input_names[0]}, but you provided {list(encoded_inputs.keys())}" ) required_input = encoded_inputs[self.model_input_names[0]] if required_input is None or (isinstance(required_input, Sized) and not required_input): if return_attention_mask: encoded_inputs["attention_mask"] = [] return encoded_inputs # If we have PyTorch/TF/NumPy tensors/arrays as inputs, we cast them as python objects # and rebuild them afterwards if no return_tensors is specified # Note that we lose the specific device the tensor may be on for PyTorch first_element = required_input[0] if isinstance(first_element, (list, tuple)): # first_element might be an empty list/tuple in some edge cases so we grab the first non empty element. for item in required_input: if item: first_element = item[0] break # At this state, if `first_element` is still a list/tuple, it's an empty one so there is nothing to do. if not isinstance(first_element, (int, list, tuple)): if isinstance(first_element, ms.Tensor): return_tensors = "ms" if return_tensors is None else return_tensors elif isinstance(first_element, np.ndarray): return_tensors = "np" if return_tensors is None else return_tensors else: raise ValueError( f"type of {first_element} unknown: {type(first_element)}. " "Should be one of a python, mindspore or numpy object." ) for key, value in encoded_inputs.items(): encoded_inputs[key] = to_py_obj(value) # Convert padding_strategy in PaddingStrategy padding_strategy, _, max_length, _ = self._get_padding_truncation_strategies( padding=padding, max_length=max_length, verbose=verbose ) required_input = encoded_inputs[self.model_input_names[0]] if required_input and not isinstance(required_input[0], (list, tuple)): encoded_inputs = self._pad( encoded_inputs, max_length=max_length, padding_strategy=padding_strategy, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, ) return BatchEncoding(encoded_inputs, tensor_type=return_tensors) batch_size = len(required_input) assert all( len(v) == batch_size for v in encoded_inputs.values() ), "Some items in the output dictionary have a different batch size than others." if padding_strategy == PaddingStrategy.LONGEST: max_length = max(len(inputs) for inputs in required_input) padding_strategy = PaddingStrategy.MAX_LENGTH batch_outputs = {} for i in range(batch_size): inputs = {k: v[i] for k, v in encoded_inputs.items()} outputs = self._pad( inputs, max_length=max_length, padding_strategy=padding_strategy, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, ) for key, value in outputs.items(): if key not in batch_outputs: batch_outputs[key] = [] batch_outputs[key].append(value) return BatchEncoding(batch_outputs, tensor_type=return_tensors) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids) Should be overridden in a subclass if the model has a special way of building those. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The token type ids. """ if token_ids_1 is None: return len(token_ids_0) * [0] return [0] * len(token_ids_0) + [1] * len(token_ids_1) def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. This implementation does not add special tokens and this method should be overridden in a subclass. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The model input with special tokens. """ if token_ids_1 is None: return token_ids_0 return token_ids_0 + token_ids_1 @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def prepare_for_model( self, ids: List[int], pair_ids: Optional[List[int]] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_length: bool = False, verbose: bool = True, prepend_batch_axis: bool = False, **kwargs, ) -> BatchEncoding: """ Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens. Please Note, for *pair_ids* different than `None` and *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a combination of arguments will raise an error. Args: ids (`List[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. """ # Backward compatibility for 'truncation_strategy', 'pad_to_max_length' padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( padding=padding, truncation=truncation, max_length=max_length, pad_to_multiple_of=pad_to_multiple_of, verbose=verbose, **kwargs, ) pair = bool(pair_ids is not None) len_ids = len(ids) len_pair_ids = len(pair_ids) if pair else 0 if return_token_type_ids and not add_special_tokens: raise ValueError( "Asking to return token_type_ids while setting add_special_tokens to False " "results in an undefined behavior. Please set add_special_tokens to True or " "set return_token_type_ids to None." ) if ( return_overflowing_tokens and truncation_strategy == TruncationStrategy.LONGEST_FIRST and pair_ids is not None ): raise ValueError( "Not possible to return overflowing tokens for pair of sequences with the " "`longest_first`. Please select another truncation strategy than `longest_first`, " "for instance `only_second` or `only_first`." ) # Load from model defaults if return_token_type_ids is None: return_token_type_ids = "token_type_ids" in self.model_input_names if return_attention_mask is None: return_attention_mask = "attention_mask" in self.model_input_names encoded_inputs = {} # Compute the total size of the returned encodings total_len = len_ids + len_pair_ids + (self.num_special_tokens_to_add(pair=pair) if add_special_tokens else 0) # Truncation: Handle max sequence length overflowing_tokens = [] if truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE and max_length and total_len > max_length: ids, pair_ids, overflowing_tokens = self.truncate_sequences( ids, pair_ids=pair_ids, num_tokens_to_remove=total_len - max_length, truncation_strategy=truncation_strategy, stride=stride, ) if return_overflowing_tokens: encoded_inputs["overflowing_tokens"] = overflowing_tokens encoded_inputs["num_truncated_tokens"] = total_len - max_length # Add special tokens if add_special_tokens: sequence = self.build_inputs_with_special_tokens(ids, pair_ids) token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids) else: sequence = ids + pair_ids if pair else ids token_type_ids = [0] * len(ids) + ([0] * len(pair_ids) if pair else []) # Build output dictionary encoded_inputs["input_ids"] = sequence if return_token_type_ids: encoded_inputs["token_type_ids"] = token_type_ids if return_special_tokens_mask: if add_special_tokens: encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids) else: encoded_inputs["special_tokens_mask"] = [0] * len(sequence) # Check lengths self._eventual_warn_about_too_long_sequence(encoded_inputs["input_ids"], max_length, verbose) # Padding if padding_strategy != PaddingStrategy.DO_NOT_PAD or return_attention_mask: encoded_inputs = self.pad( encoded_inputs, max_length=max_length, padding=padding_strategy.value, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, ) if return_length: encoded_inputs["length"] = len(encoded_inputs["input_ids"]) batch_outputs = BatchEncoding( encoded_inputs, tensor_type=return_tensors, prepend_batch_axis=prepend_batch_axis ) return batch_outputs def truncate_sequences( self, ids: List[int], pair_ids: Optional[List[int]] = None, num_tokens_to_remove: int = 0, truncation_strategy: Union[str, TruncationStrategy] = "longest_first", stride: int = 0, ) -> Tuple[List[int], List[int], List[int]]: """ Truncates a sequence pair in-place following the strategy. Args: ids (`List[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. num_tokens_to_remove (`int`, *optional*, defaults to 0): Number of tokens to remove using the truncation strategy. truncation_strategy (`str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): The strategy to follow for truncation. Can be: - `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). stride (`int`, *optional*, defaults to 0): If set to a positive number, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. Returns: `Tuple[List[int], List[int], List[int]]`: The truncated `ids`, the truncated `pair_ids` and the list of overflowing tokens. Note: The *longest_first* strategy returns empty list of overflowing tokens if a pair of sequences (or a batch of pairs) is provided. """ if num_tokens_to_remove <= 0: return ids, pair_ids, [] if not isinstance(truncation_strategy, TruncationStrategy): truncation_strategy = TruncationStrategy(truncation_strategy) overflowing_tokens = [] if truncation_strategy == TruncationStrategy.ONLY_FIRST or ( truncation_strategy == TruncationStrategy.LONGEST_FIRST and pair_ids is None ): if len(ids) > num_tokens_to_remove: window_len = min(len(ids), stride + num_tokens_to_remove) if self.truncation_side == "left": overflowing_tokens = ids[:window_len] ids = ids[num_tokens_to_remove:] elif self.truncation_side == "right": overflowing_tokens = ids[-window_len:] ids = ids[:-num_tokens_to_remove] else: raise ValueError(f"invalid truncation strategy: {self.truncation_side}, use 'left' or 'right'.") else: error_msg = ( f"We need to remove {num_tokens_to_remove} to truncate the input " f"but the first sequence has a length {len(ids)}. " ) if truncation_strategy == TruncationStrategy.ONLY_FIRST: error_msg = ( error_msg + "Please select another truncation strategy than " f"{truncation_strategy}, for instance 'longest_first' or 'only_second'." ) logger.error(error_msg) elif truncation_strategy == TruncationStrategy.LONGEST_FIRST: logger.warning( "Be aware, overflowing tokens are not returned for the setting you have chosen," " i.e. sequence pairs with the %s " "truncation strategy. So the returned list will always be empty even if some " "tokens have been removed.", TruncationStrategy.LONGEST_FIRST.value ) for _ in range(num_tokens_to_remove): if pair_ids is None or len(ids) > len(pair_ids): if self.truncation_side == "right": ids = ids[:-1] elif self.truncation_side == "left": ids = ids[1:] else: raise ValueError("invalid truncation strategy:" + str(self.truncation_side)) else: if self.truncation_side == "right": pair_ids = pair_ids[:-1] elif self.truncation_side == "left": pair_ids = pair_ids[1:] else: raise ValueError("invalid truncation strategy:" + str(self.truncation_side)) elif truncation_strategy == TruncationStrategy.ONLY_SECOND and pair_ids is not None: if len(pair_ids) > num_tokens_to_remove: window_len = min(len(pair_ids), stride + num_tokens_to_remove) if self.truncation_side == "right": overflowing_tokens = pair_ids[-window_len:] pair_ids = pair_ids[:-num_tokens_to_remove] elif self.truncation_side == "left": overflowing_tokens = pair_ids[:window_len] pair_ids = pair_ids[num_tokens_to_remove:] else: raise ValueError("invalid truncation strategy:" + str(self.truncation_side)) else: logger.error( "We need to remove %s to truncate the input but the second sequence has a length %s. " "Please select another truncation strategy than %s, for instance 'longest_first' or 'only_first'.", num_tokens_to_remove, len(pair_ids), truncation_strategy ) return (ids, pair_ids, overflowing_tokens) def _pad( self, encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding], max_length: Optional[int] = None, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, pad_to_multiple_of: Optional[int] = None, return_attention_mask: Optional[bool] = None, ) -> dict: """ Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) """ # Load from model defaults if return_attention_mask is None: return_attention_mask = "attention_mask" in self.model_input_names required_input = encoded_inputs[self.model_input_names[0]] if padding_strategy == PaddingStrategy.LONGEST: max_length = len(required_input) if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0): max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length # Initialize attention mask if not present. if return_attention_mask and "attention_mask" not in encoded_inputs: encoded_inputs["attention_mask"] = [1] * len(required_input) if needs_to_be_padded: difference = max_length - len(required_input) if self.padding_side == "right": if return_attention_mask: encoded_inputs["attention_mask"] = encoded_inputs["attention_mask"] + [0] * difference if "token_type_ids" in encoded_inputs: encoded_inputs["token_type_ids"] = ( encoded_inputs["token_type_ids"] + [self.pad_token_type_id] * difference ) if "special_tokens_mask" in encoded_inputs: encoded_inputs["special_tokens_mask"] = encoded_inputs["special_tokens_mask"] + [1] * difference encoded_inputs[self.model_input_names[0]] = required_input + [self.pad_token_id] * difference elif self.padding_side == "left": if return_attention_mask: encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"] if "token_type_ids" in encoded_inputs: encoded_inputs["token_type_ids"] = [self.pad_token_type_id] * difference + encoded_inputs[ "token_type_ids" ] if "special_tokens_mask" in encoded_inputs: encoded_inputs["special_tokens_mask"] = [1] * difference + encoded_inputs["special_tokens_mask"] encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input else: raise ValueError("Invalid padding strategy:" + str(self.padding_side)) return encoded_inputs def convert_tokens_to_string(self, tokens: List[str]) -> str: """ Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove sub-word tokenization artifacts at the same time. Args: tokens (`List[str]`): The token to join in a string. Returns: `str`: The joined tokens. """ raise NotImplementedError def batch_decode( self, sequences: Union[List[int], List[List[int]], "np.ndarray", "ms.Tensor"], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs, ) -> List[str]: """ Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, ms.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. """ return [ self.decode( seq, skip_special_tokens=skip_special_tokens, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs, ) if seq else "" for seq in sequences ] def decode( self, token_ids: Union[int, List[int], "np.ndarray", "ms.Tensor"], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs, ) -> str: """ Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, ms.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. """ # Convert inputs to python lists token_ids = to_py_obj(token_ids) if isinstance(token_ids[0], list): output = [] for item in token_ids: new_strs = self._decode( token_ids=item, skip_special_tokens=skip_special_tokens, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs) output.append(new_strs) else: output = self._decode( token_ids=token_ids, skip_special_tokens=skip_special_tokens, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs) return output def _decode( self, token_ids: Union[int, List[int]], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs, ) -> str: raise NotImplementedError def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ assert already_has_special_tokens and token_ids_1 is None, ( "You cannot use ``already_has_special_tokens=False`` with this tokenizer. " "Please use a slow (full python) tokenizer to activate this argument. " "Or set `return_special_tokens_mask=True` when calling the encoding method " "to get the special tokens mask in any tokenizer. " ) all_special_ids = self.all_special_ids # cache the property special_tokens_mask = [1 if token in all_special_ids else 0 for token in token_ids_0] return special_tokens_mask @staticmethod def clean_up_tokenization(out_string: str) -> str: """ Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms. Args: out_string (`str`): The text to clean up. Returns: `str`: The cleaned-up string. """ out_string = ( out_string.replace(" .", ".") .replace(" ?", "?") .replace(" !", "!") .replace(" ,", ",") .replace(" ' ", "'") .replace(" n't", "n't") .replace(" 'm", "'m") .replace(" 's", "'s") .replace(" 've", "'ve") .replace(" 're", "'re") ) return out_string def _eventual_warn_about_too_long_sequence(self, ids: List[int], max_length: Optional[int], verbose: bool): """ Depending on the input and internal state we might trigger a warning about a sequence that is too long for its corresponding model Args: ids (`List[str]`): The ids produced by the tokenization max_length (`int`, *optional*): The max_length desired (does not trigger a warning if it is set) verbose (`bool`): Whether or not to print more information and warnings. """ if max_length is None and len(ids) > self.model_max_length and verbose: if not self.deprecation_warnings.get("sequence-length-is-longer-than-the-specified-maximum", False): logger.warning( "Token indices sequence length is longer than the specified maximum sequence length " "for this model (%s > %s). Running this sequence through the model " "will result in indexing errors", len(ids), self.model_max_length ) self.deprecation_warnings["sequence-length-is-longer-than-the-specified-maximum"] = True def _switch_to_input_mode(self): """ Private method to put the tokenizer in input mode (when it has different modes for input/outputs) """ def _switch_to_target_mode(self): """ Private method to put the tokenizer in target mode (when it has different modes for input/outputs) """ @contextmanager def as_target_tokenizer(self): """ Temporarily sets the tokenizer for encoding the targets. Useful for tokenizer associated to sequence-to-sequence models that need a slightly different processing for the labels. """ warnings.warn( "`as_target_tokenizer` is deprecated and will be removed in v5 of Transformers. You can tokenize your " "labels by using the argument `text_target` of the regular `__call__` method (either in the same call as " "your input texts if you use the same keyword arguments, or in a separate call." ) self._switch_to_target_mode() self._in_target_context_manager = True yield self._in_target_context_manager = False self._switch_to_input_mode() def prepare_seq2seq_batch( self, src_texts: List[str], tgt_texts: Optional[List[str]] = None, max_length: Optional[int] = None, max_target_length: Optional[int] = None, padding: str = "longest", return_tensors: str = None, truncation: bool = True, **kwargs, ) -> BatchEncoding: """ Prepare model inputs for translation. For best performance, translate one sentence at a time. Arguments: src_texts (`List[str]`): List of documents to summarize or source language texts. tgt_texts (`list`, *optional*): List of summaries or target language texts. max_length (`int`, *optional*): Controls the maximum length for encoder inputs (documents to summarize or source language texts) If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. max_target_length (`int`, *optional*): Controls the maximum length of decoder inputs (target language texts or summaries) If left unset or set to `None`, this will use the max_length value. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return PyTorch `ms.Tensor` objects. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `True`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). **kwargs: Additional keyword arguments passed along to `self.__call__`. Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to the encoder. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model. - **labels** -- List of token ids for tgt_texts. The full set of keys `[input_ids, attention_mask, labels]`, will only be returned if tgt_texts is passed. Otherwise, input_ids, attention_mask will be the only keys. """ # docstyle-ignore formatted_warning = """ `prepare_seq2seq_batch` is deprecated and will be removed in version 5 of HuggingFace Transformers. Use the regular `__call__` method to prepare your inputs and targets. Here is a short example: model_inputs = tokenizer(src_texts, text_target=tgt_texts, ...) If you either need to use different keyword arguments for the source and target texts, you should do two calls like this: model_inputs = tokenizer(src_texts, ...) labels = tokenizer(text_target=tgt_texts, ...) model_inputs["labels"] = labels["input_ids"] See the documentation of your specific tokenizer for more details on the specific arguments to the tokenizer of choice. For a more complete example, see the implementation of `prepare_seq2seq_batch`. """ warnings.warn(formatted_warning, FutureWarning) # mBART-specific kwargs that should be ignored by other models. kwargs.pop("src_lang", None) kwargs.pop("tgt_lang", None) if max_length is None: max_length = self.model_max_length model_inputs = self( src_texts, add_special_tokens=True, return_tensors=return_tensors, max_length=max_length, padding=padding, truncation=truncation, **kwargs, ) if tgt_texts is None: return model_inputs # Process tgt_texts if max_target_length is None: max_target_length = max_length with self.as_target_tokenizer(): labels = self( tgt_texts, add_special_tokens=True, return_tensors=return_tensors, padding=padding, max_length=max_target_length, truncation=truncation, **kwargs, ) model_inputs["labels"] = labels["input_ids"] return model_inputs @add_end_docstrings(INIT_TOKENIZER_DOCSTRING) @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class Tokenizer(BaseTokenizer): """ Base class for all slow tokenizers. Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary. This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). """ _support_list = [] def __init__(self, **kwargs): super().__init__(**kwargs) # Added tokens - We store this for both slow and fast tokenizers # until the serialization of Fast tokenizers is updated self.added_tokens_encoder: Dict[str, int] = {} self.added_tokens_decoder: Dict[int, str] = {} self.unique_no_split_tokens: List[str] = [] self.tokens_trie = Trie() self._decode_use_source_tokenizer = False @property def is_fast(self) -> bool: return False @property def vocab_size(self) -> int: """ `int`: Size of the base vocabulary (without the added tokens). """ raise NotImplementedError def get_added_vocab(self) -> Dict[str, int]: """ Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. """ return self.added_tokens_encoder def __len__(self): """ Size of the full vocabulary with the added tokens. """ return self.vocab_size + len(self.added_tokens_encoder) def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int: """ Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Args: new_tokens (`List[str]`or `List[tokenizers.AddedToken]`): Token(s) to add in vocabulary. A token is only added if it's not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the tokens should be added as special tokens. Returns: `int`: The number of tokens actually added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```""" new_tokens = [str(tok) for tok in new_tokens] tokens_to_add = [] for token in new_tokens: if not isinstance(token, str): raise TypeError(f"Token {token} is not a string but a {type(token)}.") if not special_tokens and hasattr(self, "do_lower_case") and self.do_lower_case: token = token.lower() if ( token != self.unk_token and self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token) and token not in tokens_to_add ): tokens_to_add.append(token) if self.verbose: logger.info("Adding %s to the vocabulary", token) added_tok_encoder = {tok: len(self) + i for i, tok in enumerate(tokens_to_add)} added_tok_decoder = {v: k for k, v in added_tok_encoder.items()} self.added_tokens_encoder.update(added_tok_encoder) self.added_tokens_decoder.update(added_tok_decoder) # Make sure we don't split on any special tokens (even they were already in the vocab before e.g. for Albert) if special_tokens: if len(new_tokens) == 1: _insert_one_token_to_ordered_list(self.unique_no_split_tokens, new_tokens[0]) else: self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(new_tokens))) else: # Or on the newly added tokens if len(tokens_to_add) == 1: _insert_one_token_to_ordered_list(self.unique_no_split_tokens, tokens_to_add[0]) else: self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(tokens_to_add))) self._create_trie(self.unique_no_split_tokens) return len(tokens_to_add) def _create_trie(self, unique_no_split_tokens): trie = Trie() for token in unique_no_split_tokens: if hasattr(self, "do_lower_case") and self.do_lower_case and token not in self.all_special_tokens: trie.add(token.lower()) else: trie.add(token) self.tokens_trie = trie def num_special_tokens_to_add(self, pair: bool = False) -> int: """ Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. """ token_ids_0 = [] token_ids_1 = [] return len(self.build_inputs_with_special_tokens(token_ids_0, token_ids_1 if pair else None)) def tokenize( self, text: TextInput, pair: Optional[str] = None, add_special_tokens: bool = True, **kwargs ) -> List[str]: """ Converts a string in a sequence of tokens, using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Takes care of added tokens. Args: text (`str`): The sequence to be encoded. **kwargs (additional keyword arguments): Passed along to the model-specific `prepare_for_tokenization` preprocessing method. Returns: `List[str]`: The list of tokens. """ # Simple mapping string => AddedToken for special tokens with specific tokenization behaviors all_special_tokens_extended = { str(t): t for t in self.all_special_tokens_extended if isinstance(t, AddedToken) } text, kwargs = self.prepare_for_tokenization(text, **kwargs) if kwargs: logger.warning("Keyword arguments %s not recognized.", kwargs) # TODO: should this be in the base class? if hasattr(self, "do_lower_case") and self.do_lower_case: # convert non-special tokens to lowercase escaped_special_toks = [ re.escape(s_tok) for s_tok in (self.unique_no_split_tokens + self.all_special_tokens) ] pattern = r"(" + r"|".join(escaped_special_toks) + r")|" + r"(.+?)" text = re.sub(pattern, lambda m: m.groups()[0] or m.groups()[1].lower(), text) no_split_token = set(self.unique_no_split_tokens) tokens = self.tokens_trie.split(text) # ["This is something", "", " else"] for i, token in enumerate(tokens): if token in no_split_token: tok_extended = all_special_tokens_extended.get(token, None) left = tokens[i - 1] if i > 0 else None right = tokens[i + 1] if i < len(tokens) - 1 else None if isinstance(tok_extended, AddedToken): if tok_extended.rstrip and right: # A bit counter-intuitive but we strip the left of the string # since tok_extended.rstrip means the special token is eating all white spaces on its right tokens[i + 1] = right.lstrip() # Strip white spaces on the left if tok_extended.lstrip and left: tokens[i - 1] = left.rstrip() # Opposite here else: # We strip left and right by default if right: tokens[i + 1] = right.lstrip() if left: tokens[i - 1] = left.rstrip() # ["This is something", "", "else"] tokenized_text = [] for token in tokens: # Need to skip eventual empty (fully stripped) tokens if not token: continue if token in no_split_token: tokenized_text.append(token) else: tokenized_text.extend(self._tokenize(token)) # ["This", " is", " something", "", "else"] return tokenized_text def _tokenize(self, text, **kwargs): """ Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Do NOT take care of added tokens. """ raise NotImplementedError def convert_tokens_to_ids(self, tokens: Union[str, List[str]]) -> Union[int, List[int]]: """ Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. """ if tokens is None: return None if isinstance(tokens, str): return self._convert_token_to_id_with_added_voc(tokens) ids = [] for token in tokens: ids.append(self._convert_token_to_id_with_added_voc(token)) return ids def _convert_token_to_id_with_added_voc(self, token): if token is None: return None if token in self.added_tokens_encoder: return self.added_tokens_encoder[token] return self._convert_token_to_id(token) def _convert_token_to_id(self, token): raise NotImplementedError def _encode_plus( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None, add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: def get_input_ids(text): if isinstance(text, str): tokens = self.tokenize(text, **kwargs) return self.convert_tokens_to_ids(tokens) if isinstance(text, (list, tuple)) and text and isinstance(text[0], str): if is_split_into_words: tokens = list( itertools.chain(*(self.tokenize(t, is_split_into_words=True, **kwargs) for t in text)) ) return self.convert_tokens_to_ids(tokens) return self.convert_tokens_to_ids(text) if isinstance(text, (list, tuple)) and text and isinstance(text[0], int): return text if is_split_into_words: raise ValueError( f"Input {text} is not valid. Should be a string or a list/tuple of strings when" " `is_split_into_words=True`." ) raise ValueError( f"Input {text} is not valid. Should be a string, a list/tuple of strings or a list/tuple of" " integers." ) if return_offsets_mapping: raise NotImplementedError( "return_offset_mapping is not available when using Python tokenizers. " "To use this feature, change your tokenizer to one deriving from " "transformers.PreTrainedTokenizerFast. " "More information on available tokenizers at " "https://github.com/huggingface/transformers/pull/2674" ) first_ids = get_input_ids(text) second_ids = get_input_ids(text_pair) if text_pair is not None else None return self.prepare_for_model( first_ids, pair_ids=second_ids, add_special_tokens=add_special_tokens, padding=padding_strategy.value, truncation=truncation_strategy.value, max_length=max_length, stride=stride, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, prepend_batch_axis=True, return_attention_mask=return_attention_mask, return_token_type_ids=return_token_type_ids, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_length=return_length, verbose=verbose, ) def _batch_encode_plus( self, batch_text_or_text_pairs: Union[ List[TextInput], List[TextInputPair], List[PreTokenizedInput], List[PreTokenizedInputPair], List[EncodedInput], List[EncodedInputPair], ], add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: def get_input_ids(text): if isinstance(text, str): tokens = self.tokenize(text, **kwargs) return self.convert_tokens_to_ids(tokens) if isinstance(text, (list, tuple)) and text and isinstance(text[0], str): if is_split_into_words: tokens = list( itertools.chain(*(self.tokenize(t, is_split_into_words=True, **kwargs) for t in text)) ) return self.convert_tokens_to_ids(tokens) return self.convert_tokens_to_ids(text) if isinstance(text, (list, tuple)) and text and isinstance(text[0], int): return text raise ValueError( "Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers." ) if return_offsets_mapping: raise NotImplementedError( "return_offset_mapping is not available when using Python tokenizers. " "To use this feature, change your tokenizer to one deriving from " "transformers.PreTrainedTokenizerFast." ) input_ids = [] for ids_or_pair_ids in batch_text_or_text_pairs: if not isinstance(ids_or_pair_ids, (list, tuple)): ids, pair_ids = ids_or_pair_ids, None elif is_split_into_words and not isinstance(ids_or_pair_ids[0], (list, tuple)): ids, pair_ids = ids_or_pair_ids, None else: ids, pair_ids = ids_or_pair_ids first_ids = get_input_ids(ids) second_ids = get_input_ids(pair_ids) if pair_ids is not None else None input_ids.append((first_ids, second_ids)) batch_outputs = self._batch_prepare_for_model( input_ids, add_special_tokens=add_special_tokens, padding_strategy=padding_strategy, truncation_strategy=truncation_strategy, max_length=max_length, stride=stride, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, return_token_type_ids=return_token_type_ids, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_length=return_length, return_tensors=return_tensors, verbose=verbose, ) return BatchEncoding(batch_outputs) @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def _batch_prepare_for_model( self, batch_ids_pairs: List[Union[PreTokenizedInputPair, Tuple[List[int], None]]], add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[str] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_length: bool = False, verbose: bool = True, ) -> BatchEncoding: """ Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens Args: batch_ids_pairs: list of tokenized input ids or input ids pairs """ batch_outputs = {} for first_ids, second_ids in batch_ids_pairs: outputs = self.prepare_for_model( first_ids, second_ids, add_special_tokens=add_special_tokens, padding=PaddingStrategy.DO_NOT_PAD.value, # we pad in batch afterward truncation=truncation_strategy.value, max_length=max_length, stride=stride, pad_to_multiple_of=None, # we pad in batch afterward return_attention_mask=False, # we pad in batch afterward return_token_type_ids=return_token_type_ids, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_length=return_length, return_tensors=None, # We convert the whole batch to tensors at the end prepend_batch_axis=False, verbose=verbose, ) for key, value in outputs.items(): if key not in batch_outputs: batch_outputs[key] = [] batch_outputs[key].append(value) batch_outputs = self.pad( batch_outputs, padding=padding_strategy.value, max_length=max_length, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, ) batch_outputs = BatchEncoding(batch_outputs, tensor_type=return_tensors) return batch_outputs def prepare_for_tokenization( self, text: str, **kwargs ) -> Tuple[str, Dict[str, Any]]: """ Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. kwargs: Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. """ return (text, kwargs) def get_special_tokens_mask( self, token_ids_0: List, token_ids_1: Optional[List] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: if token_ids_1 is not None: raise ValueError( "You should not supply a second sequence if the provided sequence of " "ids is already formatted with special tokens for the model." ) return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) return [0] * ((len(token_ids_1) if token_ids_1 else 0) + len(token_ids_0)) def convert_ids_to_tokens( self, ids: Union[int, List[int]], skip_special_tokens: bool = False ) -> Union[str, List[str]]: """ Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). """ if isinstance(ids, int): if ids in self.added_tokens_decoder: return self.added_tokens_decoder[ids] if ids >= self.vocab_size: raise IndexError(f"The token id {ids} is out of the size of vocabulary, please check your tokenizer " f"and corresponding vocabulary files.") return self._convert_id_to_token(ids) tokens = [] for index in ids: index = int(index) if skip_special_tokens and index in self.all_special_ids: continue if index in self.added_tokens_decoder: tokens.append(self.added_tokens_decoder[index]) else: if index >= self.vocab_size: raise IndexError( f"The token id {index} is out of the size of vocabulary, please check your tokenizer " f"and corresponding vocabulary files.") tokens.append(self._convert_id_to_token(index)) return tokens def _convert_id_to_token(self, index: int) -> str: raise NotImplementedError def convert_tokens_to_string(self, tokens: List[str]) -> str: return " ".join(tokens) def _decode( self, token_ids: List[int], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs, ) -> str: self._decode_use_source_tokenizer = kwargs.pop("use_source_tokenizer", False) spaces_between_special_tokens = kwargs.pop("spaces_between_special_tokens", False) filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens) # To avoid mixing byte-level and unicode for byte-level BPT # we need to build string separately for added tokens and byte-level tokens # cf. https://github.com/huggingface/transformers/issues/1133 sub_texts = [] current_sub_text = [] for token in filtered_tokens: if skip_special_tokens and token in self.all_special_ids: continue if token in self.added_tokens_encoder: if current_sub_text: sub_texts.append(self.convert_tokens_to_string(current_sub_text)) current_sub_text = [] sub_texts.append(token) else: current_sub_text.append(token) if current_sub_text: sub_texts.append(self.convert_tokens_to_string(current_sub_text)) if spaces_between_special_tokens: text = " ".join(sub_texts) else: text = "".join(sub_texts) clean_up_tokenization_spaces = ( clean_up_tokenization_spaces if clean_up_tokenization_spaces is not None else self.clean_up_tokenization_spaces ) if clean_up_tokenization_spaces: clean_text = self.clean_up_tokenization(text) return clean_text return text @classmethod def show_support_list(cls): """show_support_list method""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get_support_list method""" return cls._support_list pengcheng.mind-test/mindformers/models/bert/000077500000000000000000000000001471400134000214555ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/bert/__init__.py000066400000000000000000000021101471400134000235600ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for bert""" from .bert_tokenizer import BertTokenizer, BasicTokenizer from .bert_config import BertConfig from .bert import ( BertForPreTraining, BertModel, BertForTokenClassification, BertForMultipleChoice, BertForQuestionAnswering) from .bert_processor import BertProcessor __all__ = [] __all__.extend(bert_tokenizer.__all__) __all__.extend(bert.__all__) __all__.extend(bert_processor.__all__) pengcheng.mind-test/mindformers/models/bert/__pycache__/000077500000000000000000000000001471400134000236655ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/bert/__pycache__/__init__.cpython-39.pyc000066400000000000000000000011561471400134000300600ustar00rootroot00000000000000a ?cfH@stdZddlmZmZddlmZddlmZmZm Z m Z m Z ddl m Z gZeejeejee jdS)zThe export function for bert) BertTokenizerBasicTokenizer) BertConfig)BertForPreTraining BertModelBertForTokenClassificationBertForMultipleChoiceBertForQuestionAnswering) BertProcessorN)__doc__Zbert_tokenizerrrZ bert_configrbertrrrrr Zbert_processorr __all__extendrrK/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/__init__.pys    pengcheng.mind-test/mindformers/models/bert/__pycache__/bert.cpython-39.pyc000066400000000000000000000557111471400134000272630ustar00rootroot00000000000000a ?cf@sdZddlZddlZddlmmZddlm Z ddl m Z ddl m m ZddlmZmZddlmZmZddl mZddlmZddlmZmZddlmZdd lmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%gd Z&e'ej(Gddde!Z)e'ej(Gddde!Z*Gddde j+Z,e'ej(Gddde!Z-e'ej(Gddde!Z.Gddde j+Z/Gddde j+Z0Gddde j+Z1Gddde j+Z2Gd d!d!e j+Z3Gd"d#d#e j+Z4dS)$z BERT Model.N)Tensor Parameter)TruncatedNormal initializer) operations) LayerNorm) TransformerVocabEmbedding)default_moe_config)MindFormerRegisterMindFormerModuleType) BaseModel)MindFormerBook) BertConfig)r BertModelBertForPreTrainingBertForTokenClassificationBertForMultipleChoiceBertForQuestionAnsweringcs@eZdZdZeddZeffdd Zd ddZ Z S) ru Bert with dense layer for name entity recoginition task. Args: config (BertConfig): The config of BertForTokenClassification. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForTokenClassification, BertTokenizer >>> model = BertForTokenClassification.from_pretrained('tokcls_bert_base_chinese') >>> tokenizer = BertTokenizer.from_pretrained('tokcls_bert_base_chinese') >>> data = tokenizer(["我在杭州华为工作。"], return_tensors='ms', max_length=128, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) [[[ 0.0886748 -0.23066735 -0.03969013 ... 0.07333283 -0.02968273 0.06125224] [-0.03115104 -0.47599065 0.01928361 ... 0.22821501 0.42415133 0.14563856] [ 0.05570185 -0.36168078 0.00884905 ... -0.15357454 -0.33119604 -0.09889631] ... [ 0.01028904 -0.20403272 -0.1117363 ... -0.03601318 0.034153 0.12492958] [-0.00420664 -0.1786235 -0.11332294 ... -0.07119732 0.02407441 0.14775723] [-0.04674841 -0.1608555 -0.0781534 ... -0.06064402 0.0439388 0.20419256]]] tokclsbertcsxtt|||j|_t||j|j|_t d|j |_ t |j |j|j|_t|_t|_||dSN)superr__init__ num_labelsr is_traininguse_one_hot_embeddingsrnnDropouthidden_dropout_probdropoutDense hidden_sizeto_float compute_dtype classifierCrossEntropyLosscross_entropy_lossPReshapereshapeload_checkpointselfconfig __class__G/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/bert.pyrLs  z#BertForTokenClassification.__init__Nc Csx|j|||d}|d}||}||}|jr^||d|jf}||d}|||}n|durl|}n||f}|S)Get Training Loss or Logits input_ids input_masktoken_type_idsrr9Nrr!r&trainingr+rr() r.r6r7r8 label_ids bert_outputssequence_outputlogitsoutputr2r2r3 constructVs   z$BertForTokenClassification.construct)N __name__ __module__ __qualname____doc__rget_model_support_list _support_listrrrB __classcell__r2r2r0r3r&s" rcsFeZdZdZedZeffdd Zd ddZ d dd Z Z S) ra Provide bert pre-training loss through network. Args: config (BertConfig): The config of BertForPreTraining. is_training (bool): Specifies whether to use the training mode. use_one_hot_embeddings (bool): Specifies whether to use one-hot for embeddings. Default: False. Returns: Tensor, the loss of the network. Examples: >>> from mindspore import Tensor >>> import mindspore.common.dtype as mstype >>> from mindformers import BertForPreTraining, BertTokenizer >>> model = BertForPreTraining.from_pretrained('bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('bert_base_uncased') >>> data = tokenizer(["Paris is the [MASK] of France."], ... return_tensors='ms', max_length=128, padding="max_length") >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> masked_lm_positions = Tensor([[4]], mstype.int32) >>> next_sentence_labels = Tensor([[1]], mstype.int32) >>> masked_lm_weights = Tensor([[1]], mstype.int32) >>> masked_lm_ids = Tensor([[3007]], mstype.int32) >>> output = model(input_ids, attention_mask, token_type_ids, next_sentence_labels, ... masked_lm_positions, masked_lm_ids, masked_lm_weights) >>> print(output) [0.6706] rcstt|||j|_t||j|j|_t||_t ||_ t ||_ t |_|jjjdk|_t d|_||dS)Nrrr2)rrrrrrrGetMaskedLMOutputmlmlossGetNextSentenceOutputnsplossBertLosslossr)Castcastparallel_config moe_config expert_numuse_moeAddshardaddr,r-r0r2r3rs    zBertForPreTraining.__init__Nc Csjd}|jr"||||\}}}}n||||\}}}||||} ||} |js`||| | fS| | |fS)zconnect backbone and heads.r)rXrrNrPr<) r.r6r7 token_type_idmasked_lm_positionsmoe_lossr? pooled_outputembedding_tableprediction_scoresseq_relationship_scorer2r2r3 bert_forwards    zBertForPreTraining.bert_forwardc Cs^|js|||||S|||||\}} } ||| |||} |jrP|| | } || tjS)zGet pre-training loss)r<rcrRrXr[rTmstypefloat32) r.r6r7r\next_sentence_labelsr] masked_lm_idsmasked_lm_weightsrarbr^ total_lossr2r2r3rBs  zBertForPreTraining.construct)N)NNNN) rDrErFrGrrHrIrrrcrBrJr2r2r0r3rjs  rcs*eZdZdZdfdd ZddZZS) ra2 Bidirectional Encoder Representations from Transformers. Args: config (Class): Configuration for BertModel. is_training (bool): True for training mode. False for eval mode. use_one_hot_embeddings (bool): Specifies whether to use one hot encoding form. Default: False. NFcstt||sd|_d|_t||_|j|_|j|_|j|_ d|_ t |j dsZt |j _|j j}|j jjdk|_t|j|jtd|j|jgtjd|j jd|_d|j|j g}t||j ||jd|j||j|jd |_t|j|j|j |j|j|j!|j|j d ||j"|j#|j$|j%|j|jd |_&t'(|_)|j*|_*t+|j"d |_,t'-.d |_/t'j0dd|_1t2j3|j|jdt4|j5d6|j"|_7t'8|_9dS)NrVrtruncatedNormaldtype vocab_sizeembedding_size param_initrUr9T)rpembedding_shapeuse_relative_positionsuse_token_typetoken_type_vocab_sizermax_position_embeddings dropout_probr)r# batch_sizeffn_hidden_sizesrc_seq_lengthtgt_seq_length num_headsencoder_layersrUdecoder_layersrVparam_init_typelayernorm_compute_typesoftmax_compute_typepost_layernorm_residualattention_dropout_ratehidden_dropout_rate)dst_type)rrraxistanh) activation weight_init):rrrr attention_probs_dropout_prob CreateAttentionMaskFromInputMaskget_attention_maskr#num_hidden_layersrpr8hasattrrUr rVrWrXr rorrdreembedding_dp_mp_configword_embedding seq_lengthEmbeddingPostprocessorrstype_vocab_sizervembedding_postprocessorrrxintermediate_sizenum_attention_headsr%layernorm_dtype softmax_dtyper bert_encoderr)rSrTrm SaturateCastcast_compute_type StridedSlicerZsliceSqueeze squeeze_1rr"rinitializer_ranger$densePrintprint)r.r/rrrVZoutput_embedding_shaper0r2r3rs|      zBertModel.__init__cCs||\}}|||}t||j}||}||||}|d} t|d} | | d| d|j fd} | | } | | } | | |j} d}|jr|d}| | ||fS| | |fS)z8Bidirectional Encoder Representations from Transformers.r)rrrrrr9)rrr)rSrmrrrShaperr#rrrTrX)r.r6r8r7word_embeddingsZembedding_tablesZembedding_outputattention_maskencoder_outputr?rxsequence_slice first_tokenr_r^r2r2r3rBs(      zBertModel.construct)NFFrDrErFrGrrBrJr2r2r0r3rs  Drcs@eZdZdZeddZeffdd Zd ddZ Z S) ra Bert with dense layer for txt classification task. Args: config (BertConfig): The config of BertForMultipleChoice. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForMultipleChoice, BertTokenizer >>> model = BertForMultipleChoice.from_pretrained('txtcls_bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('txtcls_bert_base_uncased') >>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "], ... return_tensors='ms', max_length=128, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) [[0.05121157 0.3273056 0.31614417]] txtclsrcstt|||j|_t||j|j|_t|j |_ t j |j |j|j dd|j|_t d|j|_t |_t|_t|_|j|_|j|_||dS)NTrhas_biasr)rrrrrrrrrrrrr"r#r$r%r&rrwr!r'r(r)rSrTr*r+rmr,r-r0r2r3rEs     zBertForMultipleChoice.__init__Nc Cs||||\}}}||}||}|jrZ||d|jf}||d}|||}n"|durh|}n||d}||f}|S)r4r9r:Nr;) r.r6r7r\r=_r_r@rAr2r2r3rBTs    zBertForMultipleChoice.construct)NrCr2r2r0r3r,srcs@eZdZdZeddZeffdd Zd ddZ Z S) ra Bert with dense layer for question answering task. Args: config (BertConfig): The config of BertModel. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForQuestionAnswering, BertTokenizer >>> model = BertForQuestionAnswering.from_pretrained('qa_bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased') >>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "], ... return_tensors='ms', max_length=384, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) (Tensor(shape=[1, 384], dtype=Float32, value= [[ 5.30713797e-02, 1.08105361e-01, 1.55855455e-02 ... -3.73562761e-02, -1.89271923e-02, -2.99206823e-02]]), Tensor(shape=[1, 384], dtype=Float32, value= [[ 2.39873882e-02, 2.35675126e-01, 1.52210891e-01 ... -2.11214088e-02, 4.84664962e-02, -4.52579707e-02]])) qarcsXtt||t||j|j|_t|j d |j |_ | |tjddd|_dS)Nr9) output_numr)rrrrrrrrr"r#r$r% qa_outputsr,r)Splitsplitr-r0r2r3rs  z!BertForQuestionAnswering.__init__NcCs|j|||d}|d}||} || \} } | d} | d} |jrt|jdkrb|d}t|jdkrz|d}| jd} t|d| }t|d| }t j | d} | | |}| | |}||d}|}n || | f}|S)r4r5rr9r) ignore_indexr) rrrsqueezer<lenshapeops clip_by_valuerr')r.r6r7r\start_position end_position unique_idr>r?r@ start_logits end_logitsZ ignored_indexZloss_fctZ start_lossZend_lossrirAr2r2r3rBs*           z"BertForQuestionAnswering.construct)NNNrCr2r2r0r3rhsrcs*eZdZdZd fdd Zdd ZZS) rab Postprocessors apply positional and token type embeddings to word embeddings. Args: embedding_size (int): The size of each embedding vector. embedding_shape (list): [batch_size, seq_length, embedding_size], the shape of each embedding vector. use_token_type (bool): Specifies whether to use token type embeddings. Default: False. token_type_vocab_size (int): Size of token type vocab. Default: 16. use_one_hot_embeddings (bool): Specifies whether to use one hot encoding form. Default: False. initializer_range (float): Initialization value of TruncatedNormal. Default: 0.02. max_position_embeddings (int): Maximum length of sequences used in this model. Default: 512. dropout_prob (float): The dropout probability. Default: 0.1. F皙?c stt|||_||_||_||_t||td||gt j d|j j d|_ d|_t|_tdt j |_tdt j |_t|_t|_t||_td| |_t|_||_ t!"d|_#|j\} } } t|j|td|j|gt j d|j j d|_$t%|f&t j |_'|j'"|j j(ddfftt)*| d | +t)j,|_-t."|j j(ddf|j j(ddff|_/t!"d|_#dS) Nrkrlrnr:?rr)rrr9)0rrrrtrurrvr rrdrerUrtoken_type_embeddingZ shape_flatr)OneHotone_hotron_value off_valueMatMul array_mulr*r+tuplerrrr!GathergatherrsrrZrfull_position_embeddingrr$ layernorm data_parallelnparangeastypeint32 position_idsrYr[) r.r/rprrrsrtrurrvrwrseqr0r2r3rsT        zEmbeddingPostprocessor.__init__c Cs|}|jr$||\}}|||}|jsxt|}t|j}||jd|d|dfd}||\} }||| }| |}| |}|S)zMPostprocessors apply positional and token type embeddings to word embeddings.)rrrrr) rtrr[rsFrrrrrr!) r.r8rrAZtoken_type_embeddingsrrZshape_positionrZposition_embeddingsr2r2r3rBs      z EmbeddingPostprocessor.construct)FFrFrrrr2r2r0r3rs/rcs.eZdZdZejffdd ZddZZS)ra Performs a safe saturating cast. This operation applies proper clamping before casting to prevent the danger that the value will overflow or underflow. Args: src_type (:class:`mindspore.dtype`): The type of the elements of the input tensor. Default: mstype.float32. dst_type (:class:`mindspore.dtype`): The type of the elements of the output tensor. Default: mstype.float32. csdtt|t|}tt|j|_ tt|j |_ t |_t |_t |_||_dSN)rrrrddtype_to_nptypefloatrfinfomintensor_min_typemaxtensor_max_typer)Minimummin_opMaximummax_oprSrTr)r.rnp_typer0r2r3rs    zSaturateCast.__init__cCs*|||j}|||j}|||jSr)rrrrrTr)r.xoutr2r2r3rBszSaturateCast.construct) rDrErFrGrdrerrBrJr2r2r0r3rs rcs(eZdZdZfddZddZZS)rz Create attention mask according to input mask. Args: config (Class): Configuration for BertForPreTraining. csJtt|d|_t|_t|_t j |j jddff|_dSr)rrrr7r)rSrTr*r+ mindsporerTilerZrUrtiler-r0r2r3rs   z)CreateAttentionMaskFromInputMask.__init__cCs@t|d}|||dd|ftj}||d|df}|S)Nrr9)rrrTr+rdfloat16r)r.r7rrr2r2r3rB!sz*CreateAttentionMaskFromInputMask.constructrr2r2r0r3rs rcs(eZdZdZfddZddZZS)rQz Provide bert pre-training loss. Args: config (BertConfig): The config of BertForPreTraining. Returns: Tensor, total loss. csDtt||j|_t|jjdfddf|_ t dt j |_ t dt j |_t|jjff|_t|jjdff|_t|jjff|_td|_t|_d|_t|jjff|_t|_td|_td|_t |jjdf|jjdff|_!t |jjf|jjff|_"dS) Nrr2rrj)rLr:)r2rLrK)#rrQrror)rrZrUronehotrrdrerr ReduceSum reduce_sum reduce_sum1 ReduceMean reduce_mean reduce_mean2r*r+last_idxNegnegrSrTRealDivdivrYr[Mulmulmul2r-r0r2r3r2s"  $zBertLoss.__init__cCs|||j}||||jtj}|||j|j|j}| | | |||j} | | || d} | |d|tdtj} || | } |||j} || d|j|j}| | | |||j} || |j}|| |}|S)z"Defines the computation performed.r2)gh㈵>r)r+rrTrdrerrorrrrrrrrtuple_to_arrayrrr[)r.rarbrgrhrfr=Z label_weightsZone_hot_labelsZper_example_loss numerator denominatorZmasked_lm_losslabelsZnext_sentence_lossrir2r2r3rBEs   zBertLoss.constructrr2r2r0r3rQ's rQcs(eZdZdZfddZddZZS)rMz Get masked lm output. Args: config (BertConfig): The config of BertForPreTraining. Returns: Tensor, masked lm output. cshtt||j|_t|_t|_ |j dt |j }t j|j|j||jd|j|_|jjr|jj |jjdfdfn"|jj |jjdfd|jjfft|jf|j|_|j |jjdffttd|j|_tjdd|_|j |jjdfdft jdd |_ |j j |jjdffd |_!d |_"d|jf|_#t$|_%|j|_|j&|_&dS) N)rrL)rrrrzeroT) transpose_br9r)r9rr:)'rrMrr#widthr)r*r+rrrZrrrr" hidden_actr$r%rrU vocab_emb_dpmatmulrmodel_parallelrrrrro output_biasr LogSoftmax log_softmaxshape_flat_offsetsrshape_flat_sequence_tensorrSrTrmr.r/rr0r2r3rhsH      zGetMaskedLMOutput.__init__c Cst|}tt|d}|||d|j}||||j}|||j}| ||d}| ||j }| ||j }| |}| |}|||} | | |j} | |j} || } | S)zGet output log_probsrr)r)rrr make_ranger+r rr rrTr%rrrrmrr ) r. input_tensorZoutput_weights positions input_shaperngZ flat_offsetsZ flat_positionZflat_sequence_tensorr@ log_probsr2r2r3rBs      zGetMaskedLMOutput.constructrr2r2r0r3rM]s %rMcs(eZdZdZfddZddZZS)rOz Get next sentence output. Args: config (BertConfig): The config of Bert. Returns: Tensor, next sentence output. cstt|t|_|j|jjdfft |j }t j |j d|dd|j|_|jjrz|jj|jjdfdfn&|jj|jj|jjf|jjdff|j|_t|_dS)NrrTrr)rrOrr)r r rZrUrrrrr"r#r$r%rrrrrmrSrTr r0r2r3rs     zGetNextSentenceOutput.__init__cCs&||}|||j}||}|Sr)rrTrmr )r.rr@log_probr2r2r3rBs  zGetNextSentenceOutput.constructrr2r2r0r3rOs rO)5rGnumpyrrZmindspore.common.dtypecommonrmrd mindspore.nnr mindspore.opsrZmindspore.ops.functional functionalrrrmindspore.common.initializerrrrr)mindformers.modules.layersrmindformers.modules.transformerrr #mindformers.modules.transformer.moer mindformers.tools.registerr r Zmindformers.models.base_modelr mindformers.mindformer_bookrZ#mindformers.models.bert.bert_configr__all__registerMODELSrrCellrrrrrrrQrMrOr2r2r2r3s@         C Xi ; @P6Fpengcheng.mind-test/mindformers/models/bert/__pycache__/bert_config.cpython-39.pyc000066400000000000000000000053651471400134000306100ustar00rootroot00000000000000a ?cf'@stdZddlmmZddlmZmZddlm Z m Z ddl m Z ddl mZd gZe e jGd d d e ZdS) zBert Config API.N)default_transformer_configdefault_moe_config)MindFormerRegisterMindFormerModuleType) BaseConfig)MindFormerBook BertConfigcseZdZdZedZeeddeeddeedddee e ee e e e e e e e ee e e e e e e eeeee eeee dfdd Z Z S)r z8 BERT config class which defines the model size berttokclstxtclsqaF皙?:w geluTr{Gz?float32float16default) model_typeuse_one_hot_embeddings num_labelsassessment_method dropout_prob batch_size seq_length vocab_size hidden_sizenum_hidden_layersnum_attention_headsintermediate_size hidden_actpost_layernorm_residualhidden_dropout_probattention_probs_dropout_probmax_position_embeddingstype_vocab_sizeinitializer_rangeuse_relative_positionsdtypelayernorm_dtype softmax_dtype compute_dtypeuse_pastparallel_configcheckpoint_name_or_path moe_config is_trainingc s"tt|jfi|||_||_||_||_||_||_||_ ||_ | |_ | |_ | |_ | |_| |_||_||_||_||_||_||_||_|dkrtjntj|_|dkrtjntj|_|dkrtjntj|_|dkrtjntj|_||_||_|dkrt n||_!|dkrt"n||_#||_$dS)Nrr)%superr __init__rrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0mstyperrr1r2r3r4r5r7rr6rr8r9)selfrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9kwargs __class__N/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/bert_config.pyr;#s<zBertConfig.__init__)r FrrrrrrrrrrrTrrrrrFrrrrFrrrT)__name__ __module__ __qualname____doc__r get_config_support_list _support_listextendstrboolintfloatr; __classcell__rArAr?rBr s )rFZmindspore.common.dtypecommonr1r<Z+mindformers.modules.transformer.transformerrrmindformers.tools.registerrr base_configrmindformer_bookr __all__registerCONFIGr rArArArBs   pengcheng.mind-test/mindformers/models/bert/__pycache__/bert_processor.cpython-39.pyc000066400000000000000000000032611471400134000313530ustar00rootroot00000000000000a ?cf @s^dZddlmZddlmZddlmZddlmZm Z dgZ e e j Gd ddeZ d S) z BertProcessor )MindFormerBook) BaseTokenizer) BaseProcessor)MindFormerRegisterMindFormerModuleType BertProcessorcsdeZdZdZedZeeddeeddd fd d Zdd d Z Z S)r zU Bert processor, consists of a tokenizer (BaseTokenizer) for text input. berttokclstxtclsN max_lengthmscstt|j||||ddS)N) tokenizerrpaddingreturn_tensors)superr __init__)selfrrrr __class__Q/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/bert_processor.pyr$s  zBertProcessor.__init__cCsli}|durh|jrht|jts4tdt|jdt|trD|g}|j||j|j|jdd}||d<|S)z call functionNz;tokenizer should inherited from the BaseTokenizer, but got .)rrr input_idstext) r isinstancer TypeErrortypestrrrr)rZ text_inputZ image_inputoutputZ text_outputrrr__call__-s     zBertProcessor.__call__)Nr rr)NN) __name__ __module__ __qualname____doc__rget_processor_support_list _support_listextendrr" __classcell__rrrrr s  N)r&mindformers.mindformer_bookrbase_tokenizerrbase_processorrZtools.registerrr__all__register PROCESSORr rrrrs    pengcheng.mind-test/mindformers/models/bert/__pycache__/bert_tokenizer.cpython-39.pyc000066400000000000000000000443711471400134000313550ustar00rootroot00000000000000a ?cfZ@sdZddlZddlZddlZddlmZmZddlmZddl m Z m Z ddl m Z ddlmZd d gZd d iZd dZddZddZddZddZe e jGdd d e ZGdd d ZGdddZdS)zThe bert tokenizerN)ListOptional)logger)MindFormerRegisterMindFormerModuleType) Tokenizer)MindFormerBook BertTokenizerBasicTokenizer vocab_filez vocab.txtcCs&|dvr dSt|}|dkr"dSdS)z0Checks whether `char` is a whitespace character.)    TZZsF) unicodedatacategorycharcatrQ/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/bert_tokenizer.py_is_whitespace s  rcCs(|dvr dSt|}|dr$dSdS)z-Checks whether `char` is a control character.)rrrFCT)rr startswithrrrr _is_control,s   rcCst|}d|krdksbnd|kr2dksbnd|krHdksbnd|kr^dkrfnnd St|}|d r~d Sd S) z1Checks whether `char` is a punctuation character.!/:@[`{~TPF)ordrrr)rcprrrr_is_punctuation8sZ  r'cCsft}t|ddd}|}Wdn1s40Yt|D]\}}|d}|||<qF|S)z*Loads a vocabulary file into a dictionary.rutf-8encodingNr) collections OrderedDictopen readlines enumeraterstrip)r vocabreadertokensindextokenrrr load_vocabGs&  r7cCs|}|sgS|}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)stripsplit)textr4rrrwhitespace_tokenizeRs r;c sHeZdZdZeZddgZgdZe dZ e e dde e dde e ddd-fdd Z eddZeddZddZddZddZddZdd Zd.eeeeeeed!d"d#Zd/eeeeeeeed$fd%d& Zd0eeeeeeed!d'd(Zd1d)d*Zd+d,ZZS)2r a Construct a BERT tokenizer. Based on WordPiece. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): File containing the vocabulary. do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. do_basic_tokenize (`bool`, *optional*, defaults to `True`): Whether or not to do basic tokenization before WordPiece. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` unk_token (`str`, *optional*, defaults to `"[UNK]"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. sep_token (`str`, *optional*, defaults to `"[SEP]"`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `"[PAD]"`): The token used for padding, for example when batching sequences of different lengths. cls_token (`str`, *optional*, defaults to `"[CLS]"`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. mask_token (`str`, *optional*, defaults to `"[MASK]"`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. is_tokenize_char (`bool`, *optional*, defaults to `False`): Whether or not to tokenize characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). ztokenizer_config.jsonzspecial_tokens_map.json) input_idsZtoken_type_idsattention_maskbertZtokclsZtxtclsqaTN[UNK][SEP][PAD][CLS][MASK]Fc  stjf|||||||| | | d | tj|sDtd|dt||_t dd|j D|_ ||_ |rt ||| | d|_t|j|jd|_g|_| |_dS)N) do_lower_casedo_basic_tokenize never_split unk_token sep_token pad_token cls_token mask_tokentokenize_chinese_chars strip_accentsz&Can't find a vocabulary file at path 'z'. To load the vocabulary from a Google pretrained model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`cSsg|]\}}||fqSrr).0tokidsrrr z*BertTokenizer.__init__..)rErGrMrN)r2rH)super__init__ospathisfile ValueErrorr7 vocab_dictr,r-items ids_to_tokensrFr basic_tokenizerWordpieceTokenizerrHwordpiece_tokenizer mask_indexis_tokenize_char)selfr rErFrGrHrIrJrKrLrMrNrakwargs __class__rrrUs>    zBertTokenizer.__init__cCs|jjSN)r]rErbrrrrEszBertTokenizer.do_lower_casecCs t|jSrf)lenrZrgrrr vocab_sizeszBertTokenizer.vocab_sizecCst|jfi|jSrf)dictrZZadded_tokens_encoderrgrrr get_vocabszBertTokenizer.get_vocabcKsg}|jrF|D]4}|jr |}||jvr6||q||jqnL|jr||D],}||jj vrr||qV||j |7}qVn |j |}|Srf) rarElowerrZappendrHrF_process_mask_tokensr]rGr_tokenize)rbr:rc split_tokens characterr6rrr _tokenizes     zBertTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)rZgetrH)rbr6rrr_convert_token_to_idsz"BertTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r\rsrH)rbr5rrr_convert_id_to_tokensz"BertTokenizer._convert_id_to_tokencCsd|dd}|S)z:Converts a sequence of tokens (string) in a single string.r z ##)joinreplacer8)rbr4Z out_stringrrrconvert_tokens_to_stringsz&BertTokenizer.convert_tokens_to_string) token_ids_0 token_ids_1returncCs@|dur|jg||jgS|jg}|jg}|||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) cls_token_id sep_token_id)rbrzr{clsseprrr build_inputs_with_special_tokenss z.BertTokenizer.build_inputs_with_special_tokens)rzr{already_has_special_tokensr|csf|rtj||ddS|durLdgdgt|dgdgt|dgSdgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rzr{rNr)rTget_special_tokens_maskrh)rbrzr{rrdrrrs.z%BertTokenizer.get_special_tokens_maskcCsV|jg}|jg}|dur.t|||dgSt|||dgt||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Nrr)r~r}rh)rbrzr{rrrrr$create_token_type_ids_from_sequences's z2BertTokenizer.create_token_type_ids_from_sequencescCsd}tj|r4tj||r$|dndtd}n|r@|dnd|}t|ddd^}t|jdd d D]6\}}||krt d ||}| |d |d 7}qnWdn1s0Y|S)Nr-rvr wr)r*cSs|dS)Nrr)kvrrrMrSz/BertTokenizer.save_vocabulary..)keyzsSaving vocabulary to %s: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!rr) rVrWisdirrwVOCAB_FILES_NAMESr.sortedrZr[rwarningwrite)rbZsave_directoryZfilename_prefixr5r writerr6 token_indexrrrsave_vocabularyDs" (zBertTokenizer.save_vocabularycsg}j|vrj|vrb|j}|j|d||j||tjd}q|j|fddt|D_njj|j d}|S)zprocess mask tokens in textNcsg|]\}}|jkr|qSr) _mask_token)rOindxrgrrrRbrSz6BertTokenizer._process_mask_tokens..)rG) rr5extendr]rormrhr0r`Zall_special_tokens)rbr:Z text_tokenizerrrgrrnXs    z"BertTokenizer._process_mask_tokens) TTNr@rArBrCrDTNF)N)NF)N)N) __name__ __module__ __qualname____doc__rZvocab_files_namesZ FILE_LISTZmodel_input_namesr Zget_tokenizer_support_listZ _support_listrZget_config_support_listrUpropertyrErirkrrrtruryrintrrboolrrrrn __classcell__rrrdrr [sb+ 1       c@sNeZdZdZdddZdddZdd Zdd d Zd d ZddZ ddZ dS)r a Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). TNcCs,|dur g}||_t||_||_||_dSrf)rEsetrGrMrN)rbrErGrMrNrrrrU|s  zBasicTokenizer.__init__cCs|r|jt|n|j}||}|jr4||}t|}g}|D]R}||vr|jrt|}|j dur| |}n|j r| |}| | ||qDtd |}|S)a Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. Fr )rGunionr _clean_textrM_tokenize_chinese_charsr;rErlrN_run_strip_accentsr_run_split_on_puncrw)rbr:rG orig_tokensrpr6 output_tokensrrrros"      zBasicTokenizer.tokenizecCsBtd|}g}|D]"}t|}|dkr,q||qd|S)z$Strips accents from a piece of text.NFDZMnrv)r normalizerrmrw)rbr:outputrrrrrrs   z!BasicTokenizer._run_strip_accentscCs|dur||vr|gSt|}d}d}g}|t|kr||}t|rX||gd}n |rf|gd}|d||d7}q*dd|DS) z&Splits punctuation on a piece of text.NrTFrcSsg|]}d|qS)rv)rw)rOrrrrrRrSz5BasicTokenizer._run_split_on_punc..)listrhr'rm)rbr:rGcharsiZstart_new_wordrrrrrrs"    z!BasicTokenizer._run_split_on_punccCsTg}|D]@}t|}||r>|d|||dq||qd|S)z)Adds whitespace around any CJK character.r rv)r%_is_chinese_charrmrwrbr:rrr&rrrrs     z&BasicTokenizer._tokenize_chinese_charscCsd|krdksnd|kr*dksnd|kr@dksnd|krVdksnd |krld ksnd |krd ksnd |krdksnd|krdkrnndSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)rbr&rrrrsD  zBasicTokenizer._is_chinese_charcCsXg}|D]D}t|}|dks|dkst|r.qt|rB|dq||qd|S)zBPerforms invalid character removal and whitespace cleanup on text.rir rv)r%rrrmrwrrrrrs  zBasicTokenizer._clean_text)TNTN)N)N) rrrrrUrorrrrrrrrrr hs  %  c@s"eZdZdZdddZddZdS) r^zRuns WordPiece tokenization.dcCs||_||_||_dSrf)rZrHmax_input_chars_per_word)rbr2rHrrrrrUszWordpieceTokenizer.__init__c Csg}t|D]}t|}t||jkr4||jq d}d}g}|t|krt|}d} ||krd|||} |dkrd| } | |jvr| } q|d8}qX| durd}q|| |}q@|r||jq ||q |S)a Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example, `input = "unaffable"` wil return as output `["un", "##aff", "##able"]`. Args: text: A single token or whitespace separated tokens. This should have already been passed through *BasicTokenizer*. Returns: A list of wordpiece tokens. FrNrvz##rT) r;rrhrrmrHrwrZr) rbr:rr6rZis_badstart sub_tokensendZ cur_substrsubstrrrrros:       zWordpieceTokenizer.tokenizeN)r)rrrrrUrorrrrr^s r^)rr,rVrtypingrrmindformers.toolsrmindformers.tools.registerrrZ!mindformers.models.base_tokenizerrmindformer_bookr __all__rrrr'r7r;register TOKENIZERr r r^rrrrs,       pengcheng.mind-test/mindformers/models/bert/bert.py000066400000000000000000001006441471400134000227700ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://gitee.com/mindspore/models/blob/master/official/nlp/Bert/src/bert_model.py # ============================================================================ """BERT Model.""" import numpy as np import mindspore import mindspore.common.dtype as mstype import mindspore.nn as nn import mindspore.ops as ops import mindspore.ops.functional as F from mindspore import Tensor, Parameter from mindspore.common.initializer import TruncatedNormal, initializer from mindspore.ops import operations as P from mindformers.modules.layers import LayerNorm from mindformers.modules.transformer import Transformer, VocabEmbedding from mindformers.modules.transformer.moe import default_moe_config from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.mindformer_book import MindFormerBook from mindformers.models.bert.bert_config import BertConfig __all__ = ['BertConfig', 'BertModel', 'BertForPreTraining', 'BertForTokenClassification', 'BertForMultipleChoice', 'BertForQuestionAnswering'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertForTokenClassification(BaseModel): """ Bert with dense layer for name entity recoginition task. Args: config (BertConfig): The config of BertForTokenClassification. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForTokenClassification, BertTokenizer >>> model = BertForTokenClassification.from_pretrained('tokcls_bert_base_chinese') >>> tokenizer = BertTokenizer.from_pretrained('tokcls_bert_base_chinese') >>> data = tokenizer(["我在杭州华为工作。"], return_tensors='ms', max_length=128, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) [[[ 0.0886748 -0.23066735 -0.03969013 ... 0.07333283 -0.02968273 0.06125224] [-0.03115104 -0.47599065 0.01928361 ... 0.22821501 0.42415133 0.14563856] [ 0.05570185 -0.36168078 0.00884905 ... -0.15357454 -0.33119604 -0.09889631] ... [ 0.01028904 -0.20403272 -0.1117363 ... -0.03601318 0.034153 0.12492958] [-0.00420664 -0.1786235 -0.11332294 ... -0.07119732 0.02407441 0.14775723] [-0.04674841 -0.1608555 -0.0781534 ... -0.06064402 0.0439388 0.20419256]]] """ _support_list = MindFormerBook.get_model_support_list()['tokcls']['bert'] def __init__(self, config=BertConfig()): super(BertForTokenClassification, self).__init__(config) self.num_labels = config.num_labels self.bert = BertModel(config, config.is_training, config.use_one_hot_embeddings) self.dropout = nn.Dropout(1 - config.hidden_dropout_prob) self.classifier = nn.Dense(config.hidden_size, config.num_labels).to_float(config.compute_dtype) self.cross_entropy_loss = nn.CrossEntropyLoss() self.reshape = P.Reshape() self.load_checkpoint(config) def construct(self, input_ids, input_mask, token_type_ids, label_ids=None): """Get Training Loss or Logits""" bert_outputs = self.bert(input_ids=input_ids, input_mask=input_mask, token_type_ids=token_type_ids) sequence_output = bert_outputs[0] sequence_output = self.dropout(sequence_output) logits = self.classifier(sequence_output) if self.training: logits = self.reshape(logits, (-1, self.num_labels)) label_ids = self.reshape(label_ids, (-1,)) output = self.cross_entropy_loss(logits, label_ids) else: if label_ids is None: output = logits else: output = (logits, label_ids) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertForPreTraining(BaseModel): """ Provide bert pre-training loss through network. Args: config (BertConfig): The config of BertForPreTraining. is_training (bool): Specifies whether to use the training mode. use_one_hot_embeddings (bool): Specifies whether to use one-hot for embeddings. Default: False. Returns: Tensor, the loss of the network. Examples: >>> from mindspore import Tensor >>> import mindspore.common.dtype as mstype >>> from mindformers import BertForPreTraining, BertTokenizer >>> model = BertForPreTraining.from_pretrained('bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('bert_base_uncased') >>> data = tokenizer(["Paris is the [MASK] of France."], ... return_tensors='ms', max_length=128, padding="max_length") >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> masked_lm_positions = Tensor([[4]], mstype.int32) >>> next_sentence_labels = Tensor([[1]], mstype.int32) >>> masked_lm_weights = Tensor([[1]], mstype.int32) >>> masked_lm_ids = Tensor([[3007]], mstype.int32) >>> output = model(input_ids, attention_mask, token_type_ids, next_sentence_labels, ... masked_lm_positions, masked_lm_ids, masked_lm_weights) >>> print(output) [0.6706] """ _support_list = MindFormerBook.get_model_support_list()['bert'] def __init__(self, config=BertConfig()): super(BertForPreTraining, self).__init__(config) self.is_training = config.is_training # self.bert = BertScore(config, config.is_training, config.use_one_hot_embeddings) self.bert = BertModel(config, config.is_training, config.use_one_hot_embeddings) self.mlmloss = GetMaskedLMOutput(config) self.nsploss = GetNextSentenceOutput(config) self.loss = BertLoss(config) self.cast = P.Cast() self.use_moe = (config.parallel_config.moe_config.expert_num > 1) self.add = P.Add().shard(((1,), ())) self.load_checkpoint(config) def bert_forward(self, input_ids, input_mask, token_type_id, masked_lm_positions=None): """connect backbone and heads.""" moe_loss = 0 if self.use_moe: sequence_output, pooled_output, embedding_table, moe_loss = \ self.bert(input_ids, token_type_id, input_mask) else: sequence_output, pooled_output, embedding_table = \ self.bert(input_ids, token_type_id, input_mask) prediction_scores = self.mlmloss(sequence_output, embedding_table, masked_lm_positions) seq_relationship_score = self.nsploss(pooled_output) if not self.training: return sequence_output, pooled_output, prediction_scores, seq_relationship_score return prediction_scores, seq_relationship_score, moe_loss def construct(self, input_ids, input_mask, token_type_id, next_sentence_labels=None, masked_lm_positions=None, masked_lm_ids=None, masked_lm_weights=None): """Get pre-training loss""" if not self.training: return self.bert_forward(input_ids, input_mask, token_type_id, masked_lm_positions) prediction_scores, seq_relationship_score, moe_loss = \ self.bert_forward(input_ids, input_mask, token_type_id, masked_lm_positions) total_loss = self.loss(prediction_scores, seq_relationship_score, masked_lm_ids, masked_lm_weights, next_sentence_labels) if self.use_moe: total_loss = self.add(total_loss, moe_loss) return self.cast(total_loss, mstype.float32) class BertModel(nn.Cell): """ Bidirectional Encoder Representations from Transformers. Args: config (Class): Configuration for BertModel. is_training (bool): True for training mode. False for eval mode. use_one_hot_embeddings (bool): Specifies whether to use one hot encoding form. Default: False. """ def __init__(self, config=None, is_training=False, use_one_hot_embeddings=False): super(BertModel, self).__init__() if not is_training: config.hidden_dropout_prob = 0.0 config.attention_probs_dropout_prob = 0.0 self.get_attention_mask = CreateAttentionMaskFromInputMask(config) self.hidden_size = config.hidden_size self.num_hidden_layers = config.num_hidden_layers self.embedding_size = config.hidden_size self.token_type_ids = None if not hasattr(config.parallel_config, "moe_config"): config.parallel_config.moe_config = default_moe_config moe_config = config.parallel_config.moe_config self.use_moe = (config.parallel_config.moe_config.expert_num > 1) self.word_embedding = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, param_init=initializer("truncatedNormal", [config.vocab_size, config.hidden_size], dtype=mstype.float32), parallel_config=config.parallel_config.embedding_dp_mp_config) output_embedding_shape = [-1, config.seq_length, self.embedding_size] self.embedding_postprocessor = EmbeddingPostprocessor(config, embedding_size=self.embedding_size, embedding_shape=output_embedding_shape, use_relative_positions=config.use_relative_positions, use_token_type=True, token_type_vocab_size=config.type_vocab_size, use_one_hot_embeddings=use_one_hot_embeddings, max_position_embeddings=config.max_position_embeddings, dropout_prob=config.hidden_dropout_prob) self.bert_encoder = Transformer( hidden_size=config.hidden_size, batch_size=config.batch_size, ffn_hidden_size=config.intermediate_size, src_seq_length=config.seq_length, tgt_seq_length=config.seq_length, num_heads=config.num_attention_heads, encoder_layers=config.num_hidden_layers, parallel_config=config.parallel_config, decoder_layers=0, moe_config=moe_config, param_init_type=config.compute_dtype, layernorm_compute_type=config.layernorm_dtype, softmax_compute_type=config.softmax_dtype, post_layernorm_residual=config.post_layernorm_residual, attention_dropout_rate=config.attention_probs_dropout_prob, hidden_dropout_rate=config.hidden_dropout_prob,) self.cast = P.Cast() self.dtype = config.dtype self.cast_compute_type = SaturateCast(dst_type=config.compute_dtype) self.slice = P.StridedSlice().shard(((1, 1, 1),)) self.squeeze_1 = P.Squeeze(axis=1) self.dense = nn.Dense(self.hidden_size, self.hidden_size, activation="tanh", weight_init=TruncatedNormal(config.initializer_range)).to_float(config.compute_dtype) self.print = P.Print() def construct(self, input_ids, token_type_ids, input_mask): """Bidirectional Encoder Representations from Transformers.""" # embedding word_embeddings, embedding_tables = self.word_embedding(input_ids) embedding_output = self.embedding_postprocessor(token_type_ids, word_embeddings) # attention mask [batch_size, seq_length, seq_length](4, 1, 128) -> (4, 128, 128) input_mask = P.Cast()(input_mask, self.dtype) attention_mask = self.get_attention_mask(input_mask) # bert encoder encoder_output = self.bert_encoder(self.cast_compute_type(embedding_output), attention_mask) sequence_output = encoder_output[0] # pooler batch_size = P.Shape()(input_ids)[0] sequence_slice = self.slice(sequence_output, (0, 0, 0), (batch_size, 1, self.hidden_size), (1, 1, 1)) first_token = self.squeeze_1(sequence_slice) pooled_output = self.dense(first_token) pooled_output = self.cast(pooled_output, self.dtype) moe_loss = 0 if self.use_moe: moe_loss = encoder_output[-1] return sequence_output, pooled_output, embedding_tables, moe_loss return sequence_output, pooled_output, embedding_tables @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertForMultipleChoice(BaseModel): """ Bert with dense layer for txt classification task. Args: config (BertConfig): The config of BertForMultipleChoice. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForMultipleChoice, BertTokenizer >>> model = BertForMultipleChoice.from_pretrained('txtcls_bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('txtcls_bert_base_uncased') >>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "], ... return_tensors='ms', max_length=128, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) [[0.05121157 0.3273056 0.31614417]] """ _support_list = MindFormerBook.get_model_support_list()['txtcls']['bert'] def __init__(self, config=BertConfig()): super(BertForMultipleChoice, self).__init__(config) self.num_labels = config.num_labels self.bert = BertModel(config, config.is_training, config.use_one_hot_embeddings) self.weight_init = TruncatedNormal(config.initializer_range) self.classifier = nn.Dense(config.hidden_size, self.num_labels, weight_init=self.weight_init, has_bias=True).to_float(config.compute_dtype) self.dropout = nn.Dropout(1 - config.dropout_prob) self.cross_entropy_loss = nn.CrossEntropyLoss() self.cast = P.Cast() self.reshape = P.Reshape() self.dtype = config.dtype self.compute_dtype = config.compute_dtype self.load_checkpoint(config) def construct(self, input_ids, input_mask, token_type_id, label_ids=None): """Get Training Loss or Logits""" _, pooled_output, _ = self.bert(input_ids, token_type_id, input_mask) pooled_output = self.dropout(pooled_output) logits = self.classifier(pooled_output) if self.training: logits = self.reshape(logits, (-1, self.num_labels)) label_ids = self.reshape(label_ids, (-1,)) output = self.cross_entropy_loss(logits, label_ids) else: if label_ids is None: output = logits else: label_ids = self.reshape(label_ids, (-1,)) output = (logits, label_ids) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertForQuestionAnswering(BaseModel): """ Bert with dense layer for question answering task. Args: config (BertConfig): The config of BertModel. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForQuestionAnswering, BertTokenizer >>> model = BertForQuestionAnswering.from_pretrained('qa_bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased') >>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "], ... return_tensors='ms', max_length=384, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) (Tensor(shape=[1, 384], dtype=Float32, value= [[ 5.30713797e-02, 1.08105361e-01, 1.55855455e-02 ... -3.73562761e-02, -1.89271923e-02, -2.99206823e-02]]), Tensor(shape=[1, 384], dtype=Float32, value= [[ 2.39873882e-02, 2.35675126e-01, 1.52210891e-01 ... -2.11214088e-02, 4.84664962e-02, -4.52579707e-02]])) """ _support_list = MindFormerBook.get_model_support_list()['qa']['bert'] def __init__(self, config=BertConfig()): super(BertForQuestionAnswering, self).__init__(config) self.bert = BertModel(config, config.is_training, config.use_one_hot_embeddings) self.qa_outputs = nn.Dense(config.hidden_size, 2).to_float(config.compute_dtype) self.load_checkpoint(config) self.split = P.Split(output_num=2, axis=-1) def construct(self, input_ids, input_mask, token_type_id, start_position=None, end_position=None, unique_id=None): """Get Training Loss or Logits""" bert_outputs = self.bert(input_ids=input_ids, input_mask=input_mask, token_type_ids=token_type_id) sequence_output = bert_outputs[0] logits = self.qa_outputs(sequence_output) start_logits, end_logits = self.split(logits) start_logits = start_logits.squeeze(-1) end_logits = end_logits.squeeze(-1) if self.training: if len(start_position.shape) > 1: start_position = start_position.squeeze(-1) if len(end_position.shape) > 1: end_position = end_position.squeeze(-1) ignored_index = start_logits.shape[1] start_position = ops.clip_by_value(start_position, 0, ignored_index) end_position = ops.clip_by_value(end_position, 0, ignored_index) loss_fct = nn.CrossEntropyLoss(ignore_index=ignored_index) start_loss = loss_fct(start_logits, start_position) end_loss = loss_fct(end_logits, end_position) total_loss = (start_loss + end_loss) / 2 output = total_loss else: output = (unique_id, start_logits, end_logits) return output class EmbeddingPostprocessor(nn.Cell): """ Postprocessors apply positional and token type embeddings to word embeddings. Args: embedding_size (int): The size of each embedding vector. embedding_shape (list): [batch_size, seq_length, embedding_size], the shape of each embedding vector. use_token_type (bool): Specifies whether to use token type embeddings. Default: False. token_type_vocab_size (int): Size of token type vocab. Default: 16. use_one_hot_embeddings (bool): Specifies whether to use one hot encoding form. Default: False. initializer_range (float): Initialization value of TruncatedNormal. Default: 0.02. max_position_embeddings (int): Maximum length of sequences used in this model. Default: 512. dropout_prob (float): The dropout probability. Default: 0.1. """ def __init__(self, config, embedding_size, embedding_shape, use_relative_positions=False, use_token_type=False, token_type_vocab_size=16, use_one_hot_embeddings=False, max_position_embeddings=512, dropout_prob=0.1): super(EmbeddingPostprocessor, self).__init__() self.use_token_type = use_token_type self.token_type_vocab_size = token_type_vocab_size self.use_one_hot_embeddings = use_one_hot_embeddings self.max_position_embeddings = max_position_embeddings self.token_type_embedding = VocabEmbedding(vocab_size=token_type_vocab_size, embedding_size=embedding_size, param_init=initializer("truncatedNormal", [token_type_vocab_size, embedding_size], dtype=mstype.float32), parallel_config=config.parallel_config.embedding_dp_mp_config) self.shape_flat = (-1,) self.one_hot = P.OneHot() self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.1, mstype.float32) self.array_mul = P.MatMul() self.reshape = P.Reshape() self.shape = tuple(embedding_shape) self.dropout = nn.Dropout(1 - dropout_prob) self.gather = P.Gather() self.use_relative_positions = use_relative_positions self.slice = P.StridedSlice().shard(((1, 1),)) _, seq, _ = self.shape self.full_position_embedding = VocabEmbedding(vocab_size=config.max_position_embeddings, embedding_size=embedding_size, param_init=initializer("truncatedNormal", [config.max_position_embeddings, embedding_size], dtype=mstype.float32), parallel_config=config.parallel_config.embedding_dp_mp_config) self.layernorm = LayerNorm((embedding_size,)).to_float(mstype.float32) self.layernorm.shard(((config.parallel_config.data_parallel, 1, 1),)) self.position_ids = Tensor(np.arange(seq).reshape(-1, seq).astype(np.int32)) self.add = P.Add().shard( ((config.parallel_config.data_parallel, 1, 1), (config.parallel_config.data_parallel, 1, 1))) self.slice = P.StridedSlice().shard(((1, 1),)) def construct(self, token_type_ids, word_embeddings): """Postprocessors apply positional and token type embeddings to word embeddings.""" output = word_embeddings if self.use_token_type: token_type_embeddings, _ = self.token_type_embedding(token_type_ids) output = self.add(output, token_type_embeddings) if not self.use_relative_positions: shape = F.shape(output) shape_position = F.shape(self.position_ids) position_ids = self.slice(self.position_ids, (0, 0), (shape_position[0], shape[1]), (1, 1)) position_embeddings, _ = self.full_position_embedding(position_ids) output = self.add(output, position_embeddings) output = self.layernorm(output) output = self.dropout(output) return output class SaturateCast(nn.Cell): """ Performs a safe saturating cast. This operation applies proper clamping before casting to prevent the danger that the value will overflow or underflow. Args: src_type (:class:`mindspore.dtype`): The type of the elements of the input tensor. Default: mstype.float32. dst_type (:class:`mindspore.dtype`): The type of the elements of the output tensor. Default: mstype.float32. """ def __init__(self, dst_type=mstype.float32): super(SaturateCast, self).__init__() np_type = mstype.dtype_to_nptype(dst_type) self.tensor_min_type = float(np.finfo(np_type).min) self.tensor_max_type = float(np.finfo(np_type).max) self.min_op = P.Minimum() self.max_op = P.Maximum() self.cast = P.Cast() self.dst_type = dst_type def construct(self, x): out = self.max_op(x, self.tensor_min_type) out = self.min_op(out, self.tensor_max_type) return self.cast(out, self.dst_type) class CreateAttentionMaskFromInputMask(nn.Cell): """ Create attention mask according to input mask. Args: config (Class): Configuration for BertForPreTraining. """ def __init__(self, config): super(CreateAttentionMaskFromInputMask, self).__init__() self.input_mask = None self.cast = P.Cast() self.reshape = P.Reshape() self.tile = mindspore.ops.Tile().shard(((config.parallel_config.data_parallel, 1, 1),)) def construct(self, input_mask): seq_length = F.shape(input_mask)[1] attention_mask = self.cast(self.reshape(input_mask, (-1, 1, seq_length)), mstype.float16) attention_mask = self.tile(attention_mask, (1, seq_length, 1)) return attention_mask class BertLoss(nn.Cell): """ Provide bert pre-training loss. Args: config (BertConfig): The config of BertForPreTraining. Returns: Tensor, total loss. """ def __init__(self, config): super(BertLoss, self).__init__() self.vocab_size = config.vocab_size self.onehot = P.OneHot().shard(((config.parallel_config.data_parallel, 1), (), ())) self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_sum = P.ReduceSum().shard(((config.parallel_config.data_parallel,),)) self.reduce_sum1 = P.ReduceSum().shard(((config.parallel_config.data_parallel, 1),)) self.reduce_mean = P.ReduceMean().shard(((config.parallel_config.data_parallel,),)) self.reduce_mean2 = P.ReduceMean().shard(((1,),)) self.reshape = P.Reshape() self.last_idx = (-1,) self.neg = P.Neg().shard(((config.parallel_config.data_parallel,),)) self.cast = P.Cast() self.div = P.RealDiv().shard(((), (1,))) self.add = P.Add().shard(((1,), ())) self.mul = P.Mul().shard(((config.parallel_config.data_parallel, 1), (config.parallel_config.data_parallel, 1))) self.mul2 = P.Mul().shard(((config.parallel_config.data_parallel,), (config.parallel_config.data_parallel,))) def construct(self, prediction_scores, seq_relationship_score, masked_lm_ids, masked_lm_weights, next_sentence_labels): """Defines the computation performed.""" label_ids = self.reshape(masked_lm_ids, self.last_idx) label_weights = self.cast(self.reshape(masked_lm_weights, self.last_idx), mstype.float32) one_hot_labels = self.onehot(label_ids, self.vocab_size, self.on_value, self.off_value) per_example_loss = self.neg(self.reduce_sum1(self.mul(prediction_scores, one_hot_labels), self.last_idx)) numerator = self.reduce_sum(self.mul2(label_weights, per_example_loss), ()) denominator = self.reduce_sum(label_weights, ()) + self.cast(F.tuple_to_array((1e-5,)), mstype.float32) masked_lm_loss = self.div(numerator, denominator) # next_sentence_loss labels = self.reshape(next_sentence_labels, self.last_idx) one_hot_labels = self.onehot(labels, 2, self.on_value, self.off_value) per_example_loss = self.neg(self.reduce_sum1( self.mul(one_hot_labels, seq_relationship_score), self.last_idx)) next_sentence_loss = self.reduce_mean2(per_example_loss, self.last_idx) # total_loss total_loss = self.add(masked_lm_loss, next_sentence_loss) return total_loss class GetMaskedLMOutput(nn.Cell): """ Get masked lm output. Args: config (BertConfig): The config of BertForPreTraining. Returns: Tensor, masked lm output. """ def __init__(self, config): super(GetMaskedLMOutput, self).__init__() self.width = config.hidden_size self.reshape = P.Reshape() self.gather = P.Gather() self.gather.shard(((1, 1), (1,))) weight_init = TruncatedNormal(config.initializer_range) self.dense = nn.Dense(self.width, config.hidden_size, weight_init=weight_init, activation=config.hidden_act).to_float(config.compute_dtype) if config.parallel_config.vocab_emb_dp: self.dense.matmul.shard(((config.parallel_config.data_parallel, 1), (1, 1))) else: self.dense.matmul.shard(((config.parallel_config.data_parallel, 1), ( 1, config.parallel_config.model_parallel))) self.layernorm = LayerNorm((config.hidden_size,)).to_float(config.compute_dtype) self.layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.output_bias = Parameter( initializer( 'zero', config.vocab_size)) self.matmul = P.MatMul(transpose_b=True) self.matmul.shard(((config.parallel_config.data_parallel, 1), (1, 1))) self.log_softmax = nn.LogSoftmax(axis=-1) self.log_softmax.log_softmax.shard(((config.parallel_config.data_parallel, 1),)) self.shape_flat_offsets = (-1, 1) self.last_idx = (-1,) self.shape_flat_sequence_tensor = (-1, self.width) self.cast = P.Cast() self.compute_dtype = config.compute_dtype self.dtype = config.dtype def construct(self, input_tensor, output_weights, positions): """Get output log_probs""" input_shape = P.Shape()(input_tensor) rng = F.tuple_to_array(F.make_range(input_shape[0])) flat_offsets = self.reshape(rng * input_shape[1], self.shape_flat_offsets) flat_position = self.reshape(positions + flat_offsets, self.last_idx) flat_sequence_tensor = self.reshape(input_tensor, self.shape_flat_sequence_tensor) input_tensor = self.gather(flat_sequence_tensor, flat_position, 0) input_tensor = self.cast(input_tensor, self.compute_dtype) output_weights = self.cast(output_weights, self.compute_dtype) input_tensor = self.dense(input_tensor) input_tensor = self.layernorm(input_tensor) logits = self.matmul(input_tensor, output_weights) logits = self.cast(logits, self.dtype) logits = logits + self.output_bias log_probs = self.log_softmax(logits) return log_probs class GetNextSentenceOutput(nn.Cell): """ Get next sentence output. Args: config (BertConfig): The config of Bert. Returns: Tensor, next sentence output. """ def __init__(self, config): super(GetNextSentenceOutput, self).__init__() self.log_softmax = P.LogSoftmax() self.log_softmax.shard(((config.parallel_config.data_parallel, 1),)) weight_init = TruncatedNormal(config.initializer_range) self.dense = nn.Dense(config.hidden_size, 2, weight_init=weight_init, has_bias=True).to_float(config.compute_dtype) if config.parallel_config.vocab_emb_dp: self.dense.matmul.shard(((config.parallel_config.data_parallel, 1), (1, 1))) else: self.dense.matmul.shard(((config.parallel_config.data_parallel, config.parallel_config.model_parallel), (config.parallel_config.model_parallel, 1))) self.dtype = config.dtype self.cast = P.Cast() def construct(self, input_tensor): logits = self.dense(input_tensor) logits = self.cast(logits, self.dtype) log_prob = self.log_softmax(logits) return log_prob pengcheng.mind-test/mindformers/models/bert/bert_config.py000066400000000000000000000110471471400134000243130ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Bert Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['BertConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class BertConfig(BaseConfig): """ BERT config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['bert'] _support_list.extend(MindFormerBook.get_config_support_list()['tokcls']['bert']) _support_list.extend(MindFormerBook.get_config_support_list()['txtcls']['bert']) _support_list.extend(MindFormerBook.get_config_support_list()['qa']['bert']) def __init__(self, model_type: str = "bert", use_one_hot_embeddings: bool = False, num_labels: int = 1, assessment_method: str = "", dropout_prob: float = 0.1, batch_size: int = 16, seq_length: int = 128, vocab_size: int = 30522, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: str = "gelu", post_layernorm_residual: bool = True, hidden_dropout_prob: float = 0.1, attention_probs_dropout_prob: float = 0.1, max_position_embeddings: int = 128, type_vocab_size: int = 2, initializer_range: float = 0.02, use_relative_positions: bool = False, dtype: str = "float32", layernorm_dtype: str = "float32", softmax_dtype: str = "float32", compute_dtype: str = "float16", use_past: bool = False, parallel_config: str = "default", checkpoint_name_or_path: str = "", moe_config: str = "default", is_training: bool = True, **kwargs): super(BertConfig, self).__init__(**kwargs) self.model_type = model_type self.use_one_hot_embeddings = use_one_hot_embeddings self.num_labels = num_labels self.assessment_method = assessment_method self.dropout_prob = dropout_prob self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.intermediate_size = intermediate_size self.hidden_act = hidden_act self.post_layernorm_residual = post_layernorm_residual self.hidden_dropout_prob = hidden_dropout_prob self.attention_probs_dropout_prob = attention_probs_dropout_prob self.max_position_embeddings = max_position_embeddings self.type_vocab_size = type_vocab_size self.initializer_range = initializer_range self.use_relative_positions = use_relative_positions self.dtype = mstype.float32 if dtype == "float32" else mstype.float16 self.layernorm_dtype = mstype.float32 if layernorm_dtype == "float32" else mstype.float16 self.softmax_dtype = mstype.float32 if softmax_dtype == "float32" else mstype.float16 self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.use_past = use_past self.checkpoint_name_or_path = checkpoint_name_or_path self.parallel_config = default_transformer_config if parallel_config == "default" else parallel_config self.moe_config = default_moe_config if moe_config == "default" else moe_config self.is_training = is_training pengcheng.mind-test/mindformers/models/bert/bert_processor.py000066400000000000000000000046721471400134000250730ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BertProcessor """ from mindformers.mindformer_book import MindFormerBook from ..base_tokenizer import BaseTokenizer from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['BertProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class BertProcessor(BaseProcessor): """ Bert processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['bert'] _support_list.extend(MindFormerBook.get_processor_support_list()['tokcls']['bert']) _support_list.extend(MindFormerBook.get_processor_support_list()['txtcls']['bert']) def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(BertProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-test/mindformers/models/bert/bert_tokenizer.py000066400000000000000000000552021471400134000250610ustar00rootroot00000000000000# coding=utf-8 # Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The bert tokenizer""" import collections import os import unicodedata from typing import List, Optional from mindformers.tools import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_tokenizer import Tokenizer from ...mindformer_book import MindFormerBook __all__ = ['BertTokenizer', 'BasicTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} def _is_whitespace(char): """Checks whether `char` is a whitespace character.""" # \t, \n, and \r are technically control characters but we treat them # as whitespace since they are generally considered as such. if char in (' ', '\t', '\n', '\r'): return True cat = unicodedata.category(char) if cat == "Zs": return True return False def _is_control(char): """Checks whether `char` is a control character.""" # These are technically control characters but we count them as whitespace # characters. if char in ('\t', '\n', '\r'): return False cat = unicodedata.category(char) if cat.startswith("C"): return True return False def _is_punctuation(char): """Checks whether `char` is a punctuation character.""" cp = ord(char) # We treat all non-letter/number ASCII as punctuation. # Characters such as "^", "$", and "`" are not in the Unicode # Punctuation class but we treat them as punctuation anyways, for # consistency. if 33 <= cp <= 47 or 58 <= cp <= 64 or 91 <= cp <= 96 or 123 <= cp <= 126: return True cat = unicodedata.category(char) if cat.startswith("P"): return True return False def load_vocab(vocab_file): """Loads a vocabulary file into a dictionary.""" vocab = collections.OrderedDict() with open(vocab_file, "r", encoding="utf-8") as reader: tokens = reader.readlines() for index, token in enumerate(tokens): token = token.rstrip("\n") vocab[token] = index return vocab def whitespace_tokenize(text): """Runs basic whitespace cleaning and splitting on a piece of text.""" text = text.strip() if not text: return [] tokens = text.split() return tokens @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class BertTokenizer(Tokenizer): r""" Construct a BERT tokenizer. Based on WordPiece. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): File containing the vocabulary. do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. do_basic_tokenize (`bool`, *optional*, defaults to `True`): Whether or not to do basic tokenization before WordPiece. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` unk_token (`str`, *optional*, defaults to `"[UNK]"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. sep_token (`str`, *optional*, defaults to `"[SEP]"`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `"[PAD]"`): The token used for padding, for example when batching sequences of different lengths. cls_token (`str`, *optional*, defaults to `"[CLS]"`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. mask_token (`str`, *optional*, defaults to `"[MASK]"`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. is_tokenize_char (`bool`, *optional*, defaults to `False`): Whether or not to tokenize characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). """ vocab_files_names = VOCAB_FILES_NAMES FILE_LIST = ['tokenizer_config.json', 'special_tokens_map.json'] model_input_names = ["input_ids", "token_type_ids", "attention_mask"] _support_list = MindFormerBook.get_tokenizer_support_list()['bert'] _support_list.extend(MindFormerBook.get_config_support_list()['tokcls']['bert']) _support_list.extend(MindFormerBook.get_config_support_list()['txtcls']['bert']) _support_list.extend(MindFormerBook.get_config_support_list()['qa']['bert']) def __init__( self, vocab_file, do_lower_case=True, do_basic_tokenize=True, never_split=None, unk_token="[UNK]", sep_token="[SEP]", pad_token="[PAD]", cls_token="[CLS]", mask_token="[MASK]", tokenize_chinese_chars=True, strip_accents=None, is_tokenize_char=False, **kwargs, ): super().__init__( do_lower_case=do_lower_case, do_basic_tokenize=do_basic_tokenize, never_split=never_split, unk_token=unk_token, sep_token=sep_token, pad_token=pad_token, cls_token=cls_token, mask_token=mask_token, tokenize_chinese_chars=tokenize_chinese_chars, strip_accents=strip_accents, **kwargs, ) if not os.path.isfile(vocab_file): raise ValueError( f"Can't find a vocabulary file at path '{vocab_file}'. To load the vocabulary from a Google pretrained" " model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`" ) self.vocab_dict = load_vocab(vocab_file) self.ids_to_tokens = collections.OrderedDict([(ids, tok) for tok, ids in self.vocab_dict.items()]) self.do_basic_tokenize = do_basic_tokenize if do_basic_tokenize: self.basic_tokenizer = BasicTokenizer( do_lower_case=do_lower_case, never_split=never_split, tokenize_chinese_chars=tokenize_chinese_chars, strip_accents=strip_accents, ) self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab_dict, unk_token=self.unk_token) self.mask_index = [] self.is_tokenize_char = is_tokenize_char @property def do_lower_case(self): return self.basic_tokenizer.do_lower_case @property def vocab_size(self): return len(self.vocab_dict) def get_vocab(self): return dict(self.vocab_dict, **self.added_tokens_encoder) def _tokenize(self, text, **kwargs): split_tokens = [] if self.is_tokenize_char: for character in text: if self.do_lower_case: character = character.lower() if character in self.vocab_dict: split_tokens.append(character) else: split_tokens.append(self.unk_token) else: if self.do_basic_tokenize: for token in self._process_mask_tokens(text): # If the token is part of the never_split set if token in self.basic_tokenizer.never_split: split_tokens.append(token) else: split_tokens += self.wordpiece_tokenizer.tokenize(token) else: split_tokens = self.wordpiece_tokenizer.tokenize(text) return split_tokens def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.vocab_dict.get(token, self.vocab_dict.get(self.unk_token)) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" return self.ids_to_tokens.get(index, self.unk_token) def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" out_string = " ".join(tokens).replace(" ##", "").strip() return out_string def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. """ if token_ids_1 is None: return [self.cls_token_id] + token_ids_0 + [self.sep_token_id] cls = [self.cls_token_id] sep = [self.sep_token_id] return cls + token_ids_0 + sep + token_ids_1 + sep def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) if token_ids_1 is not None: return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1] return [1] + ([0] * len(token_ids_0)) + [1] def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ sep = [self.sep_token_id] cls = [self.cls_token_id] if token_ids_1 is None: return len(cls + token_ids_0 + sep) * [0] return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1] def save_vocabulary(self, save_directory, filename_prefix=None): index = 0 if os.path.isdir(save_directory): vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) else: vocab_file = (filename_prefix + "-" if filename_prefix else "") + save_directory with open(vocab_file, "w", encoding="utf-8") as writer: for token, token_index in sorted(self.vocab_dict.items(), key=lambda kv: kv[1]): if index != token_index: logger.warning( "Saving vocabulary to %s: vocabulary indices are not consecutive." " Please check that the vocabulary is not corrupted!", vocab_file ) index = token_index writer.write(token + "\n") index += 1 return vocab_file def _process_mask_tokens(self, text): """process mask tokens in text""" text_tokenize = [] if self._mask_token in text: while self._mask_token in text: ind = text.index(self._mask_token) text_tokenize.extend(self.basic_tokenizer.tokenize(text[:ind])) text_tokenize.append(self._mask_token) text = text[ind + len(self._mask_token):] text_tokenize.extend(self.basic_tokenizer.tokenize(text)) self.mask_index = [ind for ind, x in enumerate(text_tokenize) if x == self._mask_token] else: text_tokenize = self.basic_tokenizer.tokenize(text, never_split=self.all_special_tokens) return text_tokenize class BasicTokenizer: """ Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). """ def __init__(self, do_lower_case=True, never_split=None, tokenize_chinese_chars=True, strip_accents=None): if never_split is None: never_split = [] self.do_lower_case = do_lower_case self.never_split = set(never_split) self.tokenize_chinese_chars = tokenize_chinese_chars self.strip_accents = strip_accents def tokenize(self, text, never_split=None): """ Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. """ # union() returns a new set by concatenating the two sets. never_split = self.never_split.union(set(never_split)) if never_split else self.never_split text = self._clean_text(text) # This was added on November 1st, 2018 for the multilingual and Chinese # models. This is also applied to the English models now, but it doesn't # matter since the English models were not trained on any Chinese data # and generally don't have any Chinese data in them (there are Chinese # characters in the vocabulary because Wikipedia does have some Chinese # words in the English Wikipedia.). if self.tokenize_chinese_chars: text = self._tokenize_chinese_chars(text) orig_tokens = whitespace_tokenize(text) split_tokens = [] for token in orig_tokens: if token not in never_split: if self.do_lower_case: token = token.lower() if self.strip_accents is not False: token = self._run_strip_accents(token) elif self.strip_accents: token = self._run_strip_accents(token) split_tokens.extend(self._run_split_on_punc(token, never_split)) output_tokens = whitespace_tokenize(" ".join(split_tokens)) return output_tokens def _run_strip_accents(self, text): """Strips accents from a piece of text.""" text = unicodedata.normalize("NFD", text) output = [] for char in text: cat = unicodedata.category(char) if cat == "Mn": continue output.append(char) return "".join(output) def _run_split_on_punc(self, text, never_split=None): """Splits punctuation on a piece of text.""" if never_split is not None and text in never_split: return [text] chars = list(text) i = 0 start_new_word = True output = [] while i < len(chars): char = chars[i] if _is_punctuation(char): output.append([char]) start_new_word = True else: if start_new_word: output.append([]) start_new_word = False output[-1].append(char) i += 1 return ["".join(x) for x in output] def _tokenize_chinese_chars(self, text): """Adds whitespace around any CJK character.""" output = [] for char in text: cp = ord(char) if self._is_chinese_char(cp): output.append(" ") output.append(char) output.append(" ") else: output.append(char) return "".join(output) def _is_chinese_char(self, cp): """Checks whether CP is the codepoint of a CJK character.""" # This defines a "chinese character" as anything in the CJK Unicode block: # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block) # # Note that the CJK Unicode block is NOT all Japanese and Korean characters, # despite its name. The modern Korean Hangul alphabet is a different block, # as is Japanese Hiragana and Katakana. Those alphabets are used to write # space-separated words, so they are not treated specially and handled # like the all of the other languages. if ( (0x4E00 <= cp <= 0x9FFF) or (0x3400 <= cp <= 0x4DBF) or (0x20000 <= cp <= 0x2A6DF) or (0x2A700 <= cp <= 0x2B73F) or (0x2B740 <= cp <= 0x2B81F) or (0x2B820 <= cp <= 0x2CEAF) or (0xF900 <= cp <= 0xFAFF) or (0x2F800 <= cp <= 0x2FA1F) ): return True return False def _clean_text(self, text): """Performs invalid character removal and whitespace cleanup on text.""" output = [] for char in text: cp = ord(char) if cp == 0 or cp == 0xFFFD or _is_control(char): continue if _is_whitespace(char): output.append(" ") else: output.append(char) return "".join(output) class WordpieceTokenizer: """Runs WordPiece tokenization.""" def __init__(self, vocab, unk_token, max_input_chars_per_word=100): self.vocab_dict = vocab self.unk_token = unk_token self.max_input_chars_per_word = max_input_chars_per_word def tokenize(self, text): """ Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example, `input = "unaffable"` wil return as output `["un", "##aff", "##able"]`. Args: text: A single token or whitespace separated tokens. This should have already been passed through *BasicTokenizer*. Returns: A list of wordpiece tokens. """ output_tokens = [] for token in whitespace_tokenize(text): chars = list(token) if len(chars) > self.max_input_chars_per_word: output_tokens.append(self.unk_token) continue is_bad = False start = 0 sub_tokens = [] while start < len(chars): end = len(chars) cur_substr = None while start < end: substr = "".join(chars[start:end]) if start > 0: substr = "##" + substr if substr in self.vocab_dict: cur_substr = substr break end -= 1 if cur_substr is None: is_bad = True break sub_tokens.append(cur_substr) start = end if is_bad: output_tokens.append(self.unk_token) else: output_tokens.extend(sub_tokens) return output_tokens pengcheng.mind-test/mindformers/models/bert/convert_weight.py000066400000000000000000000206761471400134000250710ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/huggingface""" import argparse import numpy as np import torch from mindspore import save_checkpoint, Tensor def generate_params_dict(total_layers, mindspore_params_per_layer, torch_params_per_layer, mindspore_additional_params, torch_additional_params): """ Generate the total parameter mapping of mindspore and pytorch. Args: total_layers(int): The total layers of the net. mindspore_params_per_layer(list): The list of params per layer for the net of mindspore. torch_params_per_layer(list): The list of params per layer for the net of pytorch. mindspore_additional_params(list): The list of params outside the layer for the net of mindspore torch_additional_params(list): The list of params outside the layer for the net of pytorch. Returns: A list of tuple. The first element is the parameter name of mindspore, the another is the parameter name of pytorch. """ mapped_params = list(zip(mindspore_params_per_layer, torch_params_per_layer)) ms_extend_param_list = [] torch_extend_param_list = [] for i in range(total_layers): for ms_para, torch_para in mapped_params: src = ms_para.format(i) tgt = torch_para.format(i) ms_extend_param_list.append(src) torch_extend_param_list.append(tgt) mapped_params = list(zip(mindspore_additional_params, torch_additional_params)) for ms_para, torch_para in mapped_params: ms_extend_param_list.append(ms_para) torch_extend_param_list.append(torch_para) return list(zip(ms_extend_param_list, torch_extend_param_list)) def get_converted_ckpt(mapped_params, weight_dict): """ Print the keys of the loaded checkpoint Args: mapped_params(dict): The loaded checkpoint. The key is parameter name and value is the numpy array. weight_dict(dict): The loaded pytorch checkpoint. Returns: None """ new_ckpt_list = [] # Currently, the ms_extend_param the torch_extend_param is the full parameters. for src, tgt in mapped_params: if tgt not in weight_dict: if "LayerNorm.gamma" in tgt: tgt = tgt.replace("gamma", "weight") if "LayerNorm.beta" in tgt: tgt = tgt.replace("beta", "bias") value = weight_dict[tgt].numpy() if 'output.dense.weight' in tgt or 'intermediate.dense.weight' in tgt: value = np.transpose(value, [1, 0]) print(f"Mapping table Mindspore:{src:<30} \t Torch:{tgt:<30} with shape {value.shape}") new_ckpt_list.append({"data": Tensor(value), "name": src}) return new_ckpt_list def split_torch_attention(state): s = list(state.keys()) for name in s: if name.endswith('attn.c_attn.weight') or name.endswith('attn.c_attn.bias'): value = state.pop(name) q, k, v = np.split(value.numpy(), 3, -1) state[name + '.q'] = torch.tensor(q, dtype=value.dtype) state[name + '.k'] = torch.tensor(k) state[name + '.v'] = torch.tensor(v) if __name__ == '__main__': parser = argparse.ArgumentParser(description="BERT convert script") parser.add_argument('--layers', type=int, default=1, help="The number of layers of the model to be converted.") parser.add_argument("--torch_path", type=str, default=None, required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="The output mindspore checkpoint path.", help="Use device nums, default is 128.") opt = parser.parse_args() state_dict = torch.load(opt.torch_path, map_location='cpu') ms_name = [ "bert.bert_encoder.encoder.blocks.{}.attention.dense1.weight", "bert.bert_encoder.encoder.blocks.{}.attention.dense1.bias", "bert.bert_encoder.encoder.blocks.{}.attention.dense2.weight", "bert.bert_encoder.encoder.blocks.{}.attention.dense2.bias", "bert.bert_encoder.encoder.blocks.{}.attention.dense3.weight", "bert.bert_encoder.encoder.blocks.{}.attention.dense3.bias", "bert.bert_encoder.encoder.blocks.{}.attention.projection.weight", "bert.bert_encoder.encoder.blocks.{}.attention.projection.bias", "bert.bert_encoder.encoder.blocks.{}.layernorm2.gamma", "bert.bert_encoder.encoder.blocks.{}.layernorm2.beta", "bert.bert_encoder.encoder.blocks.{}.output.mapping.weight", "bert.bert_encoder.encoder.blocks.{}.output.mapping.bias", "bert.bert_encoder.encoder.blocks.{}.output.projection.weight", "bert.bert_encoder.encoder.blocks.{}.output.projection.bias", "bert.bert_encoder.encoder.blocks.{}.layernorm1.gamma", "bert.bert_encoder.encoder.blocks.{}.layernorm1.beta", ] torch_name = [ "bert.encoder.layer.{}.attention.self.query.weight", "bert.encoder.layer.{}.attention.self.query.bias", "bert.encoder.layer.{}.attention.self.key.weight", "bert.encoder.layer.{}.attention.self.key.bias", "bert.encoder.layer.{}.attention.self.value.weight", "bert.encoder.layer.{}.attention.self.value.bias", "bert.encoder.layer.{}.attention.output.dense.weight", "bert.encoder.layer.{}.attention.output.dense.bias", "bert.encoder.layer.{}.attention.output.LayerNorm.gamma", "bert.encoder.layer.{}.attention.output.LayerNorm.beta", "bert.encoder.layer.{}.intermediate.dense.weight", "bert.encoder.layer.{}.intermediate.dense.bias", "bert.encoder.layer.{}.output.dense.weight", "bert.encoder.layer.{}.output.dense.bias", "bert.encoder.layer.{}.output.LayerNorm.gamma", "bert.encoder.layer.{}.output.LayerNorm.beta", ] addition_mindspore = [ "bert.word_embedding.embedding_table", "bert.embedding_postprocessor.full_position_embedding.embedding_table", "bert.embedding_postprocessor.token_type_embedding.embedding_table", "bert.embedding_postprocessor.layernorm.gamma", "bert.embedding_postprocessor.layernorm.beta", "bert.dense.weight", "bert.dense.bias", "bert.mlmloss.dense.weight", "bert.mlmloss.dense.bias", "bert.mlmloss.layernorm.gamma", "bert.mlmloss.layernorm.beta", "bert.mlmloss.vocab_dense.weight", ] addition_torch = [ "bert.embeddings.word_embeddings.weight", "bert.embeddings.position_embeddings.weight", "bert.embeddings.token_type_embeddings.weight", "bert.embeddings.LayerNorm.gamma", "bert.embeddings.LayerNorm.beta", "bert.pooler.dense.weight", "bert.pooler.dense.bias", "cls.predictions.transform.dense.weight", "cls.predictions.transform.dense.bias", "cls.predictions.transform.LayerNorm.gamma", "cls.predictions.transform.LayerNorm.beta", "cls.predictions.decoder.weight" ] mapped_param = generate_params_dict(total_layers=opt.layers, mindspore_params_per_layer=ms_name, torch_params_per_layer=torch_name, mindspore_additional_params=addition_mindspore, torch_additional_params=addition_torch) split_torch_attention(state_dict) new_ckpt = get_converted_ckpt(mapped_param, state_dict) save_checkpoint(new_ckpt, opt.mindspore_path) print(f"Convert finished, the output is saved to {opt.mindspore_path}") pengcheng.mind-test/mindformers/models/blip2/000077500000000000000000000000001471400134000215315ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/blip2/__init__.py000066400000000000000000000021431471400134000236420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """blip2 init""" from .blip2_config import Blip2Config from .blip2_llama import Blip2Llama, Blip2ImageToTextGeneration from .blip2_qformer import Blip2Qformer, Blip2Classifier from .blip2_processor import Blip2ImageProcessor, Blip2Processor from .qformer import BertLMHeadModel __all__ = ['BertLMHeadModel', 'Blip2Config', 'Blip2Classifier', 'Blip2Llama', 'Blip2ImageToTextGeneration', 'Blip2Qformer', 'Blip2ImageProcessor', 'Blip2Processor'] pengcheng.mind-test/mindformers/models/blip2/blip2.py000066400000000000000000000171031471400134000231150ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/salesforce/LAVIS/tree/main/lavis/models/blip2_models # ============================================================================ """ BLIP2 Base Model, contains Blip2Base, ViTModelForBlip2, as well as itm computing procedures. """ import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops from mindspore.common.initializer import initializer, Normal from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from mindformers.models.blip2.qformer import BertLMHeadModel from mindformers.models.vit.vit import ViTModel, ViTConfig from mindformers.modules.layers import LayerNorm class ViTModelForBlip2(ViTModel): """ ViTModel For Blip2 Models, loading a pretrained weight. forward will return the penultimate output. """ _support_list = MindFormerBook.get_config_support_list()['vit'] def __init__(self, config: ViTConfig): super(ViTModelForBlip2, self).__init__(config) self.load_checkpoint(config) def construct(self, image): return self.construct_without_pool(image) class Blip2Base(BaseModel): """ BLIP2 BaseModel, all BLIP2 models inherit this class. """ _support_list = MindFormerBook.get_model_support_list()['blip2'] def init_qformer(self): """ Init qformer for blip2 model Raises: ValueError: qformer config wrong Returns: qformer, query_tokens """ qformer_config = self.config.qformer_config qformer_config.parallel_config = self.config.parallel_config qformer = BertLMHeadModel(qformer_config) if qformer is None: raise ValueError("qformer configuration is wrong. \ please check 'qformer_config' is set in Blip2Config") query_tokens = ms.Parameter(initializer( Normal(mean=0.0, sigma=qformer_config.initializer_range), [1, qformer_config.query_length, qformer_config.hidden_size])) return qformer, query_tokens def init_vision_encoder(self): """ init vision encoder for blip2 model Raises: ValueError: vit config wrong Returns: visual_encoder, ln_vision """ vision_config = self.config.vision_config if vision_config is not None: vision_config.parallel_config = self.config.parallel_config visual_encoder = ViTModelForBlip2(vision_config) if visual_encoder is None: raise ValueError("visual_encoder configuration is wrong. \ please check 'vision_config' is set in Blip2Config") for block in visual_encoder.blocks: mapping = block.output.mapping if mapping.activation_flag and isinstance(mapping.activation, nn.GELU): mapping.activation = nn.GELU(approximate=False) ln_vision = LayerNorm(visual_encoder.config.embed_dim) return visual_encoder, ln_vision def append_text_outputs(self, text_input, text_feats, text_ids): """append_text_outputs Args: text_input: text_input, can be: 1) 2-dimension: 1 image 1 caption 2) 3-dimension: 1 image multiple captions text_feats: output to append text_ids: output to append """ if text_input.ndim == 3: for input_ids in text_input: text_embed, _ = self.forward_text(input_ids) text_feat = ops.L2Normalize(axis=1, epsilon=1e-12)(self.text_proj(text_embed[:, 0, :])) text_feats.append(text_feat) text_ids.append(input_ids) else: text_embed, _ = self.forward_text(text_input) text_feat = ops.L2Normalize(axis=1, epsilon=1e-12)(self.text_proj(text_embed[:, 0, :])) text_feats.append(text_feat) text_ids.append(text_input) def append_image_outputs(self, image_input, image_feats, vit_outputs): """append_image_outputs Args: image_input: image_input, can be: 1) 4-dimension: 1 caption 1 image 2) 5-dimension: 1 caption multiple images image_feats: output to append vit_outputs: output to append """ if image_input.ndim == 5: for image in image_input: image_embed, vit_output = self.forward_image(image) image_embed = self.vision_proj(image_embed) image_feat = ops.L2Normalize(axis=-1, epsilon=1e-12)(image_embed) image_feats.append(image_feat) vit_outputs.append(vit_output) else: image_embed, vit_output = self.forward_image(image_input) image_embed = self.vision_proj(image_embed) image_feat = ops.L2Normalize(axis=-1, epsilon=1e-12)(image_embed) image_feats.append(image_feat) vit_outputs.append(vit_output) def prepare_inputs_for_itm_eval(self, dataloader): """ prepare inputs of BLIP-2 for image_to_text_reatrieval task. Args: dataloader (GeneratorDataset): image-caption pair dataloader Returns: image_feats, text_feats, vit_outputs, text_ids, text_atts """ image_feats = [] text_feats = [] vit_outputs = [] text_ids = [] for image_input, text_input in dataloader: self.append_text_outputs(text_input, text_feats, text_ids) self.append_image_outputs(image_input, image_feats, vit_outputs) image_feats = ops.concat(image_feats, axis=0) text_feats = ops.concat(text_feats, axis=0) vit_outputs = ops.concat(vit_outputs, axis=0) text_ids = ops.concat(text_ids, axis=0) return ( image_feats, text_feats, vit_outputs, text_ids, ) def compute_extra_itm(self, extra_args, i, k_test, topk_idx, i2t=True): """ compute extra_itm score, for those model have its own itm computing method, like blip-2. Args: extra_args (tuple): required args for this method. i (int): index in the sim_matrix k_test (int): k_test number topk_idx (Tensor): topk_idx i2t (bool, optional): is image-to-text or text-to-image matching. Returns: _type_: extra itm score """ vit_outputs, text_ids = extra_args if i2t: image_inputs = ms.numpy.tile(vit_outputs[i], (k_test, 1, 1)) score = self.compute_itm( image_inputs=image_inputs, text_ids=text_ids[topk_idx], vit_computed=True ).astype(ms.float32) else: image_inputs = vit_outputs[topk_idx] score = self.compute_itm( image_inputs=image_inputs, text_ids=ms.numpy.tile(text_ids[i], (k_test, 1)), vit_computed=True).astype(ms.float32) return score pengcheng.mind-test/mindformers/models/blip2/blip2_config.py000066400000000000000000000145261471400134000244500ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Blip2 Config API""" from typing import Optional import mindspore.common.dtype as mstype from mindformers.models.blip2.qformer_config import QFormerConfig from mindformers.models.llama import LlamaConfig from mindformers.models.vit import ViTConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_config import BaseConfig from mindformers.mindformer_book import MindFormerBook from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig __all__ = ['Blip2Config'] default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class Blip2Config(BaseConfig): r""" Config For BLIP2 Module Args: model_type (Optional[int]): model type for blip2 model, default is 'blip2'. batch_size (Optional[int]): batch size for the model freeze_vision (Optional[bool]): whether to freeze vit weights, default is True. freeze_text (Optional[bool]): whether to freeze LLM weights, default is True. max_txt_len (Optional[int]): max text length for llama model. prompt (Optional[str]): prompt for llama model. prompt_length (Optional[int]): prompt length for llama model. checkpoint_name_or_path (Optional[str]): checkpoint path or name. dtype (Optional[str]): layer digital type, default is "float32". compute_dtype (Optional[str]): Linear layer compute dtype, default is "float16". layernorm_compute_type (Optional[str]): layernorm compute dtype, default is "float32". softmax_compute_type (Optional[str]):softmax compute dtype, default is "float32". vision_config (Optional[ViTConfig]): config for ViTModel. qformer_config (Optional[QFormerConfig]): config for qformer. text_config (Optional[LlamaConfig]): config for LLM model, like llama. parallel_config (TransformerOpParallelConfig): parallel_config. is_training (Optional[bool]): whether the model is in training state. Examples: >>> from mindformers import Blip2Config >>> Blip2Config() {'model_type': 'blip2', 'batch_size': 8, 'freeze_vision': True, 'freeze_text': True, 'max_txt_len': 32, 'checkpoint_name_or_path': None, 'prompt': '', 'prompt_length': 0 ....} """ _support_list = MindFormerBook.get_config_support_list()['blip2'] def __init__(self, model_type: str = "blip2", batch_size: int = 8, freeze_vision: bool = True, freeze_text: bool = True, max_txt_len: int = 32, prompt: bool = False, prompt_length: int = 0, checkpoint_name_or_path: str = None, dtype: str = "float32", compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", vision_config: Optional[ViTConfig] = None, qformer_config: Optional[QFormerConfig] = None, text_config: Optional[LlamaConfig] = None, parallel_config: TransformerOpParallelConfig = default_parallel_config, is_training: bool = True, **kwargs): super(Blip2Config, self).__init__(**kwargs) self.model_type = model_type self.batch_size = batch_size self.freeze_vision = freeze_vision self.freeze_text = freeze_text self.max_txt_len = max_txt_len self.checkpoint_name_or_path = checkpoint_name_or_path self.prompt = prompt self.prompt_length = prompt_length self.parallel_config = parallel_config self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.layernorm_compute_type = mstype.float32 if layernorm_compute_type == "float32" else mstype.float16 self.softmax_compute_type = mstype.float32 if softmax_compute_type == "float32" else mstype.float16 self.dtype = mstype.float32 if dtype == "float32" else mstype.float16 self.is_training = is_training self.vision_config = vision_config if vision_config is not None else ViTConfig() self.qformer_config = QFormerConfig(**qformer_config) if qformer_config is not None else QFormerConfig() self.text_config = text_config # pass configs to submodule config self.qformer_config.encoder_width = self.vision_config.embed_dim self.qformer_config.parallel_config = parallel_config self.qformer_config.compute_dtype = self.compute_dtype self.qformer_config.layernorm_dtype = self.layernorm_compute_type self.qformer_config.softmax_dtype = self.softmax_compute_type self.qformer_config.dtype = self.dtype self.vision_config.parallel_config = parallel_config self.vision_config.compute_dtype = self.compute_dtype self.vision_config.layernorm_compute_type = self.layernorm_compute_type self.vision_config.softmax_compute_type = self.softmax_compute_type self.vision_config.dtype = self.dtype # first stage is without text config if self.text_config is not None: self.text_config.parallel_config = parallel_config self.text_config.compute_dtype = self.compute_dtype self.text_config.layernorm_compute_type = self.layernorm_compute_type self.text_config.softmax_compute_type = self.softmax_compute_type self.text_config.dtype = self.dtype pengcheng.mind-test/mindformers/models/blip2/blip2_llama.py000066400000000000000000000410571471400134000242700ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/salesforce/LAVIS/tree/main/lavis/models/blip2_models # ============================================================================ """Blip2 second stage pretrain and infer with Llama model""" import mindspore as ms import mindspore.common.dtype as mstype from mindspore import ops, Tensor from mindspore.ops import operations as P from mindformers.mindformer_book import MindFormerBook from mindformers.models.blip2.blip2 import Blip2Base from mindformers.models.blip2.blip2_config import Blip2Config from mindformers.models.llama import LlamaConfig from mindformers.models.llama.llama import LlamaForCausalLM, LlamaModel from mindformers.modules.layers import Linear from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['Blip2Llama', 'Blip2ImageToTextGeneration'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class LlamaModelForBlip2(LlamaModel): r""" Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`], it inherits from LlamaModel and is used to adapt blip2. Args: config(LlamaConfig): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 Returns: output: Tensor, the output of llama decoder layer """ def __init__(self, config): super().__init__(config) # pylint: disable=W0221 def construct(self, input_embeddings: Tensor, input_attention_masks: Tensor, input_position=None, init_reset=True, batch_valid_length=None): bs, seq_len, _ = input_embeddings.shape if self.use_past: if not isinstance(init_reset, Tensor): init_reset = Tensor([init_reset], mstype.bool_) if not isinstance(batch_valid_length, Tensor): batch_valid_length = self.ones((bs, 1), mstype.int32) if self.is_first_iteration: freqs_cis = (self.tile(self.reshape(self.freqs_cos, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.tile(self.reshape(self.freqs_sin, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.swap_mask) mask = self.get_attention_mask(input_attention_masks) # mask: [bs, seq, seq] else: cur_pos = batch_valid_length - 1 valid_length = self.reshape(cur_pos, (-1, 1, 1)) freqs_cis = (self.reshape(self.gather_past(self.freqs_cos, cur_pos, 0), (bs, 1, seq_len, -1)), self.reshape(self.gather_past(self.freqs_sin, cur_pos, 0), (bs, 1, seq_len, -1)), self.swap_mask) mask = self.cast(self.le_past(self.range, valid_length), self.dtype) # mask: [bs, 1, 1] mask = self.sub(self.one, self.cast(mask, self.dtype)) if not self.use_flash_attention: mask = self.expand_dims(mask, 1) mask = self.mul_mask(mask, self.multiply_data) h = input_embeddings # h: [bs, seq/1, hidden_dim] for i in range(self.num_layers): h, _ = self.layers[i](h, freqs_cis, mask, init_reset=init_reset, batch_valid_length=batch_valid_length) output = self.norm_out(h) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class LlamaForBlip2(LlamaForCausalLM): r""" Provide llama inference loss or logits through network for blip2. Args: config (LlamaConfig): The config of llama model.` Inputs: input_embeddings(Tensor): the embeddings of inputs, shape :math:`(batch, seq\_length, hidden\_size)` input_ids(Tensor): the input ids of inputs, shape: math:`(batch, seq\_length)` label_ids(Tensor): Labels for computing loss. Tensor of shape :math:`(batch, seq\_length)` attention_mask(Tensor): attention_mask, used by model.construct Returns: Tensor, the loss of the network. Examples: >>> from mindformers.models.llama import LlamaConfig >>> from mindformers.models.blip2.blip2_llama import LlamaForBlip2 >>> config = LlamaConfig(batch_size=1) >>> network = LlamaForBlip2(config=config) """ def __init__(self, config: LlamaConfig = None): checkpoint_name_or_path = config.checkpoint_name_or_path config.checkpoint_name_or_path = "" super().__init__(config) self.model = LlamaModelForBlip2(config) self.config.checkpoint_name_or_path = checkpoint_name_or_path self.load_checkpoint(config) self.concat_2d = P.Concat(axis=1) self.concat_3d = P.Concat(axis=1) self.not_equal = P.NotEqual() self.ones = P.Ones() self.cast = P.Cast() self.is_first_iteration = True # pylint: disable=W0221 def construct(self, input_embeddings=None, input_ids=None, labels=None, input_position=None, attention_mask=None, init_reset=True, batch_valid_length=None): """LlamaForBlip2 forward.""" if input_embeddings is None and input_ids is not None: # for incremental infer input_embeddings = self.model.tok_embeddings(input_ids) batch_size, seq_length, _ = input_embeddings.shape output = self.model(input_embeddings=input_embeddings, input_attention_masks=attention_mask, input_position=input_position, init_reset=init_reset, batch_valid_length=batch_valid_length) logits = self.lm_head(output) logits = self.cast(logits, mstype.float32) if labels is None: # inference logits = self.reshape(logits, (batch_size, seq_length, -1)) return logits, attention_mask if logits.ndim > 2: logits = self.reshape(logits, (-1, logits.shape[-1])) labels = self.reshape(labels, (-1,)) label_mask = self.cast(self.not_equal(labels, self.ignore_token_id), mstype.float32) attention_mask = self.reshape(attention_mask, (-1,)) input_mask = self.mul(attention_mask, label_mask) input_mask = self.reshape(input_mask, (-1,)) loss = self.loss(logits, labels, input_mask) return loss def prepare_inputs_for_generation(self, input_ids, **kwargs): if self.is_first_iteration or not self.config.use_past: image_embeddings = kwargs.pop("image_embeds") image_embeddings_atts = self.ones(image_embeddings.shape[:-1], mstype.float32) image_embeddings_length = image_embeddings.shape[1] text_input_ids = Tensor(input_ids[:, image_embeddings_length:], mstype.int32) text_embeddings = self.model.tok_embeddings(text_input_ids) text_embeddings = self.cast(text_embeddings, mstype.float32) text_embeddings_atts = self.cast(self.not_equal(text_input_ids, self.pad_token_id), mstype.float32) llama_inputs_embeds = self.concat_3d([image_embeddings, text_embeddings]) llama_inputs_attention_mask = self.concat_2d([image_embeddings_atts, text_embeddings_atts]) return { "input_ids": Tensor(input_ids, mstype.int32), "input_embeddings": llama_inputs_embeds, "attention_mask": llama_inputs_attention_mask } return { "input_ids": Tensor(input_ids, mstype.int32), "input_embeddings": None, "attention_mask": None } @MindFormerRegister.register(MindFormerModuleType.MODELS) class Blip2Llama(Blip2Base): """ BLIP2 Llama model """ _support_list = MindFormerBook.get_model_support_list()['blip2']['2-stg'] def __init__(self, config: Blip2Config, **kwargs): super(Blip2Llama, self).__init__(config, **kwargs) self.config = config if config is not None else Blip2Config() self.visual_encoder, self.ln_vision = self.init_vision_encoder() if config.freeze_vision: logger.info("freeze vision encoder") for param in self.visual_encoder.trainable_params(): param.requires_grad = False self.visual_encoder.set_train(False) self.qformer, self.query_tokens = self.init_qformer() self.qformer.cls = None self.qformer.bert.embeddings.word_embeddings = None self.qformer.bert.embeddings.position_embeddings = None for layer in self.qformer.bert.encoder.layer: layer.output = None layer.intermediate = None if config.text_config.seq_length < config.max_txt_len + config.qformer_config.query_length: raise ValueError( "seq_length should be greater than sum of max_text_len and num_query_token %d, but got %d" % (config.max_txt_len + config.qformer_config.query_length, config.text_config.seq_length)) self.llama_model = LlamaForBlip2(config.text_config) if config.freeze_text: logger.info("freeze llm model") for param in self.llama_model.trainable_params(): param.requires_grad = False self.llama_model.set_train(False) dp = config.parallel_config.data_parallel self.llama_proj = Linear(in_channels=self.config.qformer_config.hidden_size, out_channels=self.config.text_config.hidden_size, param_init_type=config.dtype, compute_dtype=config.compute_dtype) if config.checkpoint_name_or_path: logger.info( "load blip2 first stage pretrained model for vision model and qformer, checkpoint_name_or_path: " "%s. pretrained llm model: %s", config.checkpoint_name_or_path, config.text_config.checkpoint_name_or_path) self.load_checkpoint(config) else: if config.vision_config.checkpoint_name_or_path: vision_checkpoint = config.vision_config.checkpoint_name_or_path else: vision_checkpoint = 'not configured' if config.text_config.checkpoint_name_or_path: text_checkpoint = config.text_config.checkpoint_name_or_path else: text_checkpoint = 'not configured' if config.qformer_config.checkpoint_name_or_path: qformer_checkpoint = config.qformer_config.checkpoint_name_or_path else: qformer_checkpoint = 'not configured' logger.info("training blip2 second stage, pretrained vision model: %s, pretrained llm model: %s, " "pretrained qformer: %s", vision_checkpoint, text_checkpoint, qformer_checkpoint) self.eos_token_id = config.text_config.eos_token_id self.pad_token_id = config.text_config.pad_token_id self.ignore_token_id = config.text_config.ignore_token_id self.max_txt_len = config.max_txt_len self.prompt = config.prompt self.prompt_length = config.prompt_length self.broadcast_to = P.BroadcastTo((dp * self.config.batch_size, self.config.qformer_config.query_length, self.config.qformer_config.hidden_size)).shard(((1, 1, 1),)) self.fill = P.Fill().shard(((dp, 1),)) self.masked_fill = P.MaskedFill().shard(((dp, 1), ())) self.ones = P.Ones().shard(((dp, 1),)) self.concat_2d = P.Concat(axis=1).shard(((dp, 1), (dp, 1))) self.concat_3d = P.Concat(axis=1).shard(((dp, 1, 1), (dp, 1, 1))) self.not_equal = P.NotEqual().shard(((dp, 1), ())) self.slice = P.StridedSlice().shard(((dp, 1),)) self.cast = P.Cast() # pylint: disable=W0221 def construct(self, image: ms.Tensor, text_input_ids: ms.Tensor): """Blip2Llama forward.""" projected_qformer_output = self.forward_qformer_and_proj(image) projected_qformer_output_atts = self.ones(projected_qformer_output.shape[:-1], mstype.float32) batch_size, seq_length = text_input_ids.shape tokens = self.slice(text_input_ids, (0, 0), (batch_size, seq_length - 1), (1, 1)) text_label_ids = self.slice(text_input_ids, (0, 1), (batch_size, seq_length), (1, 1)) targets = ops.masked_fill(text_label_ids, text_label_ids == self.pad_token_id, self.ignore_token_id) if self.prompt: prompt_label = self.ignore_token_id * self.ones((batch_size, self.prompt_length), mstype.int32) targets = self.concat_2d([prompt_label, targets[:, self.prompt_length:]]) empty_targets = self.fill(mstype.int32, projected_qformer_output_atts.shape, self.ignore_token_id) targets = self.concat_2d([empty_targets, targets]) # [batch_size, 1, query_size + max_txt_length] # [batch_size, max_txt_length, llama_hidden_size] text_inputs_embeds = self.llama_model.model.tok_embeddings(tokens) text_inputs_embeds = self.cast(text_inputs_embeds, mstype.float32) text_inputs_atts = self.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) # [batch_size, query_size + max_txt_length, llama_hidden_size] llama_inputs_embeds = self.concat_3d([projected_qformer_output, text_inputs_embeds]) # [batch_size, query_size + max_txt_length] llama_inputs_attention_mask = self.concat_2d([projected_qformer_output_atts, text_inputs_atts]) loss = self.llama_model( input_embeddings=llama_inputs_embeds, labels=targets, attention_mask=llama_inputs_attention_mask ) return loss def forward_qformer_and_proj(self, image: ms.Tensor): """forward the image tensor to the qformer, then project the output to adapt the dimension""" image_embeds = self.visual_encoder(image) image_embeds = self.ln_vision(image_embeds) # [batch_size, vit_seq_length, vit_encoder_hidden_width] image_atts = self.ones(image_embeds.shape[:-1], mstype.float32) # [batch_size, vit_seq_length] query_tokens = self.broadcast_to(self.query_tokens) # [batch_size, query_size, qformer_hidden_size] query_output = self.qformer.bert(query_embeds=query_tokens, encoder_hidden_states=image_embeds, encoder_attention_mask=image_atts, use_cache=True, return_dict=True) # [batch_size, query_size, qformer_hidden_size] -> [batch_size, query_size, llama_hidden_size] return self.llama_proj(query_output[0]) @MindFormerRegister.register(MindFormerModuleType.MODELS) class Blip2ImageToTextGeneration(Blip2Llama): """ Blip2ImageToTextGeneration rely on Blip2Llama, used for image to text genearation. Usage: >>> from mindformers import AutoModel >>> model_type = 'itt_blip2_stage2_vit_g_llama_7b' >>> model = AutoModel.from_pretrained(model_type) """ _support_list = MindFormerBook.get_model_support_list()['itt']['blip2'] def __init__(self, config: Blip2Config, **kwargs): super(Blip2ImageToTextGeneration, self).__init__(config, **kwargs) self.llama_model.set_train(False) self.one_prefix = ops.Ones() def generate_text_for_image(self, image: ms.Tensor, prompt_input_ids: ms.Tensor): batch_size = image.shape[0] prefix_ones = self.one_prefix((batch_size, self.config.qformer_config.query_length), mstype.int32) text_input_ids = self.concat_2d([prefix_ones, prompt_input_ids]) projected_qformer_output = self.forward_qformer_and_proj(image) output_ids = self.llama_model.generate(input_ids=text_input_ids.asnumpy(), image_embeds=projected_qformer_output) return output_ids pengcheng.mind-test/mindformers/models/blip2/blip2_processor.py000066400000000000000000000157521471400134000252240ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BLIP2Processor """ from typing import Optional, Union, List import numpy as np import PIL import PIL.Image import mindspore as ms from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.dataset.transforms.vision_transforms import ( BatchPILize, BatchResize, BatchToTensor, BatchNormalize ) from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class Blip2ImageProcessor(BaseImageProcessor): """ Blip2ImageProcessor. Args: image_resolution (int): The target size. Examples: >>> from mindformers import Blip2ImageProcessor >>> from mindformers.tools.image_tools import load_image >>> processor = Blip2ImageProcessor(image_size=224) >>> image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> processor(image) Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[-1.55868769e+00, -1.52949083e+00, ... -1.48569560e+00, -1.48569560e+00], [-1.54408932e+00, -1.52949083e+00, ... -1.50029397e+00, -1.50029397e+00], [-1.52949083e+00, -1.52949083e+00, ... -1.50029397e+00, -1.50029397e+00], ... [-1.38067937e+00, -1.48021984e+00, ... -1.30957901e+00, -1.40911949e+00], [-1.46599972e+00, -1.43755960e+00, ... -1.48021984e+00, -1.43755960e+00], [-1.40911949e+00, -1.28113890e+00, ... -1.48021984e+00, -1.43755960e+00]]]]) """ def __init__(self, image_size: Optional[int] = 224, interpolation: Optional[str] = 'bicubic', mean=None, std=None, is_hwc=False): self.pilize = BatchPILize() super(Blip2ImageProcessor, self).__init__() if isinstance(image_size, int): image_size = (image_size,) * 2 self.resize = BatchResize(image_size, interpolation=interpolation) self.to_tensor = BatchToTensor() if mean is None: mean = (0.48145466, 0.4578275, 0.40821073) if std is None: std = (0.26862954, 0.26130258, 0.27577711) self.normalize = BatchNormalize(mean, std, is_hwc) def preprocess(self, images: Union[ms.Tensor, PIL.Image.Image, np.ndarray, List[PIL.Image.Image]], **kwargs): r""" Preprocess Required By Base Processor. Args: images (ms.Tensor, PIL.Image, numpy.array, List[PIL.Image]): A batch of images. Return: A 4-rank tensor for a batch of images. """ images = self.pilize(images) images = self.resize(images) images = self.to_tensor(images) images = self.normalize(images) kwargs.pop("other", None) if isinstance(images, list): return ms.Tensor(np.row_stack([np.expand_dims(item, axis=0) for item in images])) if len(images.shape) == 4: return ms.Tensor(images) return ms.Tensor(np.expand_dims(images, axis=0)) def _bhwc_check(self, image_batch: Union[ms.Tensor, PIL.Image.Image, np.ndarray, List[PIL.Image.Image]]): r"""Bhwc_check""" if isinstance(image_batch, np.ndarray): if image_batch.shape[-1] == 3: return True if isinstance(image_batch, ms.Tensor): if image_batch.asnumpy().shape[-1] == 3: return True if isinstance(image_batch, (list, PIL.Image.Image)): return True return False @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class Blip2Processor(BaseProcessor): r"""Blip2 Processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. Args: image_processor (BaseImageProcessor): Used for process image data. tokenizer (BaseTokenizer): Used for process text data. max_length (Optional[int]): The length of text tokens. padding (Optional[str]): The padding strategy of tokenizer, [None, "max_length"]. return_tensors (Optional[str]): The type of returned tensors for tokenizer, [None, "ms"]. Examples: >>> from mindformers import Blip2Processor >>> from mindformers.tools.image_tools import load_image >>> image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> text = ["a boy", "a girl"] >>> Blip2Processor.show_support_list() INFO - support list of Blip2Processor is: INFO - ['blip2_stage1_vit_g', 'blip2_stage1_classification'] INFO - ------------------------------------- >>> processor = Blip2Processor.from_pretrained('blip2_stage1_vit_g') INFO - processor built successfully! >>> processor(image, text) {'image': Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[-1.55868769e+00, -1.52949083e+00, -1.55868769e+00 ... -1.48569560e+00, -1.48569560e+00], [-1.54408932e+00, -1.52949083e+00, -1.54408932e+00 ... -1.50029397e+00, -1.50029397e+00], [-1.52949083e+00, -1.52949083e+00, -1.52949083e+00 ... -1.50029397e+00, -1.50029397e+00], ... [-1.38067937e+00, -1.48021984e+00, -1.38067937e+00 ... -1.30957901e+00, -1.40911949e+00], [-1.46599972e+00, -1.43755960e+00, -1.26691878e+00 ... -1.48021984e+00, -1.43755960e+00], [-1.40911949e+00, -1.28113890e+00, -1.30957901e+00 ... -1.48021984e+00, -1.43755960e+00] ]]]), 'text': Tensor(shape=[2, 32], dtype=Int32, value= [[ 101, 1037, 2879 ... 0, 0, 0], [ 101, 1037, 2611 ... 0, 0, 0]])} """ _support_list = MindFormerBook.get_processor_support_list()['blip2'] def __init__(self, image_processor, tokenizer, max_length=32, padding='max_length', return_tensors='ms'): super(Blip2Processor, self).__init__( image_processor=image_processor, tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors) pengcheng.mind-test/mindformers/models/blip2/blip2_qformer.py000066400000000000000000000531171471400134000246550ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/salesforce/LAVIS/tree/main/lavis/models/blip2_models # ============================================================================ """ Blip2 Qformer, link to ViT. the main model for image-text pretraining. """ import os import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops import mindspore.common.dtype as mstype from mindspore import Tensor import mindspore.numpy as np from mindspore.ops import operations as P from mindformers.mindformer_book import MindFormerBook from mindformers.modules.layers import Linear from mindformers.models.blip2.blip2 import Blip2Base from mindformers.models.blip2.blip2_config import Blip2Config from mindformers.models.blip2.qformer import CrossEntropyLoss from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger def locate_card(rank_id: Tensor, group_size: int = 1): """ locate card number for ops.BroadCast, for graph mode. """ for i in range(0, group_size): if i == rank_id: return i return 0 def choose_idx_with_prob(weight: Tensor, batch_size: int = 1): """ choose idx depend on probability, replace torch.multinomial """ weight_acc = ops.cumsum(weight, -1) rand_x = np.rand([1], dtype=weight_acc.dtype) * weight_acc[-1] idx = np.argmax(weight_acc > rand_x) rank_id = idx // batch_size data_id = idx % batch_size return rank_id, data_id @MindFormerRegister.register(MindFormerModuleType.MODELS) class Blip2Qformer(Blip2Base): """ BLIP2 first-stage model with Q-former and ViT. Usage: >>> from mindformers.models.blip2 import Blip2Qformer >>> model = Blip2Qformer.from_pretrained("blip2_stage1_vit_g") """ _support_list = MindFormerBook.get_model_support_list()['blip2']['1-stg'] def __init__(self, config: Blip2Config, **kwargs): super(Blip2Qformer, self).__init__(config, **kwargs) self.config = config if config is not None else Blip2Config() self.group_size = int(os.getenv('RANK_SIZE', '1')) self.rank = int(os.getenv('RANK_ID', '0')) self.visual_encoder, self.ln_vision = self.init_vision_encoder() if config.freeze_vision: for _, cell in self.visual_encoder.cells_and_names(): params = cell.get_parameters() for param in params: param.requires_grad = False self.visual_encoder.set_train(False) logger.info("freeze vision encoder") self.qformer, self.query_tokens = self.init_qformer() qformer_config = self.config.qformer_config # note special token added: bos_token -> [DEC] self.qformer.resize_token_embeddings(qformer_config.vocab_size + 1) params = self.qformer.get_parameters() # modify layer names for param in params: if "_query" in param.name: key_orig = param.name.replace("_query", "") param.set_data(self.qformer.parameters_dict().get(key_orig)) # parallel settings if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.vision_proj = Linear(in_channels=qformer_config.hidden_size, out_channels=qformer_config.head_embed_dim, param_init_type=config.dtype, compute_dtype=config.compute_dtype) self.vision_proj.shard(strategy_matmul=((dp, mp), (1, mp))) self.text_proj = Linear(in_channels=qformer_config.hidden_size, out_channels=qformer_config.head_embed_dim, param_init_type=config.dtype, compute_dtype=config.compute_dtype) self.text_proj.shard(strategy_matmul=((dp, mp), (1, mp))) self.itm_head = Linear(in_channels=qformer_config.hidden_size, out_channels=2, param_init_type=config.dtype, compute_dtype=config.compute_dtype) self.itm_head.shard(strategy_matmul=((dp, mp), (1, mp))) self.matmul = P.BatchMatMul() self.matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.concat = ops.concat self.expand_dims = ops.expand_dims self.transpose = ops.transpose self.zeros = ops.zeros self.ones = ops.ones self.linspace = ops.linspace self.floor = ops.floor self.softmax = nn.Softmax(axis=1) self.softmax.softmax.shard(((dp, mp, 1),)) self.eye = ops.eye self.masked_fill = ops.masked_fill self.stack = ops.stack self.broadcast_to = ops.broadcast_to self.tile = P.Tile() self.tile.shard(((dp, mp, 1, 1),)) self.normalize = ops.L2Normalize(axis=-1, epsilon=1e-12) self.temp = ms.Parameter( Tensor(0.07, dtype=config.compute_dtype), requires_grad=True) self.max_txt_len = config.max_txt_len self.bos_token_id = qformer_config.bos_token_id self.pad_token_id = qformer_config.pad_token_id if self.group_size > 1: self.all_gather = ops.AllGather() self.broadcast = ops.Broadcast self.not_equal = P.NotEqual() self.cast = P.Cast() self.itc_loss = CrossEntropyLoss(label_smoothing=0.1) self.itm_loss = CrossEntropyLoss() def construct(self, image: ms.Tensor, text_input_ids: ms.Tensor, return_tuple: bool = False): """ forwarding image and text, compute itc, itm and lm losses. """ image_feats, image_embeds, past_key_values = self.forward_image( image, use_cache=True) image_feats = self.normalize(self.vision_proj(image_feats)) text_embeds, text_attention_mask = self.forward_text(text_input_ids) text_feat = self.normalize(self.text_proj(text_embeds[:, 0, :])) image_feats = self.cast(image_feats, mstype.float16) text_feat = self.cast(text_feat, mstype.float16) ### ============== Image-text Contrastive ===================### # if/else branch: distribute setting if self.group_size > 1: # [batch_size*num_gpu, num_query_tokens, embed_dim] image_feats_all = self.all_gather(image_feats) # [batch_size*num_gpu, embed_dim] text_feat_all = self.all_gather(text_feat) else: image_feats_all = image_feats text_feat_all = text_feat batch_size = image.shape[0] sim_q2t = [] for i in range(self.group_size): text_feat_part = text_feat_all[i * batch_size: (i + 1) * batch_size] sim_temp = self.matmul(self.expand_dims(image_feats, 1), self.expand_dims( self.expand_dims(text_feat_part, -1), 0)).squeeze(-1) sim_q2t.append(sim_temp.max(-1)) # query-text similarity: [batch_size, batch_size*num_gpu] sim_q2t = self.concat(sim_q2t, axis=1) # image-text similarity: aggregate across all query tokens sim_i2t = sim_q2t / self.temp sim_t2q = [] # align with ops.matmul, x1 -> [batch_size, batch_size, 1, embed_dim] text_feat = self.tile(self.expand_dims( self.expand_dims(text_feat, 1), 1), (1, batch_size, 1, 1)) for i in range(self.group_size): image_feats_part = image_feats_all[i * batch_size: (i + 1) * batch_size] # align with ops.matmul, x2 -> [batch_size, batch_size, embed_dim, num_query_tokens] image_feats_part = self.tile(self.expand_dims( self.transpose(image_feats_part, (0, 2, 1)), 0), (batch_size, 1, 1, 1)) # compute similarity same as ops.matmul sim_temp = self.matmul(text_feat, image_feats_part).squeeze(2) sim_t2q.append(sim_temp.max(-1)) # text-query similarity: [batch_size, batch_size*num_gpu] sim_t2q = self.concat(sim_t2q, axis=1) # text-image similarity: aggregate across all query tokens sim_t2i = sim_t2q / self.temp targets = self.floor(self.linspace(ms.Tensor(self.rank * batch_size, mstype.float32), ms.Tensor(self.rank * batch_size + batch_size - 1, mstype.float32), batch_size)).astype(mstype.int32) sim_i2t = self.cast(sim_i2t, mstype.float32) sim_t2i = self.cast(sim_t2i, mstype.float32) loss_itc = (self.itc_loss(sim_i2t, targets) + self.itc_loss(sim_t2i, targets)) / 2 # ============== Image-text Matching =================== # mask text-image similarity as weights weights_t2i, weights_i2t = self.fill_masked_weight(sim_t2i, sim_i2t, batch_size) # choose negative image/text for each text/image image_embeds_neg, text_ids_neg = self.choose_negative_targets(weights_t2i, weights_i2t, batch_size, image_embeds, text_input_ids) text_ids_all = self.concat( [text_input_ids, text_input_ids, text_ids_neg], axis=0) # pos, pos, neg image_embeds_all = self.concat( [image_embeds, image_embeds_neg, image_embeds], axis=0) # pos, neg, pos vl_embeddings = self.forward_text_and_image( image_embeds_all, text_ids_all, vit_computed=True) vl_output = self.itm_head(vl_embeddings) logits = vl_output.mean(axis=1) itm_labels = self.concat( [self.ones(batch_size, mstype.int32), self.zeros(2 * batch_size, mstype.int32)], axis=0 ) loss_itm = self.itm_loss(logits, itm_labels) # ================= Image Captioning ======================== decoder_input_ids = text_input_ids.copy().astype(mstype.float32) decoder_input_ids[:, 0] = self.bos_token_id decoder_input_ids = decoder_input_ids.astype(mstype.int32) labels = decoder_input_ids.masked_fill( decoder_input_ids == self.pad_token_id, -100 ) query_tokens = self.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1)) query_atts = self.ones(query_tokens.shape[:-1], mstype.float32) attention_mask = self.concat( [query_atts, text_attention_mask.astype(mstype.float32)], axis=1) lm_output = self.qformer( decoder_input_ids, attention_mask=attention_mask, past_key_values=past_key_values, return_dict=True, labels=labels, ) loss_lm = lm_output[0] loss = loss_itc + loss_itm + loss_lm if return_tuple: return ( loss, loss_itc, loss_itm, loss_lm ) return loss def forward_image(self, image, use_cache=False): """ forawrd image through vit and the bert model. Args: image (Tensor): input image use_cache (bool, optional): whether to return past_key_values. Returns: hidden_states, image_embeds_frozen, past_key_values (optional) """ image_embeds_frozen = self.ln_vision(self.visual_encoder(image)) image_atts = self.ones( image_embeds_frozen.shape[:-1], mstype.float32) query_tokens = self.broadcast_to( self.query_tokens, (image_embeds_frozen.shape[0], -1, -1)) query_output = self.qformer.bert( query_embeds=query_tokens, encoder_hidden_states=image_embeds_frozen, encoder_attention_mask=image_atts, use_cache=use_cache, ) if use_cache: return query_output[0], image_embeds_frozen, query_output[1] return query_output[0], image_embeds_frozen def forward_text(self, text_input_ids): """ forawrd text_ids through the bert model. Args: text_input_ids (Tensor): input text_ids Returns: text embeddings and mask """ attention_mask = self.cast(self.not_equal( text_input_ids, self.pad_token_id), mstype.float32) text_output = self.qformer.bert( text_input_ids, attention_mask=attention_mask, return_dict=True ) # text embeddings and mask return text_output[0], attention_mask def forward_text_and_image(self, image_inputs, text_ids, vit_computed=False): """ forward text and image at the same time to the bert model. Args: image_inputs(Tensor): input image or image embeds (computed) text_input_ids (Tensor): input text_ids vit_computed (bool, optional): whether image embeds is computed Returns: multimodal embeddings """ if not vit_computed: image_embeds_frozen = self.ln_vision( self.visual_encoder(image_inputs)) else: image_embeds_frozen = image_inputs text_atts = self.cast(self.not_equal( text_ids, self.pad_token_id), mstype.float32) image_atts = self.ones( image_embeds_frozen.shape[:-1], mstype.float32) query_tokens = self.broadcast_to( self.query_tokens, (image_embeds_frozen.shape[0], -1, -1)) query_atts = self.ones(query_tokens.shape[:-1], mstype.float32) attention_mask = self.concat([query_atts, text_atts], axis=1) output_itm = self.qformer.bert( text_ids, query_embeds=query_tokens, attention_mask=attention_mask, encoder_hidden_states=image_embeds_frozen, encoder_attention_mask=image_atts, return_dict=True ) # multimodal embeddings multimodal_embeds = output_itm[0][:, : query_tokens.shape[1], :] return multimodal_embeds def fill_masked_weight(self, sim_t2i, sim_i2t, batch_size): """return masked weights based on similarity Args: sim_t2i (Tensor): text-to-image similarity sim_i2t (Tensor): image-to-text similarity batch_size (int): current batch size """ weights_t2i = self.softmax(sim_t2i) + 1e-4 diag_fill_mask_t2i = self.eye(weights_t2i.shape[0], batch_size, mstype.bool_) filled_weights_t2i = self.masked_fill( weights_t2i[:, self.rank * batch_size: self.rank * batch_size + batch_size], diag_fill_mask_t2i, 0) weights_t2i[:, self.rank * batch_size: self.rank * batch_size + batch_size] = filled_weights_t2i weights_i2t = self.softmax(sim_i2t) + 1e-4 diag_fill_mask_i2t = self.eye(weights_i2t.shape[0], batch_size, mstype.bool_) filled_weights_i2t = self.masked_fill( weights_i2t[:, self.rank * batch_size: self.rank * batch_size + batch_size], diag_fill_mask_i2t, 0) weights_i2t[:, self.rank * batch_size: self.rank * batch_size + batch_size] = filled_weights_i2t return weights_t2i, weights_i2t def choose_negative_targets(self, weights_t2i, weights_i2t, batch_size, image_embeds, text_input_ids): """choose negative targets for each image/text. Args: weights_t2i (Tensor): masked text-to-image weights weights_i2t (Tensor): masked image-to-text weights batch_size (int): current batch size image_embeds (Tensor): image embeddings text_input_ids (Tensor): text ids Returns: image_embeds_neg (Tensor): negative image_embeds text_ids_neg (Tensor): negative text ids """ # select a negative image for each text image_embeds_neg = [] for i in range(batch_size): rank_id, data_id = choose_idx_with_prob(weights_t2i[i], batch_size) if self.group_size > 1: card_id = locate_card(rank_id, self.group_size) image_embeds_temp = self.broadcast(card_id)((image_embeds,))[0] else: image_embeds_temp = image_embeds image_embeds_neg.append(image_embeds_temp[data_id]) image_embeds_neg = self.stack(image_embeds_neg, axis=0) # select a negative text for each image text_ids_neg = [] for i in range(batch_size): rank_id, data_id = choose_idx_with_prob(weights_i2t[i], batch_size) if self.group_size > 1: card_id = locate_card(rank_id, self.group_size) text_input_ids_temp = self.broadcast( card_id)((text_input_ids,))[0] else: text_input_ids_temp = text_input_ids text_ids_neg.append(text_input_ids_temp[data_id]) text_ids_neg = self.stack(text_ids_neg, axis=0) return image_embeds_neg, text_ids_neg def compute_itm(self, image_inputs, text_ids, vit_computed=False): """ compute image-text matching scores for the model. Args: image_inputs (Tensor): input image or image embeds (computed) text_ids (Tensor): input text_ids vit_computed (bool, optional): whether image embeds is computed Returns: itm_logit """ vl_embeddings = self.forward_text_and_image( image_inputs, text_ids, vit_computed) itm_logit = self.itm_head(vl_embeddings) itm_logit = itm_logit[:, :, 1].mean(axis=1) return itm_logit def get_image_feature(self, image, output_past_keys=False): """extract image feature""" forward_image_outputs = self.forward_image(image, output_past_keys) image_features = ops.L2Normalize( axis=-1, epsilon=1e-12)(self.vision_proj(forward_image_outputs[0])) return image_features def get_text_feature(self, input_ids): """extract text feature""" forward_text_outputs = self.forward_text(input_ids) text_features = ops.L2Normalize( axis=-1, epsilon=1e-12)(self.text_proj(forward_text_outputs[0])) return text_features def extract_features(self, samples, mode="multimodal"): """ extract feature as well as embeds by given mode, Args: samples (tuple of Tensors): image/text input mode (str): [image, text, multimodal] """ image = samples.get("image") text_ids = samples.get("text_input") # assert mode is one of "image", "text", "multimodal" assert mode in [ "image", "text", "multimodal", ], "mode must be one of 'image', 'text', 'multimodal'" # initialize output image_embeds, text_embeds, multimodal_embeds = None, None, None image_features, text_features = None, None if mode == "image": assert ( image is not None ), "Image is not provided for mode 'image' or 'multimodal'" # return query features forward_image_outputs = self.forward_image(image, use_cache=False) image_embeds = forward_image_outputs[0] image_features = ops.L2Normalize( axis=-1, epsilon=1e-12)(self.vision_proj(image_embeds)) elif mode == "text": assert text_ids is not None, "text input is None for mode 'text' or 'multimodal'" forward_text_outputs = self.forward_text(text_ids) text_embeds = forward_text_outputs[0] text_features = ops.L2Normalize( axis=-1, epsilon=1e-12)(self.text_proj(text_embeds)) elif mode == "multimodal": # return multimodal query features multimodal_embeds = self.forward_text_and_image( image, text_ids, vit_computed=False) return (image_embeds, image_features, text_embeds, text_features, multimodal_embeds) @MindFormerRegister.register(MindFormerModuleType.MODELS) class Blip2Classifier(Blip2Qformer): """ Blip2Classifier rely on Blip2Qformer, used for zero-shot classification. Usage: >>> from mindformers import AutoModel >>> model_type = 'blip2_stage1_classification' >>> model = AutoModel.from_pretrained(model_type) """ def __init__(self, config: Blip2Config, **kwargs): super(Blip2Classifier, self).__init__(config, **kwargs) self.load_checkpoint(config) def construct(self, image: ms.Tensor, text_input_ids: ms.Tensor, return_tuple: bool = False): image_features = self.get_image_feature(image)[:, 0] text_features = self.get_text_feature(text_input_ids)[:, 0] sims = ops.matmul(image_features, text_features.T) / self.temp return sims, sims.T # no label as input (compare to CLIP) pengcheng.mind-test/mindformers/models/blip2/convert_weight.py000066400000000000000000000126171471400134000251410ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Convert checkpoint from salesforce.""" from collections import OrderedDict import argparse import mindspore as ms import mindspore.ops as P from mindspore import Tensor import torch def convert_weight(torch_path="blip2_pretrain.pth", mindspore_path="blip2_pretrain.ckpt"): r"""Convert Weight Convert blip2_qformer weights from pytorch to mindspore, pytorch (CPU) required. Args: torch_path: The torch path to blip2_qformer. mindspore_path: The save path for blip2_qformer. Returns: the converted mindspore_model_weight for Blip2Qformer class. Notice: when loading, warning of: ""2 parameters 'visual_encoder.fc_norm.gamma', 'visual_encoder.fc_norm.beta' ara not loaded. " is normal, because bilp2 utilizes the penultimate layer output of the eva_vit. """ pt_params = torch.load(torch_path, map_location='cpu') if not isinstance(pt_params, OrderedDict): if isinstance(pt_params, dict) and 'model' in pt_params.keys(): pt_params = pt_params['model'] else: raise ValueError("wrong torch state_dict format when loading {},\ please check.".format(torch_path)) name_pt2ms = { "cls_token": "cls_tokens", "attn.proj": "attention.projection", "attn.q_bias": "attention.dense1.bias", "attn.v_bias": "attention.dense3.bias", "norm1.weight": "layernorm1.gamma", "norm1.bias": "layernorm1.beta", "norm2.weight": "layernorm2.gamma", "norm2.bias": "layernorm2.beta", "fc_norm.weight": "fc_norm.gamma", "fc_norm.bias": "fc_norm.beta", "ln_vision.weight": "ln_vision.gamma", "ln_vision.bias": "ln_vision.beta", "mlp.fc2.weight": "output.mapping.weight", "mlp.fc1.bias": "output.mapping.bias", "mlp.fc1.weight": "output.projection.weight", "mlp.fc2.bias": "output.projection.bias", "LayerNorm.": "layernorm.", "layernorm.weight": "layernorm.gamma", "layernorm.bias": "layernorm.beta", "embeddings.weight": "embeddings.embedding_table", "self": "self_att", } ms_param_dict = [] for pt_name, pt_tensor in pt_params.items(): # initial name assign ms_name = pt_name # extract data numpy_value = pt_params[pt_name].detach().numpy() data = Tensor.from_numpy(numpy_value) # split qkv weights if "qkv.weight" in pt_name: length = pt_tensor.shape[0] // 3 ms_name1 = pt_name.replace("attn.qkv", "attention.dense1") ms_name2 = pt_name.replace("attn.qkv", "attention.dense2") ms_name3 = pt_name.replace("attn.qkv", "attention.dense3") ms_param_dict.append({"name": ms_name1, "data": data[:length]}) ms_param_dict.append({"name": ms_name2, "data": data[length:length*2]}) ms_param_dict.append({"name": ms_name3, "data": data[length*2:length*3]}) print("rename {} to {}, {} and {}".format(pt_name, ms_name1, ms_name2, ms_name3)) else: # Rename for replace_from, replace_to in name_pt2ms.items(): ms_name = ms_name.replace(replace_from, replace_to) ms_name = ms_name.replace("Qformer.", "qformer.") if ms_name.endswith("output.mapping.weight") or \ ms_name.endswith("output.projection.weight") or \ ms_name.endswith("attention.projection.weight"): data = data.T ms_param_dict.append({"name": ms_name, "data": data}) # when loading each query-bias, append a zero value key-bias. if ms_name.endswith("attention.dense1.bias"): ms_param_dict.append({ "name": ms_name.replace("dense1", "dense2"), "data": P.zeros_like(data) }) if ms_name != pt_name: print(f"rename {pt_name} to {ms_name}") print(f"\n----------------- convert {torch_path} to {mindspore_path} Finished! -----------------\n") ms.save_checkpoint(ms_param_dict, mindspore_path) if __name__ == '__main__': parser = argparse.ArgumentParser(description="blip2 weight convert script") parser.add_argument("--torch_path", type=str, default="blip2_pretrain.pth", required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="blip2_pretrain.ckpt", help="The output mindspore checkpoint path.") opt = parser.parse_args() convert_weight(opt.torch_path, opt.mindspore_path) pengcheng.mind-test/mindformers/models/blip2/qformer.py000066400000000000000000002061451471400134000235660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """qformer implementation.""" import math from collections import OrderedDict from typing import Optional import os import mindspore.common.dtype as mstype import mindspore.numpy as np import mindspore.ops as P from mindspore.train.serialization import load_checkpoint from mindspore import nn, Parameter, Tensor from mindspore.common.initializer import initializer, Normal, Zero from mindspore.nn import LossBase from mindformers.mindformer_book import MindFormerBook from mindformers.tools.download_tools import download_with_progress_bar from mindformers.tools.logger import logger from mindformers.tools.utils import try_sync_file from mindformers.models.base_model import BaseModel from mindformers.models.bert.bert_config import BertConfig from mindformers.models.blip2.qformer_config import QFormerConfig from mindformers.modules.layers import Dropout, LayerNorm, Linear class CrossEntropyLoss(LossBase): """Label smoothing align with torch implementation.""" def __init__(self, weight=None, ignore_index=-100, reduction='mean', label_smoothing=0.0): super().__init__(reduction) self.weight = weight self.ignore_index = ignore_index self.reduction = reduction self.label_smoothing = label_smoothing self.log_softmax = nn.LogSoftmax(axis=-1) self.cross_entropy = nn.CrossEntropyLoss( reduction=self.reduction, label_smoothing=0.0) @staticmethod def reduce_loss(loss, reduction='mean'): """reduce_loss. Args: loss (float): loss reduction (str, optional): reduction method. Defaults to 'mean'. Returns: reduced_loss: reduced_loss """ if reduction == 'mean': return loss.mean() if reduction == 'sum': return loss.sum() return loss def construct(self, logits, labels): loss_fct = self.cross_entropy(logits, labels) log_preds = self.log_softmax(logits) loss = self.reduce_loss(-log_preds.sum(axis=-1), reduction=self.reduction) n_class = logits.shape[-1] return (loss / n_class) * self.label_smoothing + loss_fct * (1 - self.label_smoothing) ACT2CLS = { "gelu": nn.GELU, "gelu_fast": nn.FastGelu, "relu": nn.ReLU, "relu6": nn.ReLU6, "sigmoid": nn.Sigmoid, "tanh": nn.Tanh, } class ClassInstanter(OrderedDict): """ClassInstanter for OrderedDict func-mapping input. Args: OrderedDict : function mapping. """ def __getitem__(self, key): content = super().__getitem__(key) cls, kwargs = content if isinstance(content, tuple) else (content, {}) return cls(**kwargs) ACT2FN = ClassInstanter(ACT2CLS) def recursive_apply(module: nn.Cell, function_call): """apply cetain function to a nn.Cell module, recursively. Args: module (nn.Cell): model input. fn (function): function call """ for submodule in module.cells(): recursive_apply(submodule, function_call) function_call(module) class BertEmbeddings(nn.Cell): """forward the embeddings from word and position embeddings.""" def __init__(self, config): super().__init__() self.word_embeddings = nn.Embedding( config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id ) self.position_embeddings = nn.Embedding( config.max_position_embeddings, config.hidden_size ) # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load # any TensorFlow checkpoint file self.layernorm = LayerNorm( (config.hidden_size,), eps=config.layer_norm_eps) self.layernorm.shard(((config.parallel_config.data_parallel, 1, 1),)) self.dropout = Dropout(1. - config.hidden_dropout_prob) self.dropout.shard(((config.parallel_config.data_parallel, 1, 1),)) self.concat = P.Concat(axis=1) # position_ids (1, len position emb) is contiguous in memory and exported when serialized self.expand_x = P.BroadcastTo((1, config.max_position_embeddings)) position_embeds = Tensor( [i for i in range(config.max_position_embeddings)], dtype=mstype.int32) self.position_ids = Parameter( self.expand_x(position_embeds), requires_grad=False ) self.position_embedding_type = getattr( config, "position_embedding_type", "absolute" ) def construct(self, input_ids=None, position_ids=None, query_embeds=None, past_key_values_length=0): """forward the embeddings from word and position embeddings.""" if input_ids is not None: seq_length = input_ids.shape[1] else: seq_length = 0 if position_ids is None: position_ids = self.position_ids[:, past_key_values_length: seq_length + past_key_values_length].copy() if input_ids is not None: embeddings = self.word_embeddings(input_ids) if self.position_embedding_type == "absolute": position_embeddings = self.position_embeddings(position_ids) embeddings = embeddings + position_embeddings if query_embeds is not None: embeddings = self.concat((query_embeds, embeddings)) else: embeddings = query_embeds # [bz, query_size, qformer_hidden_size] embeddings = self.layernorm(embeddings) embeddings = self.dropout(embeddings) return embeddings class BertSelfAttention(nn.Cell): """ BertSelfAttention """ def __init__(self, config, is_cross_attention): super().__init__() self.config = config self.dtype = config.dtype self.softmax_dtype = config.softmax_dtype self.compute_dtype = config.compute_dtype if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"): raise ValueError( "The hidden size (%d) is not a multiple of the number of attention " "heads (%d)" % (config.hidden_size, config.num_attention_heads) ) self.num_attention_heads = config.num_attention_heads self.attention_head_size = int( config.hidden_size / config.num_attention_heads) self.all_head_size = self.num_attention_heads * self.attention_head_size if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.query = Linear(in_channels=config.hidden_size, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype ) self.query.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) if is_cross_attention: self.key = Linear( in_channels=config.encoder_width, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.value = Linear( in_channels=config.encoder_width, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) else: self.key = Linear( in_channels=config.hidden_size, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.value = Linear( in_channels=config.hidden_size, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.key.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.value.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.dropout = Dropout(1. - config.attention_probs_dropout_prob) self.position_embedding_type = getattr( config, "position_embedding_type", "absolute") if self.position_embedding_type == "relative_key" \ or self.position_embedding_type == "relative_key_query": self.max_position_embeddings = config.max_position_embeddings self.distance_embedding = nn.Embedding(2 * config.max_position_embeddings - 1, self.attention_head_size) self.save_attention = False self.einsum = P.Einsum("bhld,lrd->bhlr") self.einsum2 = P.Einsum("bhrd,lrd->bhlr") self.divider = math.sqrt(self.attention_head_size) self.cast = P.Cast() self.concat = P.Concat(axis=2) self.batch_matmul = P.BatchMatMul().shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.softmax = nn.Softmax(axis=-1) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.transpose = P.Transpose().shard(((1, 1, 1, 1),)) def transpose_for_scores(self, x: Tensor) -> Tensor: """ transpose input for scores output. Args: x (Tensor): input Returns: Tensor: output """ new_x_shape = x.shape[:-1] + ( self.num_attention_heads, self.attention_head_size, ) x = x.view(*new_x_shape) return self.transpose(x, (0, 2, 1, 3)) def construct( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_value=None, output_attentions=False, ): """ BertSelfAttention forwarding """ # If this is instantiated as a cross-attention module, the keys # and values come from an encoder; the attention mask needs to be # such that the encoder's padding tokens are not attended to. is_cross_attention = encoder_hidden_states is not None hidden_states = self.cast(hidden_states, self.compute_dtype) if is_cross_attention: # [batch_size, vit_seq_length, encoder_hidden_width] encoder_hidden_states = self.cast( encoder_hidden_states, self.compute_dtype) # [batch_size, num_attention_heads, vit_seq_length, attention_head_size] key_layer = self.transpose_for_scores( self.key(encoder_hidden_states)) # [batch_size, num_attention_heads, vit_seq_length, attention_head_size] value_layer = self.transpose_for_scores( self.value(encoder_hidden_states)) attention_mask = encoder_attention_mask elif past_key_value is not None: key_layer = self.transpose_for_scores(self.key(hidden_states)) value_layer = self.transpose_for_scores(self.value(hidden_states)) key_layer = self.concat([past_key_value[0], key_layer]) value_layer = self.concat([past_key_value[1], value_layer]) else: # [batch_size, num_attention_heads, query_size, attention_head_size] key_layer = self.transpose_for_scores(self.key(hidden_states)) # [batch_size, num_attention_heads, query_size, attention_head_size] value_layer = self.transpose_for_scores(self.value(hidden_states)) # [batch_size, query_size, qformer_hidden_size] mixed_query_layer = self.query(hidden_states) # [batch_size, num_attention_heads, query_size, attention_head_size] query_layer = self.transpose_for_scores(mixed_query_layer) past_key_value = (key_layer, value_layer) # Take the dot product between "query" and "key" to get the raw attention scores. # key_layer.transpose(tmp_shape) [batch_size, num_attention_heads, attention_head_size, query_size] # query_layer: [batch_size, num_attention_heads, query_size, attention_head_size] trans_key_layer = self.transpose(key_layer, (0, 1, 3, 2)) attention_scores = self.batch_matmul(query_layer, trans_key_layer) if ( self.position_embedding_type == "relative_key" or self.position_embedding_type == "relative_key_query" ): seq_length = hidden_states.shape[1] position_ids_l = Tensor( [i for i in range(seq_length)], dtype=mstype.int32).view(-1, 1) position_ids_r = Tensor( [i for i in range(seq_length)], dtype=mstype.int32).view(1, -1) distance = position_ids_l - position_ids_r positional_embedding = self.distance_embedding( distance + self.max_position_embeddings - 1 ) if self.position_embedding_type == "relative_key": relative_position_scores = self.einsum( query_layer, positional_embedding) attention_scores = attention_scores + relative_position_scores elif self.position_embedding_type == "relative_key_query": relative_position_scores_query = self.einsum( query_layer, positional_embedding) relative_position_scores_key = self.einsum2( key_layer, positional_embedding) attention_scores = ( attention_scores + relative_position_scores_query + relative_position_scores_key ) attention_scores /= self.divider if attention_mask is not None: # Apply the attention mask is (precomputed for all layers in BertModel forward() function) attention_scores = attention_scores + attention_mask # Normalize the attention scores to probabilities. [batch_size, num_heads, query_size, query_size] attention_scores = self.cast(attention_scores, self.softmax_dtype) attention_probs = self.softmax(attention_scores) # This is actually dropping out entire tokens to attend to, which might # seem a bit unusual, but is taken from the original Transformer paper. attention_probs_dropped = self.dropout(attention_probs) # Mask heads if we want to if head_mask is not None: attention_probs_dropped = attention_probs_dropped * head_mask attention_probs_dropped = self.cast( attention_probs_dropped, self.compute_dtype) context_layer = self.batch_matmul(attention_probs_dropped, value_layer) # [batch_size, num_heads, query_size, attention_head_size] context_layer = self.transpose(context_layer, (0, 2, 1, 3)).copy() # [batch_size, query_size, all_head_size] new_context_layer_shape = context_layer.shape[:-2] + ( self.all_head_size,) context_layer = context_layer.view(*new_context_layer_shape) outputs = ( (context_layer, attention_probs) if output_attentions else (context_layer,) ) outputs = outputs + (past_key_value,) return outputs class BertSelfOutput(nn.Cell): """ BertSelfOutput """ def __init__(self, config): super().__init__() if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.dtype = config.dtype self.dense = Linear( in_channels=config.hidden_size, out_channels=config.hidden_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype ) self.dense.shard(strategy_matmul=((dp, mp), (1, mp))) self.layernorm = LayerNorm( (config.hidden_size,), eps=config.layer_norm_eps).shard(((dp, mp, 1),)) self.dropout = Dropout(1. - config.hidden_dropout_prob) self.cast = P.Cast() def construct(self, hidden_states, input_tensor): hidden_states = self.dense(hidden_states) hidden_states = self.cast(hidden_states, self.dtype) hidden_states = self.dropout(hidden_states) hidden_states = self.layernorm(hidden_states + input_tensor) return hidden_states class BertAttention(nn.Cell): """ BertAttention """ def __init__(self, config, is_cross_attention=False): super().__init__() self.self_att = BertSelfAttention(config, is_cross_attention) self.output = BertSelfOutput(config) def construct( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_value=None, output_attentions=False, ): """ hidden_states: [batch_size, query_size, qformer_hidden_size] attention_mask: [batch_size, 1, 1, query_size] encoder_hidden_states: [batch_size, vit_seq_length, vit_hidden_size] encoder_attention_mask: [batch_size, 1, 1, vit_seq_length] """ # self_outputs.shape ([batch_size, query_size, qformer_hidden_size], # ([batch_size, num_head, query_size, head_size], [batch_size, num_head, query_size, head_size])) self_outputs = self.self_att( hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions, ) attention_output = self.output(self_outputs[0], hidden_states) # add attentions if we output them outputs = (attention_output,) + self_outputs[1:] return outputs class BertIntermediate(nn.Cell): """ BertIntermediate """ def __init__(self, config): super().__init__() if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.dense = Linear( in_channels=config.hidden_size, out_channels=config.intermediate_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype ) self.dense.shard(strategy_matmul=((dp, mp), (1, mp))) if isinstance(config.hidden_act, str): self.intermediate_act_fn = ACT2FN[config.hidden_act] if isinstance(self.intermediate_act_fn, nn.GELU): self.transform_act_fn = nn.GELU(approximate=False) else: self.intermediate_act_fn = config.hidden_act def construct(self, hidden_states): hidden_states = self.dense(hidden_states) hidden_states = self.intermediate_act_fn(hidden_states) return hidden_states class BertOutput(nn.Cell): """ BertOutput """ def __init__(self, config): super().__init__() if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.dense = Linear( in_channels=config.intermediate_size, out_channels=config.hidden_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.dense.shard(strategy_matmul=((dp, mp), (1, mp))) self.layernorm = LayerNorm( (config.hidden_size,), eps=config.layer_norm_eps).shard(((dp, 1, 1),)) self.dropout = Dropout(1. - config.hidden_dropout_prob) def construct(self, hidden_states, input_tensor): hidden_states = self.dense(hidden_states) hidden_states = self.dropout(hidden_states) hidden_states = self.layernorm(hidden_states + input_tensor) return hidden_states class BertLayer(nn.Cell): """ BertLayer """ def __init__(self, config, layer_num): super().__init__() self.config = config self.chunk_size_feed_forward = config.chunk_size_feed_forward self.seq_len_dim = 1 self.attention = BertAttention(config) self.layer_num = layer_num if self.config.add_cross_attention and layer_num % self.config.cross_attention_freq == 0: self.crossattention = BertAttention(config, is_cross_attention=self.config.add_cross_attention) self.has_cross_attention = True else: self.has_cross_attention = False self.intermediate = BertIntermediate(config) self.output = BertOutput(config) self.intermediate_query = BertIntermediate(config) self.output_query = BertOutput(config) self.concat = P.Concat(axis=1) self.concat_seq = P.Concat(axis=self.seq_len_dim) def construct( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_value=None, output_attentions=False, query_length=0, ): """ hidden_states: [batch_size, query_size, qformer_hidden_size] attention_mask: [batch_size, 1, 1, query_size] encoder_hidden_states: [batch_size, vit_seq_length, vit_hidden_size ] encoder_attention_mask: [batch_size, 1, 1, vit_seq_length] """ # decoder uni-directional self-attention cached key/values tuple is at positions 1,2 self_attn_past_key_value = ( past_key_value[:2] if past_key_value is not None else None ) self_attention_outputs = self.attention( hidden_states, attention_mask, head_mask, None, None, self_attn_past_key_value, output_attentions, ) # [batch_size, query_size, qformer_hidden_size] attention_output = self_attention_outputs[0] # ([batch_size, num_head, query_size, head_size], [batch_size, num_head, query_size, head_size])) outputs = self_attention_outputs[1:-1] present_key_value = self_attention_outputs[-1] if query_length > 0: # [batch_size, query_size, qformer_hidden_size] query_attention_output = attention_output[:, :query_length, :] if self.has_cross_attention: assert encoder_hidden_states is not None, \ "encoder_hidden_states must be given for cross-attention layers" cross_attention_outputs = self.crossattention( query_attention_output, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, None, output_attentions, ) # [batch_size, query_size, qformer_hidden_size] query_attention_output = cross_attention_outputs[0] # add cross attentions if we output attention weights outputs = (outputs + cross_attention_outputs[1:-1]) # [batch_size, query_size, qformer_hidden_size] layer_output = self.apply_chunking_to_forward(self.feed_forward_chunk_query, query_attention_output) if attention_output.shape[1] > query_length: layer_output_text = self.apply_chunking_to_forward(self.feed_forward_chunk, attention_output[:, query_length:, :]) layer_output = self.concat([layer_output, layer_output_text]) else: layer_output = self.apply_chunking_to_forward( self.feed_forward_chunk, attention_output) outputs = (layer_output,) + outputs outputs = outputs + (present_key_value,) return outputs def feed_forward_chunk(self, attention_output): """ apply feed_forward with chunks """ intermediate_output = self.intermediate(attention_output) layer_output = self.output(intermediate_output, attention_output) return layer_output def feed_forward_chunk_query(self, attention_output): """ apply feed_forward with chunks (query) """ intermediate_output = self.intermediate_query(attention_output) layer_output = self.output_query(intermediate_output, attention_output) return layer_output def apply_chunking_to_forward(self, forward_fn, *input_tensors): """ apply chunking to forward computation """ assert input_tensors, f"{input_tensors} has to be a tuple/list of tensors" if self.chunk_size_feed_forward > 0: tensor_shape = input_tensors[0].shape[self.seq_len_dim] for input_tensor in input_tensors: if input_tensor.shape[self.seq_len_dim] != tensor_shape: raise ValueError( f"All input tenors have to be of the same shape: {tensor_shape}, " f"found shape {input_tensor.shape[self.seq_len_dim]}" ) if input_tensors[0].shape[self.seq_len_dim] % self.chunk_size_feed_forward != 0: raise ValueError( f"The dimension to be chunked {input_tensors[0].shape[self.seq_len_dim]} " f"has to be a multiple of the chunk size {self.chunk_size_feed_forward}" ) num_chunks = input_tensors[0].shape[self.seq_len_dim] // self.chunk_size_feed_forward # chunk input tensor into tuples input_tensors_chunks = tuple(input_tensor.chunk(num_chunks, dim=self.seq_len_dim) for input_tensor in input_tensors) # apply forward fn to every tuple output_chunks = tuple(forward_fn(*input_tensors_chunk) for input_tensors_chunk in zip(*input_tensors_chunks)) # concatenate output at same dimension return self.concat_seq(output_chunks) return forward_fn(*input_tensors) class BertEncoder(nn.Cell): """ BertEncoder """ def __init__(self, config): super(BertEncoder, self).__init__() self.config = config self.layer = nn.CellList( [BertLayer(config, i) for i in range(self.config.num_hidden_layers)] ) self.num_hidden_layers = [ i for i in range(self.config.num_hidden_layers)] def construct( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_values=None, use_cache=None, output_attentions=False, output_hidden_states=False, return_dict=True, query_length=0, ): """ attention_mask: [batch_size, 1, 1, query_size] encoder_hidden_states: [batch_size, vit_seq_length, encoder_hidden_width] encoder_attention_mask: [batch_size, 1, 1, vit_seq_length] """ all_hidden_states = () if output_hidden_states else None all_self_attentions = () if output_attentions else None all_cross_attentions = ( () if output_attentions and self.config.add_cross_attention else None ) next_decoder_cache = () if use_cache else None for i in self.num_hidden_layers: layer_module = self.layer[i] if output_hidden_states: all_hidden_states = all_hidden_states + (hidden_states,) layer_head_mask = head_mask[i] if head_mask is not None else None past_key_value = past_key_values[i] if past_key_values is not None else None # layer_outputs shape ([batch_size, query_size, qformer_hidden_size], # ([batch_size, num_head, query_size, head_size], [batch_size, num_head, query_size, head_size])) layer_outputs = layer_module( hidden_states, attention_mask, layer_head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions, query_length, ) hidden_states = layer_outputs[0] if use_cache: next_decoder_cache += (layer_outputs[-1],) if output_attentions: all_self_attentions = all_self_attentions + (layer_outputs[1],) all_cross_attentions = all_cross_attentions + \ (layer_outputs[2],) if output_hidden_states: all_hidden_states = all_hidden_states + (hidden_states,) value_list = [hidden_states, next_decoder_cache, all_hidden_states, all_self_attentions, all_cross_attentions] if not return_dict: return tuple(v for v in value_list if v is not None) return tuple(value_list) class BertPredictionHeadTransform(nn.Cell): """ BertPredictionHeadTransform """ def __init__(self, config): super(BertPredictionHeadTransform, self).__init__() self.dense = Linear( in_channels=config.hidden_size, out_channels=config.hidden_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype ) if isinstance(config.hidden_act, str): self.transform_act_fn = ACT2FN[config.hidden_act] if isinstance(self.transform_act_fn, nn.GELU): self.transform_act_fn = nn.GELU(approximate=False) else: self.transform_act_fn = config.hidden_act self.layernorm = LayerNorm( (config.hidden_size,), eps=config.layer_norm_eps) def construct(self, hidden_states): hidden_states = self.dense(hidden_states) hidden_states = self.transform_act_fn(hidden_states) hidden_states = self.layernorm(hidden_states) return hidden_states class BertLMPredictionHead(nn.Cell): """ BertLMPredictionHead """ def __init__(self, config): super(BertLMPredictionHead, self).__init__() self.transform = BertPredictionHeadTransform(config) # The output weights are the same as the input embeddings, but there is # an output-only bias for each token. self.decoder = Linear(in_channels=config.hidden_size, out_channels=config.vocab_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.bias = Parameter(initializer( Zero(), config.vocab_size, dtype=config.dtype)) # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings` self.decoder.bias = self.bias def construct(self, hidden_states): hidden_states = self.transform(hidden_states) hidden_states = self.decoder(hidden_states) return hidden_states class BertOnlyMLMHead(nn.Cell): """ BertOnlyMLMHead """ def __init__(self, config): super(BertOnlyMLMHead, self).__init__() self.predictions = BertLMPredictionHead(config) def construct(self, sequence_output): prediction_scores = self.predictions(sequence_output) return prediction_scores class BertPreTrainedModel(BaseModel, nn.Cell): """ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. """ def __init__(self, config: BertConfig, **kwargs): super(BertPreTrainedModel, self).__init__(config) if not isinstance(config, BertConfig): raise ValueError( f"Parameter config in `{self.__class__.__name__}(config)` should be an instance of class " "`BaseConfig`. To create a model from a pretrained model use " f"`model = {self.__class__.__name__}.from_pretrained(PRETRAINED_MODEL_NAME)`" ) # Save config and origin of the pretrained weights if given in model self.config = config self.name_or_path = config.name_or_path self.warnings_issued = {} self.ones_like = P.OnesLike() self.zeros_like = P.ZerosLike() self.expand_dims = P.ExpandDims().shard(((1, 1, 1, 1, 1),)) def init_weights(self): """ init_weights recursively for BertPreTrainedModel. """ # Initialize weights, recursively. recursive_apply(self, self._init_weights) def _init_weights(self, module): """Initialize the weights""" if isinstance(module, nn.Embedding): # Slightly different from the TF version which uses truncated_normal for initialization normed = initializer(Normal(sigma=self.config.initializer_range, mean=0.0), shape=module.embedding_table.data.shape) module.embedding_table.set_data(normed) elif isinstance(module, Linear): normed = initializer(Normal(sigma=self.config.initializer_range, mean=0.0), shape=module.weight.data.shape, dtype=self.config.compute_dtype) module.weight.set_data(normed) if module.bias is not None: module.bias.set_data(self.zeros_like(module.bias.value())) elif isinstance(module, LayerNorm): module.gamma.set_data(self.ones_like(module.gamma.value())) module.beta.set_data(self.zeros_like(module.beta.value())) @classmethod def from_config(cls, config, **kwargs): """ All context managers that the model should be initialized under go here. """ model = cls(config, **kwargs) return model def get_input_embeddings(self) -> nn.Cell: """ get input embeddings. """ raise NotImplementedError def set_input_embeddings(self, value: nn.Cell): """ set input embeddings. """ raise NotImplementedError def get_output_embeddings(self) -> nn.Cell: """ get_output_embeddings """ raise NotImplementedError def set_output_embeddings(self, value): """ set_output_embeddings """ raise NotImplementedError def tie_weights(self): """ Tie the weights between the input embeddings and the output embeddings. """ if getattr(self.config, "tie_word_embeddings", True): output_embeddings = self.get_output_embeddings() if output_embeddings is not None: self.tie_or_clone_weights( output_embeddings, self.get_input_embeddings()) for module in self.cells(): if hasattr(module, "tie_weights"): module.tie_weights() def tie_or_clone_weights(self, output_embeddings: Linear, input_embeddings: nn.Embedding): """Tie or clone module weights depending of whether we are using TorchScript or not""" output_embeddings.weight.assign_value(input_embeddings.embedding_table.value()) if getattr(output_embeddings, "bias", None) is not None: paddings = [ [0] * 2 for _ in range(output_embeddings.bias.value().ndim)] paddings[-1][-1] = output_embeddings.weight.shape[0] - \ output_embeddings.bias.shape[0] pad_op = P.Pad(paddings=tuple(paddings)) output_embeddings.bias.assign_value( pad_op(output_embeddings.bias.value()) ) if hasattr(output_embeddings, "out_channels") and hasattr(input_embeddings, "vocab_size"): output_embeddings.out_channels = input_embeddings.vocab_size def resize_token_embeddings(self, new_num_tokens: Optional[int] = None) -> nn.Embedding: """ Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. """ model_embeds = self.resize_model_embeds(new_num_tokens) if new_num_tokens is None or model_embeds is None: return model_embeds # Update base model and current model config self.config.vocab_size = new_num_tokens self.vocab_size = new_num_tokens # Tie weights again if needed self.tie_weights() return model_embeds def resize_model_embeds(self, new_num_tokens): """ resize input embeddings and output_embeddings. """ old_embeddings = self.get_input_embeddings() new_embeddings = self._get_resized_embeddings( old_embeddings, new_num_tokens) self.set_input_embeddings(new_embeddings) # if word embeddings are not tied, make sure that lm head is resized as well if self.get_output_embeddings() is not None and not self.config.tie_word_embeddings: old_lm_head = self.get_output_embeddings() new_lm_head = self._get_resized_lm_head( old_lm_head, new_num_tokens) self.set_output_embeddings(new_lm_head) return self.get_input_embeddings() def _get_resized_embeddings(self, old_embeddings: nn.Embedding, new_num_tokens: Optional[int] = None) -> nn.Embedding: """ internal function, get new embeddings with inited weights. """ if new_num_tokens is None: return old_embeddings old_num_tokens, old_embedding_dim = old_embeddings.embedding_table.shape if old_num_tokens == new_num_tokens: return old_embeddings if not isinstance(old_embeddings, nn.Embedding): raise TypeError( f"Old embeddings are of type {type(old_embeddings)}, which is not an instance of {nn.Embedding}. You" " should either use a different resize function or make sure that `old_embeddings` are an instance of" f" {nn.Embedding}." ) # Build new embeddings new_embeddings = nn.Embedding(new_num_tokens, old_embedding_dim) new_embeddings.embedding_table.name = old_embeddings.embedding_table.name # initialize all new embeddings (in particular added tokens) self._init_weights(new_embeddings) # Copy token embeddings from the previous weights # numbers of tokens to copy remain_num = min(old_num_tokens, new_num_tokens) new_embeddings.embedding_table.data[:remain_num, :] = old_embeddings.embedding_table.data[:remain_num, :] return new_embeddings def _get_resized_lm_head(self, old_lm_head: Linear, new_num_tokens: Optional[int] = None, transposed: Optional[bool] = False) -> Linear: """ Build a resized Linear Module from a provided old Linear Module. Increasing the size will add newly initialized """ if new_num_tokens is None: return old_lm_head old_num_tokens, old_lm_head_dim = ( old_lm_head.weight.shape if not transposed else old_lm_head.weight.T.shape ) if old_num_tokens == new_num_tokens: return old_lm_head if not isinstance(old_lm_head, Linear): raise TypeError( f"Old language model head is of type {type(old_lm_head)}, which is not an instance of {Linear}. You" " should either use a different resize function or make sure that `old_lm_head` are an instance of" f" {Linear}." ) # Build new lm head new_lm_head_shape = (old_lm_head_dim, new_num_tokens) if not transposed else ( new_num_tokens, old_lm_head_dim) has_new_lm_head_bias = old_lm_head.bias is not None new_lm_head = Linear(in_channels=new_lm_head_shape[0], out_channels=new_lm_head_shape[1], has_bias=has_new_lm_head_bias, compute_dtype=self.config.compute_dtype, param_init_type=self.config.dtype) # initialize new lm head (in particular added tokens) self._init_weights(new_lm_head) num_tokens_to_copy = min(old_num_tokens, new_num_tokens) # Copy old lm head weights to new lm head if not transposed: new_lm_head.weight.data[:num_tokens_to_copy, :] = old_lm_head.weight.data[:num_tokens_to_copy, :] else: new_lm_head.weight.data[:, :num_tokens_to_copy] = old_lm_head.weight.data[:, :num_tokens_to_copy] # Copy bias weights to new lm head if has_new_lm_head_bias: new_lm_head.bias.data[:num_tokens_to_copy] = old_lm_head.bias.data[:num_tokens_to_copy] return new_lm_head def invert_attention_mask(self, encoder_attention_mask): """ Invert an attention mask (e.g., switches 0. and 1.). """ encoder_extended_attention_mask = encoder_attention_mask for _ in range(4 - encoder_attention_mask.ndim): encoder_extended_attention_mask = self.expand_dims( encoder_extended_attention_mask, 1) encoder_extended_attention_mask = ( 1.0 - encoder_extended_attention_mask) return encoder_extended_attention_mask def get_head_mask(self, head_mask: Optional[Tensor], num_hidden_layers: int, is_attention_chunked: bool = False) -> Tensor: """ Prepare the head mask if needed. """ if head_mask is not None: head_mask = self._convert_head_mask_to_5d(head_mask) if is_attention_chunked is True: head_mask = self.expand_dims(head_mask, -1) else: head_mask = [None] * num_hidden_layers return head_mask def _convert_head_mask_to_5d(self, head_mask): """-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]""" if head_mask.dim() == 1: head_mask = self.expand_dims(self.expand_dims(head_mask, 0), 0) head_mask = self.expand_dims(self.expand_dims(head_mask, -1), -1) head_mask = self.broadcast_to(head_mask) elif head_mask.dim() == 2: head_mask = self.expand_dims(self.expand_dims(head_mask, 1), -1) # We can specify head_mask for each layer head_mask = self.expand_dims(head_mask, -1) assert head_mask.dim( ) == 5, f"head_mask.dim != 5, instead {head_mask.dim()}" return head_mask class BertModel(BertPreTrainedModel): """ The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in `Attention is all you need `__ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and :obj:`add_cross_attention` set to :obj:`True`; an :obj:`encoder_hidden_states` is then expected as an input to the forward pass. """ def __init__(self, config): super(BertModel, self).__init__(config) self.config = config self.num_hidden_layers = self.config.num_hidden_layers self.output_attentions = self.config.output_attentions self.output_hidden_states = self.config.output_hidden_states self.use_return_dict = self.config.use_return_dict self.query_length = self.config.query_length self.embeddings = BertEmbeddings(config) self.encoder = BertEncoder(config) self.ones = P.Ones().shard(((self.config.parallel_config.data_parallel, 1),)) self.zeros = P.Zeros() self.broadcast_to = P.BroadcastTo( (self.num_hidden_layers, -1, -1, -1, -1)) self.concat_one = P.Concat(axis=1) self.concat_minus_one = P.Concat(axis=-1) self.init_weights() def get_input_embeddings(self) -> nn.Cell: return self.embeddings.word_embeddings def set_input_embeddings(self, value): if not isinstance(value, nn.Embedding): raise ValueError( "expect new input_embeddings to be of type: ", nn.Embedding) self.embeddings.word_embeddings = value def get_output_embeddings(self) -> nn.Cell: return None # no output embeddings def set_output_embeddings(self, value): return # no output embeddings def tie_weights(self): return # no output embeddings for tie def get_extended_attention_mask( self, attention_mask, input_shape, is_decoder, has_query=False, ): """ Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (:obj:`mindspore.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (:obj:`Tuple[int]`): The shape of the input to the model. is_decoder (:obj:`Bool`): has_query (:obj:`Bool`): Returns: :obj:`mindspore.Tensor` The extended attention mask, with the same dtype as :obj:`attention_mask.dtype`. """ # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length] # ourselves in which case we just need to make it broadcastable to all heads. if attention_mask.ndim == 3: extended_attention_mask = attention_mask[:, None, :, :] elif attention_mask.ndim == 2: # Provided a padding mask of dimensions [batch_size, seq_length] # - if the model is a decoder, apply a causal mask in addition to the padding mask # - if the model is an encoder, make the mask broadcastable to # [batch_size, num_heads, seq_length, seq_length] if is_decoder: batch_size, seq_length = input_shape seq_ids = Tensor( [i for i in range(seq_length)], dtype=mstype.int32) causal_mask = ( np.tile(seq_ids[None, None, :], (batch_size, seq_length, 1)) <= seq_ids[None, :, None] ) # add a prefix ones mask to the causal mask # causal and attention masks must have same type with pytorch version < 1.3 causal_mask = causal_mask.astype(attention_mask.dtype) if causal_mask.shape[1] < attention_mask.shape[1]: prefix_seq_len = attention_mask.shape[1] - \ causal_mask.shape[1] if has_query: # UniLM style attention mask causal_mask = self.concat_one( [self.zeros((batch_size, prefix_seq_len, seq_length), causal_mask.dtype), causal_mask]) causal_mask = self.concat_minus_one( [self.ones((batch_size, causal_mask.shape[1], prefix_seq_len), causal_mask.dtype), causal_mask]) extended_attention_mask = ( causal_mask[:, None, :, :] * attention_mask[:, None, None, :]) else: extended_attention_mask = attention_mask[:, None, None, :] else: raise ValueError( "Wrong shape for input_ids (shape {}) or attention_mask (shape {})".format( input_shape, attention_mask.shape ) ) # Since attention_mask is 1.0 for positions we want to attend and 0.0 for # masked positions, this operation will create a tensor which is 0.0 for # positions we want to attend and -10000.0 for masked positions. # Since we are adding it to the raw scores before the softmax, this is # effectively the same as removing these entirely. extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0 return extended_attention_mask def construct( self, input_ids=None, attention_mask=None, position_ids=None, head_mask=None, query_embeds=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_values=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None, is_decoder=False, ): r""" encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). """ output_attentions = ( output_attentions if output_attentions is not None else self.output_attentions ) output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.output_hidden_states ) return_dict = ( return_dict if return_dict is not None else self.use_return_dict ) if input_ids is None: assert query_embeds is not None, "You have to specify query_embeds when input_ids is None" # past_key_values_length past_key_values_length = ( past_key_values[0][0].shape[2] - self.query_length if past_key_values is not None else 0 ) query_length = query_embeds.shape[1] if query_embeds is not None else 0 # [bz, query_size/seq_length, qformer_hidden_size] dp embedding_output = self.embeddings( input_ids=input_ids, position_ids=position_ids, query_embeds=query_embeds, past_key_values_length=past_key_values_length, ) # [bz, query_size/seq_length] dp input_shape = embedding_output.shape[:-1] batch_size, seq_length = input_shape if attention_mask is None: attention_mask = self.ones( (batch_size, seq_length + past_key_values_length), mstype.float32 ) # [bz, seq_length] # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length] # ourselves in which case we just need to make it broadcastable to all heads. if is_decoder: extended_attention_mask = self.get_extended_attention_mask( attention_mask, input_ids.shape, is_decoder, has_query=(query_embeds is not None), ) else: extended_attention_mask = self.get_extended_attention_mask( attention_mask, input_shape, is_decoder, False ) # If a 2D or 3D attention mask is provided for the cross-attention # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length] if encoder_hidden_states is not None: if isinstance(encoder_hidden_states, list): encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states[0].shape else: encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.shape encoder_hidden_shape = ( encoder_batch_size, encoder_sequence_length) if isinstance(encoder_attention_mask, list): encoder_extended_attention_mask = [ self.invert_attention_mask(mask) for mask in encoder_attention_mask] elif encoder_attention_mask is None: encoder_attention_mask = self.ones(encoder_hidden_shape) encoder_extended_attention_mask = self.invert_attention_mask( encoder_attention_mask) else: encoder_extended_attention_mask = self.invert_attention_mask( encoder_attention_mask) else: encoder_extended_attention_mask = None # Prepare head mask if needed # 1.0 in head_mask indicate we keep the head # attention_probs has shape bsz x n_heads x N x N # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads] # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length] head_mask = self.get_head_mask(head_mask, self.num_hidden_layers) encoder_outputs = self.encoder( embedding_output, attention_mask=extended_attention_mask, head_mask=head_mask, encoder_hidden_states=encoder_hidden_states, encoder_attention_mask=encoder_extended_attention_mask, past_key_values=past_key_values, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, query_length=query_length, ) sequence_output = encoder_outputs[0] if not return_dict: return (sequence_output,) + encoder_outputs[1:] return ( sequence_output, encoder_outputs[1], encoder_outputs[2], encoder_outputs[3], encoder_outputs[4], ) class BertLMHeadModel(BertPreTrainedModel): """ BertLMHeadModel, the main model for Qformer Args: config (QFormerConfig): config for qformer, see qformer_config.py. Raises: ValueError: config type Error. Returns: a BertLMHeadModel instance. """ def __init__(self, config: QFormerConfig): super(BertLMHeadModel, self).__init__(config) if not isinstance(config, QFormerConfig): raise ValueError( f"Parameter config in `{self.__class__.__name__}(config)` " "should be an instance of class `QFormerConfig`. " "To create a model from a pretrained model use " f"`model = {self.__class__.__name__}.from_pretrained(PRETRAINED_MODEL_NAME)`" ) self.config = config self.bert = BertModel(config) self.cls = BertOnlyMLMHead(config) self.init_weights() if self.config.checkpoint_name_or_path: self.load_checkpoint(config) # for lm_loss reduction - GRAPH_MODE self.reduction = config.loss_reduction self.loss = CrossEntropyLoss( reduction=self.reduction, label_smoothing=0.1) self.vocab_size = self.config.vocab_size def convert_bert_model_params(self, bert_model_params: OrderedDict): """ convert params from BertModel in MindFormers, some param names are altered. """ dict_mapping = {'layer.': 'blocks.', 'encoder.': 'bert_encoder.encoder.', 'self_att.query.': 'dense1.', 'self_att.key.': 'dense2.', 'self_att.value.': 'dense3.', 'attention.output.dense.': 'attention.projection.', 'attention.output.layernorm.gamma': 'layernorm2.gamma', 'attention.output.layernorm.beta': 'layernorm2.beta', 'intermediate.dense.weight': 'output.mapping.weight', 'intermediate.dense.bias': 'output.mapping.bias', 'output.dense.weight': 'output.projection.weight', 'output.dense.bias': 'output.projection.bias', 'output.layernorm.gamma': 'layernorm1.gamma', 'output.layernorm.beta': 'layernorm1.beta', 'embeddings.position_embeddings.embedding_table': \ 'embedding_postprocessor.full_position_embedding.embedding_table', 'embeddings.layernorm.gamma': 'embedding_postprocessor.layernorm.gamma', 'embeddings.layernorm.beta': 'embedding_postprocessor.layernorm.beta', 'embeddings.word_embeddings.embedding_table': 'word_embedding.embedding_table', 'cls.predictions.transform.dense': 'bert.mlmloss.dense', 'cls.predictions.transform.layernorm': 'bert.mlmloss.layernorm', 'cls.predictions.decoder': 'bert.mlmloss.vocab_dense'} param_dict = self.parameters_dict() for name, data in param_dict.items(): new_name = name for replace_from, replace_to in dict_mapping.items(): new_name = new_name.replace(replace_from, replace_to) if new_name not in bert_model_params.keys(): logger.warning("%s not loaded.", name) continue new_data = bert_model_params[new_name] if name.endswith("intermediate.dense.weight") or name.endswith("output.dense.weight"): new_data = new_data.T data.assign_value(new_data) def load_checkpoint(self, config: QFormerConfig): """ load checkpoint for BertLMHeadModel. (we can use the param for BertModel on obs, but we need to alter the names of some param) Args: config (ModelConfig): QFormerConfig instance, which could have attribute "checkpoint_name_or_path (str)". set checkpoint_name_or_path to a supported model name or a path to checkpoint, to load model weights. """ checkpoint_name_or_path = config.checkpoint_name_or_path # direct loading, no param conversion if not config.convert_param_from_bert: super(BertLMHeadModel, self).load_checkpoint(config) # convert params and load from BertModel else: # the relevant file will be downloaded from the Obs platform. if not os.path.exists(checkpoint_name_or_path): support_list = MindFormerBook.get_model_support_list()['bert'] if checkpoint_name_or_path not in support_list: raise ValueError(f"{checkpoint_name_or_path} is not a supported default model" f" or a valid path to checkpoint," f" please select from {support_list}.") checkpoint_name = checkpoint_name_or_path default_checkpoint_download_folder = os.path.join( MindFormerBook.get_default_checkpoint_download_folder(), checkpoint_name_or_path.split("_")[0]) if not os.path.exists(default_checkpoint_download_folder): os.makedirs(default_checkpoint_download_folder, exist_ok=True) ckpt_file = os.path.join(default_checkpoint_download_folder, checkpoint_name + ".ckpt") if not os.path.exists(ckpt_file): url = MindFormerBook.get_model_ckpt_url_list()[checkpoint_name_or_path][0] succeed = download_with_progress_bar(url, ckpt_file) if not succeed: logger.info("checkpoint download failed, and pretrained weights are unloaded.") return try_sync_file(ckpt_file) self.default_checkpoint_download_path = ckpt_file logger.info("start to read the ckpt file: %s", os.path.getsize(ckpt_file)) else: ckpt_file = checkpoint_name_or_path param = load_checkpoint(ckpt_file) self.convert_bert_model_params(param) logger.info("weights in %s are loaded", ckpt_file) def get_input_embeddings(self) -> nn.Cell: return self.bert.get_input_embeddings() def set_input_embeddings(self, value): return self.bert.set_input_embeddings(value) def get_output_embeddings(self) -> nn.Cell: return self.cls.predictions.decoder def set_output_embeddings(self, value): self.cls.predictions.decoder = value def resize_token_embeddings(self, new_num_tokens: Optional[int] = None) -> nn.Embedding: """ resize token embeddings, inherit from super class. """ old_num_tokens = self.get_input_embeddings().embedding_table.shape[0] logger.info("resize_token_embeddings from %d to %d.", old_num_tokens, new_num_tokens) super(BertLMHeadModel, self).resize_token_embeddings(new_num_tokens) def tie_weights(self): """ tie encoder and decoder weights, inherit from super class. """ logger.info("weights tied.") super(BertLMHeadModel, self).tie_weights() def construct( self, input_ids=None, attention_mask=None, position_ids=None, head_mask=None, query_embeds=None, encoder_hidden_states=None, encoder_attention_mask=None, labels=None, past_key_values=None, use_cache=True, output_attentions=None, output_hidden_states=None, return_dict=None, return_logits=False, is_decoder=True ): r""" encoder_hidden_states `Tensor` of shape : (batch_size, sequence_length, hidden_size)` Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask `Tensor` of shape : (batch_size, sequence_length)`, `optional`) Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels `Tensor(mstype.int32)` of shape : (batch_size, sequence_length)`, `optional`) Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels n ``[0, ..., config.vocab_size]``, past_key_values (:obj:`tuple(tuple(Tensor(mstype.float)))` of length: `config.n_layers` with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)), Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape (batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape (batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). Returns: Example:: >>> from mindformers import BertTokenizer, Blip2Config, BertLMHeadModel >>> tokenizer = BertTokenizer.from_pretrained('bert-base-cased') >>> config = Blip2Config.from_pretrained("blip2_stage1_vit_g") >>> model = BertLMHeadModel(config=config) >>> inputs = tokenizer("Hello, my dog is cute") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits """ return_dict = ( return_dict if return_dict is not None else self.config.use_return_dict ) if labels is not None: use_cache = False if past_key_values is not None: query_embeds = None outputs = self.bert( input_ids, attention_mask=attention_mask, position_ids=position_ids, head_mask=head_mask, query_embeds=query_embeds, encoder_hidden_states=encoder_hidden_states, encoder_attention_mask=encoder_attention_mask, past_key_values=past_key_values, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, is_decoder=is_decoder, ) sequence_output = outputs[0] if query_embeds is not None: sequence_output = outputs[0][:, query_embeds.shape[1]:, :] prediction_scores = self.cls(sequence_output) if return_logits: return prediction_scores[:, :-1, :].copy() lm_loss = None if labels is not None: # we are doing next-token prediction; shift prediction scores and input ids by one shifted_prediction_scores = prediction_scores[:, :-1, :].copy() labels = labels[:, 1:].copy() lm_loss = self.loss( shifted_prediction_scores.view(-1, self.vocab_size), labels.view(-1), ) if self.reduction == "none": lm_loss = lm_loss.view(prediction_scores.size(0), -1).sum(1) if not return_dict: output = (prediction_scores,) + outputs[1:] return ((lm_loss,) + output) if lm_loss is not None else output return ( lm_loss, prediction_scores, outputs[1], outputs[2], outputs[3], outputs[4], ) pengcheng.mind-test/mindformers/models/blip2/qformer_config.py000066400000000000000000000121631471400134000251060ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Qformer Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.bert import BertConfig __all__ = ['QFormerConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class QFormerConfig(BertConfig): """ Qformer config class which defines the model size """ def __init__(self, model_type: str = "Qformer", num_hidden_layers: int = 12, num_attention_heads: int = 12, query_length: int = 32, vocab_size: int = 30523, hidden_size: int = 768, encoder_width: int = 1408, head_embed_dim: int = 256, bos_token_id: int = 30522, sep_token_id: int = 102, pad_token_id: int = 0, max_position_embeddings: int = 512, layer_norm_eps: float = 1.e-12, hidden_dropout_prob: float = 0.1, attention_probs_dropout_prob: float = 0.1, chunk_size_feed_forward: int = 0, cross_attention_freq: int = 2, intermediate_size: int = 3072, initializer_range: float = 0.02, hidden_act: str = "gelu", dtype: str = "float32", layernorm_dtype: str = "float32", softmax_dtype: str = "float32", compute_dtype: str = "float16", add_cross_attention: bool = True, use_relative_positions: bool = False, tie_word_embeddings: bool = True, output_attentions: bool = False, output_hidden_states: bool = False, use_return_dict: bool = False, convert_param_from_bert: bool = False, parallel_config: str = "default", moe_config: str = "default", **kwargs): super(QFormerConfig, self).__init__(**kwargs) self.model_type = model_type self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.query_length = query_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.encoder_width = encoder_width self.head_embed_dim = head_embed_dim self.bos_token_id = bos_token_id self.sep_token_id = sep_token_id self.pad_token_id = pad_token_id self.max_position_embeddings = max_position_embeddings self.layer_norm_eps = layer_norm_eps self.hidden_dropout_prob = hidden_dropout_prob self.attention_probs_dropout_prob = attention_probs_dropout_prob self.chunk_size_feed_forward = chunk_size_feed_forward self.cross_attention_freq = cross_attention_freq self.intermediate_size = intermediate_size self.initializer_range = initializer_range self.hidden_act = hidden_act self.dtype = mstype.float32 if dtype == "float32" else mstype.float16 self.layernorm_dtype = mstype.float32 if layernorm_dtype == "float32" else mstype.float16 self.softmax_dtype = mstype.float32 if softmax_dtype == "float32" else mstype.float16 self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.add_cross_attention = add_cross_attention self.use_relative_positions = use_relative_positions self.tie_word_embeddings = tie_word_embeddings self.output_attentions = output_attentions self.output_hidden_states = output_hidden_states self.use_return_dict = use_return_dict self.convert_param_from_bert = convert_param_from_bert self.parallel_config = default_transformer_config if parallel_config == "default" \ else parallel_config self.moe_config = default_moe_config if moe_config == "default" else moe_config self.chunk_size_feed_forward = kwargs.pop("chunk_size_feed_forward", 0) self.position_embedding_type = "absolute" if not use_relative_positions else "relative" # Whether input and output word embeddings should be tied. self.tie_word_embeddings = kwargs.pop( "tie_word_embeddings", True ) self.loss_reduction = kwargs.pop("loss_reduction", "mean") pengcheng.mind-test/mindformers/models/bloom/000077500000000000000000000000001471400134000216315ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/bloom/__init__.py000066400000000000000000000020461471400134000237440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for bloom""" from .bloom_config import BloomConfig from .bloom import BloomModel, BloomLMHeadModel from .bloom_tokenizer import BloomTokenizer from .bloom_processor import BloomProcessor from .bloom_reward import * __all__ = ['BloomConfig', 'BloomModel', 'BloomLMHeadModel', 'BloomTokenizer', 'BloomProcessor'] __all__.extend(bloom_reward.__all__) pengcheng.mind-test/mindformers/models/bloom/bloom.py000066400000000000000000000377301471400134000233250ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Bloom model""" import copy import os import numpy as np import mindspore.common.dtype as mstype from mindspore import nn from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.ops import operations as P from mindformers.modules.transformer import VocabEmbedding from mindformers.modules.layers import LayerNorm, AlibiTensor from mindformers.core.loss import CrossEntropyLoss from mindformers.models.base_model import BaseModel from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.mindformer_book import MindFormerBook from .layers import BloomBlocks, CausalMask from .bloom_config import BloomConfig from ..utils import convert_mstype, cell_reuse def jit_inference_with_condition(): """allow jit inference""" def decorator(func): if os.getenv("JIT_INFERENCE", "NOT_FOUND") == "NOT_FOUND": return func from mindspore import jit, JitConfig dec = jit(jit_config=JitConfig(jit_level="O2")) return dec(func) return decorator class BloomEmbeddingLayer(nn.Cell): """The Embedding Layer of Bloom network.""" def __init__(self, config=None): super(BloomEmbeddingLayer, self).__init__(auto_prefix=False) parallel_config = copy.deepcopy(config.parallel_config) embedding_mp = config.parallel_config.embedding_dp_mp_config.model_parallel vocab_size = config.vocab_size if vocab_size % embedding_mp != 0: logger.warning("The vocab size of embedding layer is: %s, it is not divide by model_parallel: %s", vocab_size, embedding_mp) logger.warning("Now, model_parallel will be changed: mp = 1") parallel_config.embedding_dp_mp_config.model_parallel = 1 self.word_embedding = VocabEmbedding(vocab_size=vocab_size, embedding_size=config.hidden_size, param_init=initializer("normal", [vocab_size, config.hidden_size], dtype=config.embedding_init_type), parallel_config=config.parallel_config.embedding_dp_mp_config) self.norm = LayerNorm((config.hidden_size,)).shard(((1, 1, 1), (1,), (1,))) def construct(self, input_ids): """The forward compute of Embedding Layer.""" word_embedding, word_table = self.word_embedding(input_ids) embedding = self.norm(word_embedding) embedding = embedding.astype(mstype.float16) return embedding, word_table def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers, use_select_recompute=False): """ Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. use_select_recompute(bool) - Indicates whether to use the select recompute mode. """ pp_dis = max(int((layers + 1) / parallel_config.pipeline_stage), 1) pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not use_select_recompute: network.recompute() else: if parallel_config.recompute.recompute and not use_select_recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class BloomModel(nn.Cell): """ The backbone of Bloom network Args: config(BloomConfig): The config of network Inputs: input_ids(Tensor): The tokenized inputs with datatype int32 input_mask(Tensor): The mask indicating whether each position is a valid input Returns: output_state(Tensor): The output logit of backbone embedding_table(Tensor): The embedding table for the vocabulary """ def __init__(self, config): super(BloomModel, self).__init__() self.embedding = BloomEmbeddingLayer(config) self.embedding.pipeline_stage = 0 self.make_causal_attention = CausalMask(seq_length=config.seq_length, parallel_config=config.parallel_config.dp_mp_config) self.build_alibi_tensor = AlibiTensor(seq_length=config.seq_length, num_heads=config.num_heads, parallel_config=config.parallel_config) self.blocks = BloomBlocks(hidden_size=config.hidden_size, batch_size=config.batch_size, ffn_hidden_size=config.hidden_size * config.expand_ratio, seq_length=config.seq_length, num_layers=config.num_layers, num_heads=config.num_heads, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, hidden_act=config.hidden_act, lambda_func=set_parallel_configure_for_layer, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, use_past=config.use_past, use_seq_parallel=config.use_seq_parallel, use_select_recompute=config.use_select_recompute, parallel_config=config.parallel_config).blocks self.num_layers = config.num_layers self.ln_f = LayerNorm((config.hidden_size,)).to_float(config.layernorm_compute_type) if config.parallel_config.pipeline_stage > 1: self.ln_f.set_comm_fusion(2) else: self.ln_f.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.ln_f.shard(((config.parallel_config.data_parallel, 1, 1),)) self.ln_f.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.use_past = config.use_past self.dtype = convert_mstype(config.param_init_type) self.mul_init_reset = P.Mul().shard( ((config.parallel_config.data_parallel, config.parallel_config.model_parallel, 1, 1), (1,))) def construct(self, input_ids, input_mask, init_reset=True, batch_valid_length=None): """Bloom model""" input_embedding, embedding_table = self.embedding(input_ids) hidden_states = input_embedding hidden_states_shape = hidden_states.shape hidden_states = hidden_states.reshape((-1, hidden_states_shape[-1])) causal_mask = self.make_causal_attention(input_mask) alibi_tensor = self.build_alibi_tensor(input_mask, hidden_states.dtype) if self.use_past: init_reset = self.mul_init_reset(self.blocks[0].key_past, init_reset.astype(self.dtype)) for i in range(self.num_layers): hidden_states, _ = self.blocks[i](hidden_states, alibi_tensor, causal_mask, init_reset, batch_valid_length) hidden_states = hidden_states.reshape(hidden_states_shape) output_state = self.ln_f(hidden_states) return output_state, embedding_table class BloomHead(nn.Cell): """Head for Bloom to get the logits of each token in the vocab.""" def __init__(self, hidden_size, vocab_size, compute_type="float16", parallel_config=None): super(BloomHead, self).__init__() copied_parallel_config = copy.deepcopy(parallel_config) mp = copied_parallel_config.model_parallel if vocab_size % mp != 0: logger.warning("The vocab size of BloomHead MatMul is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of BloomHead MatMul will be changed: mp = 1") copied_parallel_config.model_parallel = 1 if copied_parallel_config.pipeline_stage > 1: copied_parallel_config.vocab_emb_dp = False if copied_parallel_config.vocab_emb_dp: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), ( copied_parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = convert_mstype(compute_type) def construct(self, state, embedding_table): ori_dtype = state.dtype state = state.reshape((-1, self.hidden_size)) logits = self.matmul(state.astype(self.dtype), embedding_table.astype(self.dtype)) return logits.astype(ori_dtype) @MindFormerRegister.register(MindFormerModuleType.MODELS) class BloomLMHeadModel(BaseModel): """ Provide bloom training loss or logits through network. Args: config (BloomConfig): The config of BloomModel. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['bloom'] @cell_reuse() def __init__(self, config=None): config = config if config is not None else BloomConfig() super(BloomLMHeadModel, self).__init__(config, auto_prefix=False) self.use_past = self.config.use_past self.is_sample_acceleration = self.config.is_sample_acceleration if self.use_past: self.input_mask_all_ones = Tensor( np.ones((self.config.batch_size, self.config.seq_length), np.float32), mstype.float32) if self.is_sample_acceleration: self.p_all_ones = Tensor(np.ones((self.config.batch_size, 1), np.float32), mstype.float32) self.eos_token_id = self.config.eos_token_id parallel_config = self.config.parallel_config self.stridedslice = P.StridedSlice().shard(((parallel_config.data_parallel, 1),)) self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.gt = P.Greater().shard(((parallel_config.data_parallel, 1), ())) self.mul = P.Mul().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.abs = P.Abs().shard(((parallel_config.data_parallel, 1),)) self.transformer = BloomModel(self.config) self.head = BloomHead(hidden_size=config.hidden_size, vocab_size=config.vocab_size, parallel_config=self.config.parallel_config) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.transformer.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) mp = config.parallel_config.model_parallel vocab_size = config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of Bloom Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of Bloom Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = CrossEntropyLoss(parallel_config=loss_parallel_config) self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 @jit_inference_with_condition() def construct(self, input_ids, input_position=None, position_ids=None, attention_mask=None, input_embeds=None, labels=None, init_reset=True, batch_valid_length=None): """ construct function for Language Modeling Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary. input_position(Tensor): current position, used by model.predict. Default None. position_ids(Tensor): Reserved param, not used. attention_mask(Tensor): Reserved param, not used. input_embeds(Tensor): Reserved param, not used. labels(Tensor): Reserved param, not used. init_reset(bool, optional): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Default True. batch_valid_length(Tensor): the past calculated the index with datatype int32, used for incremental prediction. Tensor of shape :math:`(batch_size,)`. Default None. Returns: logits (Tensor) or loss (mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ batch_size, seq_length = input_ids.shape if self.training: tokens = self.stridedslice(input_ids, (0, 0), (batch_size, seq_length - 1), (1, 1)) else: tokens = input_ids input_mask = self.not_equal(tokens, self.eos_token_id).astype(mstype.float32) \ if not self.use_past else self.input_mask_all_ones loss_mask = self.mul(input_mask, self.gt(tokens, 0).astype(mstype.float32)) tokens = self.abs(tokens) # [batch_size, seq_length, vocab_size] output_states, embedding_table = self.transformer(tokens, input_mask, init_reset, batch_valid_length) logits = self.head(output_states, embedding_table) if not self.training: if self.is_sample_acceleration: return self.get_top_token_id(logits, current_index=input_position) return logits, tokens, input_mask labels = self.stridedslice(input_ids, (0, 1), (batch_size, seq_length), (1, 1)) labels = labels.reshape((-1,)) loss_mask = loss_mask.reshape((-1,)) loss = self.loss(logits, labels, loss_mask) return loss def get_top_token_id(self, logits, current_index=None): """get_top_token_id""" logits = logits.reshape(-1, logits.shape[-1]) if self.use_past and not self.is_first_iteration: logits = logits elif current_index is not None: index = current_index.view(-1,) logits = P.Gather()(logits, index, 0) probabilities = P.Softmax(-1)(logits) top_token_id = P.Argmax(-1)(probabilities) top_token_id = top_token_id.view(-1, 1) return self.p_all_ones, top_token_id pengcheng.mind-test/mindformers/models/bloom/bloom_config.py000066400000000000000000000111321471400134000246360ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Bloom Config API""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.utils import convert_mstype from mindformers.models.base_config import BaseConfig from mindformers.mindformer_book import MindFormerBook __all__ = ['BloomConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class BloomConfig(BaseConfig): """ Bloom config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['bloom'] def __init__(self, embedding_dropout_prob: float = 0.0, batch_size: int = None, seq_length: int = 1024, vocab_size: int = 250880, hidden_size: int = 64, num_layers: int = 2, num_heads: int = 8, expand_ratio: int = 4, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, unk_token_id: int = 0, bos_token_id: int = 1, eos_token_id: int = 2, pad_token_id: int = 3, param_init_type: str = "float32", embedding_init_type: str = "float32", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", compute_dtype: str = "float16", hidden_act: str = 'gelu', parallel_config: TransformerOpParallelConfig = default_transformer_config, checkpoint_name_or_path: str = "", moe_config: MoEConfig = default_moe_config, use_past: bool = False, use_seq_parallel: bool = False, use_select_recompute: bool = False, repetition_penalty: int = 1, max_decode_length: int = 1024, top_k: int = 5, top_p: int = 1, do_sample: bool = True, is_sample_acceleration: bool = False, **kwargs): super().__init__(**kwargs) self.embedding_dropout_prob = embedding_dropout_prob self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.expand_ratio = expand_ratio self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.param_init_type = convert_mstype(param_init_type) self.embedding_init_type = convert_mstype(embedding_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.compute_dtype = convert_mstype(compute_dtype) self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path self.moe_config = moe_config self.use_past = use_past self.unk_token_id = unk_token_id self.bos_token_id = bos_token_id self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.hidden_act = hidden_act self.use_seq_parallel = use_seq_parallel self.use_select_recompute = use_select_recompute self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample self.is_sample_acceleration = is_sample_acceleration if self.batch_size is None: self.use_past = False # currently require batch_size = 1 self.is_sample_acceleration = False # currently require batch_size = 1 pengcheng.mind-test/mindformers/models/bloom/bloom_processor.py000066400000000000000000000044741471400134000254230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BloomProcessor """ from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import BaseTokenizer from mindformers.models.base_processor import BaseProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['BloomProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class BloomProcessor(BaseProcessor): """ Bloom processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['bloom'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(BloomProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-test/mindformers/models/bloom/bloom_reward.py000066400000000000000000000107631471400134000246660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """bloom reward model""" import copy import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.modules.layers import Linear from mindformers.core.loss import CompareLoss from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.tools.logger import logger from .bloom_config import BloomConfig from .bloom import BloomModel __all__ = ['BloomRewardModel', 'VHead'] class VHead(nn.Cell): r"""Head for Bloom to get the logits of each token in the vocab.""" def __init__(self, config=None): super().__init__() dp = config.parallel_config.data_parallel mp = 1 self.vhead = Linear(in_channels=config.hidden_size, out_channels=1, has_bias=False).to_float(mstype.float16) self.vhead.shard(strategy_matmul=((dp, mp), (mp, 1))) self.vhead.pipeline_stage = config.parallel_config.pipeline_stage - 1 def construct(self, output_states): """ construct function for vhead """ return self.vhead(output_states) @MindFormerRegister.register(MindFormerModuleType.MODELS) class BloomRewardModel(BaseModel): r""" Provide bloom reward model training loss or logits through network. Args: config (BloomConfig): The config of BloomModel. Returns: Tensor, the loss or logits of the network. """ def __init__(self, config=None): config = config if config is not None else BloomConfig() super(BloomRewardModel, self).__init__(config, auto_prefix=False) mp = config.parallel_config.model_parallel self.eos_token_id = self.config.eos_token_id self.seq_length = config.seq_length self.eos_token = self.config.eos_token parallel_config = self.config.parallel_config self.stridedslice = P.StridedSlice().shard(((parallel_config.data_parallel, 1),)) self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.transformer = BloomModel(self.config) self.vhead = VHead(self.config) if parallel_config.pipeline_stage > 1: self.vhead.pipeline_stage = parallel_config.pipeline_stage - 1 self.transformer.embedding.word_embedding.embedding_table.add_pipeline_stage(self.vhead.pipeline_stage) vocab_size = config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of Bloom Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of Bloom Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = CompareLoss(config=loss_parallel_config) self.load_checkpoint(config) def construct(self, input_ids, position_id=None, attention_mask=None, loss_mask=None, end_ind=None): """ construct function for reward model """ _ = position_id if attention_mask is None: attention_mask = self.not_equal(input_ids, self.eos_token_id).astype(mstype.float32) output_states, _ = self.transformer(input_ids, attention_mask) # [bs, seq, hidden_size] logits = self.vhead(output_states) # [bs, seq, 1] logits = logits.squeeze(-1) logits = F.reshape(logits, (-1, self.seq_length)) loss, chosen_end_scores, reject_end_scores = self.loss(logits, loss_mask, end_ind) if self.training: return loss return loss, chosen_end_scores, reject_end_scores pengcheng.mind-test/mindformers/models/bloom/bloom_tokenizer.py000066400000000000000000000247401471400134000254140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Bloom Tokenzier """ import json import os from functools import lru_cache from typing import List, Optional import regex as re from mindformers.models.base_tokenizer import Tokenizer from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.mindformer_book import MindFormerBook __all__ = ['BloomTokenizer'] VOCAB_FILES_NAMES = {'vocab_file': 'tokenizer.json'} @lru_cache() def bytes_to_unicode(): """ bytes to unicode """ bs = list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1)) cs = bs[:] n = 0 for b in range(2 ** 8): if b not in bs: bs.append(b) cs.append(2 ** 8 + n) n += 1 cs = [chr(i) for i in cs] return dict(zip(bs, cs)) def get_pairs(word): """ Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). """ pairs = set() prev_char = word[0] for char in word[1:]: pairs.add((prev_char, char)) prev_char = char return pairs @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class BloomTokenizer(Tokenizer): r""" Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Args: vocab_file(str): The vocabulary file path. unk_token(str): The token that represents the unknown. Default "<|unk|>". bos_token(str): The token that represents the begin-of-sentence. Default "<|s|>"". eos_token(str): The token that represents the end-of-sentence. Default "<|/s|>". pad_token(str): The token that represents the pad. Default "<|pad|>". add_prefix_space(bool): whether to add a whitespace in the front of text. Default "False" add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import BloomTokenizer >>> tokenizer = BloomTokenizer.from_pretrained("bloom_560m") >>> res = tokenizer("Hello world", add_special_tokens=False) >>> print(res) {'input_ids': [59414, 8876], 'token_type_ids': [0, 0], 'attention_mask': [1, 1]} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES FILE_LIST = ['tokenizer_config.json'] model_input_names = ["input_ids", "token_type_ids", "attention_mask"] _support_list = MindFormerBook.get_tokenizer_support_list()['bloom'] def __init__( self, vocab_file, unk_token="", bos_token="", eos_token="", pad_token="", add_prefix_space=False, add_bos_token=False, add_eos_token=False, **kwargs): super(BloomTokenizer, self).__init__( unk_token=unk_token, bos_token=bos_token, eos_token=eos_token, pad_token=pad_token, **kwargs ) self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token with open(vocab_file, 'r', encoding="utf-8") as vocab_handle: self.encoder = json.load(vocab_handle)["model"]["vocab"] self.decoder = {v: k for k, v in self.encoder.items()} with open(vocab_file, 'r', encoding="utf-8") as vocab_handle: bpe_merges = json.load(vocab_handle)["model"]["merges"] self.the_unk_token = unk_token bpe_merges = [tuple(merge.split()) for merge in bpe_merges] self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges)))) self.byte_encoder = bytes_to_unicode() self.byte_decoder = {v: k for k, v in self.byte_encoder.items()} self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""") self.add_prefix_space = add_prefix_space self.cache = {} self._unk_token_id = 0 self._bos_token_id = 1 self._eos_token_id = 2 self._pad_token_id = 3 def bpe(self, token): """ bpe encode """ if token in self.cache: return self.cache[token] word = tuple(token) pairs = get_pairs(token) if not pairs: return token while True: bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf"))) if bigram not in self.bpe_ranks: break first, second = bigram new_word = [] i = 0 while i < len(word): try: j = word.index(first, i) except ValueError: new_word.extend(word[i:]) break else: new_word.extend(word[i:j]) i = j if word[i] == first and i + 1 < len(word) and word[i + 1] == second: new_word.append(first + second) i += 2 else: new_word.append(word[i]) i += 1 new_word = tuple(new_word) word = new_word if len(word) == 1: break else: pairs = get_pairs(word) word = " ".join(word) self.cache[token] = word return word def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): if not isinstance(text, str): raise ValueError("Text should be type str, but found type", type(text)) return self._tokenize(text) def _tokenize(self, text, **kwargs): """ Tokenize a string using bpe encode. """ text = self.prepare_for_tokenization(text, is_pretokenized=False) bpe_tokens = [] for token in re.findall(self.pat, text): token = "".join( self.byte_encoder[b] for b in token.encode("utf-8") ) bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(" ")) return bpe_tokens def _convert_token_to_id(self, token): """ the index of the tokens in the vocabulary. """ return self.encoder.get(token, self.encoder.get(self.the_unk_token)) def _convert_id_to_token(self, index): """ return the origin bpe tokens according to ids """ return self.decoder.get(index) def _convert_tokens_to_string(self, tokens): """ return a string according to the list of tokens""" text = "".join(tokens) text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors='ignore') return text def convert_tokens_to_string(self, tokens): """Convert the tokens to the string""" return self._convert_tokens_to_string(tokens) def prepare_for_tokenization(self, text, **kwargs): """ whether to add a whitespace in the front of text """ add_prefix_space = kwargs.pop("add_prefix_space", self.add_prefix_space) is_pretokenized = kwargs.pop("is_pretokenized", False) if is_pretokenized or add_prefix_space: text = " " + text return text def save_vocabulary(self, save_directory, filename_prefix=None): """write the word to the files""" if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None output_file_path = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]) with open(output_file_path, "w", encoding="utf-8") as f: f.write(json.dumps(self.encoder)) return output_file_path @property def vocab_size(self): """Get the vocab size of the """ return len(self.decoder) def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): """Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output def get_vocab(self): return dict(self.encoder, **self.added_tokens_encoder) pengcheng.mind-test/mindformers/models/bloom/convert_weight.py000066400000000000000000000145771471400134000252500ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from huggingface""" import os import re import argparse import torch import mindspore from mindspore import Tensor, Parameter def layer_name_mapping(key): """Convert huggingface PP weights mapping in MindSpore. return: split, new_name """ prefix = '' if 'transformer' in key: prefix = 'transformer.' key = key.replace('transformer.', '') # Handle first and last layers layer_rename_map = { "word_embeddings.weight": "embedding.word_embedding.embedding_table", "word_embeddings_layernorm.weight": "embedding.norm.gamma", "word_embeddings_layernorm.bias": "embedding.norm.beta", "ln_f.weight": "ln_f.gamma", "ln_f.bias": "ln_f.beta", "input_layernorm.weight": "layernorm1.gamma", "input_layernorm.bias": "layernorm1.beta", "self_attention.query_key_value.weight": "attention.dense{}.weight", "self_attention.query_key_value.bias": "attention.dense{}.bias", "self_attention.dense.weight": "attention.projection.weight", "self_attention.dense.bias": "attention.projection.bias", "post_attention_layernorm.weight": "layernorm2.gamma", "post_attention_layernorm.bias": "layernorm2.beta", "mlp.dense_h_to_4h.weight": "output.mapping.weight", "mlp.dense_h_to_4h.bias": "output.mapping.bias", "mlp.dense_4h_to_h.weight": "output.projection.weight", "mlp.dense_4h_to_h.bias": "output.projection.bias", "lm_head.weight": "head.weight", "lm_head.bias": "head.bias", } split = False if key in layer_rename_map: return split, prefix + layer_rename_map[key] # Handle transformer blocks match = re.match(r'^\w*\.(\d+)\.(\w+\.\w+\.\w+|\w+\.\w+)$', key) layer_number = int(match.group(1)) text = match.group(2) if "self_attention.query_key_value" in key: split = True return split, f"{prefix}blocks.{layer_number}." + layer_rename_map[text] def hf_to_ms(hf_weights, args, ms_dtype=mindspore.float32, for_save=False): """Convert hf layers to ms.""" ms_params = {} for k, v in hf_weights.items(): print(k, v.shape, v.dtype) split, new_name = layer_name_mapping(k) if split: if 'weight' in new_name: v = v.reshape(args.n_head, 3, args.hidden_size // args.n_head, v.shape[-1]) v_list = v.tensor_split(3, dim=1) for i in range(1, 4): tmp_name = new_name.format(i) print(v_list[i-1].shape) tmp_tensor = Tensor(v_list[i-1].reshape(-1, v_list[i-1].shape[-1]).float().detach().numpy(), ms_dtype) ms_params[tmp_name] = Parameter(tmp_tensor, name=tmp_name) else: v = v.reshape(args.n_head, 3, args.hidden_size // args.n_head) v_list = v.tensor_split(3, dim=1) for i in range(1, 4): tmp_name = new_name.format(i) print(v_list[i-1].shape) tmp_tensor = Tensor(v_list[i-1].reshape(-1).float().detach().numpy(), ms_dtype) ms_params[tmp_name] = Parameter(tmp_tensor, name=tmp_name) else: if ('projection' in new_name or 'mapping' in new_name) and 'weight' in new_name: new_tensor = Tensor(v.transpose(0, 1).float().detach().numpy(), ms_dtype) else: new_tensor = Tensor(v.float().detach().numpy(), ms_dtype) ms_params[new_name] = Parameter(new_tensor, name=new_name) if for_save: return [{'name': k, 'data': v} for k, v in ms_params.items()] return ms_params def process_hf_shard_files(file_list, args, save_dir=None, combine=False, ms_dtype=mindspore.float32): ''' torch ckpt files loop''' if save_dir and not os.path.exists(save_dir): os.makedirs(save_dir, exist_ok=True) combine_params = [] file = None for file in file_list: pt_states = torch.load(file, map_location='cpu') ms_params = hf_to_ms(pt_states, args, ms_dtype, True) if combine: combine_params.extend(ms_params) else: save_file = save_dir + '/' + file.split('/')[-1] if save_dir else file + '.ckpt' mindspore.save_checkpoint(ms_params, save_file) del pt_states del ms_params if combine: path = save_dir + '/' + 'combine.ckpt' if save_dir else \ '/'.join(file.split('/')[:-1]) + 'combine.ckpt' mindspore.save_checkpoint(combine_params, path) if __name__ == '__main__': parser = argparse.ArgumentParser(description="Bloom convert script") parser.add_argument('--n_head', type=int, default=32, required=True, help="The number of head of the model to be converted.") parser.add_argument('--hidden_size', type=int, default=4096, required=True, help="The number of hidden size of the model to be converted.") parser.add_argument("--torch_path", type=str, default=None, required=True, help="The input torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default=None, help="The output mindspore checkpoint path.") config = parser.parse_args() # convert hf ckpt to ms process_hf_shard_files(file_list=[config.torch_path], args=config, combine=True, save_dir=config.mindspore_path) pengcheng.mind-test/mindformers/models/bloom/layers.py000066400000000000000000000735751471400134000235230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Bloom custom layers""" import math import numpy as np from mindspore.common.tensor import Tensor from mindspore import nn, ops, Parameter import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.context import ParallelMode from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindformers.tools.logger import logger as mindformer_logger from mindformers.modules import AttentionMask from mindformers.modules.transformer.op_parallel_config import default_dpmp_config from mindformers.modules.transformer.moe import default_moe_config from mindformers.modules.transformer import MultiHeadAttention,\ TransformerEncoderLayer, TransformerEncoder from mindformers.modules.transformer.transformer import default_transformer_config, _get_lambda_func class BloomAttention(MultiHeadAttention): r"""The implementation of Bloom attention.""" def __init__(self, batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, hidden_dropout_rate=0.1, attention_dropout_rate=0.1, compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, use_past=False, use_seq_parallel=False, use_select_recompute=False, parallel_config=default_dpmp_config): super(BloomAttention, self).__init__(batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, hidden_dropout_rate, attention_dropout_rate, compute_dtype, softmax_compute_type, param_init_type, use_past, parallel_config) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.inv_norm_factor = Tensor([1.0 / math.sqrt(self.size_per_head)]) self.beta = Tensor([1.0]) else: self.add_alibi = P.Add().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.mul_alibi = P.Mul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (1,))) self.mul_alibi1 = P.Mul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (1,))) # [B, 1, S, S] + [B, H, S, S] self.add_atten_score = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # [B, H, S, D] * [B, 1, S, 1] self.mul_key = P.Mul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (1, 1, 1, 1))) # [B, H, S, D] * [B, 1, S, 1] self.mul_value = P.Mul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (1, 1, 1, 1))) # [B, H, S, D] * [B, H, S, D] self.add_key = P.Add().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # [B, H, S, D] * [B, H, S, D] self.add_value = P.Add().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.inv_norm_factor = Tensor([1.0 / math.sqrt(self.size_per_head)]) self.beta = Tensor([1.0]) self.cast = P.Cast().shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.batch_matmul_trans_b = P.BatchMatMul(transpose_b=True).shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) if use_select_recompute: mindformer_logger.info("Using select recompute mode!") self.cast.recompute() self.batch_matmul.recompute() self.batch_matmul_trans_b.recompute() self.sub.recompute() self.mul_alibi1.recompute() self.add.recompute() self.add_alibi.recompute() self.prob_dropout.recompute() self.softmax.softmax.recompute() self.softmax_3d.recompute() if use_seq_parallel: mindformer_logger.info("Using seq parallel mode!") self.dropout.dropout.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) # pylint: disable=arguments-differ def construct(self, query_tensor, key_tensor, value_tensor, alibi_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the BloomAttention""" self._check_inputs(query_tensor, key_tensor, value_tensor, attention_mask, key_past, value_past, batch_valid_length) ori_shape = query_tensor.shape batch_size = self._get_batch_size_from_query(query_tensor) query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) ori_dtype = query_tensor.dtype query_tensor = query_tensor.astype(self.dtype) key_tensor = key_tensor.astype(self.dtype) value_tensor = value_tensor.astype(self.dtype) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( query.reshape((batch_size, self._get_seq_length_under_incremental(self.src_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # return query, query # the returned shape is [bs, num_heads, size_per_head, seq_length] key = self.transpose( key.reshape((batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( value.reshape((batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if attention_mask is not None and attention_mask.ndim == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # key and value for current token(s) key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = (self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_key(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_value(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: current_index = batch_valid_length - 1 current_index = current_index.reshape((-1, 1, 1)) current_mask = (self.equal(self.range, current_index)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_key(key, self.expand_dims(current_mask, 3)) current_value = self.mul_value(value, self.expand_dims(current_mask, 3)) # Concat the previous saved state and current state key = self.add_key(key_past, current_key) value = self.add_value(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, alibi_tensor, attention_mask, batch_valid_length) # Output output = self.projection(attention) output = self.dropout(output) output = output.reshape(ori_shape) output = output.astype(ori_dtype) return output, layer_present def _softmax(self, attention_scores): """ :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ attention_probs = self.softmax(attention_scores) return attention_probs def _attn(self, query, key, value, alibi_tensor, attention_mask, batch_valid_length): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix alibi_tensor: the alibi matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] ori_dtype = query.dtype score = self.batch_matmul_trans_b(query.astype(self.dtype), key.astype(self.dtype)) score = self.add_alibi( self.mul_alibi1(score, self.inv_norm_factor.astype(ori_dtype)), self.mul_alibi(alibi_tensor, self.beta.astype(ori_dtype)) ) attention_scores = self.cast(score, self.softmax_dtype) # for input size of (bs, 1) namely the second graph, # the shape of attention_mask matrix should be (bs, 1, 1, seq_length) if attention_mask is not None: if self.use_past and not self.is_first_iteration: # Calculate the current total token # bs, 1 -> bs, 1, 1 batch_valid_length = batch_valid_length - 1 index = batch_valid_length.reshape((-1, 1, 1)) # Calculate the attention_mask matrix via the position index attention_mask = (self.tensor_le(self.range, index)).astype(mstype.int32) attention_mask = self.expand_dims(attention_mask, 2) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( Tensor((1.0,)).astype(attention_scores.dtype), attention_mask.astype(attention_scores.dtype)) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add_atten_score(adder, attention_scores) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = self.cast(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs.astype(self.dtype), value.astype(self.dtype)) weighted_values = weighted_values.astype(self.softmax_dtype) attention_merge = self._merge_heads(weighted_values) return attention_merge class BloomBlock(TransformerEncoderLayer): r"""A block of Bloom model.""" def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', use_past=False, use_seq_parallel=False, use_select_recompute=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(BloomBlock, self).__init__(batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate, hidden_dropout_rate, post_layernorm_residual, layernorm_compute_type, softmax_compute_type, param_init_type, hidden_act, use_past, moe_config, parallel_config) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.use_past = use_past attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = BloomAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, use_seq_parallel=use_seq_parallel, use_select_recompute=use_select_recompute, parallel_config=attention_parallel_config) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.use_past = use_past attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = BloomAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, use_seq_parallel=use_seq_parallel, use_select_recompute=use_select_recompute, parallel_config=attention_parallel_config) if self.use_past: size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, seq_length, size_per_head) self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") # [B, H, S, D] self.assign_key_past = P.Assign().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # [B, H, S, D] self.assign_value_past = P.Assign().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) if use_seq_parallel: self.add.shard(((parallel_config.data_parallel*parallel_config.model_parallel, 1), (parallel_config.data_parallel*parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel*parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel*parallel_config.model_parallel, 1),)) if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) # pylint: disable=arguments-differ def construct(self, x, alibi_tensor, input_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(x, input_mask, init_reset, batch_valid_length) if self.post_layernorm_residual: input_x = x else: input_x = self.layernorm1(x) input_x = input_x.astype(self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past and self.is_first_iteration: # reset states, init_reset True for reuse and False for reset self.assign_key_past(self.key_past, init_reset) key_reset = self.key_past self.assign_value_past(self.value_past, init_reset) value_reset = self.value_past # add dependency for desired execution order input_x = ops.depend(input_x, key_reset) input_x = ops.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, alibi_tensor, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = output_x.astype(self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign_key_past(self.key_past, key_present) key_update = self.key_past self.assign_value_past(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = ops.depend(key_update, key_reset) value_update = ops.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = ops.depend(mlp_logit, value_update) mlp_logit = ops.depend(mlp_logit, key_update) output = self.add(x, mlp_logit) if self.use_moe: return output, layer_present, aux_loss return output, layer_present class BloomBlocks(TransformerEncoder): r"""All blocks of Bloom model.""" def __init__(self, batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, offset=0, use_past=False, use_seq_parallel=False, use_select_recompute=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(BloomBlocks, self).__init__(batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, attention_dropout_rate, hidden_dropout_rate, hidden_act, post_layernorm_residual, layernorm_compute_type, softmax_compute_type, param_init_type, lambda_func, offset, use_past, moe_config, parallel_config) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = BloomBlock(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, use_seq_parallel=use_seq_parallel, use_select_recompute=use_select_recompute, moe_config=moe_config, parallel_config=config_to_layer) if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config, use_select_recompute=use_select_recompute) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = BloomBlock(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, use_seq_parallel=use_seq_parallel, use_select_recompute=use_select_recompute, moe_config=moe_config, parallel_config=config_to_layer) if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config, use_select_recompute=use_select_recompute) self.blocks.append(block) # pylint: disable=arguments-differ def construct(self, hidden_states, alibi_tensor, attention_mask, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, alibi_tensor, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class CausalMask(AttentionMask): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length) with 1 and 0, where 1 indicates the current position is a valid token, otherwise not. Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). """ def __init__(self, seq_length, parallel_config=default_dpmp_config): super(CausalMask, self).__init__(seq_length, parallel_config) self.seq_length = seq_length self.parallel_config = parallel_config def construct(self, input_mask): """Forward process of the CausalMask""" input_mask = self.not_equal(input_mask, 0).astype(mstype.float32) input_shape = input_mask.shape shape_right = (input_shape[0], 1, input_shape[1]) # Mask the padded inputs attention_mask = input_mask.reshape(shape_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask pengcheng.mind-test/mindformers/models/build_config.py000066400000000000000000000051061471400134000235210ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ build model config modules """ from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_model_config( config: dict = None, default_args: dict = None, module_type: str = 'config', class_name: str = None, **kwargs): r"""Build model config For MindFormer. Instantiate the model config from MindFormerRegister's registry. Args: config (dict): The task model config's config. Default: None. default_args (dict): The default argument of model config API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'config'. class_name (str): The class name of model config API. Default: None. Return: The function instance of model config API. Examples: >>> from mindformers import build_model_config >>> model_config = {'type': 'ViTConfig'} >>> # 1) use config dict to build model >>> model_config_from_config = build_model_config(model_config) >>> # 2) use class name to build model >>> model_config_class_name = build_model_config(class_name='ViTConfig') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.text_config is not None: config.text_config = build_model_config(config.text_config) if config.vision_config is not None: config.vision_config = build_model_config(config.vision_config) if config.head_config is not None: config.head_config = build_model_config(config.head_config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.CONFIG, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-test/mindformers/models/build_model.py000066400000000000000000000102021471400134000233450ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Model API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .build_config import build_model_config def build_model( config: dict = None, default_args: dict = None, module_type: str = 'models', class_name: str = None, **kwargs): r"""Build model For MindFormer. Instantiate the model from MindFormerRegister's registry. Args: config (dict): The task model's config. Default: None. default_args (dict): The default argument of model API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'model'. class_name (str): The class name of model API. Default: None. Return: The function instance of model API. Examples: >>> from mindformers import build_model >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> model_from_config = build_model(config.model) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if default_args is None: default_args = {} model_config = build_model_config(config.model_config, default_args=default_args) if model_config is not None: if default_args is not None: for key, value in default_args.items(): model_config.__setattr__(key, value) default_args = {} default_args.setdefault('config', model_config) return MindFormerRegister.get_instance_from_cfg( config.arch, MindFormerModuleType.MODELS, default_args=default_args) return None return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def build_encoder( config: dict = None, default_args: dict = None, module_type: str = 'encoder', class_name: str = None, **kwargs): """Build encoder API.""" if config is None and class_name is None: return None if config is not None: cfg_encoders = config if not isinstance(cfg_encoders, list): return MindFormerRegister.get_instance_from_cfg( cfg_encoders, MindFormerModuleType.ENCODER, default_args=default_args) encoders = [] for encoder in cfg_encoders: encoder_op = MindFormerRegister.get_instance_from_cfg( encoder, MindFormerModuleType.ENCODER) encoders.append(encoder_op) return encoders return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def build_head( config: dict = None, default_args: dict = None, module_type: str = 'head', class_name: str = None, **kwargs): """Build head API.""" if config is None and class_name is None: return None if config is not None: cfg_heads = config if not isinstance(cfg_heads, list): return MindFormerRegister.get_instance_from_cfg( cfg_heads, MindFormerModuleType.HEAD, default_args=default_args) heads = [] for head in cfg_heads: head_op = MindFormerRegister.get_instance_from_cfg( head, MindFormerModuleType.HEAD) heads.append(head_op) return heads return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-test/mindformers/models/build_processor.py000066400000000000000000000046351471400134000243010ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Processor API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .build_tokenizer import build_tokenizer def build_processor( config: dict = None, default_args: dict = None, module_type: str = 'processor', class_name: str = None, **kwargs): r"""Build processor For MindFormer. Instantiate the processor from MindFormerRegister's registry. Args: config (dict): The task processor's config. Default: None. default_args (dict): The default argument of processor API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'processor'. class_name (str): The class name of processor API. Default: None. Return: The function instance of processor API. Examples: >>> from mindformers import build_processor >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> processor_from_config = build_processor(config.processor) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.image_processor is not None: config.image_processor = build_processor(config.image_processor) if config.tokenizer is not None: config.tokenizer = build_tokenizer(config.tokenizer, **kwargs) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.PROCESSOR, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-test/mindformers/models/build_tokenizer.py000066400000000000000000000070201471400134000242630ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Tokenizer API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from ..mindformer_book import MindFormerBook def check_and_add_vocab_file_path(config, **kwargs): """And the vocab file path to the config if there is not vocab file in the config""" if 'vocab_file' in config: return class_name = config['type'] dynamic_class = MindFormerRegister.get_cls(module_type='tokenizer', class_name=class_name) # If the tokenizer does not require the vocab_file, just stop if not dynamic_class.vocab_files_names: return name_or_path = class_name path = kwargs.pop('lib_path', None) remote_tokenizer_support_list = MindFormerBook.get_tokenizer_url_support_list().keys() if name_or_path not in remote_tokenizer_support_list and path: read_vocab_file_dict, read_tokenizer_file_dict = \ dynamic_class.read_files_according_specific_by_tokenizer(name_or_path=path) config.update(read_vocab_file_dict) config.update(read_tokenizer_file_dict) else: checkpoint_name_or_path = config.get("checkpoint_name_or_path") if checkpoint_name_or_path in dynamic_class.get_support_list(): support_name = checkpoint_name_or_path else: support_name = dynamic_class.get_support_list()[0] vocab_file = dynamic_class.cache_vocab_files(name_or_path=support_name) config.update(vocab_file) def build_tokenizer( config: dict = None, default_args: dict = None, module_type: str = 'tokenizer', class_name: str = None, **kwargs): r"""Build tokenizer For MindFormer. Instantiate the tokenizer from MindFormerRegister's registry. Args: config (dict): The task tokenizer's config. Default: None. default_args (dict): The default argument of tokenizer API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'tokenizer'. class_name (str): The class name of tokenizer API. Default: None. Return: The function instance of tokenizer API. Examples: >>> from mindformers import build_tokenizer >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> tokenizer_from_config = build_tokenizer(config.processor.tokenizer) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) check_and_add_vocab_file_path(config, **kwargs) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.TOKENIZER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-test/mindformers/models/clip/000077500000000000000000000000001471400134000214505ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/clip/__init__.py000066400000000000000000000020061471400134000235570ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """clip init""" from .clip_config import CLIPTextConfig, CLIPVisionConfig, CLIPConfig from .clip import CLIPModel from .clip_tokenizer import CLIPTokenizer from .clip_processor import CLIPProcessor, CLIPImageProcessor __all__ = ['CLIPModel', 'CLIPConfig', 'CLIPVisionConfig', 'CLIPImageProcessor', 'CLIPTextConfig', 'CLIPTokenizer', 'CLIPProcessor'] pengcheng.mind-test/mindformers/models/clip/clip.py000066400000000000000000000261311471400134000227540ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ CLIPModel """ from typing import Optional, Union import numpy as np import mindspore as ms from mindspore import nn from mindspore.ops import functional as F from mindspore.common.initializer import Normal, initializer from mindspore import Parameter, Tensor import mindspore.ops as ops from mindformers.version_control import get_norm from ...mindformer_book import MindFormerBook from ..base_model import BaseModel from .clip_modules import VisionTransformer, Transformer, LayerNorm from .clip_config import CLIPConfig from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.MODELS) class CLIPModel(BaseModel): r"""CLIPModel. The supported model name could be selected from CLIPModel.show_support_list(). Args: config (CLIPConfig): The config of clip model, which could be obtained by CLIPConfig class. Examples: >>> from mindformers import CLIPModel >>> CLIPModel.show_support_list() INFO - support list of CLIPModel is: INFO - ['clip_vit_b_32'] INFO - ------------------------------------- >>> model = CLIPModel.from_pretrained('clip_vit_b_32') >>> type(model) """ _support_list = MindFormerBook.get_model_support_list()['clip'] def __init__(self, config: CLIPConfig): super(CLIPModel, self).__init__(config) self.dtype = self.get_dtype(config.dtype) self.cross_entropy = nn.SoftmaxCrossEntropyWithLogits(reduction="mean", sparse=True) self.max_position_embeddings = config.text_config.max_position_embeddings self.visual = VisionTransformer( input_resolution=config.vision_config.image_size, patch_size=config.vision_config.patch_size, width=config.vision_config.hidden_size, layers=config.vision_config.num_hidden_layers, heads=config.vision_config.num_attention_heads, output_dim=config.projection_dim, dtype=self.dtype ) self.transformer = Transformer( width=config.text_config.hidden_size, layers=config.text_config.num_hidden_layers, heads=config.text_config.num_attention_heads, dtype=self.dtype, attn_mask=self.build_attention_mask() ) self.token_embedding = \ nn.Embedding(config.text_config.vocab_size, config.text_config.hidden_size, embedding_table=Normal(mean=0.0, sigma=0.02)) self.positional_embedding = Parameter(initializer( Normal(mean=0.0, sigma=0.01), [config.text_config.max_position_embeddings, config.text_config.hidden_size])) self.ln_final = LayerNorm([config.text_config.hidden_size]) self.text_projection = Parameter(initializer( Normal(mean=0.0, sigma=config.text_config.hidden_size ** -0.5), [config.text_config.hidden_size, config.projection_dim], ms.float32)) self.logit_scale = Parameter(Tensor(np.log(1 / 0.07)).astype(ms.float32)) self.exp = ops.Exp() self.norm = get_norm() self.load_checkpoint(config) def get_dtype(self, dtype: str): """Get_dtype""" if dtype == "float16": return ms.float16 if dtype == "float32": return ms.float32 raise TypeError("unsupported data type.") def construct(self, image: ms.Tensor, text: ms.Tensor, label: Optional[Union[ms.Tensor, np.ndarray]] = None, input_ids: Optional[ms.Tensor] = None, pixel_values: Optional[ms.Tensor] = None): r"""Construct Args: image (Tensor): A image tensor processed by image_processor. text (Tensor): A text id tensor processed by tokenizer. input_ids (Optional[ms.Tensor]): Equal to "text", if "input_ids" is set, "text" is useless. pixel_values (Optional[ms.Tensor]): Equal to "image", if "pixel_values" is set, "image" is useless. label (Optional[Union[ms.Tensor, np.ndarray]]): The classification label. Returns: if not self.trainining: if label is None: logits_per_image: Similarity between image and text. logits_per_text: Similarity between text and image. else: logits_per_image: Similarity between image and text. label: The classification label. else: loss: Constructive language image pretraining loss. Examples: >>> import numpy as np >>> from mindformers import CLIPModel, CLIPProcessor >>> processor = CLIPProcessor.from_pretrained('clip_vit_b_32') >>> model = CLIPModel.from_pretrained('clip_vit_b_32') >>> fake_image_batch = np.random.random((5, 3, 578, 213)) >>> fake_text_batch = ["a boy", "a girl", "a women", "a men"] >>> model(**processor(fake_image_batch, fake_text_batch)) (Tensor(shape=[5, 4], dtype=Float32, value= [[2.26097965e+001, 2.29247952e+001, 2.40179482e+001, 2.30396290e+001], [2.26102257e+001, 2.29256859e+001, 2.40180817e+001, 2.30393028e+001], [2.26097965e+001, 2.29247952e+001, 2.40179482e+001, 2.30396290e+001], [2.26109924e+001, 2.29261818e+001, 2.40193062e+001, 2.30404854e+001], [2.26097965e+001, 2.29247952e+001, 2.40179482e+001, 2.30396290e+001]]), Tensor(shape=[4, 5], dtype=Float32, value= ...)) """ if pixel_values is not None: image = pixel_values if input_ids is not None: text = input_ids if len(text.shape) == 3: text = text[0].squeeze() image_features = self.get_image_features(image) text_features = self. get_text_features(text) image_features = image_features / self.norm(image_features, dim=1, keepdim=True) text_features = text_features / self.norm(text_features, dim=1, keepdim=True) logit_scale = self.exp(self.logit_scale) if not self.training: if label is None: logits_per_image = ops.matmul(logit_scale * image_features, text_features.T) logits_per_text = logits_per_image.T return logits_per_image, logits_per_text logits_per_image = ops.matmul(logit_scale * image_features, text_features.T) return logits_per_image, label logits = ops.matmul(logit_scale * image_features, text_features.T) batch_size, _ = F.shape(logits) labels = ms.Tensor(np.arange(batch_size)) images_loss = self.cross_entropy(logits, labels) texts_loss = self.cross_entropy(logits.T, labels) loss = (images_loss + texts_loss) / 2 return loss def build_attention_mask(self): """Build_attention_mask""" mask = np.ones((self.max_position_embeddings, self.max_position_embeddings)) mask = np.triu(mask * float("-inf"), k=1) return Tensor(mask).astype(self.dtype) def get_image_features(self, image: ms.Tensor, pixel_values: Optional[ms.Tensor] = None): r"""Get_image_features Args: image (ms.Tensor): A image tensor processed by image_processor. pixel_values (Optional[ms.Tensor]): Equal to "image", if "pixel_values" is set, "image" is useless. Returns: Image feature. Examples: >>> import numpy as np >>> from mindformers import CLIPModel, CLIPProcessor >>> processor = CLIPProcessor.from_pretrained('clip_vit_b_32') >>> model = CLIPModel.from_pretrained('clip_vit_b_32') >>> fake_image_batch = np.random.random((5, 3, 578, 213)) >>> model.get_image_features(processor.image_processor(fake_image_batch)) Tensor(shape=[5, 512], dtype=Float32, value= [[-1.50102973e-001, -2.63687313e-001, -5.65953791e-001 ... -2.93511450e-001], [-1.50103331e-001, -2.63622820e-001, -5.65623760e-001 ... -2.93337226e-001], [-1.50102973e-001, -2.63687313e-001, -5.65953791e-001 ... -2.93511450e-001], [-1.49712294e-001, -2.64100820e-001, -5.65740824e-001 ... -2.93599486e-001], [-1.50102973e-001, -2.63687313e-001, -5.65953791e-001 ... -2.93511450e-001]]) """ if pixel_values is not None: image = pixel_values image = image.astype(self.dtype) return self.visual(image) def get_text_features(self, text: ms.Tensor, input_ids: Optional[ms.Tensor] = None): r"""Get_text_features Args: text (ms.Tensor): A text id tensor processed by tokenizer. input_ids (Optional[ms.Tensor]): Equal to "text", if "input_ids" is set, "text" is useless. Returns: Text feature. Examples: >>> from mindformers import CLIPModel, CLIPProcessor >>> processor = CLIPProcessor.from_pretrained('clip_vit_b_32') >>> model = CLIPModel.from_pretrained('clip_vit_b_32') >>> fake_text_batch = ["a boy", "a girl", "a women", "a men"] >>> text = processor.tokenizer( ... fake_text_batch, max_length=77, padding="max_length", return_tensors="ms" ... )["input_ids"] >>> model.get_text_features(text) Tensor(shape=[4, 512], dtype=Float32, value= [[6.03631809e-002, 1.79528534e-001, ... -2.23753393e-001, 1.42413378e-002], [1.28974199e-001, 7.46373609e-002, ... -3.68579805e-001, 1.53980583e-001], [9.89909172e-002, 2.01410800e-002, ... -2.54495114e-001, 7.68117979e-002], [3.16975415e-002, 2.26992741e-001, ... -5.22942394e-002, 1.98922127e-001]]) """ if input_ids is not None: text = input_ids text_ = self.token_embedding(text).astype(self.dtype) text_ = ops.Add()(text_, self.positional_embedding).astype(self.dtype) text_ = text_.transpose(1, 0, 2) text_ = self.transformer(text_) text_ = text_.transpose(1, 0, 2) text_ = self.ln_final(text_).astype(ms.float32) text_ = ops.MatMul()( text_[ms.numpy.arange(text_.shape[0]), text.argmax(-1)], self.text_projection) return text_ pengcheng.mind-test/mindformers/models/clip/clip_config.py000066400000000000000000000246571471400134000243140ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ CLIPConfig class, which consists of CLIPTextConfig and CLIPVisionConfig All configs here are inherited from BaseConfig """ from typing import Optional from ..base_config import BaseConfig from ...tools.register import MindFormerRegister, MindFormerModuleType from ...mindformer_book import MindFormerBook from ...tools import logger @MindFormerRegister.register(MindFormerModuleType.CONFIG) class CLIPTextConfig(BaseConfig): r""" Config For CLIP Text Module Args: vocab_size (Optional[int]): Vocabulary size of the CLIP text model. hidden_size (Optional[int]): The dims of text features. intermediate_size (Optional[int]): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. num_hidden_layers (Optional[int]): The number of transformer layers in text encoder. num_attention_heads (Optional[int]): Number of attention heads for each attention layer in the Transformer encoder. max_position_embeddings (Optional[int]): The maximum sequence length that this model might ever be used with. hidden_act (Optional[str]): The non-linear activation function (function or string) in the encoder and pooler. Only "quick_gelu" supported currently. attention_dropout (Optional[float]): The dropout ratio for the attention probabilities. dropout (Optional[float]): The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler. initializer_range (Optional[float]): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_factor (Optional[float]): A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). Examples: >>> from mindformers import CLIPTextConfig >>> CLIPTextConfig(hidden_size=256, vocab_size=40000) {'vocab_size': 40000, 'hidden_size': 256, 'intermediate_size': 2048, 'num_hidden_layers': 12, 'num_attention_heads': 8, 'max_position_embeddings': 77, 'hidden_act': 'quick_gelu', 'attention_dropout': 0.0, 'initializer_range': 0.02, 'initializer_factor': 1.0} """ def __init__(self, vocab_size: Optional[int] = 49408, hidden_size: Optional[int] = 512, intermediate_size: Optional[int] = 2048, num_hidden_layers: Optional[int] = 12, num_attention_heads: Optional[int] = 8, max_position_embeddings: Optional[int] = 77, hidden_act: Optional[str] = "quick_gelu", attention_dropout: Optional[float] = 0.0, drop_out: Optional[float] = 0.0, initializer_range: Optional[float] = 0.02, initializer_factor: Optional[float] = 1.0, **kwargs): super(CLIPTextConfig, self).__init__(**kwargs) self.vocab_size = vocab_size self.hidden_size = hidden_size self.intermediate_size = intermediate_size self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.max_position_embeddings = max_position_embeddings self.hidden_act = hidden_act self.attention_dropout = attention_dropout self.dropout = drop_out self.initializer_range = initializer_range self.initializer_factor = initializer_factor @MindFormerRegister.register(MindFormerModuleType.CONFIG) class CLIPVisionConfig(BaseConfig): r""" Config For CLIP Vision Module Args: hidden_size (Optional[int]): Dimensionality of the encoder layers and the pooler layer. intermediate_size (Optional[int]): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. num_hidden_layers (Optional[int]): Number of hidden layers in the Transformer encoder. num_attention_heads (Optional[int]): Number of attention heads for each attention layer in the Transformer encoder. image_size (Optional[int]): The size (resolution) of each image. patch_size (Optional[int]): The size (resolution) of each patch. hidden_act (Optional[str]): The non-linear activation function (function or string) in the encoder and pooler. Only "quick_gelu" supported currently. dropout (Optional[float]): The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler. attention_dropout (Optional[float]): The dropout ratio for the attention probabilities. initializer_range (Optional[float]): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_factor (Optional[float]): A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). Examples: >>> from mindformers import CLIPVisionConfig >>> CLIPVisionConfig(hidden_size=512, image_size=256) {'hidden_size': 512, 'intermediate_size': 3072, 'num_hidden_layers': 12, 'num_attention_heads': 16, 'image_size': 256, 'patch_size': 32, 'hidden_act': 'quick_gelu', 'dropout': 0.0, 'attention_dropout': 0.0, 'initializer_range': 0.02, 'initializer_factor': 1.0} """ def __init__(self, hidden_size: Optional[int] = 768, intermediate_size: Optional[int] = 3072, num_hidden_layers: Optional[int] = 12, num_attention_heads: Optional[int] = 12, image_size: Optional[int] = 224, patch_size: Optional[int] = 32, hidden_act: Optional[str] = "quick_gelu", dropout: Optional[float] = 0.0, attention_dropout: Optional[float] = 0.0, initializer_range: Optional[float] = 0.02, initializer_factor: Optional[float] = 1.0, **kwargs): super(CLIPVisionConfig, self).__init__(**kwargs) self.hidden_size = hidden_size self.intermediate_size = intermediate_size self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.image_size = image_size self.patch_size = patch_size self.hidden_act = hidden_act self.dropout = dropout self.attention_dropout = attention_dropout self.initializer_range = initializer_range self.initializer_factor = initializer_factor @MindFormerRegister.register(MindFormerModuleType.CONFIG) class CLIPConfig(BaseConfig): r""" Config For CLIP Model Args: text_config (Optional[CLIPTextConfig]): The config of text transformer. vision_config (Optional[CLIPVisionConfig]): The config of vision transformer. projection_dim (Optional[int]): The dims of projected featrues. logit_scale_init_value (Optional[float]): The initial value of the *logit_scale* parameter. checkpoint_name_or_path (Optional[str]): The path of checkpoint(.ckpt) or a support model name in CLIPConfig.show_support_list() dtype (Optional[str]): The type of tensors in model, ["float16", "float32"]. Raises: TypeError: If the type of text_config is not CLIPTextConfig or the type of vision_config is not CLIPVisionConfig Examples: >>> from mindformers import CLIPConfig >>> CLIPConfig.show_support_list() INFO - support list of CLIPConfig is: INFO - ['clip_vit_b_32'] INFO - ------------------------------------- >>> config = CLIPConfig.from_pretrained('clip_vit_b_32') >>> config {'text_config': {'hidden_size': 512, 'vocab_size': 49408, 'max_position_embeddings': 77, 'num_hidden_layers': 12}, 'vision_config': {'hidden_size': 768, 'image_size': 224, 'patch_size': 32, 'num_hidden_layers': 12}, 'projection_dim': 512, 'ratio': 64, 'checkpoint_name_or_path': 'clip_vit_b_32', 'dtype': 'float16'} >>> config.save_pretrained(save_directory="./", save_name="clip_config") INFO - config saved successfully! """ _support_list = MindFormerBook.get_config_support_list()['clip'] def __init__(self, text_config: Optional[CLIPTextConfig] = None, vision_config: Optional[CLIPVisionConfig] = None, projection_dim: Optional[int] = 512, logit_scale_init_value: Optional[float] = 2.6592, checkpoint_name_or_path: Optional[str] = "", dtype: Optional[str] = "float16", **kwargs): if text_config is None: text_config = CLIPTextConfig() logger.info("text_config is None. Initializing the CLIPTextConfig with default values.") elif isinstance(text_config, CLIPTextConfig): pass else: raise TypeError(f"text_config should be a " f"CLIpTextConfig class, but got {type(CLIPTextConfig)}") if vision_config is None: vision_config = CLIPVisionConfig() logger.info("vision_config is None." " Initializing the CLIPTextConfig with default values.") elif isinstance(vision_config, CLIPVisionConfig): pass else: raise TypeError("text_config should be a CLIPVisionConfig" f" class, but got {type(CLIPVisionConfig)}") super(CLIPConfig, self).__init__(**kwargs) self.text_config = text_config self.vision_config = vision_config self.projection_dim = projection_dim self.logit_scale_init_value = logit_scale_init_value self.checkpoint_name_or_path = checkpoint_name_or_path self.dtype = dtype pengcheng.mind-test/mindformers/models/clip/clip_modules.py000066400000000000000000000236131471400134000245060ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/openai/CLIP/blob/main/clip/clip.py # ============================================================================ """ Modules of CLIPModel, including MultiheadAttention，VisionTransformer, QuickGELU, ResidualAttentionBlock, Transformer """ from collections import OrderedDict from typing import Optional import numpy as np import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops from mindspore import dtype as mstype from mindspore.ops import operations as P from mindspore import Parameter, Tensor from mindspore.common.initializer import Normal class LayerNorm(nn.LayerNorm): r"""Implementation That Supports Fp16 Inputs But Fp32 Gains Biases. Args: x (ms.Tensor): Input tensor. The detailed function could refer to mindspore.nn.LayerNorm. Return: y (ms.Tensor): Normalized tensor. """ def construct(self, x: ms.Tensor): """construct""" y = super().construct(P.Cast()(x, ms.float32)) y = P.Cast()(y, x.dtype) return y class MultiheadAttention(nn.Cell): r"""MultiheadAttention, With Layers As Input For Initialization Args: d_model (int): The feature dimension n_head (int): The number of attention heads layers (int): The number of transformers, used for weight initialization dtype (mstype): The type of calculation, [mstype.float32, mstype.float16]. """ def __init__(self, d_model: int, n_head: int, layers: int, dtype: mstype): super(MultiheadAttention, self).__init__() self.num_heads = n_head self.head_dim = d_model // n_head self.softmax = nn.Softmax(-1) self.transpose = ops.Transpose() self.scaling = self.head_dim ** -0.5 proj_std = (d_model ** -0.5) * ((2 * layers) ** -0.5) attn_std = d_model ** -0.5 self.out_proj = nn.Dense(d_model, d_model, weight_init=Normal(mean=0.0, sigma=proj_std)).to_float(dtype) self.in_proj = nn.Dense(d_model, 3 * d_model, weight_init=Normal(mean=0.0, sigma=attn_std)).to_float(dtype) def construct(self, query: ms.Tensor, attn_mask: Optional[ms.Tensor] = None): r"""Construct Args: query (ms.Tensor): query of attention. attn_mask (Optional[ms.Tensor]): attention mask. Returns: attn_output (ms.Tensor): attention output. """ len_tgt, batch_size, width = query.shape qkv = self.in_proj(query).view(len_tgt, batch_size, 3, width).transpose((2, 0, 1, 3)) att_q = qkv[0:1] att_q = ops.Squeeze(0)(att_q) att_q = att_q * self.scaling att_q = att_q.view(len_tgt, batch_size * self.num_heads, self.head_dim).transpose((1, 0, 2)) att_k = qkv[1:2] att_k = ops.Squeeze(0)(att_k) att_k = att_k.view(-1, batch_size * self.num_heads, self.head_dim).transpose((1, 0, 2)) att_v = qkv[2:3] att_v = ops.Squeeze(0)(att_v) att_v = att_v.view(-1, batch_size * self.num_heads, self.head_dim).transpose((1, 0, 2)) if attn_mask is not None: attn_output_weights = attn_mask + ops.matmul(att_q, att_k.transpose((0, 2, 1))) else: attn_output_weights = ops.matmul(att_q, att_k.transpose((0, 2, 1))) attn_output_weights = self.softmax(attn_output_weights) attn_output = ops.matmul(attn_output_weights, att_v) attn_output = self.transpose(attn_output, (1, 0, 2)) attn_output = attn_output.view(len_tgt, batch_size, width) attn_output = self.out_proj(attn_output) return attn_output class VisionTransformer(nn.Cell): r"""VisionTransformer Of CLIPModel Args: input_resolution (int): The image size of input. patch_size (int): The patch size of vision transformer. width (int): The dimension of vision transformer. layers (int): The number of layers of vision transformer. heads (int): The number of attention heads. output_dim (int): The output dimension of vision transformer. dtype (mstype): The type of calculation, [mstype.float32, mstype.float16]. """ def __init__(self, input_resolution: int, patch_size: int, width: int, layers: int, heads: int, output_dim: int, dtype: mstype): super(VisionTransformer, self).__init__() self.conv1 = \ nn.Conv2d( in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, has_bias=False).to_float(dtype) scale = width ** -0.5 self.class_embedding = \ Parameter(scale * Tensor(np.random.normal(0, 1, size=(width))).astype(ms.float32)) self.positional_embedding = \ Parameter(scale * Tensor( np.random.normal(0, 1, size=( (input_resolution // patch_size) ** 2 + 1, width))).astype(ms.float32)) self.ln_pre = LayerNorm([width], epsilon=1e-5) self.transformer = Transformer(width, layers, heads, dtype) self.ln_post = LayerNorm([width], epsilon=1e-5) self.proj = \ Parameter(scale * Tensor(np.random.normal(0, 1, size=(width, output_dim))).astype(ms.float32)) self.cat = ops.Concat(1) self.tile = ops.Tile() self.slice = P.StridedSlice() self.dtype = dtype def construct(self, input_x: ms.Tensor): r"""Construct Args: input_x (ms.Tensor): Input tensor. Returns: input_x (ms.Tensor): Output tensor. """ input_x = self.conv1(input_x) input_x = input_x.reshape(input_x.shape[0], input_x.shape[1], -1) input_x = input_x.transpose(0, 2, 1) class_embedding = self.tile(self.class_embedding, (input_x.shape[0]+1, 1, 1)).astype(self.dtype) input_x = self.cat([ self.slice(class_embedding, (0, 0, 0), (-1, class_embedding.shape[1], class_embedding.shape[2]), (1, 1, 1)), input_x ]) input_x = ops.Add()(input_x, self.positional_embedding) input_x = self.ln_pre(input_x) input_x = input_x.transpose(1, 0, 2) input_x = self.transformer(input_x) input_x = input_x.transpose(1, 0, 2) input_x = self.ln_post(input_x[:, 0, :]) input_x = ops.matmul(input_x, self.proj) return input_x class QuickGELU(nn.Cell): r"""QuickGELU of CLIP""" def __init__(self, ratio: Optional[int] = 1.702): super(QuickGELU, self).__init__() self.ratio = ratio self.sigmoid = nn.Sigmoid() def construct(self, input_x: ms.Tensor): '''construct''' return input_x * self.sigmoid(self.ratio * input_x) class ResidualAttentionBlock(nn.Cell): r""" ResidualAttentionBlock of CLIP Args: d_model (int): The dimension of features. n_head (int): The number of attention heads. layers (int): The number of transformer layers for weight initialization. dtype (mstype): The type of calculation, [mstype.float32, mstype.float16]. attn_mask (Optional[ms.Tensor]): attention mask. """ def __init__(self, d_model: int, n_head: int, layers: int, dtype: mstype, attn_mask: Optional[ms.Tensor] = None): super(ResidualAttentionBlock, self).__init__() proj_std = (d_model ** -0.5) * ((2 * layers) ** -0.5) fc_std = (2 * d_model) ** -0.5 self.dtype = dtype self.attn = MultiheadAttention(d_model, n_head, layers, dtype) self.ln_1 = LayerNorm([d_model], epsilon=1e-5) self.mlp = nn.SequentialCell(OrderedDict([ ("c_fc", nn.Dense(d_model, d_model * 4, weight_init=Normal(mean=0.0, sigma=fc_std)).to_float(dtype)), ("gelu", QuickGELU()), ("c_proj", nn.Dense(d_model * 4, d_model, weight_init=Normal(mean=0.0, sigma=proj_std)).to_float(dtype)) ])) self.ln_2 = LayerNorm([d_model], epsilon=1e-5) self.attn_mask = attn_mask def construct(self, input_x: ms.Tensor): r"""Construct""" input_x = ops.Add()(input_x, self.attention(self.ln_1(input_x))) input_x = ops.Add()(input_x, self.mlp(self.ln_2(input_x))) return input_x def attention(self, input_x: ms.Tensor): r"""Attention""" return self.attn(input_x, self.attn_mask) class Transformer(nn.Cell): r""" Text Transformer of CLIP Args: width (int): The dimension of input features. layers (int): The number of transformer layers. heads (int): The number of attention heads. attn_mask (ms.Tensor): Attention mask. dtype (mstype): The type of calculation, [mstype.float32, mstype.float16]. """ def __init__(self, width, layers, heads, dtype, attn_mask=None): super(Transformer, self).__init__() self.width = width self.layers = layers self.resblocks = nn.SequentialCell( *[ResidualAttentionBlock(width, heads, layers, dtype, attn_mask) for _ in range(layers)] ) def construct(self, input_x): r"""Construct""" return self.resblocks(input_x) pengcheng.mind-test/mindformers/models/clip/clip_processor.py000066400000000000000000000154001471400134000250500ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ CLIPProcessor """ from typing import Optional, Union, List import numpy as np import PIL import mindspore as ms from mindformers.dataset import ( BCHW2BHWC, BatchResize, BatchToTensor, BatchNormalize, BatchCenterCrop, BatchPILize ) from mindformers.mindformer_book import MindFormerBook from ..base_processor import BaseImageProcessor from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class CLIPImageProcessor(BaseImageProcessor): """ CLIPImageProcessor. Args: image_resolution (int): The target size. Examples: >>> from mindformers import CLIPImageProcessor >>> from mindformers.tools.image_tools import load_image >>> processor = CLIPImageProcessor(image_resolution=256) >>> image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." ... "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> processor(image) Tensor(shape=[1, 3, 256, 256], dtype=Float32, value= [[[[-1.52949083e+000, -1.52949083e+000, ... -1.48569560e+000, -1.50029397e+000], [-1.52949083e+000, -1.52949083e+000, ... -1.50029397e+000, -1.50029397e+000], [-1.51489246e+000, -1.51489246e+000, ... -1.48569560e+000, -1.48569560e+000], ... ... [8.66091192e-001, 8.80311251e-001, ... -1.36645925e+000, -1.45177972e+000], [8.09210956e-001, 8.23431015e-001, ... -1.29535890e+000, -1.43755960e+000], [7.09670484e-001, 7.94990897e-001, ... -1.26691878e+000, -1.42333949e+000]]]]) """ def __init__(self, image_resolution: Optional[int] = 224): super(CLIPImageProcessor, self).__init__( image_resolution=image_resolution) self.bchw2bhwc = BCHW2BHWC() self.batch_pilizer = BatchPILize() self.batch_resizer = BatchResize(image_resolution) self.batch_crop = BatchCenterCrop(image_resolution) self.batch_totensor = BatchToTensor() self.batch_normalizer = BatchNormalize() def preprocess(self, images: Union[ms.Tensor, PIL.Image.Image, np.ndarray, List[PIL.Image.Image]], **kwargs): r""" Preprocess Required By Base Processor. Args: images (ms.Tensor, PIL.Image, numpy.array, List[PIL.Image]): A batch of images. Return: A 4-rank tensor for a batch of images. """ if not self._bhwc_check(images): images = self.bchw2bhwc(images) images = self.batch_pilizer(images) images = self.batch_resizer(images) images = self.batch_crop(images) images = self.batch_totensor(images) images = self.batch_normalizer(images) kwargs.pop("other", None) if isinstance(images, list): return ms.Tensor(np.row_stack([np.expand_dims(item, axis=0) for item in images])) if len(images.shape) == 4: return ms.Tensor(images) return ms.Tensor(np.expand_dims(images, axis=0)) def _bhwc_check(self, image_batch: Union[ms.Tensor, PIL.Image.Image, np.ndarray, List[PIL.Image.Image]]): r"""Bhwc_check""" if isinstance(image_batch, np.ndarray): if image_batch.shape[-1] == 3: return True if isinstance(image_batch, ms.Tensor): if image_batch.asnumpy().shape[-1] == 3: return True if isinstance(image_batch, (list, PIL.Image.Image)): return True return False @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class CLIPProcessor(BaseProcessor): r"""CLIP Processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. Args: image_processor (BaseImageProcessor): Used for process image data. tokenizer (BaseTokenizer): Used for process text data. max_length (Optional[int]): The length of text tokens. padding (Optional[str]): The padding strategy of tokenizer, [None, "max_length"]. return_tensors (Optional[str]): The type of returned tensors for tokenizer, [None, "ms"]. Examples: >>> from mindformers import CLIPProcessor >>> from mindformers.tools.image_tools import load_image >>> image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." ... "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> text = ["a boy", "a girl"] >>> CLIPProcessor.show_support_list() INFO - support list of CLIP Processor is: INFO - ['clip_vit_b_32'] INFO - ------------------------------------- >>> processor = CLIPProcessor.from_pretrained('clip_vit_b_32') >>> processor(image, text) {'image': Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[-1.52949083e+000, -1.52949083e+000,... -1.48569560e+000, -1.50029397e+000], [-1.52949083e+000, -1.52949083e+000, ... -1.50029397e+000, -1.50029397e+000], [-1.50029397e+000, -1.50029397e+000 ... -1.48569560e+000, -1.50029397e+000], ... [8.23431015e-001, 8.80311251e-001, ... -1.33801913e+000, -1.43755960e+000], [7.80770779e-001, 8.37651074e-001, ... -1.23847866e+000, -1.39489937e+000], [6.10130012e-001, 7.66550720e-001, ... -1.19581854e+000, -1.38067937e+000]]]]), 'text': Tensor(shape=[2, 77], dtype=Int32, value= [[49406, 320, 1876 ... 0, 0, 0], [49406, 320, 1611 ... 0, 0, 0]])} """ _support_list = MindFormerBook.get_processor_support_list()['clip'] def __init__(self, image_processor, tokenizer, max_length=77, padding='max_length', return_tensors='ms'): super(CLIPProcessor, self).__init__( image_processor=image_processor, tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors) pengcheng.mind-test/mindformers/models/clip/clip_tokenizer.py000066400000000000000000000275051471400134000250540ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ This is a temporary version of clip tokenizer """ import gzip import html import os import shutil from functools import lru_cache from typing import List, Optional import ftfy import regex as re from mindformers.tools import logger from mindformers.tools.utils import try_sync_file from ...mindformer_book import MindFormerBook from ...tools.register import MindFormerRegister, MindFormerModuleType from ...tools.download_tools import download_with_progress_bar from ..base_tokenizer import Tokenizer __all__ = ['CLIPTokenizer'] VOCAB_FILES_NAMES = {'vocab_file': ['vocab.txt', 'bpe_simple_vocab_16e6.txt.gz']} @lru_cache() def default_bpe(): r"""Bpe path""" path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip', "bpe_simple_vocab_16e6.txt.gz") if not os.path.exists(path): url = "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" \ "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" download_with_progress_bar(url, path) try_sync_file(path) return path def get_pairs(input_wd): r"""Get_pairs""" output = set() prev_char = input_wd[0] for char in input_wd[1:]: output.add((prev_char, char)) prev_char = char return output @lru_cache() def bytes_to_unicode(): r"""Bytes_to_unicode""" input_bt = list(range(ord("!"), ord("~")+1))\ +list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1)) output_cd = input_bt[:] num = 0 for item in range(2**8): if item not in input_bt: input_bt.append(item) output_cd.append(2**8+num) num += 1 output_cd = [chr(item) for item in output_cd] return dict(zip(input_bt, output_cd)) def whitespace_clean(input_text): r"""Whitespace clean""" input_text = re.sub(r'\s+', ' ', input_text) input_text = input_text.strip() return input_text def basic_clean(input_text): r"""Basic_clean""" input_text = ftfy.fix_text(input_text) input_text = html.unescape(html.unescape(input_text)) return input_text.strip() class TempTokenizer: r"""Simple Tokenizer""" def __init__(self, merges, vocab, flag_dict): self.byte_encoder = bytes_to_unicode() self.byte_decoder = {v: k for k, v in self.byte_encoder.items()} self.bpe_ranks = dict(zip(merges, range(len(merges)))) self.flag_dict = flag_dict self.encoder = dict(zip(vocab, range(len(vocab)))) self.decoder = {v: k for k, v in self.encoder.items()} def tokenize_alg(self, input_tk): r"""Bpe""" if input_tk in self.flag_dict: return self.flag_dict[input_tk] word = tuple(input_tk[:-1]) + (input_tk[-1] + '',) pairs = get_pairs(word) if not pairs: return input_tk+'' while True: bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float('inf'))) if bigram not in self.bpe_ranks: break first, second = bigram new_word = [] i = 0 while i < len(word): try: j = word.index(first, i) new_word.extend(word[i:j]) i = j except ValueError: new_word.extend(word[i:]) break if word[i] == first and i < len(word)-1 and word[i+1] == second: new_word.append(first+second) i += 2 else: new_word.append(word[i]) i += 1 new_word = tuple(new_word) word = new_word if len(word) == 1: break pairs = get_pairs(word) word = ' '.join(word) self.flag_dict[input_tk] = word return word def decode(self, input_ids): r"""Decode""" output_text = ''.join([self.decoder[input_id] for input_id in input_ids]) output_text = bytearray([self.byte_decoder[c] for c in output_text]).decode('utf-8', errors="replace").replace('', ' ') return output_text def encode(self, content): r"""Encode""" output_ids = [] content = whitespace_clean(basic_clean(content)).lower() for token in re.findall(self.pat, content): token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8')) output_ids.extend(self.encoder[bpe_token] for bpe_token in self.tokenize_alg(token).split(' ')) print("res is:", output_ids) return output_ids @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class CLIPTokenizer(Tokenizer): r""" CLIP Tokenizer Args: vocab_file(str): The vocabulary file path. eos_token(str): The token that represents the end-of-sentence. Default "<|endoftext|>". bos_token(str): The token that represents the begin-of-sentence. Default "<|startoftext|>"". pad_token(str): The token that represents the pad. Default "<|endoftext|>". unk_token(str): The token that represents the unknown. Default "<|endoftext|>". add_prefix_space(bool): whether to add a whitespace in the front of text. Default "False" add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import CLIPTokenizer >>> CLIPTokenizer.show_support_list() INFO - support list of CLIPTokenizer is: INFO - ['clip_vit_b_32'] INFO - ------------------------------------- >>> tokenizer = CLIPTokenizer.from_pretrained('clip_vit_b_32') >>> tokenizer("a boy") {'input_ids': [49406, 320, 1876, 49407], 'attention_mask': [1, 1, 1, 1]} """ model_input_names = ["input_ids", "attention_mask"] vocab_files_names = VOCAB_FILES_NAMES FILE_LIST = ['tokenizer_config.json'] '''clip tokenizer''' _support_list = MindFormerBook.get_tokenizer_support_list()['clip'] def __init__(self, vocab_file, eos_token="<|endoftext|>", bos_token="<|startoftext|>", pad_token="<|endoftext|>", unk_token="<|endoftext|>", add_bos_token=True, add_eos_token=True ): super(CLIPTokenizer, self).__init__( eos_token=eos_token, bos_token=bos_token, pad_token=pad_token, unk_token=unk_token ) self.path = vocab_file self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token merges = self._read_merge_files(vocab_file) vocab = list(bytes_to_unicode().values()) vocab = vocab + [v + '' for v in vocab] for merge in merges: vocab.append(''.join(merge)) vocab.extend([bos_token, eos_token]) flag_dict = {bos_token: bos_token, eos_token: eos_token} self.tool = TempTokenizer(merges, vocab, flag_dict) self.pat = re.compile(r"""<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re| 've|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+""", re.IGNORECASE) @staticmethod def _read_merge_files(text_path, start_pos=1, end_pos=49152-256-2+1): r"""Read the merge files""" with gzip.open(text_path) as fp: data = fp.read() merges = data.decode("utf-8").split('\n') merges = merges[start_pos: end_pos] new_list = [] for item in merges: new_list.append(tuple(item.split())) return new_list def _tokenize(self, text, **kwargs): r"""Tokenize""" output_ids = [] content = whitespace_clean(basic_clean(text)).lower() for token in re.findall(self.pat, content): token = ''.join(self.tool.byte_encoder[b] for b in token.encode('utf-8')) output_ids.extend(self.tool.tokenize_alg(token).split(' ')) return output_ids def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): r""" Insert the special tokens to the input_ids. Currently, we support token_ids_0 is a list of ids. """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def save_vocabulary(self, save_directory, filename_prefix=None): r"""Save_vocabulary""" if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None output_file_path = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"][0]) shutil.copy(self.path, output_file_path) return output_file_path def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): r"""Tokenizer the input_text""" if not isinstance(text, str): raise ValueError("Text should be type str, but found type", type(text)) return self._tokenize(text) def _convert_token_to_id(self, token): r"""Convert_token_to_id""" return self.tool.encoder[token] def _convert_id_to_token(self, index): r"""Convert_id_to_token""" return self.tool.decoder[index] @property def vocab_size(self): r"""Get the vocab size""" return len(self.tool.encoder) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab pengcheng.mind-test/mindformers/models/clip/convert_weight.py000066400000000000000000000056551471400134000250640ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Convert checkpoint from torch/huggingface """ import argparse import torch from mindspore import Tensor, Parameter import mindspore as ms def convert_weight(torch_path="ViT-B-32.pt", mindspore_path="clip_vit_b_32.ckpt"): r"""Convert Weight Convert clip_vit_b_32 weights from pytorch to mindspore pytorch and GPU required. Args: torch_path: The path to ViT-B-32.pt. mindspore_path： The save path for clip_vit_b_32.ckpt. """ param_dict = torch.load(torch_path) new_dict = [] for name, param in param_dict.items(): if "ln_pre.weight" in name or "ln_1.weight" in name or\ "ln_2.weight" in name or "ln_post.weight" in name or\ "ln_final.weight" in name: new_name = name.replace('weight', 'gamma') elif "ln_pre.bias" in name or "ln_1.bias" in name or\ "ln_2.bias" in name or "ln_post.bias" in name or\ "ln_final.bias" in name: new_name = name.replace('bias', 'beta') elif "in_proj_weight" in name: new_name = name.replace('in_proj_weight', 'in_proj.weight') elif "in_proj_bias" in name: new_name = name.replace('in_proj_bias', 'in_proj.bias') elif "token_embedding.weight" in name: new_name = name.replace('token_embedding.weight', 'token_embedding.embedding_table') else: new_name = name new_dict.append({ "name": new_name, "data": Parameter(Tensor(param.detach().numpy()), name=new_name) }) ms.save_checkpoint(new_dict, mindspore_path) if __name__ == '__main__': parser = argparse.ArgumentParser(description="clip weight convert script") parser.add_argument("--torch_path", type=str, default="ViT-B-32.pt", required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="clip_vit_b_32.ckpt", help="The output mindspore checkpoint path.") opt = parser.parse_args() convert_weight(opt.torch_path, opt.mindspore_path) pengcheng.mind-test/mindformers/models/filip/000077500000000000000000000000001471400134000216245ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/filip/__init__.py000066400000000000000000000015461471400134000237430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """filip init""" from .filip_config import FilipTextConfig, FilipVisionConfig, FilipConfig from .filip import FilipModel __all__ = ['FilipModel', 'FilipConfig', 'FilipVisionConfig', 'FilipTextConfig'] pengcheng.mind-test/mindformers/models/filip/filip.py000066400000000000000000000344111471400134000233040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ FilipModel """ import math import os import numpy as np import mindspore as ms import mindspore.ops as ops from mindspore.ops import functional as F from mindspore import nn, Parameter, Tensor from mindspore.common import dtype as mstype from mindspore.communication.management import get_group_size, get_rank, create_group from mindspore.nn.wrap.grad_reducer import DistributedGradReducer from mindspore.ops.primitive import constexpr from ...mindformer_book import MindFormerBook from ..base_model import BaseModel from .filip_modules import VisualTransformer, TextTransformer from ...tools.register import MindFormerRegister, MindFormerModuleType @constexpr def int_num(num): return int(num) class SoftCrossEntropyLoss(nn.Cell): """ SoftCrossEntropyLoss. """ def __init__(self, smooth=0.1): super(SoftCrossEntropyLoss, self).__init__() self.smooth = smooth self.expand_op = ops.ExpandDims() self.concat_op = ops.Concat(1) self.ones = ops.Ones() self.on_value = Tensor(1.0, ms.float32) self.off_value = Tensor(0.0, ms.float32) self.logsoftmax = nn.LogSoftmax() self.onehot = ops.OneHot() def construct(self, logit, target): target_onehot = self.onehot(target, logit.shape[1], self.on_value, self.off_value) target_onehot = (1 - self.smooth) * target_onehot + self.smooth / logit.shape[1] log_probs = self.logsoftmax(logit) loss = (- target_onehot * log_probs).mean(0).sum() return loss class FilipImgProcess(nn.Cell): """ FilipImgProcess, norm the input image features. """ def __init__(self): super(FilipImgProcess, self).__init__() self.image_norm = nn.Norm(axis=-1, keep_dims=True) def construct(self, image_features): image_features = image_features[:, 1:, :] image_features = image_features / self.image_norm(image_features) return image_features class FilipTextProcess(nn.Cell): """ FilipTextProcess, norm the input text features and get text padding mask. """ def __init__(self): super(FilipTextProcess, self).__init__() self.text_norm = nn.Norm(axis=-1, keep_dims=True) self.expand_dims = ops.ExpandDims() def construct(self, text_features, text): text_features = text_features / self.text_norm(text_features) text_pad_mask = text > 0 text_pad_mask_expand = self.expand_dims(text_pad_mask, -1) text_features = text_features * text_pad_mask_expand return text_features, text_pad_mask class FilipTopKFeature(nn.Cell): """ FilipTopKFeature, used in filip loss. """ def __init__(self, top_token_percent, local_group_size): super(FilipTopKFeature, self).__init__() self.percent_i, self.percent_t = top_token_percent self.local_group_size = local_group_size self.rank_size = int(os.getenv("RANK_SIZE", '1')) if self.rank_size > 1: rank_id = get_rank() sub_group = rank_id // local_group_size group = "{}-{}".format(sub_group * local_group_size, (sub_group + 1) * local_group_size) rank_ids = list(range(sub_group * local_group_size, (sub_group + 1) * local_group_size)) create_group(group, rank_ids) self.local_gather = ops.AllGather(group) self.matmul = ops.MatMul(transpose_b=True) self.topk = ops.TopK(sorted=True) self.expand_dim = ops.ExpandDims() def construct(self, image_feature, text_feature, text_pad_mask): """construct of FilipTopKFeature""" if self.rank_size > 1: text_local_gather = self.local_gather(text_feature) image_local_gather = self.local_gather(image_feature) else: text_local_gather = text_feature image_local_gather = image_feature # get image features batch_size, n_token1, feat_dim = image_feature.shape output = self.matmul(image_feature.reshape((-1, feat_dim)), text_local_gather.reshape((-1, feat_dim))) token_rep = output.reshape((batch_size, n_token1, -1)).max(2) max_token_idx = self.topk(token_rep, int_num(n_token1 * self.percent_i))[1] max_token_idx = ops.stop_gradient(max_token_idx) bs_index = nn.Range(batch_size)() bs_index = self.expand_dim(bs_index, 1) image_feature = image_feature[bs_index, max_token_idx] # get text features batch_size, n_token1, feat_dim = text_feature.shape output = self.matmul(text_feature.reshape((-1, feat_dim)), image_local_gather.reshape((-1, feat_dim))) token_rep = output.reshape((batch_size, n_token1, -1)).max(2) max_token_idx = self.topk(token_rep, int_num(n_token1 * self.percent_i))[1] max_token_idx = ops.stop_gradient(max_token_idx) bs_index = nn.Range(batch_size)() bs_index = self.expand_dim(bs_index, 1) text_feature = text_feature[bs_index, max_token_idx] text_pad_mask = text_pad_mask[bs_index, max_token_idx] return image_feature, text_feature, text_pad_mask class FilipLogit(nn.Cell): """ FilipLogit, compute logit. """ def __init__(self, use_mask_flag=False): super(FilipLogit, self).__init__() self.use_mask_flag = use_mask_flag self.matmul = ops.MatMul(transpose_b=True) logit_value = math.exp(3.8665097) self.logit_scale = Parameter(Tensor(np.log(logit_value), dtype=ms.float32)) self.exp = ops.Exp() self.cast = ops.Cast() def construct(self, rep1, rep2, mask=None): """construct of FilipLogit""" logit_scale = self.exp(self.logit_scale) batch_size1, n_token1, feat_dim = rep1.shape _, n_token2, _ = rep2.shape rep1 = rep1.reshape((-1, feat_dim)) rep2 = rep2.reshape((-1, feat_dim)) out = self.matmul(rep1, rep2) out = out.reshape((batch_size1, n_token1, -1, n_token2)).max(3) if self.use_mask_flag: out = out.sum(1) mask = self.cast(mask, ms.float32) mask_sum = mask.sum(axis=1, keepdims=True).clip(min=1.0, max=None) logits = out / mask_sum else: logits = out.mean(1) logits = logit_scale * logits logits = logits.clip(min=-100, max=100) return logits class FilipGather(nn.Cell): """ FilipGather, input image features, text features, and text padding mask, return logits. """ def __init__(self, top_token_percent, local_group_size): super(FilipGather, self).__init__() self.top_k_feature = FilipTopKFeature(top_token_percent, local_group_size) self.get_img_logits = FilipLogit() self.get_txt_logits = FilipLogit(True) self.rank_size = int(os.getenv("RANK_SIZE", '1')) if self.rank_size > 1: self.all_gather = ops.AllGather() def construct(self, image_features, text_features, text_pad_mask): """ construct of FilipGather """ if self.rank_size > 1: image_gather = self.all_gather(image_features) text_gather = self.all_gather(text_features) else: image_gather = image_features text_gather = text_features logits_per_image = self.get_img_logits(image_features, text_gather) logits_per_text = self.get_txt_logits(text_features, image_gather, text_pad_mask) return logits_per_image, logits_per_text class FilipLossCell(nn.Cell): """ FilipLossCell. """ def __init__(self, label_smooth=0.1, local_group_size=1, top_token_percent=(1, 1), ): super(FilipLossCell, self).__init__() self.local_group_size = local_group_size self.top_token_percent = top_token_percent self.process_img_features = FilipImgProcess() self.process_text_features = FilipTextProcess() self.rank_size = int(os.getenv("RANK_SIZE", '1')) if self.rank_size > 1: self.rank = get_rank() else: self.rank = 0 self.two = Tensor(2.0, dtype=mstype.float32) # self.all_gather = ops.AllGather() self.equal = ops.Equal() self.cast = ops.Cast() self.logsoftmax = nn.LogSoftmax() self.loss = SoftCrossEntropyLoss(label_smooth) self.gather_and_compute = FilipGather(top_token_percent, local_group_size) def construct(self, image_features, text_features, text): """construct of FilipLossCell""" image_features = self.process_img_features(image_features) text_features, text_pad_mask = self.process_text_features(text_features, text) batch_size = image_features.shape[0] logits_per_image, logits_per_text = self.gather_and_compute( image_features, text_features, text_pad_mask ) target = nn.Range(self.rank * batch_size, (self.rank + 1) * batch_size)() target = F.cast(target, ms.int32) loss = (self.loss(logits_per_image, target) + self.loss(logits_per_text, target)) / self.two return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class FilipModel(BaseModel): """ FilipModel. The supported model name could be selected from FilipModel.show_support_list(). Args: config (FilipConfig): the config of filip model. """ _support_list = MindFormerBook.get_model_support_list()['filip'] def __init__(self, config, is_training=True): super(FilipModel, self).__init__(config) self.image_encoder = VisualTransformer(config=config) self.text_encoder = TextTransformer(config=config) self.image_norm = nn.Norm(axis=-1, keep_dims=True) self.text_norm = nn.Norm(axis=-1, keep_dims=True) self.is_training = is_training self.loss = FilipLossCell() self.cast = ops.Cast() self.load_checkpoint(config) def get_image_feature(self, image): image_features = self.image_encoder(image) return image_features def get_text_feature(self, text): text_features = self.text_encoder(text) return text_features def construct(self, image, text): image_features = self.image_encoder(image) text_features = self.text_encoder(text) if not self.is_training: return image_features, text_features total_loss = self.loss(image_features, text_features, text) total_loss = self.cast(total_loss, mstype.float32) return total_loss GRADIENT_CLIP_TYPE = 1 GRADIENT_CLIP_VALUE = 1.0 clip_grad = ops.MultitypeFuncGraph("clip_grad") @clip_grad.register("Number", "Number", "Tensor") def _clip_grad(clip_type, clip_value, grad): """ Clip gradients. Inputs: clip_type (int): The way to clip, 0 for 'value', 1 for 'norm'. clip_value (float): Specifies how much to clip. grad (tuple[Tensor]): Gradients. Outputs: tuple[Tensor], clipped gradients. """ if clip_type not in (0, 1): return grad dt = F.dtype(grad) if clip_type == 0: new_grad = ops.clip_by_value(grad, F.cast(F.tuple_to_array((-clip_value,)), dt), F.cast(F.tuple_to_array((clip_value,)), dt)) else: new_grad = nn.ClipByNorm()(grad, F.cast(F.tuple_to_array((clip_value,)), dt)) return new_grad grad_scale = ops.MultitypeFuncGraph("grad_scale") reciprocal = ops.Reciprocal() @grad_scale.register("Tensor", "Tensor") def tensor_grad_scale(scale, grad): """Construct the trainer of Bert.""" return grad * reciprocal(scale) _grad_overflow = ops.MultitypeFuncGraph("_grad_overflow") grad_overflow = ops.FloatStatus() @_grad_overflow.register("Tensor") def _tensor_grad_overflow(grad): """Construct the trainer of Bert.""" return grad_overflow(grad) @MindFormerRegister.register(MindFormerModuleType.WRAPPER, alias="FilipTrainOneStepWithLossScaleWrapper") class FilipTrainOneStepWithLossScaleCell(nn.TrainOneStepWithLossScaleCell): """ FilipTrainOneStepWithLossScaleCell wrapper """ def __init__(self, network, optimizer, scale_update_cell=None): super(FilipTrainOneStepWithLossScaleCell, self).__init__(network, optimizer, scale_update_cell) self.cast = ops.Cast() self.degree = 1 if self.reducer_flag: self.degree = get_group_size() self.grad_reducer = DistributedGradReducer(optimizer.parameters, False, self.degree) self.loss_scale = None self.loss_scaling_manager = scale_update_cell if scale_update_cell: self.loss_scale = Parameter(Tensor(scale_update_cell.get_loss_scale(), dtype=mstype.float32)) def construct(self, image, text, sens=None): """ construct of FilipTrainOneStepWithLossScaleCell """ weights = self.weights loss = self.network(image, text) if sens is None: scaling_sens = self.loss_scale else: scaling_sens = sens status, scaling_sens = self.start_overflow_check(loss, scaling_sens) grads = self.grad(self.network, weights)(image, text, self.cast(scaling_sens, mstype.float32)) # apply grad reducer on grads grads = self.grad_reducer(grads) grads = self.hyper_map(F.partial(grad_scale, scaling_sens * self.degree), grads) grads = self.hyper_map(F.partial(clip_grad, GRADIENT_CLIP_TYPE, GRADIENT_CLIP_VALUE), grads) cond = self.get_overflow_status(status, grads) overflow = cond if sens is None: overflow = self.loss_scaling_manager(self.loss_scale, cond) if not overflow: loss = F.depend(loss, self.optimizer(grads)) else: print(">>>>overflow") return loss, cond, scaling_sens pengcheng.mind-test/mindformers/models/filip/filip_config.py000066400000000000000000000075501471400134000246350ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ FilipConfig class, which consists FilipTextConfig and FilipVisionConfig All configs here are inherited from BaseConfig """ from ..base_config import BaseConfig from ...tools.register import MindFormerRegister, MindFormerModuleType from ...mindformer_book import MindFormerBook from ...tools import logger @MindFormerRegister.register(MindFormerModuleType.CONFIG) class FilipTextConfig(BaseConfig): """Config for filip text module""" def __init__(self, hidden_size=768, vocab_size=21128, max_position_embeddings=32, num_hidden_layers=12, **kwargs): super(FilipTextConfig, self).__init__(hidden_size=hidden_size, vocab_size=vocab_size, max_position_embeddings=max_position_embeddings, num_hidden_layers=num_hidden_layers, **kwargs) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class FilipVisionConfig(BaseConfig): """Config for filip vision module""" def __init__(self, hidden_size=1024, image_size=224, patch_size=14, num_hidden_layers=24, **kwargs): super(FilipVisionConfig, self).__init__(hidden_size=hidden_size, image_size=image_size, patch_size=patch_size, num_hidden_layers=num_hidden_layers, **kwargs) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class FilipConfig(BaseConfig): """Config for filip model""" _support_list = MindFormerBook.get_model_support_list()['filip'] def __init__(self, text_config=None, vision_config=None, projection_dim=256, ratio=64, checkpoint_name_or_path="", dtype="float16", **kwargs): if text_config is None: text_config = FilipTextConfig() logger.info("text_config is None. " "Initializing the FilipTextConfig with default values.") elif isinstance(text_config, FilipTextConfig): pass else: raise TypeError(f"text_config should be a " f"FiLipTextConfig class, but got {type(FilipTextConfig)}") if vision_config is None: vision_config = FilipVisionConfig() logger.info("vision_config is None. " "Initializing the FilipTextConfig with default values.") elif isinstance(vision_config, FilipVisionConfig): pass else: raise TypeError("text_config should be a FilipVisionConfig" f"class, but got {type(FilipVisionConfig)}") super(FilipConfig, self).__init__(text_config=text_config, vision_config=vision_config, projection_dim=projection_dim, ratio=ratio, checkpoint_name_or_path=checkpoint_name_or_path, dtype=dtype, **kwargs) pengcheng.mind-test/mindformers/models/filip/filip_modules.py000066400000000000000000000257211471400134000250400ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/openai/CLIP/blob/main/clip/clip.py # ============================================================================ """ Modules of FilipModel, including image encoder model, text encoder model, and token learner """ import numpy as np import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops from mindspore import Parameter, Tensor from mindspore.ops.primitive import constexpr from mindspore.common.initializer import TruncatedNormal, initializer from mindformers.modules.transformer import Transformer from mindformers.modules import TransformerOpParallelConfig, TransformerRecomputeConfig class QuickGELU(nn.Cell): """QuickGELU of Filip""" def __init__(self): super(QuickGELU, self).__init__() self.ratio = 1.702 self.sigmoid = nn.Sigmoid() def construct(self, x): return x * self.sigmoid(self.ratio * x) class VisualTransformer(nn.Cell): """VisualTransformer of Filip""" def __init__(self, config): super(VisualTransformer, self).__init__() self.batch_size = config.batch_size self.single_bs = self.batch_size == 1 self.input_resolution = config.vision_config.image_size self.patch_size = config.vision_config.patch_size self.width = config.vision_config.hidden_size self.layers = config.vision_config.num_hidden_layers self.num_tokens_side = self.input_resolution // self.patch_size self.output_dim = config.projection_dim self.heads = config.vision_config.num_heads if not self.heads: self.heads = self.width // config.ratio else: self.heads = int(self.heads) self.token_learner_flag = config.vision_config.token_learner self.output_dim = config.projection_dim self.conv1 = nn.Conv2d(3, self.width, self.patch_size, self.patch_size) scale = self.width ** -0.5 src_seq_length = (self.input_resolution // self.patch_size) ** 2 + 1 self.class_embedding = Parameter(scale * Tensor(np.random.normal(0, 1, size=(self.width)).astype(np.float32))) self.positional_embedding = Parameter(scale * Tensor( np.random.normal(size=(src_seq_length, self.width)).astype(np.float32))) self.ln_pre = nn.LayerNorm([self.width]) parallel_config = TransformerOpParallelConfig() if config.recompute: recompute_config = TransformerRecomputeConfig(recompute=False) parallel_config = TransformerOpParallelConfig(recompute=recompute_config) self.transformer = Transformer(hidden_size=self.width, batch_size=self.batch_size, num_heads=self.heads, ffn_hidden_size=self.width * 4, encoder_layers=self.layers, attention_dropout_rate=0.0, hidden_dropout_rate=0.0, decoder_layers=0, # hidden_act=QuickGELU, hidden_act='fast_gelu', layernorm_compute_type=ms.float16, softmax_compute_type=ms.float16, param_init_type=ms.float16, src_seq_length=src_seq_length, tgt_seq_length=src_seq_length, parallel_config=parallel_config) self.ln_post = nn.LayerNorm([self.width]) self.proj = Parameter(scale * Tensor(np.random.normal(0, 1, size=(self.width, self.output_dim)).astype(np.float32))) self.cat = ops.Concat(1) self.tile = ops.Tile() self.ones = ops.OnesLike() self.expand_dims = ops.ExpandDims() if self.token_learner_flag: self.token_learner = TokenLearner(in_channels=self.output_dim, num_tokens=config.vision_config.num_tokens, num_groups=config.vision_config.num_token_groups, dropout_rate=config.vision_config.token_learner_dropout) def construct(self, x): """construct of VisualTransformer""" x = self.conv1(x) x = x.reshape(x.shape[0], x.shape[1], -1) x = x.transpose(0, 2, 1) if self.single_bs: class_embedding = self.expand_dims(self.class_embedding, 0) class_embedding = self.expand_dims(class_embedding, 0) else: class_embedding = self.tile(self.class_embedding, (x.shape[0], 1, 1)) x = self.cat((class_embedding, x)) x = x + self.positional_embedding x = self.ln_pre(x) x_shape = x.shape mask = ops.fill(ops.DType()(x), (x_shape[0], x_shape[1], x_shape[1]), 1) x = self.transformer(x, mask)[0] x = self.ln_post(x) x = ops.matmul(x, self.proj) if self.token_learner_flag: cls_token, x = x[:, 0, :], x[:, 1:, :] x = x.reshape(-1, self.num_tokens_side, self.num_tokens_side, self.output_dim) x = self.token_learner(x) cls_token = self.expand_dims(cls_token, 1).astype(x.dtype) x = self.cat((cls_token, x)) return x @constexpr def build_attntion_mask(batch_size, context_length): mask = np.tril(np.full((batch_size, context_length, context_length), 1).astype(np.float32), 0) mask = Tensor(mask) return mask class TextTransformer(nn.Cell): """TextTransformer of Filip""" def __init__(self, config): super(TextTransformer, self).__init__() self.batch_size = config.batch_size self.context_length = config.text_config.max_position_embeddins self.vocab_size = config.text_config.vocab_size self.width = config.text_config.hidden_size self.heads = config.text_config.num_heads if not self.heads: self.heads = self.width // config.ratio else: self.heads = int(self.heads) self.layers = config.text_config.num_hidden_layers self.output_dim = config.projection_dim self.embedding_table = Parameter(initializer(TruncatedNormal(0.02), [self.vocab_size, self.width])) self.gather = ops.Gather() self.reshape = ops.Reshape() self.not_equal = ops.NotEqual() self.cast = ops.Cast() self.positional_embedding = Parameter(initializer(TruncatedNormal(0.01), [self.context_length, self.width])) self.ln_final = nn.LayerNorm([self.width]) self.text_projection = Parameter(Tensor(np.random.normal(0, self.width ** -0.5, size=( self.width, self.output_dim)).astype(np.float32))) parallel_config = TransformerOpParallelConfig() if config.recompute: recompute_config = TransformerRecomputeConfig(recompute=False) parallel_config = TransformerOpParallelConfig(recompute=recompute_config) self.transformer_layer = Transformer(hidden_size=self.width, batch_size=self.batch_size, num_heads=self.heads, ffn_hidden_size=self.width * 4, encoder_layers=self.layers, attention_dropout_rate=0.0, hidden_dropout_rate=0.0, decoder_layers=0, # hidden_act=QuickGELU, hidden_act='fast_gelu', layernorm_compute_type=ms.float16, softmax_compute_type=ms.float16, param_init_type=ms.float16, src_seq_length=self.context_length, tgt_seq_length=self.context_length, parallel_config=parallel_config) def construct(self, text): """construct of TextTransformer""" bsz, ctx_len = text.shape flatten_id = text.flatten() gather_result = self.gather(self.embedding_table, flatten_id, 0) x = self.reshape(gather_result, (bsz, ctx_len, -1)) x = x + self.positional_embedding mask_tensor = build_attntion_mask(bsz, self.context_length) x = self.transformer_layer(x, mask_tensor)[0] x = self.ln_final(x) x = ops.matmul(x, self.text_projection) return x class TokenLearner(nn.Cell): """TokenLearner of Filip""" def __init__(self, in_channels, num_tokens, num_groups=8, dropout_rate=0.): super(TokenLearner, self).__init__() self.in_channels = in_channels self.num_tokens = num_tokens self.num_groups = num_groups self.norm = nn.LayerNorm([self.in_channels]) self.attntion_maps = nn.SequentialCell([ nn.Conv2d(self.in_channels, self.in_channels, kernel_size=1, group=self.num_groups), nn.Conv2d(self.in_channels, self.num_tokens, kernel_size=1) ]) self.feat_conv = nn.Conv2d( self.in_channels, self.in_channels, 1, group=self.num_groups ) self.dropout = nn.Dropout(dropout_rate) self.softmax = nn.Softmax() self.matmul = ops.BatchMatMul() def construct(self, x): """construct of TokenLearner""" feature_shape = x.shape selected = x selected = self.norm(selected) selected = selected.transpose((0, 3, 1, 2)) selected = self.attntion_maps(selected) selected = selected.transpose((0, 2, 3, 1)) selected = selected.reshape((feature_shape[0], feature_shape[1] * feature_shape[2], -1)) selected = selected.transpose((0, 2, 1)) selected = self.softmax(selected) feat = x feat = feat.transpose((0, 3, 1, 2)) feat = self.feat_conv(feat) feat = feat.transpose((0, 2, 3, 1)) feat = feat.reshape((feature_shape[0], feature_shape[1] * feature_shape[2], -1)) outputs = self.matmul(selected, feat) outputs = self.dropout(outputs) return outputs pengcheng.mind-test/mindformers/models/glm/000077500000000000000000000000001471400134000213005ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/glm/__init__.py000066400000000000000000000021211471400134000234050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for glm""" from .chatglm_6b_tokenizer import ChatGLMTokenizer from .glm import (GLMChatModel, GLMChatModelWithLora, GLMForPreTraining, GLMForPreTrainingWithLora) from .glm_config import GLMConfig from .glm_processor import GLMProcessor __all__ = [] __all__.extend(chatglm_6b_tokenizer.__all__) __all__.extend(glm.__all__) __all__.extend(glm_config.__all__) __all__.extend(glm_processor.__all__) pengcheng.mind-test/mindformers/models/glm/attention.py000066400000000000000000000474331471400134000236720ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """attention modules.""" import numpy as np from mindspore import Tensor from mindspore import dtype as mstype from mindspore import nn, ops from mindspore.ops import functional as F from mindspore.ops import operations as P from mindformers.modules.transformer import OpParallelConfig from mindformers.modules.layers import Linear from mindformers.version_control import get_dropout default_dpmp_config = OpParallelConfig() def divide(numerator, denominator): """Ensure that numerator is divisible by the denominator and return the division value.""" assert numerator % denominator == 0, f'{numerator} is not divisible by {denominator}' return numerator // denominator class RotaryEmbedding(nn.Cell): """ Rotary embedding layer Args: dim (int): Hidden layer dimension number. base (int, optional): Rotary embedding base. Default: 10000. params_dtype (ms.dtype, optional): Default: mstype.float32. compute_dtype (ms.dtype, optional): Default: mstype.float16. max_seq_len (int): Max sequence length. parallel_config (optional): Operator parallel strategy. Default: `OpParallelConfig()`. """ def __init__(self, dim, base=10000, params_dtype=mstype.float32, compute_dtype=mstype.float16, max_seq_len=512, parallel_config=None): super(RotaryEmbedding, self).__init__() if not parallel_config: parallel_config = default_dpmp_config self.max_seq_len = max_seq_len self.params_dtype = params_dtype self.compute_dtype = compute_dtype inv_freq = 1. / (base ** (np.arange(0, dim, 2).astype(np.float32) / dim)) inv_freq = inv_freq.astype(np.float16) t_range = np.arange(max_seq_len, dtype=inv_freq.dtype) freqs = np.outer(t_range, inv_freq) emb = np.concatenate((freqs, freqs), axis=-1) self.emb = emb # cos_cached: [max_seq_len, 1, dim] self.cos_cached = np.expand_dims(np.cos(emb), 1) self.sin_cached = np.expand_dims(np.sin(emb), 1) self.cos_cached = Tensor(self.cos_cached, params_dtype) self.sin_cached = Tensor(self.sin_cached, params_dtype) self.stride_slice = ops.StridedSlice() self.mul = ops.Mul() self.add = ops.Add() self.squeeze = ops.Squeeze(axis=1).shard(((1, 1, 1),)) self.concat = P.Concat(axis=-1).shard(((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, 1, 1, 1))) def rotate_half(self, x): """rotate half""" x1 = x[:, :, :, : x.shape[-1] // 2] x2 = x[:, :, :, x.shape[-1] // 2:] return self.concat((-x2, x1)) def apply_rotary_pos_emb_index(self, q, k, position_id): """ apply rotary pos emb index Inputs: q, k (Tensor): Querry and key. position_id (Tensor): Used to identify each token's position in the list of tokens. returns: q, k (Tensor): Embedded querry and key layer. """ # position_id: [bs, seq_len] # cos, sin: [max_seq_len, 1, hidden_size] -> [max_seq_len, hidden_size] # -> [seq_len, bs, hidden_size] -> [seq_len, bs, 1, hidden_size] q, k = ops.cast(q, self.compute_dtype), ops.cast(k, self.compute_dtype) cos = ops.expand_dims(self.squeeze(self.cos_cached)[position_id], 2) sin = ops.expand_dims(self.squeeze(self.sin_cached)[position_id], 2) cos, sin = ops.cast(cos, q.dtype), ops.cast(sin, q.dtype) # q, k: [bs, seq_len, size_per_head, hidden_size] q = self.add(self.mul(q, cos), self.mul(self.rotate_half(q), sin)) k = self.add(self.mul(k, cos), self.mul(self.rotate_half(k), sin)) return q, k def construct(self, q, k, position_id): return self.apply_rotary_pos_emb_index(q, k, position_id) def split_tensor_along_last_dim(tensor, num_partitions): """ Split a tensor along its last dimension. Used in construct function. Arguments: tensor (Tensor): Input tensor. num_partitions (int): Number of partitions to split the tensor. """ # Get the size and dimension. last_dim = tensor.ndim - 1 # Split. tensor_list = ops.Split(axis=last_dim, output_num=num_partitions)(tensor) return tensor_list class RotaryEmbeddingFP32SoftmaxSelfAttention(nn.Cell): """ Attention layer with rotary embedding and softmax Args: hidden_size (int): Hidden layer size. batch_size (int): Batch size. num_attention_heads (int): Number of attention heads. parallel_config: Operator parallel strategy. attention_dropout_prob (float, [0, 1.0]): Attention layer dropout probability. output_dropout_prob (float, [0, 1.0]): Output dropout probability. layer_id (int): Layer id. max_seq_len (int): Max sequence length. hidden_size_per_attention_head (ms.dtype, optional): Default: None. position_encoding_2d (ms.dtype, optional): Use 2d position encoding or not. Default: True. bias (bool, optional): Use bias or not. Default: True. params_dtype (ms.dtype, optional): Parameter data type. Default: mstype.float32. softmax_dtype (ms.dtype, optional): Calculate softmax data type. Default: mstype.float32. compute_dtype (ms.dtype, optional): Other compute data type. Default: mstype.float16. use_past (bool, optional): Use infer cache or not. Default: False. """ def __init__( self, hidden_size, batch_size, num_attention_heads, parallel_config, attention_dropout_prob, output_dropout_prob, layer_id, max_seq_len=512, hidden_size_per_attention_head=None, position_encoding_2d=True, bias=True, params_dtype=mstype.float32, softmax_dtype=mstype.float32, compute_dtype=mstype.float16, use_past=False, ): super(RotaryEmbeddingFP32SoftmaxSelfAttention, self).__init__() self.params_dtype = params_dtype self.softmax_dtype = softmax_dtype self.compute_dtype = compute_dtype self.layer_id = layer_id # Per attention head values. self.hidden_size = hidden_size if hidden_size_per_attention_head: self.hidden_size_per_attention_head = hidden_size_per_attention_head else: self.hidden_size_per_attention_head = divide(hidden_size, num_attention_heads) self.num_attention_heads = num_attention_heads self.inner_hidden_size = (num_attention_heads * self.hidden_size_per_attention_head) self.use_past = use_past self.is_first_iteration = True self.position_encoding_2d = position_encoding_2d self.rotary_emb = RotaryEmbedding( self.hidden_size // (self.num_attention_heads * 2) if self.position_encoding_2d else self.hidden_size // self.num_attention_heads, base=10000, params_dtype=params_dtype, # need to use float32 for proper ops.cos/ops.sin compute_dtype=compute_dtype, max_seq_len=max_seq_len, parallel_config=parallel_config, ) # Strided linear layer. self.query_key_value = Linear( hidden_size, 3 * self.inner_hidden_size, has_bias=bias, param_init_type=params_dtype, compute_dtype=compute_dtype ) self.query_key_value.shard( strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,)) ) self.attention_dropout = get_dropout(attention_dropout_prob) self.dense = Linear( self.inner_hidden_size, hidden_size, has_bias=bias, param_init_type=params_dtype, compute_dtype=compute_dtype ) self.dense.shard( strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, 1), (1,))) self.output_dropout = get_dropout(output_dropout_prob) self.matmul = P.BatchMatMul() self.matmul.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.softmax = nn.Softmax(axis=-1) self.reduce_max = P.ReduceMax(keep_dims=False) self.split = P.Split(axis=-1, output_num=2) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.concat_query = P.Concat(axis=3).shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1)) ) self.concat_key = P.Concat(axis=3).shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1)) ) if self.use_past: # operators used for state reuse seq_range = np.arange(max_seq_len).reshape(1, 1, -1) # [1, 1, config.seq_length] self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) # [bs, 1, config.seq_len] self.seq_length = max_seq_len self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.TensorAdd().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.concat_query = P.Concat(axis=3).shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.concat_key = P.Concat(axis=3).shard(((1, 1, 1, 1), (1, 1, 1, 1))) def attention_fn( self, query_layer, key_layer, value_layer, attention_mask, layer_id, scaling_attention_score=True, ): """ calculate attention function """ # seqlen, batch, head, hidden_size query_key_layer_scaling_coeff = F.cast(layer_id + 1, self.compute_dtype) if scaling_attention_score: query_layer_dtype = query_layer.dtype query_layer = F.cast(query_layer, self.compute_dtype) sqrt_value = F.sqrt(F.cast(self.hidden_size_per_attention_head, self.compute_dtype)) query_layer = query_layer / (sqrt_value * query_key_layer_scaling_coeff) query_layer = F.cast(query_layer, query_layer_dtype) # =================================== # Raw attention scores. [b, np, s, s] # =================================== # [b, np, sq, hn] * [b * np, hn, sk] matmul_result = self.matmul( query_layer.swapaxes(1, 2), F.transpose(key_layer, (0, 2, 3, 1)) ) # record original score dtype attention_scores_dtype = matmul_result.dtype # change view to [b, np, sq, sk] attention_scores = matmul_result attention_mask = F.cast(attention_mask, mstype.bool_) attention_scores = attention_scores.masked_fill(attention_mask, -10000.0) attention_scores = F.cast(attention_scores, self.softmax_dtype) attention_scores = attention_scores * query_key_layer_scaling_coeff # softmax under `softmax_dtype` mode attention_probs = self.softmax(attention_scores) # cast back to original score dtype attention_probs = F.cast(attention_probs, attention_scores_dtype) if self.training: attention_probs = self.attention_dropout(attention_probs) # ========================= # Context layer. [sq, b, hp] # ========================= # value_layer -> context layer. # [sk, b, np, hn] --> [b, np, sq, hn] # matmul: [b * np, sq, sk] x [b * np, sk, hn] --> [b * np, sq, hn] context_layer = self.matmul(attention_probs, value_layer) context_layer = F.cast(context_layer, self.compute_dtype) context_layer = self._merge_heads(context_layer) return context_layer def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (x_shape[0], x_shape[1], x_shape[-2] * x_shape[-1]) x_merge = F.reshape(x, new_shape) return x_merge def attention_forward(self, hidden_states, mask, position_ids, layer_id, key_past=None, value_past=None, batch_valid_length=None): """ attention forward Input: hidden_states (Tensor): Hidden layer states. mask (Tensor): Same as `attention_mask`, used when batching sequences together. position_ids (Tensor): Used to identify each token's position in the list of tokens. layer_id (int): Layer id. key_past (Tensor, optional): Default: None. value_past (Tensor, optional): Default: None. batch_valid_length (bool, optional): Default: None. return: output (Tensor): Attention output. layer_present (Tensor): Layer present, used for infer cache. """ mixed_raw_layer = self.query_key_value(hidden_states) mixed_raw_layer = F.cast(mixed_raw_layer, self.compute_dtype) # [b, seq, (heads * 3 * hidden_size_per_head)] --> [b, seq, heads, 3 * hidden_size_per_head] new_tensor_shape = mixed_raw_layer.shape[:-1] + ( self.num_attention_heads, 3 * self.hidden_size_per_attention_head, ) mixed_raw_layer = mixed_raw_layer.view(*new_tensor_shape) # [b, seq, heads, hidden_size_per_head] (query_layer, key_layer, value_layer) = split_tensor_along_last_dim(mixed_raw_layer, 3) if self.position_encoding_2d: q1, q2 = self.split(query_layer) k1, k2 = self.split(key_layer) position_ids, block_position_ids = position_ids[:, 0, :], \ position_ids[:, 1, :] q1, k1 = self.rotary_emb(q1, k1, position_ids) q2, k2 = self.rotary_emb(q2, k2, block_position_ids) query_layer = self.concat_query((q1, q2)) key_layer = self.concat_query((k1, k2)) else: # apply rotary embed on q, k: [bs, seq, num_heads, hidden_size] # position_ids: bs, 2, seq_length query_layer, key_layer = self.rotary_emb(query_layer, key_layer, position_ids) # key and value for current token(s) # [bs, num_heads, hidden_size, seq_len] value_layer = F.transpose(value_layer, (0, 2, 1, 3)) key_present = key_layer value_present = value_layer if self.use_past: # reshape key_present = F.transpose(key_present, (0, 2, 3, 1)) value_present = F.transpose(value_present, (0, 1, 3, 2)) # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.params_dtype) # [bs, 1, seq_len] # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key_present, self.expand_dims(valid_length_vector, 2)) value_present = self.mul1(value_present, self.expand_dims(valid_length_vector, 2)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: # Get the current token position index # key_past: [batch_size, num_heads, size_per_head, seq_length] valid_length = batch_valid_length - 1 valid_length = F.reshape(valid_length, (-1, 1, 1)) # [bs, 1, 1] # self.range: [bs, 1, config.seq_len] valid_length_vector = F.cast(self.equal(valid_length, self.range), self.params_dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key_present, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) current_value = self.mul1(self.tile(value_present, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) # Concat the previous saved state and current state key_present = self.add(key_past, current_key) # [batch_size, num_heads, size_per_head, seq_length] value_present = self.add(value_past, current_value) # update k v for attention # [batch_size, num_heads, size_per_head, seq_length] -> [bs, num_heads, hidden_size, seq_len] key_layer = F.transpose(key_present, (0, 3, 1, 2)) # [batch_size, num_heads, size_per_head, seq_length] -> [bs, num_heads, seq_len, hidden_size] value_layer = F.transpose(value_present, (0, 1, 3, 2)) layer_present = (key_present, value_present) # [batch_size, num_heads, size_per_head, seq_length] -> [seq_len, bs, num_heads, hidden_size] query_layer = F.cast(query_layer, self.compute_dtype) key_layer = F.cast(key_layer, self.compute_dtype) value_layer = F.cast(value_layer, self.compute_dtype) context_layer = self.attention_fn(query_layer, key_layer, value_layer, mask, layer_id, True) output = self.dense(context_layer) output = F.cast(output, self.params_dtype) if self.training: output = self.output_dropout(output) return output, layer_present def construct(self, hidden_states, mask, position_ids, layer_id, key_past=None, value_past=None, batch_valid_length=None): return self.attention_forward(hidden_states, mask, position_ids, layer_id, key_past, value_past, batch_valid_length) pengcheng.mind-test/mindformers/models/glm/chatglm_6b_tokenizer.py000066400000000000000000000405571471400134000257650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Tokenization classes for ChatGLM.""" import os from typing import List, Optional, Union import sentencepiece as spm from mindformers.tools import logger from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import Tokenizer from mindformers.tools.register import MindFormerRegister, MindFormerModuleType EncodedInput = List[int] __all__ = ['ChatGLMTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "ice_text.model"} class TextTokenizer: """Base text tokenizer.""" def __init__(self, model_path): self.sp = spm.SentencePieceProcessor() self.sp.Load(model_path) self.num_tokens = self.sp.vocab_size() def convert_token_to_id(self, token): return self.sp.PieceToId(token) def convert_id_to_token(self, idx): return self.sp.IdToPiece(idx) def tokenize(self, text): return self.sp.EncodeAsPieces(text) def decode(self, ids: List[int]): return self.sp.DecodeIds(ids) def __len__(self): return self.num_tokens class SPTokenizer: """Tokenizer process special tokens.""" def __init__( self, vocab_file, num_image_tokens=20000, max_blank_length=80, byte_fallback=True, ): assert vocab_file is not None self.vocab_file = vocab_file self.num_image_tokens = num_image_tokens self.special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "", "", "", "", ""] self.max_blank_length = max_blank_length self.byte_fallback = byte_fallback self.text_tokenizer = TextTokenizer(vocab_file) @staticmethod def get_blank_token(length: int): assert length >= 2 return f"<|blank_{length}|>" @staticmethod def get_tab_token(): return f"<|tab|>" @property def num_text_tokens(self): return self.text_tokenizer.num_tokens @property def num_tokens(self): return self.num_image_tokens + self.num_text_tokens @staticmethod def _encode_whitespaces(text: str, max_len: int = 80): text = text.replace("\t", SPTokenizer.get_tab_token()) for i in range(max_len, 1, -1): text = text.replace(" " * i, SPTokenizer.get_blank_token(i)) return text def _preprocess(self, text: str, linebreak=True, whitespaces=True): if linebreak: text = text.replace("\n", "") if whitespaces: text = self._encode_whitespaces(text, max_len=self.max_blank_length) return text def encode(self, text: str, linebreak=True, whitespaces=True, add_dummy_prefix=True) -> List[int]: """ Encode text to token id. Args: text: Text to encode. linebreak: Whether to encode newline (\n) in text. whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding. add_dummy_prefix: Whether to add dummy blank space in the beginning. """ text = self._preprocess(text, linebreak, whitespaces) if not add_dummy_prefix: text = "" + text tmp = self.text_tokenizer.encode(text) tokens = [x + self.num_image_tokens for x in tmp] return tokens if add_dummy_prefix else tokens[2:] def decode(self, text_ids: List[int]) -> str: """Decode id to text.""" ids = [int(id) - self.num_image_tokens for id in text_ids] ids = [id for id in ids if id >= 0] text = self.text_tokenizer.decode(ids) text = text.replace("", "\n") text = text.replace(SPTokenizer.get_tab_token(), "\t") for i in range(2, self.max_blank_length + 1): text = text.replace(self.get_blank_token(i), " " * i) return text def tokenize(self, text: str, linebreak=True, whitespaces=True, add_dummy_prefix=True) -> List[str]: """ Encode text to id. Args: text: Text to encode. linebreak: Whether to encode newline (\n) in text. whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding. add_dummy_prefix: Whether to add dummy blank space in the beginning. """ text = self._preprocess(text, linebreak, whitespaces) if not add_dummy_prefix: text = "" + text tokens = self.text_tokenizer.tokenize(text) return tokens if add_dummy_prefix else tokens[2:] def __getitem__(self, x: Union[int, str]): if isinstance(x, int): if x < self.num_image_tokens: return "".format(x) return self.text_tokenizer.convert_id_to_token(x - self.num_image_tokens) if isinstance(x, str): if x.startswith("") and x[7:-1].isdigit(): return int(x[7:-1]) return self.text_tokenizer.convert_token_to_id(x) + self.num_image_tokens raise ValueError("The key should be str or int.") @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class ChatGLMTokenizer(Tokenizer): """ Construct a ChatGLM tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file(str): The vocabulary file path. do_lower_case(bool): Lower input text. Default False. remove_space(str): The merge file path. bos_token(str): The token that represents the begin-of-sentence. Default ''. eos_token(str): The token that represents the end-of-sentence. Default ''. end_token(str): The token that represents the end-of-sentence. Default ''. mask_token(str): The token that represents the special mask. Default '[MASK]', gmask_token(str): The token that represents the special mask. Default '[gMASK]', pad_token(str): The token that represents the pad. Default "". unk_token(str): The token that represents the unknown. Default ''. add_prefix_space(bool): whether to add a whitespace in the front of text. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import AutoTokenizer >>> tokenize = AutoTokenizer.from_pretrained('glm_6b') >>> tokenize("你好") {'input_ids': [5, 74874, 130001, 130004], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]} >>> from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer >>> tokenizer = ChatGLMTokenizer('ice_text.model') >>> prompts_list = ["晚上睡不着应该怎么办"] >>> token_id = tokenizer(prompts) >>> input_ids = token_id['input_ids'] >>> print(input_ids) [[74747, 83400, 64213, 66846, 130001, 130004]] >>> response = tokenizer.decode(input_ids) >>> print(response) ['晚上睡不着应该怎么办'] Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "token_type_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['glm'] def __init__( self, vocab_file, do_lower_case=False, remove_space=False, bos_token='', eos_token='', end_token='', mask_token='[MASK]', gmask_token='[gMASK]', pad_token="", unk_token="", num_image_tokens=0, **kwargs ) -> None: super().__init__( do_lower_case=do_lower_case, remove_space=remove_space, bos_token=bos_token, eos_token=eos_token, end_token=end_token, mask_token=mask_token, gmask_token=gmask_token, pad_token=pad_token, unk_token=unk_token, num_image_tokens=num_image_tokens, **kwargs ) self.do_lower_case = do_lower_case self.remove_space = remove_space self.vocab_file = vocab_file self._bos_token = bos_token self._eos_token = eos_token self._end_token = end_token self._mask_token = mask_token self._gmask_token = gmask_token self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens) self.added_tokens_encoder = {} self.added_tokens_decoder = {} @property def gmask_token_id(self) -> Optional[int]: if self._gmask_token is None: return None return self.convert_tokens_to_ids(self._gmask_token) @property def end_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the end of context token in the vocabulary. Returns `None` if the token has not been set. """ if self._end_token is None: return None return self.convert_tokens_to_ids(self._end_token) @property def vocab_size(self): """ Returns vocab size """ return self.sp_tokenizer.num_tokens def get_vocab(self): """ Returns vocab as a dict """ vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def preprocess_text(self, inputs): """Preprocess text.""" if self.remove_space: outputs = " ".join(inputs.strip().split()) else: outputs = inputs if self.do_lower_case: outputs = outputs.lower() return outputs def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): """ Returns a tokenized string. """ return self._tokenize(text) def _tokenize(self, text, **kwargs): """ Returns a tokenized string. """ text = self.preprocess_text(text) seq = self.sp_tokenizer.tokenize(text) return seq def _decode(self, token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=None, **kwargs): """ Decode id to text. """ # unused in this tokenizer. _, _ = skip_special_tokens, kwargs if isinstance(token_ids, int): token_ids = [token_ids] if self.pad_token_id in token_ids: # remove pad token_ids = list(filter(self.pad_token_id.__ne__, token_ids)) for token_id in token_ids: if token_id not in self.added_tokens_decoder and token_id >= self.vocab_size: raise IndexError(f"The token id {token_id} is out of the size of vocabulary, please check " f"your tokenizer and corresponding vocabulary files.") return self.sp_tokenizer.decode(token_ids) # pylint:disable=arguments-differ def convert_tokens_to_ids(self, tokens: Union[str, List[str]]) -> Union[int, List[int]]: """ Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. """ return self._convert_tokens_to_ids(tokens) def _convert_tokens_to_ids(self, tokens: Union[str, List[str]]) -> Union[int, List[int]]: """ Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. """ if tokens is None: return None if isinstance(tokens, str): return self._convert_token_to_id_with_added_voc(tokens) ids = [] for token in tokens: ids.append(self._convert_token_to_id_with_added_voc(token)) return ids def _convert_token_to_id_with_added_voc(self, token): if token is None: return None if token in self.added_tokens_encoder: return self.added_tokens_encoder[token] return self.sp_tokenizer[token] def _convert_token_to_id(self, token): """copy from _convert_token_to_id_with_added_voc""" if token is None: return None if token in self.added_tokens_encoder: return self.added_tokens_encoder[token] return self.sp_tokenizer[token] def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" return self.sp_tokenizer[index] def build_inputs_with_special_tokens(self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None): """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. """ gmask_id = self.sp_tokenizer[self._gmask_token] eos_id = self.sp_tokenizer[self._eos_token] token_ids_0 = token_ids_0 + [gmask_id, self.sp_tokenizer[self._bos_token]] if token_ids_1 is not None: token_ids_0 = token_ids_0 + token_ids_1 + [eos_id] return token_ids_0 def save_vocabulary(self, save_directory, filename_prefix=None): """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. """ if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]) with open(self.vocab_file, 'rb') as fin: proto_str = fin.read() with open(vocab_file, "wb") as writer: writer.write(proto_str) return vocab_file def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ output = [0] * (len(token_ids_0) + 1 + 1) if token_ids_1 is not None: output += [1] * (len(token_ids_1) + 1) return output pengcheng.mind-test/mindformers/models/glm/convert_weight.py000066400000000000000000000042601471400134000247030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/huggingface""" import argparse import mindspore as ms import torch as pt def convert_pt_to_ms(pt_pth_path, ms_ckpt_path): """ Convert pytorch mpdel file to MindSpore model file. """ pt_param = pt.load(pt_pth_path) print('parameter convert....') ms_param = [] for k, v in pt_param.items(): if "word_embeddings.weight" in k: k = k.replace("weight", "embedding_table") if "post_attention_layernorm" in k or "input_layernorm" in k or "final_layernorm" in k: k = k.replace("weight", "gamma") k = k.replace("bias", "beta") if "input_layernorm" in k: v = v.half() ms_param.append({"name": k, "data": ms.Tensor(v.numpy())}) print('saving ms ckpt....') ms.save_checkpoint(ms_param, ms_ckpt_path) if __name__ == '__main__': parser = argparse.ArgumentParser(description="ChatGLM6B weight convert script") parser.add_argument("--pt_ckpt_path", type=str, required=True, default="None", help="The torch checkpoint path.") parser.add_argument("--ms_ckpt_path", type=str, required=False, default="ms_glm_6b.ckpt", help='The output mindspore checkpoint path.') opt = parser.parse_args() convert_pt_to_ms(opt.pt_ckpt_path, opt.ms_ckpt_path) print(f"Convert finished, the output is saved to {opt.ms_ckpt_path}") pengcheng.mind-test/mindformers/models/glm/glm.py000066400000000000000000000557131471400134000224440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GLM model.""" import os import numpy as np import mindspore as ms from mindspore import dtype as mstype from mindspore import nn, ops, Tensor from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.common.initializer import initializer from mindspore.common.parameter import Parameter from mindformers.mindformer_book import MindFormerBook from mindformers.modules.transformer import VocabEmbedding, EmbeddingOpParallelConfig, OpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.core.loss import CrossEntropyLoss from mindformers.modules.layers import LayerNorm from mindformers.pet.tuners.pet_adapter import PetAdapter from mindformers.pet.tuners.lora_adapter import LoraAdapter from mindformers.version_control import get_dropout from .glm_config import GLMConfig from .layers import DeepNormWithGLULayer from ..base_model import BaseModel # Get MS backend: 0 vm 1 GE is_ge = os.getenv('MS_ENABLE_GE') if is_ge == '1': jit_level = "O3" else: jit_level = "O1" default_dpmp_config = OpParallelConfig() default_embedding_parallel_config = EmbeddingOpParallelConfig() __all__ = ['GLMForPreTraining', 'GLMChatModel', 'GLMForPreTrainingWithLora', 'GLMChatModelWithLora'] def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: # we give the fusion in pipeline mode a fixed value, otherwise the performance may become worse. network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class ProcessLogits(nn.Cell): r"""Process logits into probability distribution.""" def __init__(self, use_past=False): super(ProcessLogits, self).__init__() self.e = ms.Tensor(np.e) self.gather = P.Gather() self.logsoftmax = P.LogSoftmax() self.reshape = P.Reshape() self.use_past = use_past def construct(self, logits, current_index=None, is_first_iteration=False): logits = logits.reshape(-1, logits.shape[-1]) if self.use_past and not is_first_iteration: logits = logits elif current_index is not None: index = current_index.view(-1,) logits = self.gather(logits, index, 0) outputs = self.logsoftmax(logits) outputs = F.tensor_pow(self.e, outputs) return outputs class GLMModel(nn.Cell): """ The backbone of GLM network Args: config (GLMConfig): The config of network. op_parallel_config (optional): Operator parallel strategy. Default: `OpParallelConfig()`. embed_parallel_config (optional): Operator parallel strategy. Default: `EmbeddingOpParallelConfig()`. """ def __init__(self, config, op_parallel_config=default_dpmp_config, embed_parallel_config=default_embedding_parallel_config): super(GLMModel, self).__init__() # recording parameters self.num_layers = config.num_layers self.hidden_size = config.hidden_size self.num_heads = config.num_heads self.seq_length = config.seq_length self.use_past = config.use_past layernorm = LayerNorm if config.parallel_config: op_parallel_config = config.parallel_config # create embedding parameters self.embedding_dropout = get_dropout(config.embedding_dropout_prob) embed_parallel_config.data_parallel = op_parallel_config.data_parallel embed_parallel_config.model_parallel = op_parallel_config.model_parallel embed_parallel_config.vocab_emb_dp = False self.word_embeddings = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, parallel_config=embed_parallel_config) self.word_embeddings.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.matmul = ops.MatMul().shard(((1, 1), (1, embed_parallel_config.model_parallel))) self.transpose = ops.Transpose().shard(((embed_parallel_config.model_parallel, 1),)) def get_layer(layer_id): return DeepNormWithGLULayer( self.num_layers, self.hidden_size, self.num_heads, config.batch_size, config.attention_dropout_rate, config.hidden_dropout_rate, config.layernorm_epsilon, layer_id, max_seq_len=self.seq_length, inner_hidden_size=config.inner_hidden_size, hidden_size_per_attention_head=config.hidden_size_per_attention_head, layernorm_order=config.layernorm_order, layernorm=layernorm, use_bias=True, activation_func=config.activation_func, position_encoding_2d=config.position_encoding_2d, params_dtype=config.param_init_type, layernorm_dtype=config.layernorm_compute_type, softmax_dtype=config.softmax_compute_type, compute_dtype=config.compute_dtype, use_past=self.use_past, parallel_config=op_parallel_config, ) self.layers = nn.CellList() for i in range(config.num_layers): layer = get_layer(i+1) set_parallel_configure_for_layer(layer, layer_id=i, layers=config.num_layers, offset=0, parallel_config=op_parallel_config) self.layers.append(layer) # Final layer norm before output. self.use_final_layernorm = config.use_final_layernorm if config.use_final_layernorm: self.final_layernorm = layernorm(config.hidden_size, eps=config.layernorm_epsilon) self.final_layernorm.shard(((op_parallel_config.data_parallel, 1, 1),)) def construct(self, input_ids, position_ids, attention_mask, init_reset=True, batch_valid_length=None): """ Get output logits Inputs: input_ids (Tensor): The tokenized inputs with dtype int32. input_mask (Tensor): The mask indicating whether each position is a valid input. position_ids (Tensor): Used to identify each token's position in the list of tokens. attention_mask (Tensor): Used when batching sequences together. init_reset (bool, optional): Default: True. batch_valid_length (Tensor, optional): Default: None. Returns: logits (Tensor): The output logit of backbone. table (Tensor): The embedding table for the vocabulary. """ if attention_mask is None: attention_mask = ops.ones((1, 1), mstype.int32) hidden_states, table = self.word_embeddings(input_ids) hidden_states = self.embedding_dropout(hidden_states) for i in range(self.num_layers): layer_ret = self.layers[i](hidden_states, attention_mask, position_ids, init_reset, batch_valid_length) if isinstance(layer_ret, tuple): layer_ret = layer_ret[0] hidden_states = layer_ret # Final layer norm. if self.use_final_layernorm: logits = self.final_layernorm(hidden_states) else: logits = hidden_states return logits, table class GLMHead(nn.Cell): r"""Head for GLM to get the logits of each token in the vocab.""" def __init__(self, hidden_size, vocab_size, param_init_type=mstype.float32, compute_dtype=mstype.float16, embed_parallel_config=None): super(GLMHead, self).__init__() self.param_init_type = param_init_type self.compute_dtype = compute_dtype self.weight = Parameter(initializer("normal", [vocab_size, hidden_size], compute_dtype), name="weight") self.transpose = ops.Transpose().shard(((embed_parallel_config.model_parallel, 1),)) self.matmul = ops.MatMul(transpose_b=True).shard( ((embed_parallel_config.data_parallel, 1), (embed_parallel_config.model_parallel, 1))) def construct(self, state, embedding_table=None): """Get vocab probs""" state = F.reshape(state, (-1, F.shape(state)[-1])) state = ops.cast(state, self.compute_dtype) if embedding_table is None: embedding_table = self.weight embedding_table = self.cast(embedding_table, self.compute_dtype) logits_parallel = self.matmul(state, embedding_table) return logits_parallel @MindFormerRegister.register(MindFormerModuleType.MODELS) class GLMForPreTraining(BaseModel): r""" Provide glm training loss or logits through network. Args: config (GLMConfig): The config of GLMModel. """ _support_list = MindFormerBook.get_model_support_list()['glm'] def __init__(self, config: GLMConfig): super(GLMForPreTraining, self).__init__(config) self.config = config self.position_encoding_2d = config.position_encoding_2d self.transformer = GLMModel(config) self.lm_head = GLMHead( hidden_size=config.hidden_size, vocab_size=config.vocab_size, param_init_type=config.param_init_type, compute_dtype=config.compute_dtype, embed_parallel_config=config.parallel_config) self.stridedslice = ops.StridedSlice().shard(((1, 1),)) self.loss = CrossEntropyLoss(parallel_config=config.parallel_config, eps_const=3.4e-38) self.gmask = config.gmask_token_id self.bos_token_id = config.bos_token_id self.ones = P.Ones() self.not_equal = P.NotEqual() self.gather = P.Gather() self.use_past = config.use_past self.is_first_iteration = True self.ignore_index = config.ignore_index self.load_checkpoint(config) def get_masks_np(self, input_ids): """get attention mask using numpy.""" batch_size, seq_length = input_ids.shape context_lengths = [list(seq).index(self.config.bos_token_id) for seq in input_ids] attention_mask = np.tril(np.ones((batch_size, seq_length, seq_length))) for i, context_length in enumerate(context_lengths): attention_mask[i, :, :context_length] = 1 attention_mask = np.expand_dims(attention_mask, axis=1) attention_mask = np.array(attention_mask < 0.5, np.bool_) return attention_mask def get_position_ids_np(self, input_ids, mask_positions, use_gmasks=None): """Get position ids from input_ids and mask_positions with numpy""" batch_size, seq_length = input_ids.shape if use_gmasks is None: use_gmasks = [False] * batch_size context_lengths = [list(seq).index(self.config.bos_token_id) for seq in input_ids] if self.config.position_encoding_2d: position_ids = np.repeat(np.expand_dims(np.arange(seq_length), 0), batch_size, axis=0) for i, context_length in enumerate(context_lengths): position_ids[i, context_length:] = mask_positions[i] block_position_ids = [np.concatenate(( np.zeros(context_length, np.int32), np.arange(seq_length - context_length, dtype=np.int32) + 1 )) for context_length in context_lengths] block_position_ids = np.stack(block_position_ids, axis=0) position_ids = np.stack((position_ids, block_position_ids), axis=1) else: position_ids = np.repeat(np.expand_dims(np.arange(seq_length), 0), batch_size, axis=0) for i, context_length in enumerate(context_lengths): if not use_gmasks[i]: position_ids[context_length:] = mask_positions[i] return position_ids def create_position_ids_np(self, input_ids): """Get position ids from input_ids with numpy""" mask, gmask = self.config.mask_token_id, self.config.gmask_token_id seqs = list(input_ids) mask_positions, use_gmasks = [], [] for seq in seqs: mask_token = gmask if gmask in seq else mask use_gmask = mask_token == gmask mask_positions.append(list(seq).index(mask_token)) use_gmasks.append(use_gmask) position_ids = self.get_position_ids_np(input_ids, mask_positions, use_gmasks=None) return position_ids def update_model_kwargs_before_generate(self, input_ids, model_kwargs: dict): """update glm kwargs before generate.""" # for GLM `attention_mask` and `position_ids` generation attention_mask = self.get_masks_np(input_ids) position_ids = self.create_position_ids_np(input_ids) # update in model kwargs model_kwargs["attention_mask"] = attention_mask model_kwargs["position_ids"] = position_ids def prepare_inputs_for_generation(self, input_ids, **kwargs): """prepare inputs for generation.""" attention_mask = kwargs.get("attention_mask", None) position_ids = kwargs.get("position_ids", None) input_position = kwargs.get("current_index", None) if input_position is not None: input_position = Tensor(input_position, mstype.int32) return { "input_ids": Tensor(input_ids, mstype.int32), "attention_mask": Tensor(attention_mask, mstype.int32), "position_ids": Tensor(position_ids, mstype.int32), "input_position": input_position } def slice_incremental_inputs(self, model_inputs: dict, current_index): """used for non-first iterations, slice the inputs to length 1.""" input_ids = model_inputs.pop("input_ids") attention_mask = model_inputs.pop("attention_mask") position_ids = model_inputs.pop("position_ids") if isinstance(input_ids, Tensor): input_ids = input_ids.asnumpy() if isinstance(attention_mask, Tensor): attention_mask = attention_mask.asnumpy() if isinstance(position_ids, Tensor): position_ids = position_ids.asnumpy() inputs_tmp = [] position_ids_tmp = [] attention_mask_tmp = [] for i, index_value in enumerate(current_index): current_index_tmp = ( int(index_value) - i * input_ids.shape[1] ) # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp : current_index_tmp + 1]) position_ids_tmp.append(position_ids[i][..., current_index_tmp:current_index_tmp + 1]) attention_mask_tmp.append(attention_mask[i][:, current_index_tmp:current_index_tmp + 1, :]) inputs_tmp = np.array(inputs_tmp, dtype=np.int32) position_ids_tmp = np.array(position_ids_tmp, dtype=np.int32) attention_mask_tmp = np.array(attention_mask_tmp, dtype=np.int32) model_inputs["input_ids"] = Tensor(inputs_tmp, mstype.int32) model_inputs["position_ids"] = Tensor(position_ids_tmp, mstype.int32) model_inputs["attention_mask"] = Tensor(attention_mask_tmp, mstype.int32) # pylint: disable=W0613 def construct(self, input_ids, labels=None, position_ids=None, attention_mask=None, input_position=None, input_embeds=None, init_reset=True, batch_valid_length=None): """ Extract logits and calculate loss Inputs: input_ids (Tensor): the tokenized inputs with dtype int32. labels (Tensor): the indices of input sequence tokens in the vocabulary. position_ids (Tensor): used to identify each token's position in the list of tokens. attention_mask (Tensor): used when batching sequences together. input_position(Tensor): Reserved param, not used. input_embeds(Tensor): Reserved param, not used. init_reset (bool, optional): Default: True. batch_valid_length(Tensor, optional): Default: None. Returns: Training phase: loss: Training loss. Other phase: logits (Tensor): The output logit of backbone. """ batch_size, seq_length = input_ids.shape if self.training: tokens = self.stridedslice(input_ids, (0, 0), (batch_size, seq_length), (1, 1)) else: tokens = input_ids output_states, _ = self.transformer(tokens, position_ids, attention_mask, init_reset, batch_valid_length) logits = self.lm_head(output_states) logits_shape = logits.shape if not self.training: logits = logits.reshape((-1, logits_shape[-1])) # only gather in auto-aggressive generate or first iteration if (not self.use_past or self.is_first_iteration) and input_position is not None: logits = self.gather(logits, input_position, 0) return (logits,) labels = labels.reshape((-1,)) logits = logits.reshape((-1, logits_shape[-1])) input_mask = self.not_equal(labels, self.ignore_index).astype(logits.dtype) input_mask = input_mask.reshape((-1,)) loss = self.loss(logits, labels, input_mask) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class GLMChatModel(GLMForPreTraining): r""" Provide glm chat capability through network. Args: config (GLMConfig): The config of GLMModel. Returns: Tensor, the probability distribution of network loss. """ _support_list = MindFormerBook.get_model_support_list()['glm'] def __init__(self, config: GLMConfig): super(GLMChatModel, self).__init__(config) self.e = ms.Tensor(np.e, dtype=mstype.float32) self.pow = P.Pow() self.topk = P.TopK(sorted=True) self.cumsum = P.CumSum() self.sum = P.ReduceSum(keep_dims=False) self.vocab_size = config.vocab_size self.batch_size = config.batch_size self.frequency_list = ms.Tensor([[0 for _ in range(self.vocab_size)]]) self.post_logits = ProcessLogits(use_past=config.use_past) # seems not supported yet. # self.top_p = config.top_p self.top_p = 1 self.top_k = config.top_k self.repetition_penalty = config.repetition_penalty self.is_first_iteration = False self.is_sample_acceleration = config.is_sample_acceleration def sample(self, log_probs): """Convert the log_probs to probability""" if self.repetition_penalty != 1: log_probs = log_probs - self.frequency_list * self.repetition_penalty - \ (self.frequency_list > 0) * self.repetition_penalty # Process sample in graph to accelerate generate logits = self.pow(self.e, log_probs) # If top_p is less than 1.0, use top_p sampling # seems not supported yet. if self.top_p < 1.0: sorted_logits, index = self.topk(logits, 5000) cumsum_logits = self.cumsum(sorted_logits, 1) top_p_num = self.sum((cumsum_logits < self.top_p).astype(mstype.int32), -1) + 1 top_p_num = int(top_p_num) # Get the corresponding probs and indices probs = sorted_logits[:, :top_p_num] p_args = index[:, :top_p_num] p = probs / self.sum(probs, -1, keepdim=True) # if top_p is set to 1.0, use top_k sampling else: probs, p_args = self.topk(logits, self.top_k) p = probs return p, p_args # pylint:disable=arguments-differ,W0613 def construct(self, input_ids, position_ids=None, attention_mask=None, input_position=None, labels=None, input_embeds=None, init_reset=True, batch_valid_length=None): """Get probs and p_args""" # model forward output_states, _ = self.transformer(input_ids, position_ids, attention_mask, init_reset, batch_valid_length) logits = self.lm_head(output_states) if not self.is_sample_acceleration: return (logits,) # logit post process log_probs = self.post_logits(logits, input_position, self.is_first_iteration) # logit sort and sample probs, p_args = self.sample(log_probs) return probs, p_args @MindFormerRegister.register(MindFormerModuleType.MODELS) class GLMForPreTrainingWithLora(GLMForPreTraining): """GLM Model for pretraining with LoRA Args: config (GLMConfig): The config of network. """ def __init__(self, config: GLMConfig = None, **kwargs): _ = kwargs super().__init__(config) # get Pet tuning model. config.pet_config.reg_rules = r'.*query_key_value*' self.transformer = LoraAdapter.get_pet_model(self.transformer, config.pet_config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, config.pet_config.pet_type) @MindFormerRegister.register(MindFormerModuleType.MODELS) class GLMChatModelWithLora(GLMChatModel): """GLM Model for pretraining with LoRA Args: config (GLMConfig): The config of network. """ def __init__(self, config: GLMConfig = None, **kwargs): _ = kwargs ckpt_cfg = config.checkpoint_name_or_path config.checkpoint_name_or_path = None super().__init__(config) # get Pet tuning model. config.pet_config.reg_rules = r'.*query_key_value*' self.transformer = LoraAdapter.get_pet_model(self.transformer, config.pet_config) config.checkpoint_name_or_path = ckpt_cfg self.load_checkpoint(config) pengcheng.mind-test/mindformers/models/glm/glm_config.py000066400000000000000000000124601471400134000237610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GLM config""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ TransformerOpParallelConfig, OpParallelConfig, EmbeddingOpParallelConfig, default_embedding_parallel_config from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook default_dpmp_config = OpParallelConfig() __all__ = ['GLMConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class GLMConfig(BaseConfig): """ GLM config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['glm'] def __init__(self, batch_size: int = 1, vocab_size: int = 130528, hidden_size: int = 4096, num_layers: int = 28, num_heads: int = 32, inner_hidden_size: int = 16384, seq_length: int = 512, embedding_dropout_prob: float = 0.0, attention_dropout_rate: float = 0.0, hidden_dropout_rate: float = 0.0, hidden_size_per_attention_head: bool = None, layernorm_order: str = "post", layernorm_epsilon: float = 1.0e-5, use_final_layernorm: bool = True, op_parallel_config: OpParallelConfig = default_dpmp_config, embed_parallel_config: EmbeddingOpParallelConfig = default_embedding_parallel_config, parallel_config: TransformerOpParallelConfig = default_transformer_config, moe_config: MoEConfig = default_moe_config, use_past: bool = False, activation_func: str = 'GELU', position_encoding_2d: bool = True, param_init_type: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", compute_dtype: str = "float16", bos_token_id: int = 130004, eos_token_id: int = 130005, mask_token_id: int = 130000, gmask_token_id: int = 130001, pad_token_id: int = 3, is_enhanced_encoder: bool = True, is_sample_acceleration: bool = False, checkpoint_name_or_path: str = "", max_decode_length: int = 2048, top_k: int = 1, top_p: float = 1, repetition_penalty: float = 1.0, do_sample: bool = True, ignore_index: int = -100, **kwargs): super().__init__(**kwargs) self.batch_size = batch_size self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.embedding_dropout_prob = embedding_dropout_prob self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.hidden_size_per_attention_head = hidden_size_per_attention_head self.layernorm_order = layernorm_order self.layernorm_epsilon = layernorm_epsilon self.use_final_layernorm = use_final_layernorm self.op_parallel_config = op_parallel_config self.embed_parallel_config = embed_parallel_config self.moe_config = moe_config self.use_past = use_past self.parallel_config = parallel_config self.activation_func = activation_func self.inner_hidden_size = inner_hidden_size self.position_encoding_2d = position_encoding_2d self.param_init_type = convert_mstype(param_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.compute_dtype = convert_mstype(compute_dtype) self.bos_token_id = bos_token_id self.eos_token_id = eos_token_id self.mask_token_id = mask_token_id self.gmask_token_id = gmask_token_id self.pad_token_id = pad_token_id self.max_decode_length = max_decode_length self.seq_length = seq_length self.is_enhanced_encoder = is_enhanced_encoder self.is_sample_acceleration = is_sample_acceleration self.checkpoint_name_or_path = checkpoint_name_or_path self.top_k = top_k self.top_p = top_p self.repetition_penalty = repetition_penalty self.do_sample = do_sample self.ignore_index = ignore_index pengcheng.mind-test/mindformers/models/glm/glm_processor.py000066400000000000000000000055251471400134000245370ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ GLMProcessor """ import re from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import BaseTokenizer from mindformers.models.base_processor import BaseProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['GLMProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class GLMProcessor(BaseProcessor): """ GLM processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['glm'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(GLMProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output def process_response(response): """ process the response of chat glm. """ response = response.strip() response = response.replace("[[训练时间]]", "2023年") punkts = [ [",", "，"], ["!", "！"], [":", "："], [";", "；"], [r"\?", "？"], ] for item in punkts: response = re.sub(r"([\u4e00-\u9fff])%s" % item[0], r"\1%s" % item[1], response) response = re.sub(r"%s([\u4e00-\u9fff])" % item[0], r"%s\1" % item[1], response) return response pengcheng.mind-test/mindformers/models/glm/layers.py000066400000000000000000000314751471400134000231630ustar00rootroot00000000000000"""base transformer layer.""" import numpy as np from mindspore import Parameter, Tensor from mindspore import dtype as mstype from mindspore import nn from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore import ops from mindformers.modules.layers import LayerNorm, Linear from mindformers.version_control import get_dropout from mindformers.models.glm.attention import RotaryEmbeddingFP32SoftmaxSelfAttention class GEGLU(nn.Cell): """GEGLU activation""" def __init__(self, parallel_config): super(GEGLU, self).__init__() self.split = ops.Split(output_num=2, axis=-1) self.activation_fn = P.GeLU() self.parallel_config = parallel_config def construct(self, x): x1, x2 = self.split(x) return x1 * self.activation_fn(x2) class MLPWithGEGLU(nn.Cell): """MLP layer with GEGLU""" def __init__(self, hidden_size, output_dropout_prob, inner_hidden_size=None, layer_id=None, bias=True, activation_func='GELU', params_dtype=mstype.float32, compute_dtype=mstype.float16, parallel_config=None): super(MLPWithGEGLU, self).__init__() self.layer_id = layer_id # Project to 4h. self.hidden_size = hidden_size if inner_hidden_size is None: inner_hidden_size = 4 * hidden_size self.inner_hidden_size = inner_hidden_size if activation_func == 'GEGLU': self.activation_func = GEGLU(parallel_config) h_to_4h_output_channel = 2 * self.inner_hidden_size elif activation_func == 'GELU': self.activation_func = P.GeLU() self.activation_func.shard(((parallel_config.data_parallel, 1, parallel_config.model_parallel),)) h_to_4h_output_channel = self.inner_hidden_size self.dense_h_to_4h = Linear( self.hidden_size, h_to_4h_output_channel, has_bias=bias, transpose_b=True, param_init_type=params_dtype, compute_dtype=compute_dtype, ) self.dense_h_to_4h.shard( strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Project back to h. self.dense_4h_to_h = Linear( self.inner_hidden_size, self.hidden_size, has_bias=bias, param_init_type=params_dtype, compute_dtype=compute_dtype, ) self.dense_4h_to_h.shard( strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (1, parallel_config.model_parallel)), strategy_bias=((parallel_config.data_parallel, 1), (1,)) ) self.dropout = get_dropout(output_dropout_prob) self.dropout.dropout.shard(((parallel_config.data_parallel, parallel_config.model_parallel),)) def mlp_forward(self, hidden_states): """mlp forward.""" intermediate_parallel = self.dense_h_to_4h(hidden_states) intermediate_parallel = self.activation_func(intermediate_parallel) output = self.dense_4h_to_h(intermediate_parallel) return output def construct(self, hidden_states): output = self.mlp_forward(hidden_states) if self.training: output = self.dropout(output) return output class DeepNormWithGLULayer(nn.Cell): """ GLM base layer Args: num_layers (int): Number of layers. hidden_size (int): Hidden layer size. num_attention_heads (int): Number of attention heads. batch_size (int): Batch size. attention_dropout_prob (float, [0, 1.0]): Attention layer dropout probability. output_dropout_prob (float, [0, 1.0]): Output dropout probability. layernorm_epsilon (float): Layernorm epsilon. layer_id (int): Layer id. max_seq_len (int): Max sequence length. inner_hidden_size (optional): Inner hidden layer size. Default: None. hidden_size_per_attention_head (optional): Default: None. layernorm_order (str, optional): Which order to use layernorm. Default: 'pre'. layernorm (optional): Layernorm function. Default: LayerNorm. use_bias (bool, optional): Use bias or not. Default: True. activation_func (str, optional, 'GEGLU'/'GELU'): Choose activation function. Default: 'GEGLU'. position_encoding_2d (bool, optional): Use 2d position encoding or not. Default: True. params_dtype (ms.dtype, optional): Parameter data type. Default: mstype.float32. layernorm_dtype (ms.dtype, optional): Calculate layernorm data type. Default: mstype.float32. softmax_dtype (ms.dtype, optional): Calculate softmax data type. Default: mstype.float32. compute_dtype (ms.dtype, optional): Other compute data type. Default: mstype.float16. parallel_config (optional): Operator parallel strategy, Default: None. use_past (bool, optional): Use infer cache or not. Default: False. """ def __init__(self, num_layers, hidden_size, num_attention_heads, batch_size, attention_dropout_prob, output_dropout_prob, layernorm_epsilon, layer_id, max_seq_len=512, inner_hidden_size=None, hidden_size_per_attention_head=None, layernorm_order='pre', layernorm=LayerNorm, use_bias=True, activation_func='GEGLU', position_encoding_2d=True, params_dtype=mstype.float32, layernorm_dtype=mstype.float32, softmax_dtype=mstype.float32, compute_dtype=mstype.float16, parallel_config=None, use_past=False): super(DeepNormWithGLULayer, self).__init__() # Set output layer initialization if not provided. self.num_layers = num_layers self.hidden_size = hidden_size self.layer_id = layer_id if inner_hidden_size is None: inner_hidden_size = 4 * hidden_size * 2 // 3 self.inner_hidden_size = inner_hidden_size self.position_encoding_2d = position_encoding_2d self.layernorm_order = layernorm_order self.use_past = use_past self.params_dtype = params_dtype self.layernorm_dtype = layernorm_dtype self.softmax_dtype = softmax_dtype self.compute_dtype = compute_dtype # Layernorm on the input data. self.input_layernorm = layernorm(hidden_size, param_init_type=layernorm_dtype, eps=layernorm_epsilon) self.input_layernorm.shard(((parallel_config.data_parallel, 1, 1),)) self.input_layernorm.set_comm_fusion(parallel_config.gradient_aggregation_group) # Self attention. self.attention = RotaryEmbeddingFP32SoftmaxSelfAttention( hidden_size, batch_size, num_attention_heads, parallel_config, attention_dropout_prob, output_dropout_prob, layer_id, hidden_size_per_attention_head=hidden_size_per_attention_head, position_encoding_2d=self.position_encoding_2d, bias=use_bias, params_dtype=params_dtype, softmax_dtype=softmax_dtype, compute_dtype=compute_dtype, max_seq_len=max_seq_len, use_past=use_past, ) # Layernorm on the input data. self.post_attention_layernorm = layernorm(hidden_size, param_init_type=layernorm_dtype, eps=layernorm_epsilon) self.post_attention_layernorm.shard(((parallel_config.data_parallel, 1, 1),)) self.post_attention_layernorm.set_comm_fusion(parallel_config.gradient_aggregation_group) if self.layernorm_order == 'sandwich': self.third_layernorm = layernorm(hidden_size, param_init_type=layernorm_dtype, eps=layernorm_epsilon) self.fourth_layernorm = layernorm(hidden_size, param_init_type=layernorm_dtype, eps=layernorm_epsilon) # MLP self.mlp = MLPWithGEGLU( hidden_size, output_dropout_prob, inner_hidden_size=inner_hidden_size, bias=use_bias, layer_id=layer_id, activation_func=activation_func, params_dtype=params_dtype, parallel_config=parallel_config ) self.key_past = None self.value_past = None if use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = int(hidden_size / num_attention_heads) self.kv_shape = (batch_size, num_attention_heads, size_per_head, max_seq_len) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.kv_shape), self.params_dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.kv_shape), self.params_dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), ())) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.mul = P.Mul() self.mul.shard(((parallel_config.data_parallel, 1, 1), ())) self.mul_4 = P.Mul() self.mul_4.shard(((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel,))) def layer_forward(self, hidden_states, mask, position_ids, init_reset=True, batch_valid_length=None): """ hidden_states: [seq_len, batch, hidden_size] with (1, dp, 1) mask: [(1, 1), seq_len, seq_len] Inputs: hidden_states (Tensor): Hidden layer output. mask (Tensor): Used when batching sequences together. position_ids (Tensor): Used to identify each token's position in the list of tokens. init_reset (bool, optional): Default: True. batch_valid_length (Tensor, optional): Default: None. Return: output (Tensor): Layer output. """ # Layer norm at the beginning of the transformer layer. attention_input = self.input_layernorm(hidden_states) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul_4(self.key_past, F.cast(init_reset, self.params_dtype))) value_reset = self.assign(self.value_past, self.mul_4(self.value_past, F.cast(init_reset, self.params_dtype))) # add dependency for desired execution order attention_input = F.depend(attention_input, key_reset) attention_input = F.depend(attention_input, value_reset) attention_output, layer_present = self.attention(attention_input, mask, position_ids, self.layer_id, self.key_past, self.value_past, batch_valid_length) # Residual connection. alpha = (2 * self.num_layers) ** 0.5 hidden_states = self.mul(attention_input, alpha) + attention_output mlp_input = self.post_attention_layernorm(hidden_states) # MLP. mlp_output = self.mlp(mlp_input) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step key_update = self.assign(self.key_past, key_present) value_update = self.assign(self.value_past, value_present) # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_output = F.depend(mlp_output, value_update) mlp_output = F.depend(mlp_output, key_update) # Second residual connection. output = mlp_input * alpha + mlp_output return output def construct(self, hidden_states, mask, position_ids, init_reset=True, batch_valid_length=None): ''' hidden_states: [batch, seq_len, hidden_size] mask: [(1, 1), seq_len, seq_len] ''' return self.layer_forward(hidden_states, mask, position_ids, init_reset, batch_valid_length) pengcheng.mind-test/mindformers/models/glm2/000077500000000000000000000000001471400134000213625ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/glm2/__init__.py000066400000000000000000000016051471400134000234750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for ChatGLM2""" from .glm2_config import * from .glm2_tokenizer import * from .glm2 import * __all__ = [] __all__.extend(glm2_config.__all__) __all__.extend(glm2_tokenizer.__all__) __all__.extend(glm2.__all__) pengcheng.mind-test/mindformers/models/glm2/glm2.py000066400000000000000000000237231471400134000226040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 model.""" import mindspore.ops as ops import mindspore.common.dtype as mstype import mindspore.ops.operations as P from mindspore import Tensor, nn import numpy as np from mindformers.mindformer_book import MindFormerBook from mindformers.modules import VocabEmbedding, EmbeddingOpParallelConfig from mindformers.modules.layers import Linear from mindformers.tools.register import MindFormerModuleType, MindFormerRegister from mindformers.core.loss import CrossEntropyLoss from mindformers.pet.tuners.pet_adapter import PetAdapter from mindformers.pet.tuners.lora_adapter import LoraAdapter from mindformers.version_control import get_tril from ..base_model import BaseModel from .glm2_config import ChatGLM2Config from .glm2_modules import precompute_rotary_emb_cache from .glm2_transformer import ChatGLM2Transformer __all__ = ['ChatGLM2ForConditionalGeneration', 'ChatGLM2WithLora', 'ChatGLM2Model'] class ChatGLM2Model(nn.Cell): r""" The backbone of ChatGLM2 network Args: config (GLMConfig): The config of network. """ def __init__(self, config: ChatGLM2Config, **kwargs): super(ChatGLM2Model, self).__init__(config, **kwargs) self.num_layers = config.num_layers self.multi_query_group_num = config.multi_query_group_num self.kv_channels = config.kv_channels self.seq_length = config.seq_length self.compute_dtype = config.compute_dtype self.use_past = config.use_past self.is_first_iteration = True # vocab embedding embed_parallel_config = EmbeddingOpParallelConfig() embed_parallel_config.data_parallel = config.parallel_config.data_parallel embed_parallel_config.model_parallel = config.parallel_config.model_parallel embed_parallel_config.vocab_emb_dp = False self.embedding = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, parallel_config=embed_parallel_config) self.embedding.set_comm_fusion(config.parallel_config.gradient_aggregation_group) # rotary embedding rotary_dim = ( config.hidden_size // config.num_attention_heads if config.kv_channels is None else config.kv_channels ) self.rotary_pos_emb = precompute_rotary_emb_cache( seq_len=self.seq_length, dim=rotary_dim // 2 ) self.rotary_pos_emb = Tensor(self.rotary_pos_emb, config.compute_dtype) self.encoder = ChatGLM2Transformer(config) self.output_layer = Linear(config.hidden_size, config.vocab_size, has_bias=False, param_init_type=config.param_init_type, compute_dtype=config.compute_dtype) self.output_layer.shard(strategy_matmul=((config.parallel_config.data_parallel, 1), (config.parallel_config.model_parallel, 1))) self.output_layer.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.tril = get_tril() self.ones = P.Ones() self.less = P.Less() self.gather = P.Gather() self.expand_dims = P.ExpandDims() self.reshape = P.Reshape() self.mul = P.Mul() self.tile = ops.Tile() low_triangle = np.tril(np.ones((1, self.seq_length, self.seq_length))) self.low_triangle = Tensor(low_triangle, mstype.int32) def get_masks(self, batch_size, padding_mask=None, input_position=None): """Get attention mask.""" # [1, seq_length, seq_length] -> [batch_size, seq_length, seq_length] low_triangle = self.tile(self.low_triangle, (batch_size, 1, 1)) if padding_mask is not None: low_triangle = self.mul(low_triangle, self.expand_dims(padding_mask, 1)) if self.use_past and padding_mask is not None: low_triangle -= self.expand_dims(padding_mask, -1) - 1 attention_mask = self.less(low_triangle, 0.5) if self.use_past and not self.is_first_iteration: # [bs, 1, seq_len] for incremental infer attention_mask = self.gather(attention_mask.view(-1, self.seq_length), input_position, 0) # [bs, 1, seq_len, seq_len] for normal, [bs, 1, 1, seq_len] for incremental infer attention_mask = self.reshape(attention_mask, (batch_size, 1, -1, self.seq_length)) return attention_mask def construct(self, input_ids, position_ids=None, attention_mask=None, full_attention_mask=None, inputs_embeds=None, input_position=None, init_reset=True, batch_valid_length=None): """ChatGLM2 model.""" _ = position_ids batch_size, _ = input_ids.shape if inputs_embeds is None: inputs_embeds, _ = self.embedding(input_ids) # (bs, seq_len, hs) if full_attention_mask is None: # (bs, 1, seq_len, seq_len) full_attention_mask = self.get_masks(batch_size, attention_mask, input_position) # (sen length, kv_channels // 4, 2) rotary_pos_emb = self.rotary_pos_emb if self.use_past and not self.is_first_iteration and batch_valid_length is not None: # only take [bs, 1, kv_channels // 4, 2] batch_gather_position = batch_valid_length.view(-1, 1) - 1 # [bs, seq_len=1] rotary_pos_emb = self.gather(rotary_pos_emb, batch_gather_position, 0) # Run encoder. hidden_states = self.encoder( inputs_embeds, full_attention_mask, rotary_pos_emb=rotary_pos_emb, init_reset=init_reset, batch_valid_length=batch_valid_length) return hidden_states @MindFormerRegister.register(MindFormerModuleType.MODELS) class ChatGLM2ForConditionalGeneration(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GLMConfig): The config of ChatGLM2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['glm2'] def __init__(self, config: ChatGLM2Config, **kwargs): super(ChatGLM2ForConditionalGeneration, self).__init__(config, **kwargs) self.max_seq_len = config.max_length self.transformer = ChatGLM2Model(config=config) self.cast = P.Cast() self.gather = P.Gather() self.is_first_iteration = True self.loss = CrossEntropyLoss(parallel_config=config.parallel_config) self.gmask = config.gmask_token_id self.bos_token_id = config.bos_token_id self.use_past = config.use_past self.is_first_iteration = True self.not_equal = P.NotEqual() self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): """prepare inputs for generation.""" input_position = kwargs.get("current_index", None) if input_position is not None: input_position = Tensor(input_position, mstype.int32) return { "input_ids": Tensor(input_ids, mstype.int32), "input_position": input_position } def construct(self, input_ids=None, labels=None, position_ids=None, attention_mask=None, inputs_embeds=None, input_position=None, init_reset=True, batch_valid_length=None): """ChatGLM2 for conditional generation model.""" # input_ids: (bs, seq_len) # position_ids: (bs, seq_len) # attention_mask: (bs, seq_len) hidden_states = self.transformer( input_ids=input_ids, position_ids=position_ids, attention_mask=attention_mask, inputs_embeds=inputs_embeds, input_position=input_position, init_reset=init_reset, batch_valid_length=batch_valid_length, ) lm_logits = self.transformer.output_layer(hidden_states) if labels is not None: logits = self.cast(lm_logits, mstype.float32) logits_shape = logits.shape labels = labels.reshape((-1,)) logits = logits.reshape((-1, logits_shape[-1])) input_mask = self.not_equal(labels, -100).astype(logits.dtype) input_mask = input_mask.reshape((-1,)) loss = self.loss(logits, labels, input_mask) return loss lm_logits = lm_logits.reshape((-1, lm_logits.shape[-1])) if (not self.use_past or self.is_first_iteration) and input_position is not None: lm_logits = self.gather(lm_logits, input_position, 0) return (lm_logits,) @MindFormerRegister.register(MindFormerModuleType.MODELS) class ChatGLM2WithLora(ChatGLM2ForConditionalGeneration): """ChatGLM2 Model for pretraining with LoRA Args: config (ChatGLM2Config): The config of network. """ def __init__(self, config: ChatGLM2Config = None, **kwargs): _ = kwargs ckpt_cfg = config.checkpoint_name_or_path config.checkpoint_name_or_path = None super().__init__(config) # get Pet tuning model. config.pet_config.reg_rules = r'.*query_key_value*' self.transformer = LoraAdapter.get_pet_model(self.transformer, config.pet_config) if ckpt_cfg: config.checkpoint_name_or_path = ckpt_cfg self.load_checkpoint(config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, config.pet_config.pet_type) pengcheng.mind-test/mindformers/models/glm2/glm2_config.py000066400000000000000000000105171471400134000241260ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 config""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.modules.transformer.transformer import default_transformer_config from ..base_config import BaseConfig from ..utils import convert_mstype from ...mindformer_book import MindFormerBook __all__ = ['ChatGLM2Config'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class ChatGLM2Config(BaseConfig): """ ChatGLM2 model config class. """ _support_list = MindFormerBook.get_config_support_list()['glm2'] def __init__(self, batch_size=1, # only for incremental infer num_layers=28, padded_vocab_size=65024, hidden_size=4096, ffn_hidden_size=13696, kv_channels=128, num_attention_heads=32, seq_length=2048, hidden_dropout=0.0, attention_dropout=0.0, layernorm_epsilon=1e-5, rmsnorm=True, apply_residual_connection_post_layernorm=False, post_layer_norm=True, add_bias_linear=False, add_qkv_bias=True, bias_dropout_fusion=True, multi_query_attention=True, multi_query_group_num=2, apply_query_key_layer_scaling=True, attention_softmax_in_fp32=True, fp32_residual_connection=False, quantization_bit=0, pre_seq_len=None, prefix_projection=False, param_init_type: str = "float16", compute_dtype: str = "float16", layernorm_compute_type: str = "float32", use_past=False, eos_token_id=2, pad_token_id=0, repetition_penalty=1.0, parallel_config=default_transformer_config, **kwargs): super().__init__(**kwargs) self.batch_size = batch_size self.num_layers = num_layers self.vocab_size = padded_vocab_size self.padded_vocab_size = padded_vocab_size self.hidden_size = hidden_size self.ffn_hidden_size = ffn_hidden_size self.kv_channels = kv_channels self.num_attention_heads = num_attention_heads self.seq_length = seq_length self.hidden_dropout = hidden_dropout self.attention_dropout = attention_dropout self.layernorm_epsilon = layernorm_epsilon self.rmsnorm = rmsnorm self.apply_residual_connection_post_layernorm = apply_residual_connection_post_layernorm self.post_layer_norm = post_layer_norm self.add_bias_linear = add_bias_linear self.add_qkv_bias = add_qkv_bias self.bias_dropout_fusion = bias_dropout_fusion self.multi_query_attention = multi_query_attention self.multi_query_group_num = multi_query_group_num self.apply_query_key_layer_scaling = apply_query_key_layer_scaling self.attention_softmax_in_fp32 = attention_softmax_in_fp32 self.fp32_residual_connection = fp32_residual_connection self.quantization_bit = quantization_bit self.pre_seq_len = pre_seq_len self.prefix_projection = prefix_projection self.param_init_type = convert_mstype(param_init_type) self.compute_dtype = convert_mstype(compute_dtype) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.use_past = use_past self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.repetition_penalty = repetition_penalty self.parallel_config = parallel_config pengcheng.mind-test/mindformers/models/glm2/glm2_modules.py000066400000000000000000000143621471400134000243330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 Modules.""" import numpy as np import mindspore.common.dtype as mstype import mindspore.ops.operations as P from mindspore import nn, Parameter, Tensor from mindspore.common.initializer import initializer from mindformers.modules.layers import Linear from .glm2_config import ChatGLM2Config def precompute_rotary_emb_cache(seq_len: int, dim: int, dtype=np.float32, base: int = 10000): """pre compute rotary emb cache.""" # $\Theta = {\theta_i = 10000^{\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$ theta = 1.0 / (base ** (np.arange(0, dim, 2, dtype=dtype) / dim)) # Create position indexes `[0, 1, ..., seq_len - 1]` seq_idx = np.arange(seq_len, dtype=dtype) # Calculate the product of position index and $\theta_i$ idx_theta = np.outer(seq_idx, theta).astype(np.float32) cache = np.stack((np.cos(idx_theta), np.sin(idx_theta)), axis=-1).astype(dtype) return cache class ChatGLM2RMSNorm(nn.Cell): r""" A self-defined RMSNorm operation using reduce mean. Args: dim (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. param_init_type: The param init type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, dim, eps=1e-6, param_init_type=mstype.float32): super(ChatGLM2RMSNorm, self).__init__() self.eps = Tensor(float(eps), dtype=param_init_type) self.weight = Parameter(initializer('ones', (dim,), dtype=param_init_type)) self.square = P.Square() self.mean = P.ReduceMean(keep_dims=True) self.add = P.Add() self.rsqrt = P.Rsqrt() self.mul = P.Mul() self.mul2 = P.Mul() def _norm(self, x): # shard:(dp, 1, 1) norm_factor = self.square(x) norm_factor = self.mean(norm_factor, -1) norm_factor = self.add(norm_factor, self.eps) norm_factor = self.rsqrt(norm_factor) return self.mul(x, norm_factor) def construct(self, x): """Forward of RMSNorm.""" output = self._norm(x) output = self.mul2(output, self.weight) return output def shard(self, strategy): """Parallel strategy configuratiuon interface.""" self.square.shard(strategy) self.mean.shard(strategy) self.rsqrt.shard(strategy) self.add.shard((strategy[0], ())) self.mul.shard((strategy[0], strategy[0])) self.mul2.shard((strategy[0], (1,))) class ChatGLM2SiLU(nn.Cell): r""" A self-defined SwiGlu. Inputs: - **x** (Tensor) - Tensor. Outputs: Tensor. x = x * sigmod(x). """ def __init__(self): super(ChatGLM2SiLU, self).__init__() self.sigmoid = P.Sigmoid() self.mul = P.Mul() def shard(self, strategy): self.sigmoid.shard(strategy) self.mul.shard((strategy[0], strategy[0])) def construct(self, x): return self.mul(x, self.sigmoid(x)) class ChatGLM2SwiGLU(nn.Cell): """SwiGLU activation function.""" def __init__(self): super(ChatGLM2SwiGLU, self).__init__() self.split = P.Split(axis=-1, output_num=2) self.silu = ChatGLM2SiLU() self.mul = P.Mul() def construct(self, x): x0, x1 = self.split(x) return self.mul(self.silu(x0), x1) def shard(self, strategy): self.split.shard(strategy) self.silu.shard(strategy) self.mul.shard((strategy[0], strategy[0])) class ChatGLM2MLP(nn.Cell): """MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the state back into h hidden dimension. """ def __init__(self, config: ChatGLM2Config): super(ChatGLM2MLP, self).__init__() self.add_bias = config.add_bias_linear self.dense_h_to_4h = Linear( config.hidden_size, config.ffn_hidden_size * 2, has_bias=self.add_bias, param_init_type=config.param_init_type, compute_dtype=config.compute_dtype, ) self.dense_h_to_4h.shard( strategy_matmul=((config.parallel_config.data_parallel, 1), (config.parallel_config.model_parallel, 1)), strategy_bias=((config.parallel_config.data_parallel, config.parallel_config.model_parallel), (config.parallel_config.model_parallel,))) self.activation_func = ChatGLM2SwiGLU() # shard need to be checked. self.activation_func.shard(((config.parallel_config.data_parallel, 1, 1),)) # Project back to h. self.dense_4h_to_h = Linear( config.ffn_hidden_size, config.hidden_size, has_bias=self.add_bias, param_init_type=config.param_init_type, compute_dtype=config.compute_dtype, ) self.dense_4h_to_h.shard( strategy_matmul=((config.parallel_config.data_parallel, config.parallel_config.model_parallel), (1, config.parallel_config.model_parallel)), strategy_bias=((config.parallel_config.data_parallel, 1), (1,))) def construct(self, hidden_states): # [bs, seq_len, 4 * hidden_size] intermediate_parallel = self.dense_h_to_4h(hidden_states) intermediate_parallel = self.activation_func(intermediate_parallel) # [bs, seq_len, hidden_size] output = self.dense_4h_to_h(intermediate_parallel) return output pengcheng.mind-test/mindformers/models/glm2/glm2_tokenizer.py000066400000000000000000000251021471400134000246670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 Tokenizer.""" import os from typing import List, Optional, Union from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import Tokenizer from mindformers.tools.register import MindFormerModuleType, MindFormerRegister from sentencepiece import SentencePieceProcessor __all__ = ['ChatGLM2Tokenizer'] class SPTokenizer: """Tokenizer process special tokens.""" def __init__(self, model_path: str): # reload tokenizer assert os.path.isfile(model_path), model_path self.sp_model = SentencePieceProcessor(model_file=model_path) # BOS / EOS token IDs self.n_words: int = self.sp_model.vocab_size() self.bos_id: int = self.sp_model.bos_id() self.eos_id: int = self.sp_model.eos_id() self.pad_id: int = self.sp_model.unk_id() assert self.sp_model.vocab_size() == self.sp_model.get_piece_size() special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"] self.special_tokens = {} self.index_special_tokens = {} for token in special_tokens: self.special_tokens[token] = self.n_words self.index_special_tokens[self.n_words] = token self.n_words += 1 def tokenize(self, s: str, pair=None, add_special_tokens=True, **kwargs): # unused in this tokenizer. _, _, _ = pair, add_special_tokens, kwargs return self.sp_model.EncodeAsPieces(s) def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]: assert isinstance(s, str) t = self.sp_model.encode(s) if bos: t = [self.bos_id] + t if eos: t = t + [self.eos_id] return t def decode(self, t: List[int], skip_special_tokens=False, clean_up_tokenization_spaces=None, **kwargs) -> str: # unused in this tokenizer. _, _, _ = skip_special_tokens, clean_up_tokenization_spaces, kwargs return self.sp_model.decode(t) def decode_tokens(self, tokens: List[str]) -> str: text = self.sp_model.DecodePieces(tokens) return text def convert_token_to_id(self, token): """ Converts a token (str) in an id using the vocab. """ if token in self.special_tokens: return self.special_tokens[token] return self.sp_model.PieceToId(token) def convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" if index in self.index_special_tokens or index in [self.eos_id, self.bos_id, self.pad_id] or index < 0: return "" return self.sp_model.IdToPiece(index) @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class ChatGLM2Tokenizer(Tokenizer): """ Construct a ChatGLM2 tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file(str): The vocabulary file path. padding_side(str): ["left", "right"] Lower input text. Default False. **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import AutoTokenizer >>> tokenize = AutoTokenizer.from_pretrained('glm2_6b') >>> tokenize("你好") {'input_ids': [64790, 64792, 36474, 54591], 'attention_mask': [1, 1, 1, 1]} >>> from mindformers import ChatGLM2Tokenizer >>> tokenizer = ChatGLM2Tokenizer('tokenizer.model') >>> prompts = ["晚上睡不着应该怎么办"] >>> token_id = tokenizer(prompts) >>> input_ids = token_id['input_ids'] >>> print(input_ids) [[64790, 64792, 30910, 32820, 54266, 31876, 35153]] >>> response = tokenizer.decode(input_ids) >>> print(response) ['晚上睡不着应该怎么办'] Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = {"vocab_file": "tokenizer.model"} model_input_names = ["input_ids", "attention_mask", "position_ids"] _support_list = MindFormerBook.get_tokenizer_support_list()['glm2'] def __init__(self, vocab_file, bos_token='', eos_token='', end_token='', mask_token='[MASK]', gmask_token='[gMASK]', pad_token='', unk_token='', **kwargs): super().__init__(bos_token=bos_token, eos_token=eos_token, end_token=end_token, mask_token=mask_token, gmask_token=gmask_token, pad_token=pad_token, unk_token=unk_token, **kwargs) self.name = "ChatGLM2Tokenizer" self.vocab_file = vocab_file self.tokenizer = SPTokenizer(vocab_file) self.special_tokens = { "": self.tokenizer.bos_id, "": self.tokenizer.eos_id, "": self.tokenizer.pad_id } self._bos_token = bos_token self._eos_token = eos_token self._end_token = end_token self._mask_token = mask_token self._gmask_token = gmask_token def get_command(self, token): if token in self.special_tokens: return self.special_tokens[token] assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}" return self.tokenizer.special_tokens[token] @property def pad_token_id(self): return self.get_command("") @property def eos_token_id(self): return self.get_command("") @property def vocab_size(self): return self.tokenizer.n_words def get_vocab(self): """ Returns vocab as a dict """ vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def build_prompt(self, query, history=None): if history is None: history = [] prompt = "" for i, (old_query, response) in enumerate(history): prompt += "[Round {}]\n\n问：{}\n\n答：{}\n\n".format( i + 1, old_query, response) prompt += "[Round {}]\n\n问：{}\n\n答：".format(len(history) + 1, query) return prompt def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): """ Returns a tokenized string. """ return self._tokenize(text) def _tokenize(self, text, **kwargs): return self.tokenizer.tokenize(text) def _convert_token_to_id(self, token): return self.tokenizer.convert_token_to_id(token) def convert_tokens_to_ids(self, tokens: List[str]) -> list: """ Converts tokens to ids using the vocab. """ ids = [] for token in tokens: ids.append(self.tokenizer.convert_token_to_id(token)) return ids def _decode(self, token_ids: Union[int, List[int]], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs) -> str: _, _ = skip_special_tokens, clean_up_tokenization_spaces tokens = [] for token_id in token_ids: tokens.append(self._convert_id_to_token(token_id)) return self.convert_tokens_to_string(tokens) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" if index >= self.vocab_size: raise IndexError( f"The token id {index} is out of the size of vocabulary, please check your tokenizer " f"and corresponding vocabulary files.") return self.tokenizer.convert_id_to_token(index) def convert_tokens_to_string(self, tokens: List[str]) -> str: return self.tokenizer.decode_tokens(tokens) def save_vocabulary(self, save_directory, filename_prefix=None): """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. """ if os.path.isdir(save_directory): vocab_file = os.path.join( save_directory, self.vocab_files_names["vocab_file"] ) else: vocab_file = save_directory with open(self.vocab_file, 'rb') as fin: proto_str = fin.read() with open(vocab_file, "wb") as writer: writer.write(proto_str) return (vocab_file,) def get_prefix_tokens(self): prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")] return prefix_tokens def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. """ prefix_tokens = self.get_prefix_tokens() token_ids_0 = prefix_tokens + token_ids_0 if token_ids_1 is not None: token_ids_0 = token_ids_0 + token_ids_1 + [self.get_command("")] return token_ids_0 pengcheng.mind-test/mindformers/models/glm2/glm2_transformer.py000066400000000000000000000654221471400134000252300ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 Transformer.""" import math import numpy as np import mindspore.ops.functional as F import mindspore.ops.operations as P from mindspore import Parameter, Tensor, nn, ops from mindspore import dtype as mstype from mindformers.modules import LayerNorm from mindformers.modules.layers import Linear from mindformers.version_control import get_dropout from .glm2_config import ChatGLM2Config from .glm2_modules import ChatGLM2MLP, ChatGLM2RMSNorm class CoreAttention(nn.Cell): """ChatGLM2 core attention.""" def __init__(self, config: ChatGLM2Config, layer_number): super(CoreAttention, self).__init__() self.apply_query_key_layer_scaling = config.apply_query_key_layer_scaling self.attention_softmax_in_fp32 = config.attention_softmax_in_fp32 if self.apply_query_key_layer_scaling: self.attention_softmax_in_fp32 = True self.layer_number = max(1, layer_number) projection_size = config.kv_channels * config.num_attention_heads self.hidden_size_per_partition = projection_size self.hidden_size_per_attention_head = projection_size // config.num_attention_heads self.num_attention_heads_per_partition = config.num_attention_heads coeff = None self.norm_factor = math.sqrt(self.hidden_size_per_attention_head) if self.apply_query_key_layer_scaling: coeff = self.layer_number self.norm_factor *= coeff self.coeff = coeff # Strided linear layer. self.attention_dropout = get_dropout(config.attention_dropout) parallel_config = config.parallel_config self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.batch_matmul_q_k.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.softmax = nn.Softmax(axis=-1) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.compute_dtype = config.compute_dtype def construct(self, query_layer, key_layer, value_layer, attention_mask): """ calculate attention function """ # query_layer [b, heads, seq, hidden_size_per_head] # key_layer [b, heads, seq, hidden_size_per_head] # value_layer # [bs, heads, seq_len, hidden_size_per_head] # seqlen, batch, head, hidden_size if self.apply_query_key_layer_scaling: query_layer = query_layer / self.norm_factor # =================================== # Raw attention scores. [b, heads, s, s] # =================================== # [b, heads, seq_q, hidden_size_per_head] × [b, heads, seq_k, hidden_size_per_head]^T -> [b, heads, seq_q, seq_k] matmul_result = self.batch_matmul_q_k(query_layer, key_layer) # record original score dtype attention_scores_dtype = matmul_result.dtype # [b, heads, seq, seq] attention_scores = matmul_result if self.coeff is not None: attention_scores = attention_scores * self.coeff if attention_mask is None and attention_scores.shape[2] == attention_scores.shape[3]: attention_mask = ops.ones((attention_scores.shape[0], 1, attention_scores.shape[2], attention_scores.shape[3]), dtype=mstype.bool_) attention_mask.tril() attention_mask = ~attention_mask if attention_mask is not None: attention_scores = attention_scores.masked_fill(attention_mask, -10000) if self.attention_softmax_in_fp32: attention_scores = F.cast(attention_scores, mstype.float32) attention_probs = self.softmax(attention_scores) attention_probs = F.cast(attention_probs, attention_scores_dtype) attention_probs = self.attention_dropout(attention_probs) # [bs, heads, seq_q, seq_k] x [bs, heads, seq_v, hidden_size_per_head] -> [b, heads, seq_q, hidden_size_per_head] context_layer = self.batch_matmul(attention_probs, value_layer) context_layer = F.cast(context_layer, self.compute_dtype) context_layer = self._merge_heads(context_layer) return context_layer def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose(x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = x.shape new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge class ChatGLM2SelfAttention(nn.Cell): """ChatGLM2 self-attention.""" def __init__(self, config: ChatGLM2Config, layer_number): super(ChatGLM2SelfAttention, self).__init__() self.layer_number = max(1, layer_number) self.projection_size = config.kv_channels * config.num_attention_heads # Per attention head and per partition values. self.hidden_size_per_attention_head = self.projection_size // config.num_attention_heads self.num_attention_heads_per_partition = config.num_attention_heads self.params_dtype = config.param_init_type self.compute_dtype = config.compute_dtype self.batch_size = config.batch_size self.seq_length = config.seq_length self.multi_query_attention = config.multi_query_attention self.qkv_hidden_size = 3 * self.projection_size if self.multi_query_attention: self.num_multi_query_groups_per_partition = config.multi_query_group_num self.qkv_hidden_size = ( self.projection_size + 2 * self.hidden_size_per_attention_head * config.multi_query_group_num) parallel_config = config.parallel_config self.query_key_value = Linear(config.hidden_size, self.qkv_hidden_size, has_bias=config.add_bias_linear or config.add_qkv_bias, param_init_type=self.params_dtype, compute_dtype=self.compute_dtype) self.query_key_value.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,)) ) self.core_attention = CoreAttention(config, self.layer_number) self.dense = Linear(self.projection_size, config.hidden_size, has_bias=config.add_bias_linear, param_init_type=self.params_dtype, compute_dtype=self.compute_dtype) self.dense.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, 1), (1,))) self.reshape = P.Reshape() self.stack = P.Stack(axis=-1) self.gather = P.Gather() self.index_0 = Tensor(0) self.index_1 = Tensor(1) self.mul = P.Mul() self.sub = P.Sub() self.add = P.Add() self.concat = P.Concat(axis=-1) self.split_3 = P.Split(axis=-1, output_num=3) self.transpose = P.Transpose() self.use_past = config.use_past if self.use_past: self.is_first_iteration = True seq_range = np.arange(self.seq_length).reshape(1, 1, -1) self.range = Tensor( np.tile(seq_range, (self.batch_size, 1, 1)), mstype.int32) self.less = P.Less() self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.tile = P.Tile().shard(((1, 1, 1, 1),)) def apply_rotary_pos_emb(self, x: Tensor, rope_cache: Tensor) -> Tensor: """apply rotary position embedding to q,k.""" # x: [b, heads, seq, hidden_size_per_head] bs, num_heads, seq_len, _ = x.shape # 1, 32，4, 128 # rope_cache: first (seq_len, kv_channels//4, 2), other (1, kv_channels//4, 2) rot_dim = rope_cache.shape[-2] * 2 # kv_channels // 2 x, x_pass = x[..., :rot_dim], x[..., rot_dim:] # ms not support variable sizes # truncate to support variable sizes # rope_cache = rope_cache[:sq] # [bs, nh, sq, kv_channels//4, 2] xshaped = self.reshape(x, (bs, num_heads, seq_len, rot_dim // 2, 2)) # [bs, 1, sq, kv_channels//4, 2] rope_cache = self.reshape(rope_cache, (-1, 1, seq_len, xshaped.shape[3], 2)) xshaped_0, xshaped_1 = ops.split(xshaped, 1, -1) rope_cache_0, rope_cache_1 = ops.split(rope_cache, 1, -1) x_out1 = self.sub(self.mul(xshaped_0, rope_cache_0), self.mul(xshaped_1, rope_cache_1)) x_out2 = self.add(self.mul(xshaped_1, rope_cache_0), self.mul(xshaped_0, rope_cache_1)) x_out = self.stack((x_out1, x_out2)) x_out = self.reshape(x_out, (x_out.shape[0], x_out.shape[1], x_out.shape[2], -1)) # [bs, sq, nh, hidden_size_per_head] return self.concat((x_out, x_pass)) def construct(self, hidden_states, attention_mask, rotary_pos_emb, key_past=None, value_past=None, batch_valid_length=None): """Forward process of self-attention.""" # hidden_states: [bs, seq_len, hidden_size] # attention_mask (bs, 1, seq_len, seq_len) # rotary_pos_emb: first: (sen length, kv_channels//4, 2)， after:(1, kv_channels//4, 2] # [bs, seq_len, qkv_hidden_size] mixed_raw_layer = self.query_key_value(hidden_states) # not compatible with ms below 2.0 if self.multi_query_attention: (query_layer, key_layer, value_layer) = mixed_raw_layer.split( [self.num_attention_heads_per_partition * self.hidden_size_per_attention_head, self.num_multi_query_groups_per_partition * self.hidden_size_per_attention_head, self.num_multi_query_groups_per_partition * self.hidden_size_per_attention_head, ], axis=-1, ) # [bs, seq_len, nh, hidden_size_per_attention_head] -> [bs, nh, seq_len, hidden_size_per_attention_head] query_layer = query_layer.view( query_layer.shape[:-1] + (self.num_attention_heads_per_partition, self.hidden_size_per_attention_head) ) query_layer = self.transpose(query_layer, (0, 2, 1, 3)) # [bs, seq_len, multi_query_groups, hidden_size_per_attention_head] # -> [bs, multi_query_groups, seq_len, hidden_size_per_attention_head] key_layer = key_layer.view( key_layer.shape[:-1] + (self.num_multi_query_groups_per_partition, self.hidden_size_per_attention_head) ) key_layer = self.transpose(key_layer, (0, 2, 1, 3)) # [bs, seq_len, multi_query_groups, hidden_size_per_attention_head] # -> [bs, multi_query_groups, seq_len, hidden_size_per_attention_head] value_layer = value_layer.view( value_layer.shape[:-1] + (self.num_multi_query_groups_per_partition, self.hidden_size_per_attention_head) ) value_layer = self.transpose(value_layer, (0, 2, 1, 3)) else: # [b, seq, (heads * 3 * hidden_size_per_head)] --> [b, seq, heads, 3 * hidden_size_per_head] new_tensor_shape = mixed_raw_layer.shape[:-1] + ( self.num_attention_heads_per_partition, 3 * self.hidden_size_per_attention_head, ) mixed_raw_layer = mixed_raw_layer.view(*new_tensor_shape) # [b, seq, heads, hidden_size_per_head] (query_layer, key_layer, value_layer) = self.split_3(mixed_raw_layer) # [b, seq, heads, hidden_size_per_head] -> [bs, num_heads, seq_len, hidden_size_per_head] query_layer = self.transpose(query_layer, (0, 2, 1, 3)) key_layer = self.transpose(key_layer, (0, 2, 1, 3)) value_layer = self.transpose(value_layer, (0, 2, 1, 3)) # rotary_pos_emb: first: (seq_length, kv_channels//4, 2)， after:(1, kv_channels//4, 2) if rotary_pos_emb is not None: # [b, heads, seq, hidden_size_per_head] query_layer = self.apply_rotary_pos_emb(query_layer, rotary_pos_emb) # [bs, multi_query_groups, seq_len, hidden_size_per_attention_head] key_layer = self.apply_rotary_pos_emb(key_layer, rotary_pos_emb) # key and value for current token(s) # [bs, heads, seq_len, hidden_size_per_head] key_present = key_layer value_present = value_layer if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.params_dtype) # [bs, 1, seq_len] # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key_present, self.expand_dims(valid_length_vector, 3)) value_present = self.mul1(value_present, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, multi_query_groups, 1, size_per_head) # the shape of value is (bs, multi_query_groups, 1, size_per_head) else: # Get the current token position index # key_past: [batch_size, multi_query_groups, seq_length, size_per_head] valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) # [bs, 1, 1] # self.range: [bs, 1, config.seq_len] valid_length_vector = F.cast(self.equal(valid_length, self.range), self.params_dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key_present, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) current_value = self.mul1(self.tile(value_present, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state # [batch_size, multi_query_groups, seq_length, size_per_head] key_present = self.add(key_past, current_key) value_present = self.add(value_past, current_value) # update k v for attention # [batch_size, multi_query_groups, seq_length, size_per_head] key_layer = key_present # [batch_size, multi_query_groups, seq_length, size_per_head] value_layer = value_present layer_present = (key_present, value_present) # tile k,v to num_heads if self.multi_query_attention: bs, heads, _, hs_ph = key_layer.shape key_layer = key_layer.view((bs, heads, -1)) key_layer = key_layer.unsqueeze(2) key_layer = key_layer.tile( (1, 1, self.num_attention_heads_per_partition // self.num_multi_query_groups_per_partition, 1)) # [b, heads, seq, hidden_size_per_head] key_layer = key_layer.view((bs, self.num_attention_heads_per_partition, -1, hs_ph)) value_layer = value_layer.view((bs, heads, -1)) value_layer = value_layer.unsqueeze(2) value_layer = value_layer.tile( (1, 1, self.num_attention_heads_per_partition // self.num_multi_query_groups_per_partition, 1)) # [b, heads, seq, hidden_size_per_head] value_layer = value_layer.view((bs, self.num_attention_heads_per_partition, -1, hs_ph)) context_layer = self.core_attention(query_layer, key_layer, value_layer, attention_mask) # # ================= # # Output. [bs, seq_len, hidden_size] # # ================= output = self.dense(context_layer) return output, layer_present class ChatGLM2Block(nn.Cell): """A single transformer layer. Transformer layer takes input with size [s, b, h] and returns an output of the same size. """ def __init__(self, config: ChatGLM2Config, layer_number: int): super(ChatGLM2Block, self).__init__() self.layer_number = layer_number self.apply_residual_connection_post_layernorm = config.apply_residual_connection_post_layernorm self.fp32_residual_connection = config.fp32_residual_connection self.use_past = config.use_past self.params_dtype = config.param_init_type self.layernorm_dtype = config.layernorm_compute_type self.compute_dtype = config.compute_dtype self.seq_length = config.seq_length layer_norm_func = ChatGLM2RMSNorm if config.rmsnorm else LayerNorm # Layernorm on the input data. self.input_layernorm = layer_norm_func(config.hidden_size, eps=config.layernorm_epsilon, param_init_type=self.layernorm_dtype) self.input_layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) # Self attention. self.self_attention = ChatGLM2SelfAttention(config, layer_number) self.hidden_dropout = config.hidden_dropout # Layernorm on the attention output self.post_attention_layernorm = layer_norm_func(config.hidden_size, eps=config.layernorm_epsilon, param_init_type=self.layernorm_dtype) # self.post_attention_layernorm.shard() # MLP self.mlp = ChatGLM2MLP(config) self.dropout = get_dropout(self.hidden_dropout) self.dropout.dropout.shard(((config.parallel_config.data_parallel, 1, 1),)) self.cast = P.Cast() self.key_past = None self.value_past = None if self.use_past: size_per_head = config.hidden_size // config.num_attention_heads kv_num_partition = config.num_attention_heads if config.multi_query_attention: kv_num_partition = config.multi_query_group_num kv_shape = (config.batch_size, kv_num_partition, self.seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter( Tensor(np.zeros(shape=kv_shape), self.params_dtype), name="key_past") self.value_past = Parameter( Tensor(np.zeros(shape=kv_shape), self.params_dtype), name="value_past") self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) def construct(self, hidden_states, attention_mask, rotary_pos_emb, init_reset=True, batch_valid_length=None): """Forward process of the transformer layer.""" # hidden_states: [bs, seq_len, hidden_size] # attention_mask first: (bs, 1, seq_len, seq_len), after: (bs, 1, 1, seq_len) # rotary_pos_emb: first: (seq_len, kv_channels//4, 2)， after: (1, kv_channels//4, 2) # Layer norm at the beginning of the transformer layer. hidden_states = self.cast(hidden_states, self.layernorm_dtype) layernorm_output = self.input_layernorm(hidden_states) # fp32 -> fp16 layernorm_output = self.cast(layernorm_output, self.compute_dtype) key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul( self.key_past, F.cast(init_reset, self.params_dtype))) value_reset = self.assign(self.value_past, self.mul( self.value_past, F.cast(init_reset, self.params_dtype))) # add dependency for desired execution order layernorm_output = F.depend(layernorm_output, key_reset) layernorm_output = F.depend(layernorm_output, value_reset) # Self attention. attention_output, layer_present = self.self_attention( layernorm_output, attention_mask, rotary_pos_emb, self.key_past, self.value_past, batch_valid_length ) # Residual connection. # False on default. if self.apply_residual_connection_post_layernorm: residual = layernorm_output else: residual = hidden_states layernorm_input = self.dropout(attention_output) layernorm_input = residual + layernorm_input # Layer norm post the self attention. layernorm_output = self.post_attention_layernorm(layernorm_input) layernorm_output = self.cast(layernorm_output, self.compute_dtype) # MLP. mlp_output = self.mlp(layernorm_output) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_output = F.depend(mlp_output, value_update) mlp_output = F.depend(mlp_output, key_update) # Second residual connection. # False on default. if self.apply_residual_connection_post_layernorm: residual = layernorm_output else: residual = layernorm_input output = self.dropout(mlp_output) output = residual + output return output def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: # we give the fusion in pipeline mode a fixed value, otherwise the performance may become worse. network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class ChatGLM2Transformer(nn.Cell): """Transformer class.""" def __init__(self, config: ChatGLM2Config): super(ChatGLM2Transformer, self).__init__() self.post_layer_norm = config.post_layer_norm self.compute_dtype = config.compute_dtype # Number of layers. self.num_layers = config.num_layers # Transformer layers. def build_layer(layer_number): return ChatGLM2Block(config, layer_number) self.layers = nn.CellList() for i in range(self.num_layers): layer = build_layer(i + 1) set_parallel_configure_for_layer(layer, layer_id=i, layers=self.num_layers, offset=0, parallel_config=config.parallel_config) self.layers.append(layer) if self.post_layer_norm: layer_norm_func = ChatGLM2RMSNorm if config.rmsnorm else LayerNorm # Final layer norm before output. self.final_layernorm = layer_norm_func(config.hidden_size, eps=config.layernorm_epsilon, param_init_type=config.layernorm_compute_type) # self.final_layernorm.shard() self.final_layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) def construct(self, hidden_states, attention_mask, rotary_pos_emb, init_reset=True, batch_valid_length=None): """Forward process of the transformer.""" # hidden_states (bs, seq_len, hs) # attention_mask (bs, 1, seq_len, seq_len) # rotary_pos_emb: first: (sen length, kv_channels//2, 2)， after:[1, kv_channels // 2, 2] for layer in self.layers: hidden_states = layer( hidden_states, attention_mask, rotary_pos_emb, init_reset=init_reset, batch_valid_length=batch_valid_length, ) # Final layer norm. if self.post_layer_norm: hidden_states = self.final_layernorm(hidden_states) hidden_states = self.cast(hidden_states, self.compute_dtype) return hidden_states pengcheng.mind-test/mindformers/models/gpt2/000077500000000000000000000000001471400134000213755ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/gpt2/.ipynb_checkpoints/000077500000000000000000000000001471400134000251665ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/gpt2/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000020121471400134000314170ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for gpt""" from .gpt2_config import GPT2Config from .gpt2 import GPT2Model, GPT2LMHeadModel, GPT2ForSequenceClassification from .gpt2_tokenizer import GPT2Tokenizer from .gpt2_processor import GPT2Processor __all__ = ['GPT2Config', 'GPT2Model', 'GPT2LMHeadModel', 'GPT2ForSequenceClassification', 'GPT2Tokenizer', 'GPT2Processor'] pengcheng.mind-test/mindformers/models/gpt2/.ipynb_checkpoints/gpt2-checkpoint.py000066400000000000000000000627671471400134000305630ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GPT model""" import copy import numpy as np import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.modules.transformer.moe import default_moe_config from mindformers.modules.layers import LayerNorm, Dropout, Linear from mindformers.core.loss import CrossEntropyLoss from mindformers.modules.transformer import AttentionMask, VocabEmbedding from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.mindformer_book import MindFormerBook from mindformers.tools.logger import logger #from mindformers.pet import LoraAdapter, PetAdapter from .gpt2_config import GPT2Config from .gpt_modules import GPTTransformerDecoderLayer __all__ = ['GPT2LMHeadModel', 'GPT2WithLora', 'GPT2ForSequenceClassification', 'GPT2Model', 'GPTHead'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2LMHeadModel(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['gpt2'] def __init__(self, config: GPT2Config = None): config = config if config is not None else GPT2Config() super(GPT2LMHeadModel, self).__init__(config, auto_prefix=True) self.eos_token_id = self.config.eos_token_id parallel_config = self.config.parallel_config self.stridedslice = P.StridedSlice().shard(((parallel_config.data_parallel, 1),)) self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.get_attention_mask = AttentionMask(seq_length=config.seq_length, parallel_config=parallel_config.dp_mp_config) self.backbone = GPT2Model(config) self.head = GPTHead(hidden_size=config.hidden_size, vocab_size=config.vocab_size, parallel_config=self.config.parallel_config) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) mp = config.parallel_config.model_parallel vocab_size = config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of GPT Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPT Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = CrossEntropyLoss(parallel_config=loss_parallel_config) self.reshape = P.Reshape() self.cast = P.Cast() self.load_checkpoint(config) self.add = P.Add().shard(((parallel_config.data_parallel, 1), ())) self.tile = P.Tile() self.is_first_iteration = True self.all_ones_attention_mask = P.Ones()((1, 1, 1), mstype.float32) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 def construct(self, input_ids, attention_mask=None, input_embeds=None, labels=None, input_position=None, position_ids=None, init_reset=True, batch_valid_length=None): r""" construct function for Language Modeling Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. attention_mask (Tensor): input sentences padding mask, where 0 indicates padding position with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. labels (Tensor): the labels of inputs with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. input_position (Tensor): the position ids of inputs (at incremental reasoning mode) which is an increasing sequence with data type int64/int32, Tensor :math:`(bacth, seq\_length)`. position_ids (Tensor): the position ids of inputs which is an increasing sequence with data type int64/int32, Tensor :math:`(bacth, seq\_length)`. inputs_embeds (Tensor): the embedding of inputs with data type float32/float16, Tensor of shape :math:`(batch, seq\_length, hidden_size) init_reset (bool): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. batch_valid_length (Tensor): Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Returns: logits (Tensor) or loss (mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ if attention_mask is None: attention_mask = self.not_equal(input_ids, self.eos_token_id) batch_size, seq_length = input_ids.shape attention_mask = self.cast(attention_mask, mstype.float32) loss_mask = attention_mask if not self.training: tokens = input_ids else: tokens = self.stridedslice(input_ids, (0, 0), (batch_size, seq_length - 1), (1, 1)) attention_mask = self.stridedslice(attention_mask, (0, 0), (batch_size, seq_length - 1), (1, 1)) attention_mask = self.get_attention_mask(attention_mask) if not self.is_first_iteration: attention_mask = self.tile(self.all_ones_attention_mask, (batch_size, 1, 1)) # [batch_size, seq_length, vocab_size] output_states, embedding_table = self.backbone(tokens, attention_mask, input_position, init_reset, batch_valid_length) logits = self.head(output_states, embedding_table) if not self.training: logits = self.reshape(logits, (batch_size, seq_length, -1)) # makes cast effective to avoid allgather issue in Mindspore1.10 loss_mask = self.add(loss_mask, 1) return logits, tokens, loss_mask loss_mask = self.stridedslice(loss_mask, (0, 1), (batch_size, seq_length), (1, 1)) labels = self.stridedslice(input_ids, (0, 1), (batch_size, seq_length), (1, 1)) labels = self.reshape(labels, (-1,)) loss_mask = self.reshape(loss_mask, (-1,)) loss = self.loss(logits, labels, loss_mask) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2ForSequenceClassification(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['gpt2'] def __init__(self, config: GPT2Config = None): self.config = config if config is not None else GPT2Config() super(GPT2ForSequenceClassification, self).__init__(self.config, auto_prefix=True) self.eos_token = self.config.eos_token self.seq_length = self.config.seq_length self.num_labels = self.config.num_labels self.hidden_size = self.config.hidden_size parallel_config = self.config.parallel_config dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.get_attention_mask = AttentionMask(seq_length=self.seq_length, parallel_config=parallel_config.dp_mp_config) self.backbone = GPT2Model(self.config) self.score = Linear(in_channels=self.hidden_size, out_channels=self.num_labels, has_bias=False, compute_dtype=self.config.compute_dtype) self.score.shard(strategy_matmul=((dp, 1), (1, 1))) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) vocab_size = self.config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of GPT Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPT Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = nn.CrossEntropyLoss() self.reshape = P.Reshape() self.cast = P.Cast() self.load_checkpoint(config) self.reduce_sum = P.ReduceSum().shard(((dp, 1),)) self.add = P.Add().shard(((dp,), (dp,))) self.sub = P.Sub().shard(((1,), ())) self.gather = P.Gather().shard(((1, 1), (1,))) # pylint: disable=W0613 def construct(self, input_ids, attention_mask, labels=None, input_embeds=None, input_position=None, position_ids=None, init_reset=True, batch_valid_length=None): r""" construct function for GPT2 Text Classification Model Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary. attention_mask (Tensor): input sentences padding mask, where 0 indicates padding position. labels (Tensor): the labels of corresponding input sequences. input_embeds(Tensor): Reserved param, not used. input_position(Tensor): Reserved param, not used. position_ids(Tensor): Reserved param, not used. init_reset(Tensor): Reserved param, not used. batch_valid_length(Tensor): Reserved param, not used. Returns: (logits, labels) (Tensor, Tensor) or logits (Tensor) or loss (mstype.float32): in train mode, return loss; in eval mode, return logits and loss; in predict mode, return logits. """ attention_mask = self.cast(attention_mask, mstype.float32) attention_mask_lower_triangle = self.get_attention_mask(attention_mask) output_states, _ = self.backbone(input_ids, attention_mask_lower_triangle) output_states = self.reshape(output_states, (-1, self.hidden_size)) logits = self.score(output_states) # get the last logit of each sequence last_indices = self.sub(self.reduce_sum(attention_mask, -1), 1) batch_size = attention_mask.shape[0] indices_increments = Tensor(np.arange(0, self.seq_length * batch_size, self.seq_length)) last_indices = self.cast(self.add(last_indices, indices_increments), mstype.int32) pooled_logits = self.gather(logits, last_indices, 0) if labels is not None: labels = self.reshape(labels, (-1,)) if self.training: output = self.loss(pooled_logits, labels) else: if labels is not None: output = (pooled_logits, labels) else: output = pooled_logits return output class GPTEmbeddingLayer(nn.Cell): r"""The Embedding Layer of GPT-2 network.""" def __init__(self, config: GPT2Config = None): super(GPTEmbeddingLayer, self).__init__() parallel_config = copy.deepcopy(config.parallel_config) embedding_mp = config.parallel_config.embedding_dp_mp_config.model_parallel vocab_size = config.vocab_size if vocab_size % embedding_mp != 0: logger.warning("The vocab size of embedding layer is: %s, it is not divide by model_parallel: %s", vocab_size, embedding_mp) logger.warning("Now, model_parallel will be changed: mp = 1") parallel_config.embedding_dp_mp_config.model_parallel = 1 self.word_embedding = VocabEmbedding(vocab_size=vocab_size, embedding_size=config.hidden_size, param_init=initializer('normal', [vocab_size, config.hidden_size], dtype=mstype.float32), parallel_config=parallel_config.embedding_dp_mp_config) new_parallel_config = copy.deepcopy(parallel_config) new_parallel_config.vocab_emb_dp = True self.position_embedding = VocabEmbedding(vocab_size=config.seq_length, embedding_size=config.hidden_size, param_init=initializer('normal', [config.seq_length, config.hidden_size], dtype=mstype.float32), parallel_config=new_parallel_config.embedding_dp_mp_config) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dropout = Dropout(1 - config.embedding_dropout_prob) self.dropout.shard(((parallel_config.data_parallel, 1, 1),)) def construct(self, input_ids, input_position): """The forward compute of Embedding Layer.""" word_embedding, word_table = self.word_embedding(input_ids) position_embedding, _ = self.position_embedding(input_position) embedding = self.add(word_embedding, position_embedding) embedding = self.dropout(embedding) return embedding, word_table def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ pp_dis = max(int((layers + 1) / parallel_config.pipeline_stage), 1) pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class GPT2Model(nn.Cell): """ The backbone of GPT network Args: config(GPT2Config): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 input_mask: the mask indicating whether each position is a valid input Returns: output_state: Tensor, the output logit of backbone present_layer: Tensor, the current feature map embedding_table: Tensor, the embedding table for the vocabulary """ def __init__(self, config): super(GPT2Model, self).__init__() self.config = config self.embedding = GPTEmbeddingLayer(config) self.embedding.pipeline_stage = 0 self.layernorm = LayerNorm((config.hidden_size,)).to_float(config.layernorm_compute_type) if config.parallel_config.pipeline_stage > 1: self.layernorm.set_comm_fusion(2) else: self.layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.layernorm.pipeline_stage = config.parallel_config.pipeline_stage - 1 if not hasattr(config.parallel_config, "moe_config"): config.parallel_config.moe_config = default_moe_config moe_config = config.parallel_config.moe_config self.blocks = nn.CellList() for i in range(config.num_layers): block = GPTTransformerDecoderLayer( hidden_size=config.hidden_size, batch_size=config.batch_size, ffn_hidden_size=config.hidden_size * config.expand_ratio, seq_length=config.seq_length, num_heads=config.num_heads, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, hidden_act=config.hidden_act, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, parallel_config=config.parallel_config.dp_mp_config, moe_config=moe_config, use_past=config.use_past ) set_parallel_configure_for_layer( block, layer_id=i, layers=config.num_layers, offset=0, parallel_config=config.parallel_config) self.blocks.append(block) self.cast = P.Cast() self.tile = P.Tile().shard(((config.parallel_config.data_parallel,),)) self.dtype = mstype.float16 self.num_layers = config.num_layers self.position_ids = Tensor(np.arange(config.seq_length), mstype.int32) self.is_first_iteration = True def construct(self, input_ids, attention_mask, input_position=None, init_reset=True, batch_valid_length=None): """GPT model""" batch_size, seq_length = F.shape(input_ids) # When input_position is None, the phase is train mode. When the phase is train mode and is the first iteration # of incremental reasoning, it goes into the following logic. if input_position is None or self.is_first_iteration: if batch_size == 1: input_position = F.reshape(self.position_ids, (1, seq_length)) else: input_position = self.tile(self.position_ids, (batch_size, 1)) # when the phase is not train and incremental reasoning is not the first iteration, it goes into the # following logic. else: bias = Tensor(np.arange(batch_size) * self.config.seq_length, mstype.int32) input_position = F.sub(input_position, bias) input_position = F.reshape(input_position, (batch_size, 1)) input_embedding, embedding_table = self.embedding(input_ids, input_position) hidden_states = self.cast(input_embedding, self.dtype) hidden_shape = F.shape(hidden_states) hidden_states = F.reshape(hidden_states, (-1, hidden_shape[-1])) for i in range(self.num_layers): hidden_states = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) output_state = self.layernorm(hidden_states) return output_state, embedding_table class GPTHead(nn.Cell): r"""Head for GPT to get the logits of each token in the vocab.""" def __init__(self, hidden_size, vocab_size, compute_type=mstype.float16, parallel_config=None): super().__init__() copied_parallel_config = copy.deepcopy(parallel_config) mp = copied_parallel_config.model_parallel if vocab_size % mp != 0: logger.warning("The vocab size of GPTHead MatMul is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPTHead MatMul will be changed: mp = 1") copied_parallel_config.model_parallel = 1 if copied_parallel_config.pipeline_stage > 1: copied_parallel_config.vocab_emb_dp = False if copied_parallel_config.vocab_emb_dp: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), ( copied_parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = compute_type self.cast = P.Cast() def construct(self, state, embedding_table): logits = self.matmul(self.cast(state, self.dtype), self.cast(embedding_table, self.dtype)) return logits @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2WithLora(GPT2LMHeadModel): """ GPT2LMHeadModel with LoRA parameter-efficient tuning algorithm Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ def __init__(self, config: GPT2Config = None): checkpoint_name_or_path = config.pop("checkpoint_name_or_path") super().__init__(config) config.pet_config.reg_rules = r'.*dense1.*|.*dense3.*' self.backbone = LoraAdapter.get_pet_model(self.backbone, config.pet_config) config.checkpoint_name_or_path = checkpoint_name_or_path self.load_checkpoint(config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, config.pet_config.pet_type) class CrossEntropyCalculationWithMask(nn.Cell): """ Cross Entropy loss """ def __init__(self, is_training=None, num_labels=None): super(CrossEntropyCalculationWithMask, self).__init__() self.onehot = P.OneHot() self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_sum = P.ReduceSum() self.reduce_mean = P.ReduceMean() self.reshape = P.Reshape() self.last_idx = (-1,) self.neg = P.Neg() self.cast = P.Cast() self.is_training = is_training self.num_labels = num_labels self.log_softmax = P.LogSoftmax(axis=-1) def construct(self, logits, label_ids, input_mask=None): """ Calculate loss Args: logits (Tensor): the probability distribution over vocabulary. label_ids (Tensor): the indices of input sequence tokens in the vocabulary. input_mask (Tensor): input sentences padding mask, where 0 indicates padding position. Returns: return_value (Tensor, mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ # logits [batch * (seq_length-1), vocab_size] label_ids [batch, seq_length-1] logits = self.log_softmax(logits) if self.is_training: label_ids = self.reshape(label_ids, self.last_idx) # label_ids [batch * (seq_length-1)] one_hot_labels = self.onehot(label_ids, self.num_labels, self.on_value, self.off_value) # [batch * (seq_length-1), vocab_size] per_example_loss = self.neg( self.reduce_sum(one_hot_labels * logits, self.last_idx)) # [batch * (seq_length-1)] # for PPL calculation in evaluation if input_mask is not None: input_mask = self.cast(self.reshape(input_mask, self.last_idx), mstype.float32) # [batch * (seq_length-1)] valid_loss_sum = self.reduce_sum(input_mask * per_example_loss, ()) valid_element_sum = self.reduce_sum(input_mask, ()) + self.cast(F.tuple_to_array((1e-5,)), mstype.float32) loss = valid_loss_sum / valid_element_sum else: loss = self.reduce_mean(per_example_loss, self.last_idx) # a number return_value = self.cast(loss, mstype.float32) else: return_value = logits * 1.0 # [batch * (seq_length-1), vocab_size] return return_value pengcheng.mind-test/mindformers/models/gpt2/__init__.py000066400000000000000000000020121471400134000235010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for gpt""" from .gpt2_config import GPT2Config from .gpt2 import GPT2Model, GPT2LMHeadModel, GPT2ForSequenceClassification from .gpt2_tokenizer import GPT2Tokenizer from .gpt2_processor import GPT2Processor __all__ = ['GPT2Config', 'GPT2Model', 'GPT2LMHeadModel', 'GPT2ForSequenceClassification', 'GPT2Tokenizer', 'GPT2Processor'] pengcheng.mind-test/mindformers/models/gpt2/convert_weight.py000066400000000000000000000160541471400134000250040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/huggingface""" import argparse import numpy as np import torch from mindspore import save_checkpoint, Tensor def generate_params_dict(total_layers, mindspore_params_per_layer, torch_params_per_layer, mindspore_additional_params, torch_additional_params): """ Generate the total parameter mapping of mindspore and pytorch. Args: total_layers(int): The total layers of the net. mindspore_params_per_layer(list): The list of params per layer for the net of mindspore. torch_params_per_layer(list): The list of params per layer for the net of pytorch. mindspore_additional_params(list): The list of params outside the layer for the net of mindspore torch_additional_params(list): The list of params outside the layer for the net of pytorch. Returns: A list of tuple. The first element is the parameter name of mindspore, the another is the parameter name of pytorch. """ mapped_params = list(zip(mindspore_params_per_layer, torch_params_per_layer)) ms_extend_param_list = [] torch_extend_param_list = [] for i in range(total_layers): for ms_para, torch_para in mapped_params: src = ms_para.format(i) tgt = torch_para.format(i) ms_extend_param_list.append(src) torch_extend_param_list.append(tgt) mapped_params = list(zip(mindspore_additional_params, torch_additional_params)) for ms_para, torch_para in mapped_params: ms_extend_param_list.append(ms_para) torch_extend_param_list.append(torch_para) return list(zip(ms_extend_param_list, torch_extend_param_list)) def print_dict(input_dict): """ Print the keys and values of input dict Args: input_dict(dict): input dict with key and value. Returns: None """ for k, v in input_dict.items(): print(f"Param: {k} with shape {v}") def get_converted_ckpt(mapped_params, weight_dict): """ Print the keys of the loaded checkpoint Args: mapped_params(dict): The loaded checkpoint. The key is parameter name and value is the numpy array. weight_dict(dict): The loaded pytorch checkpoint. Returns: None """ new_ckpt_list = [] # Currently, the ms_extend_param the torch_extend_param is the full parameters. for src, tgt in mapped_params: value = weight_dict[tgt].numpy() # split the attention layer for q, k, v if 'c_attn.weight' in tgt: print("tgt:", tgt) value = np.transpose(value, [1, 0]) print(f"Mapping table Mindspore:{src:<30} \t Torch:{tgt:<30} with shape {value.shape}") new_ckpt_list.append({"data": Tensor(value), "name": src}) return new_ckpt_list def split_torch_attention(state): s = list(state.keys()) for name in s: if name.endswith('attn.c_attn.weight') or name.endswith('attn.c_attn.bias'): value = state.pop(name) q, k, v = np.split(value.numpy(), 3, -1) state[name + '.q'] = torch.tensor(q, dtype=value.dtype) state[name + '.k'] = torch.tensor(k) state[name + '.v'] = torch.tensor(v) if __name__ == '__main__': parser = argparse.ArgumentParser(description="OPT convert script") parser.add_argument('--layers', type=int, default=12, help="The number of layers of the model to be converted.") parser.add_argument("--torch_path", type=str, default=None, required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="The output mindspore checkpoint path.", help="Use device nums, default is 128.") opt = parser.parse_args() state_dict = torch.load(opt.torch_path, map_location='cpu') print_dict(state_dict) ms_name = [ "backbone.blocks.{}.layernorm1.gamma", "backbone.blocks.{}.layernorm1.beta", "backbone.blocks.{}.layernorm2.gamma", "backbone.blocks.{}.layernorm2.beta", "backbone.blocks.{}.attention.projection.weight", "backbone.blocks.{}.attention.projection.bias", "backbone.blocks.{}.attention.dense1.weight", "backbone.blocks.{}.attention.dense1.bias", "backbone.blocks.{}.attention.dense2.weight", "backbone.blocks.{}.attention.dense2.bias", "backbone.blocks.{}.attention.dense3.weight", "backbone.blocks.{}.attention.dense3.bias", "backbone.blocks.{}.output.mapping.weight", "backbone.blocks.{}.output.mapping.bias", "backbone.blocks.{}.output.projection.weight", "backbone.blocks.{}.output.projection.bias", ] torch_name = [ "h.{}.ln_1.weight", "h.{}.ln_1.bias", "h.{}.ln_2.weight", "h.{}.ln_2.bias", "h.{}.attn.c_proj.weight", "h.{}.attn.c_proj.bias", "h.{}.attn.c_attn.weight.q", "h.{}.attn.c_attn.bias.q", "h.{}.attn.c_attn.weight.k", "h.{}.attn.c_attn.bias.k", "h.{}.attn.c_attn.weight.v", "h.{}.attn.c_attn.bias.v", "h.{}.mlp.c_fc.weight", "h.{}.mlp.c_fc.bias", "h.{}.mlp.c_proj.weight", "h.{}.mlp.c_proj.bias" ] addition_mindspore = [ "backbone.layernorm.gamma", "backbone.layernorm.beta", "backbone.embedding.word_embedding.embedding_table", "backbone.embedding.position_embedding.embedding_table", ] addition_torch = [ "ln_f.weight", "ln_f.bias", "wte.weight", "wpe.weight", ] mapped_param = generate_params_dict(total_layers=opt.layers, mindspore_params_per_layer=ms_name, torch_params_per_layer=torch_name, mindspore_additional_params=addition_mindspore, torch_additional_params=addition_torch) split_torch_attention(state_dict) new_ckpt = get_converted_ckpt(mapped_param, state_dict) save_checkpoint(new_ckpt, opt.mindspore_path) print(f"Convert finished, the output is saved to {opt.mindspore_path}") pengcheng.mind-test/mindformers/models/gpt2/gpt2.py000066400000000000000000000627671471400134000226450ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GPT model""" import copy import numpy as np import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.modules.transformer.moe import default_moe_config from mindformers.modules.layers import LayerNorm, Dropout, Linear from mindformers.core.loss import CrossEntropyLoss from mindformers.modules.transformer import AttentionMask, VocabEmbedding from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.mindformer_book import MindFormerBook from mindformers.tools.logger import logger #from mindformers.pet import LoraAdapter, PetAdapter from .gpt2_config import GPT2Config from .gpt_modules import GPTTransformerDecoderLayer __all__ = ['GPT2LMHeadModel', 'GPT2WithLora', 'GPT2ForSequenceClassification', 'GPT2Model', 'GPTHead'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2LMHeadModel(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['gpt2'] def __init__(self, config: GPT2Config = None): config = config if config is not None else GPT2Config() super(GPT2LMHeadModel, self).__init__(config, auto_prefix=True) self.eos_token_id = self.config.eos_token_id parallel_config = self.config.parallel_config self.stridedslice = P.StridedSlice().shard(((parallel_config.data_parallel, 1),)) self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.get_attention_mask = AttentionMask(seq_length=config.seq_length, parallel_config=parallel_config.dp_mp_config) self.backbone = GPT2Model(config) self.head = GPTHead(hidden_size=config.hidden_size, vocab_size=config.vocab_size, parallel_config=self.config.parallel_config) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) mp = config.parallel_config.model_parallel vocab_size = config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of GPT Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPT Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = CrossEntropyLoss(parallel_config=loss_parallel_config) self.reshape = P.Reshape() self.cast = P.Cast() self.load_checkpoint(config) self.add = P.Add().shard(((parallel_config.data_parallel, 1), ())) self.tile = P.Tile() self.is_first_iteration = True self.all_ones_attention_mask = P.Ones()((1, 1, 1), mstype.float32) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 def construct(self, input_ids, attention_mask=None, input_embeds=None, labels=None, input_position=None, position_ids=None, init_reset=True, batch_valid_length=None): r""" construct function for Language Modeling Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. attention_mask (Tensor): input sentences padding mask, where 0 indicates padding position with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. labels (Tensor): the labels of inputs with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. input_position (Tensor): the position ids of inputs (at incremental reasoning mode) which is an increasing sequence with data type int64/int32, Tensor :math:`(bacth, seq\_length)`. position_ids (Tensor): the position ids of inputs which is an increasing sequence with data type int64/int32, Tensor :math:`(bacth, seq\_length)`. inputs_embeds (Tensor): the embedding of inputs with data type float32/float16, Tensor of shape :math:`(batch, seq\_length, hidden_size) init_reset (bool): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. batch_valid_length (Tensor): Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Returns: logits (Tensor) or loss (mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ if attention_mask is None: attention_mask = self.not_equal(input_ids, self.eos_token_id) batch_size, seq_length = input_ids.shape attention_mask = self.cast(attention_mask, mstype.float32) loss_mask = attention_mask if not self.training: tokens = input_ids else: tokens = self.stridedslice(input_ids, (0, 0), (batch_size, seq_length - 1), (1, 1)) attention_mask = self.stridedslice(attention_mask, (0, 0), (batch_size, seq_length - 1), (1, 1)) attention_mask = self.get_attention_mask(attention_mask) if not self.is_first_iteration: attention_mask = self.tile(self.all_ones_attention_mask, (batch_size, 1, 1)) # [batch_size, seq_length, vocab_size] output_states, embedding_table = self.backbone(tokens, attention_mask, input_position, init_reset, batch_valid_length) logits = self.head(output_states, embedding_table) if not self.training: logits = self.reshape(logits, (batch_size, seq_length, -1)) # makes cast effective to avoid allgather issue in Mindspore1.10 loss_mask = self.add(loss_mask, 1) return logits, tokens, loss_mask loss_mask = self.stridedslice(loss_mask, (0, 1), (batch_size, seq_length), (1, 1)) labels = self.stridedslice(input_ids, (0, 1), (batch_size, seq_length), (1, 1)) labels = self.reshape(labels, (-1,)) loss_mask = self.reshape(loss_mask, (-1,)) loss = self.loss(logits, labels, loss_mask) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2ForSequenceClassification(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['gpt2'] def __init__(self, config: GPT2Config = None): self.config = config if config is not None else GPT2Config() super(GPT2ForSequenceClassification, self).__init__(self.config, auto_prefix=True) self.eos_token = self.config.eos_token self.seq_length = self.config.seq_length self.num_labels = self.config.num_labels self.hidden_size = self.config.hidden_size parallel_config = self.config.parallel_config dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.get_attention_mask = AttentionMask(seq_length=self.seq_length, parallel_config=parallel_config.dp_mp_config) self.backbone = GPT2Model(self.config) self.score = Linear(in_channels=self.hidden_size, out_channels=self.num_labels, has_bias=False, compute_dtype=self.config.compute_dtype) self.score.shard(strategy_matmul=((dp, 1), (1, 1))) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) vocab_size = self.config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of GPT Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPT Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = nn.CrossEntropyLoss() self.reshape = P.Reshape() self.cast = P.Cast() self.load_checkpoint(config) self.reduce_sum = P.ReduceSum().shard(((dp, 1),)) self.add = P.Add().shard(((dp,), (dp,))) self.sub = P.Sub().shard(((1,), ())) self.gather = P.Gather().shard(((1, 1), (1,))) # pylint: disable=W0613 def construct(self, input_ids, attention_mask, labels=None, input_embeds=None, input_position=None, position_ids=None, init_reset=True, batch_valid_length=None): r""" construct function for GPT2 Text Classification Model Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary. attention_mask (Tensor): input sentences padding mask, where 0 indicates padding position. labels (Tensor): the labels of corresponding input sequences. input_embeds(Tensor): Reserved param, not used. input_position(Tensor): Reserved param, not used. position_ids(Tensor): Reserved param, not used. init_reset(Tensor): Reserved param, not used. batch_valid_length(Tensor): Reserved param, not used. Returns: (logits, labels) (Tensor, Tensor) or logits (Tensor) or loss (mstype.float32): in train mode, return loss; in eval mode, return logits and loss; in predict mode, return logits. """ attention_mask = self.cast(attention_mask, mstype.float32) attention_mask_lower_triangle = self.get_attention_mask(attention_mask) output_states, _ = self.backbone(input_ids, attention_mask_lower_triangle) output_states = self.reshape(output_states, (-1, self.hidden_size)) logits = self.score(output_states) # get the last logit of each sequence last_indices = self.sub(self.reduce_sum(attention_mask, -1), 1) batch_size = attention_mask.shape[0] indices_increments = Tensor(np.arange(0, self.seq_length * batch_size, self.seq_length)) last_indices = self.cast(self.add(last_indices, indices_increments), mstype.int32) pooled_logits = self.gather(logits, last_indices, 0) if labels is not None: labels = self.reshape(labels, (-1,)) if self.training: output = self.loss(pooled_logits, labels) else: if labels is not None: output = (pooled_logits, labels) else: output = pooled_logits return output class GPTEmbeddingLayer(nn.Cell): r"""The Embedding Layer of GPT-2 network.""" def __init__(self, config: GPT2Config = None): super(GPTEmbeddingLayer, self).__init__() parallel_config = copy.deepcopy(config.parallel_config) embedding_mp = config.parallel_config.embedding_dp_mp_config.model_parallel vocab_size = config.vocab_size if vocab_size % embedding_mp != 0: logger.warning("The vocab size of embedding layer is: %s, it is not divide by model_parallel: %s", vocab_size, embedding_mp) logger.warning("Now, model_parallel will be changed: mp = 1") parallel_config.embedding_dp_mp_config.model_parallel = 1 self.word_embedding = VocabEmbedding(vocab_size=vocab_size, embedding_size=config.hidden_size, param_init=initializer('normal', [vocab_size, config.hidden_size], dtype=mstype.float32), parallel_config=parallel_config.embedding_dp_mp_config) new_parallel_config = copy.deepcopy(parallel_config) new_parallel_config.vocab_emb_dp = True self.position_embedding = VocabEmbedding(vocab_size=config.seq_length, embedding_size=config.hidden_size, param_init=initializer('normal', [config.seq_length, config.hidden_size], dtype=mstype.float32), parallel_config=new_parallel_config.embedding_dp_mp_config) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dropout = Dropout(1 - config.embedding_dropout_prob) self.dropout.shard(((parallel_config.data_parallel, 1, 1),)) def construct(self, input_ids, input_position): """The forward compute of Embedding Layer.""" word_embedding, word_table = self.word_embedding(input_ids) position_embedding, _ = self.position_embedding(input_position) embedding = self.add(word_embedding, position_embedding) embedding = self.dropout(embedding) return embedding, word_table def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ pp_dis = max(int((layers + 1) / parallel_config.pipeline_stage), 1) pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class GPT2Model(nn.Cell): """ The backbone of GPT network Args: config(GPT2Config): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 input_mask: the mask indicating whether each position is a valid input Returns: output_state: Tensor, the output logit of backbone present_layer: Tensor, the current feature map embedding_table: Tensor, the embedding table for the vocabulary """ def __init__(self, config): super(GPT2Model, self).__init__() self.config = config self.embedding = GPTEmbeddingLayer(config) self.embedding.pipeline_stage = 0 self.layernorm = LayerNorm((config.hidden_size,)).to_float(config.layernorm_compute_type) if config.parallel_config.pipeline_stage > 1: self.layernorm.set_comm_fusion(2) else: self.layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.layernorm.pipeline_stage = config.parallel_config.pipeline_stage - 1 if not hasattr(config.parallel_config, "moe_config"): config.parallel_config.moe_config = default_moe_config moe_config = config.parallel_config.moe_config self.blocks = nn.CellList() for i in range(config.num_layers): block = GPTTransformerDecoderLayer( hidden_size=config.hidden_size, batch_size=config.batch_size, ffn_hidden_size=config.hidden_size * config.expand_ratio, seq_length=config.seq_length, num_heads=config.num_heads, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, hidden_act=config.hidden_act, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, parallel_config=config.parallel_config.dp_mp_config, moe_config=moe_config, use_past=config.use_past ) set_parallel_configure_for_layer( block, layer_id=i, layers=config.num_layers, offset=0, parallel_config=config.parallel_config) self.blocks.append(block) self.cast = P.Cast() self.tile = P.Tile().shard(((config.parallel_config.data_parallel,),)) self.dtype = mstype.float16 self.num_layers = config.num_layers self.position_ids = Tensor(np.arange(config.seq_length), mstype.int32) self.is_first_iteration = True def construct(self, input_ids, attention_mask, input_position=None, init_reset=True, batch_valid_length=None): """GPT model""" batch_size, seq_length = F.shape(input_ids) # When input_position is None, the phase is train mode. When the phase is train mode and is the first iteration # of incremental reasoning, it goes into the following logic. if input_position is None or self.is_first_iteration: if batch_size == 1: input_position = F.reshape(self.position_ids, (1, seq_length)) else: input_position = self.tile(self.position_ids, (batch_size, 1)) # when the phase is not train and incremental reasoning is not the first iteration, it goes into the # following logic. else: bias = Tensor(np.arange(batch_size) * self.config.seq_length, mstype.int32) input_position = F.sub(input_position, bias) input_position = F.reshape(input_position, (batch_size, 1)) input_embedding, embedding_table = self.embedding(input_ids, input_position) hidden_states = self.cast(input_embedding, self.dtype) hidden_shape = F.shape(hidden_states) hidden_states = F.reshape(hidden_states, (-1, hidden_shape[-1])) for i in range(self.num_layers): hidden_states = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) output_state = self.layernorm(hidden_states) return output_state, embedding_table class GPTHead(nn.Cell): r"""Head for GPT to get the logits of each token in the vocab.""" def __init__(self, hidden_size, vocab_size, compute_type=mstype.float16, parallel_config=None): super().__init__() copied_parallel_config = copy.deepcopy(parallel_config) mp = copied_parallel_config.model_parallel if vocab_size % mp != 0: logger.warning("The vocab size of GPTHead MatMul is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPTHead MatMul will be changed: mp = 1") copied_parallel_config.model_parallel = 1 if copied_parallel_config.pipeline_stage > 1: copied_parallel_config.vocab_emb_dp = False if copied_parallel_config.vocab_emb_dp: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), ( copied_parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = compute_type self.cast = P.Cast() def construct(self, state, embedding_table): logits = self.matmul(self.cast(state, self.dtype), self.cast(embedding_table, self.dtype)) return logits @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2WithLora(GPT2LMHeadModel): """ GPT2LMHeadModel with LoRA parameter-efficient tuning algorithm Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ def __init__(self, config: GPT2Config = None): checkpoint_name_or_path = config.pop("checkpoint_name_or_path") super().__init__(config) config.pet_config.reg_rules = r'.*dense1.*|.*dense3.*' self.backbone = LoraAdapter.get_pet_model(self.backbone, config.pet_config) config.checkpoint_name_or_path = checkpoint_name_or_path self.load_checkpoint(config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, config.pet_config.pet_type) class CrossEntropyCalculationWithMask(nn.Cell): """ Cross Entropy loss """ def __init__(self, is_training=None, num_labels=None): super(CrossEntropyCalculationWithMask, self).__init__() self.onehot = P.OneHot() self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_sum = P.ReduceSum() self.reduce_mean = P.ReduceMean() self.reshape = P.Reshape() self.last_idx = (-1,) self.neg = P.Neg() self.cast = P.Cast() self.is_training = is_training self.num_labels = num_labels self.log_softmax = P.LogSoftmax(axis=-1) def construct(self, logits, label_ids, input_mask=None): """ Calculate loss Args: logits (Tensor): the probability distribution over vocabulary. label_ids (Tensor): the indices of input sequence tokens in the vocabulary. input_mask (Tensor): input sentences padding mask, where 0 indicates padding position. Returns: return_value (Tensor, mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ # logits [batch * (seq_length-1), vocab_size] label_ids [batch, seq_length-1] logits = self.log_softmax(logits) if self.is_training: label_ids = self.reshape(label_ids, self.last_idx) # label_ids [batch * (seq_length-1)] one_hot_labels = self.onehot(label_ids, self.num_labels, self.on_value, self.off_value) # [batch * (seq_length-1), vocab_size] per_example_loss = self.neg( self.reduce_sum(one_hot_labels * logits, self.last_idx)) # [batch * (seq_length-1)] # for PPL calculation in evaluation if input_mask is not None: input_mask = self.cast(self.reshape(input_mask, self.last_idx), mstype.float32) # [batch * (seq_length-1)] valid_loss_sum = self.reduce_sum(input_mask * per_example_loss, ()) valid_element_sum = self.reduce_sum(input_mask, ()) + self.cast(F.tuple_to_array((1e-5,)), mstype.float32) loss = valid_loss_sum / valid_element_sum else: loss = self.reduce_mean(per_example_loss, self.last_idx) # a number return_value = self.cast(loss, mstype.float32) else: return_value = logits * 1.0 # [batch * (seq_length-1), vocab_size] return return_value pengcheng.mind-test/mindformers/models/gpt2/gpt2_config.py000066400000000000000000000103241471400134000241500ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Gpt Config API.""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['GPT2Config'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class GPT2Config(BaseConfig): """ Gpt config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['gpt2'] def __init__(self, batch_size: int = 1, eos_token_id: int = 50256, pad_token_id: int = 50256, bos_token_id: int = 50256, unk_token_id: int = 50256, seq_length: int = 1024, vocab_size: int = 50257, hidden_size: int = 768, num_layers: int = 12, num_heads: int = 12, num_labels: int = 2, expand_ratio: int = 4, embedding_dropout_prob: float = 0.1, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, param_init_type: str = "float32", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", compute_dtype: str = "float16", hidden_act: str = 'gelu', use_past: bool = False, post_layernorm_residual: bool = False, offset: int = 0, parallel_config: TransformerOpParallelConfig = default_transformer_config, checkpoint_name_or_path: str = "", moe_config: MoEConfig = default_moe_config, repetition_penalty: float = 1.0, max_decode_length: int = 1024, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, **kwargs): super(GPT2Config, self).__init__(**kwargs) self.batch_size = batch_size self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.bos_token_id = bos_token_id self.unk_token_id = unk_token_id self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.num_labels = num_labels self.expand_ratio = expand_ratio self.embedding_dropout_prob = embedding_dropout_prob self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.param_init_type = convert_mstype(param_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.compute_dtype = convert_mstype(compute_dtype) self.hidden_act = hidden_act self.use_past = use_past self.post_layernorm_residual = post_layernorm_residual self.offset = offset self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path self.moe_config = moe_config self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample pengcheng.mind-test/mindformers/models/gpt2/gpt2_processor.py000066400000000000000000000044121471400134000247230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ GPT2Processor """ from mindformers.mindformer_book import MindFormerBook from ..base_tokenizer import BaseTokenizer from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['GPT2Processor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class GPT2Processor(BaseProcessor): """ GPT2 processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['gpt2'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(GPT2Processor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-test/mindformers/models/gpt2/gpt2_tokenizer.py000066400000000000000000000367771471400134000247410ustar00rootroot00000000000000# coding=utf-8 # Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GPT2 Tokenizer""" import json from functools import lru_cache from typing import List, Optional import os import regex as re from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_tokenizer import Tokenizer, AddedToken from mindformers.tools import logger from ...mindformer_book import MindFormerBook __all__ = ['GPT2Tokenizer'] @lru_cache() def bytes_to_unicode(): """ Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. """ bs = ( list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1)) ) cs = bs[:] n = 0 for b in range(2**8): if b not in bs: bs.append(b) cs.append(2**8 + n) n += 1 cs = [chr(n) for n in cs] return dict(zip(bs, cs)) def get_pairs(word): """ Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). """ pairs = set() prev_char = word[0] for char in word[1:]: pairs.add((prev_char, char)) prev_char = char return pairs VOCAB_FILES_NAMES = { "vocab_file": "vocab.json", "merges_file": "merges.txt", } @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class GPT2Tokenizer(Tokenizer): """ Construct a GPT-2 tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not: ```python >>> from mindformers import GPT2Tokenizer >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> tokenizer("Hello world", add_bos_token=False, add_eos_token=False)["input_ids"] [15496, 995] >>> tokenizer(" Hello world", add_bos_token=False, add_eos_token=False)["input_ids"] [18435, 995] ``` You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance. When used with `is_split_into_words=True`, this tokenizer will add a space before each word (even the first one). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `<|endoftext|>`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `<|endoftext|>`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `<|endoftext|>`): The end of sequence token. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (GPT2 tokenizer detect beginning of words by the preceding space). add_bos_token(`bool`, *optional*, defaults to `True`): Whether or not to add the bos_token_id to the left of the input. add_eos_token(`bool`, *optional*, defaults to `True`): Whether or not to add the eos_token_id to the right of the input. Examples: >>> from mindformers import GPT2Tokenizer >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> res = tokenizer("Hello world") >>> print(res) {'input_ids': [50256, 15496, 995, 50256], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]} >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> res = tokenizer("Hello world", add_special_tokens=False) >>> print(res) {'input_ids': [15496, 995], 'token_type_ids': [0, 0], 'attention_mask': [1, 1]} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "token_type_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['gpt2'] def __init__( self, vocab_file, merges_file, errors="replace", unk_token="<|endoftext|>", bos_token="<|endoftext|>", eos_token="<|endoftext|>", pad_token="<|endoftext|>", add_prefix_space=False, add_bos_token=False, add_eos_token=False, **kwargs, ): bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token super().__init__( errors=errors, unk_token=unk_token, bos_token=bos_token, eos_token=eos_token, pad_token=pad_token, add_prefix_space=add_prefix_space, **kwargs, ) self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token with open(vocab_file, encoding="utf-8") as vocab_handle: self.encoder = json.load(vocab_handle) self.decoder = {v: k for k, v in self.encoder.items()} self.errors = errors # how to handle errors in decoding self.byte_encoder = bytes_to_unicode() self.byte_decoder = {v: k for k, v in self.byte_encoder.items()} with open(merges_file, encoding="utf-8") as merges_handle: bpe_merges = merges_handle.read().split("\n")[1:-1] bpe_merges = [tuple(merge.split()) for merge in bpe_merges] self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges)))) self.cache = {} self.add_prefix_space = add_prefix_space # Should have added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""") @property def vocab_size(self): return len(self.encoder) def get_vocab(self): return dict(self.encoder, **self.added_tokens_encoder) def bpe(self, token): """bpe encode """ if token in self.cache: return self.cache[token] word = tuple(token) pairs = get_pairs(word) if not pairs: return token while True: bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf"))) if bigram not in self.bpe_ranks: break first, second = bigram new_word = [] i = 0 while i < len(word): try: j = word.index(first, i) except ValueError: new_word.extend(word[i:]) break else: new_word.extend(word[i:j]) i = j if word[i] == first and i < len(word) - 1 and word[i + 1] == second: new_word.append(first + second) i += 2 else: new_word.append(word[i]) i += 1 new_word = tuple(new_word) word = new_word if len(word) == 1: break else: pairs = get_pairs(word) word = " ".join(word) self.cache[token] = word return word def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): """Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) if not self.add_bos_token: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=False ) if token_ids_1 is None: return [1] + ([0] * len(token_ids_0)) return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) def _tokenize(self, text, **kwargs): """Tokenize a string.""" bpe_tokens = [] for token in re.findall(self.pat, text): token = "".join( self.byte_encoder[b] for b in token.encode("utf-8") ) # Maps all our bytes to unicode strings, avoiding control tokens of the BPE (spaces in our case) bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(" ")) return bpe_tokens def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.encoder.get(token, self.encoder.get(self.unk_token)) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" return self.decoder.get(index) def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" text = "".join(tokens) text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors) return text def save_vocabulary(self, save_directory, filename_prefix=None): """write the word to the files""" if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) merge_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["merges_file"] ) with open(vocab_file, "w", encoding="utf-8") as f: # you can set sort_keys=True, then the keys of vocabulary dict is sorted, # which is different from original vocabulary file. f.write(json.dumps(self.encoder, indent=2, ensure_ascii=False) + "\n") index = 0 with open(merge_file, "w", encoding="utf-8") as writer: writer.write("#version: 0.2\n") for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]): if index != token_index: logger.warning( "Saving vocabulary to %s: BPE merge indices are not consecutive." " Please check that the tokenizer is not corrupted!", merge_file ) index = token_index writer.write(" ".join(bpe_tokens) + "\n") index += 1 return vocab_file, merge_file def prepare_for_tokenization(self, text, **kwargs): add_prefix_space = kwargs.pop("add_prefix_space", self.add_prefix_space) is_split_into_words = kwargs.pop("is_split_into_words", False) if is_split_into_words or add_prefix_space: text = " " + text return (text, kwargs) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output pengcheng.mind-test/mindformers/models/gpt2/gpt_modules.py000066400000000000000000000253621471400134000243010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GPT modules.""" import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.modules.transformer.moe import default_moe_config from mindformers.modules.transformer import TransformerEncoderLayer from mindformers.modules.transformer.op_parallel_config import default_dpmp_config class GPTTransformerDecoderLayer(TransformerEncoderLayer): r"""GPT Transformer Decoder Layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. seq_length(int): The input sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, seq_length), (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` """ def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', use_past=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(GPTTransformerDecoderLayer, self).__init__( batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, hidden_act=hidden_act, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config ) def construct(self, x, input_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(x, input_mask, init_reset, batch_valid_length) x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) if self.post_layernorm_residual: input_x = x else: input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) output = F.reshape(output, (-1, x_shape[-1])) output = self.layernorm1(output) output = F.reshape(output, x_shape) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) output = self.layernorm1(output) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) if self.use_moe: return output, aux_loss return output pengcheng.mind-test/mindformers/models/llama/000077500000000000000000000000001471400134000216075ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/llama/.ipynb_checkpoints/000077500000000000000000000000001471400134000254005ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/llama/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000017321471400134000316410ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for llama""" from .llama_config import LlamaConfig from .llama import LlamaForCausalLM, LlamaModel from .llama_tokenizer import LlamaTokenizer from .llama_processor import LlamaProcessor __all__ = ['LlamaConfig', 'LlamaModel', 'LlamaForCausalLM', 'LlamaTokenizer', 'LlamaProcessor'] pengcheng.mind-test/mindformers/models/llama/.ipynb_checkpoints/llama-checkpoint.py000066400000000000000000000456311471400134000311760ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA models' APIs.""" import math import numpy as np import mindspore.common.dtype as mstype try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import Tensor, nn from mindspore.context import ParallelMode from mindspore.ops import operations as P from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation try: # pylint: disable=W0611 from mindspore.nn.layer.flash_attention import FlashAttention FLASHATTENTION_VALID = True except ImportError: FLASHATTENTION_VALID = False from mindspore.common.initializer import Normal from mindspore import lazy_inline from mindformers.core.loss.loss import CrossEntropyLoss from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from mindformers.modules.layers import Linear from mindformers.modules.transformer.op_parallel_config import _check_config from mindformers.modules.transformer.transformer import AttentionMask from mindformers.tools.register.register import MindFormerModuleType, MindFormerRegister #from mindformers.pet.tuners.pet_adapter import PetAdapter #from mindformers.pet.tuners.lora_adapter import LoraAdapter from .llama_config import LlamaConfig from .llama_layer import LlamaEmbedding, LlamaRMSNorm, precompute_freqs_cis from .llama_transformer import LLamaDecodeLayer from ..utils import cell_reuse from ...tools.logger import logger __all__ = ['LlamaModel', 'LlamaForCausalLM'] def layer_compute_dtype(layer, layer_id, offset, parallel_config, n_layers, select_recompute=False): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: layer(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(Union[int, List[int]]) - Means the layer_index needs a offset, if there are other modules in the net. n_layers(int) - The total layers used for the model. """ pp_dis = max(int((n_layers + 1) / parallel_config.pipeline_stage), 1) if isinstance(offset, list): if len(offset) != parallel_config.pipeline_stage: raise ValueError(f"The length of `offset` {len(offset)} do not match " "`pipeline stage` {parallel_config.pipeline_stage}.") i = min(layer_id // pp_dis, parallel_config.pipeline_stage - 1) offset_layer = offset[i] elif isinstance(offset, int): offset_layer = offset else: raise TypeError(f"`offset` must be `int` of list of `int`, but got {type(offset)}.") if parallel_config.pipeline_stage == 4: pp_id_list = [10, 11, 11, 8] pp_id = 0 tmp = layer_id while tmp >= pp_id_list[pp_id]: tmp -= pp_id_list[pp_id] pp_id += 1 else: pp_id = min((layer_id + offset_layer) // pp_dis, parallel_config.pipeline_stage - 1) layer.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((n_layers + 1) / parallel_config.gradient_aggregation_group), 1) layer.set_comm_fusion(3 + layer_id) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not select_recompute: layer.recompute() else: if parallel_config.recompute.recompute and not select_recompute: layer.recompute( recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) #layer.attention.flash_attention.flash_attention.recompute(mode=False) class LlamaModel(BaseModel): r""" Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config(LlamaConfig): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 Returns: output: Tensor, the output of llama decoderlayer """ _support_list = MindFormerBook.get_model_support_list()['llama'] def __init__(self, config: LlamaConfig = None): super().__init__(config, auto_prefix=True) _check_config(config.parallel_config) if config.batch_size or config.use_past: Validator.check_positive_int(config.batch_size) self.dtype = config.compute_dtype self.num_layers = config.num_layers self.pad_token_id = config.pad_token_id self.eos_token_id = config.eos_token_id self.is_first_iteration = True self.use_past = config.use_past self.use_flash_attention = config.use_flash_attention and FLASHATTENTION_VALID if self.use_flash_attention: logger.info("Enable flash attention.") elif config.use_flash_attention: logger.info("Current MindSpore do not support flash attention.") self.freqs_cos, self.freqs_sin, self.swap_mask = precompute_freqs_cis( config.hidden_size // config.num_heads, config.seq_length, dtype=config.rotary_dtype, pretrain_seqlen=config.pretrain_seqlen, extend_method=config.extend_method) self.get_attention_mask = AttentionMask( config.seq_length, parallel_config=config.parallel_config.dp_mp_config).to_float(config.compute_dtype) self.multiply_data = Tensor([-10000.0], dtype=config.compute_dtype) self.one = Tensor([1.0], dtype=config.compute_dtype) self.reshape = P.Reshape() self.cast = P.Cast() self.tile = P.Tile() self.mul_mask = P.Mul() self.sub = P.Sub() self.expand_dims = P.ExpandDims() self.not_equal = P.NotEqual() self.gather = P.Gather() self.tok_embeddings = LlamaEmbedding( config.vocab_size, config.hidden_size, param_init_type=config.param_init_type) self.embed_scale = Tensor([config.embed_scale], dtype=config.compute_dtype) self.mul_embed = P.Mul() self.layers = nn.CellList() for layer_id in range(config.num_layers): layer = LLamaDecodeLayer(config.batch_size, config.seq_length, layer_id, config.num_layers, dim=config.hidden_size, n_heads=config.num_heads, multiple_of=config.multiple_of, n_kv_heads=config.n_kv_heads, ffn_dim_multiplier=config.ffn_dim_multiplier, mla_multiplier=config.mla_multiplier, norm_eps=config.rms_norm_eps, compute_dtype=config.compute_dtype, layernorm_compute_dtype=config.layernorm_compute_type, softmax_compute_dtype=config.softmax_compute_type, rotary_dtype=config.rotary_dtype, mup_width_base=config.mup_width_base, param_init_type=config.param_init_type, use_past=config.use_past, use_flash_attention=config.use_flash_attention, compute_in_2d=config.compute_in_2d, use_past_shard=config.use_past_shard, depth_scale=config.depth_scale, init_std=config.init_std, parallel_config=config.parallel_config) layer_compute_dtype(layer, layer_id, config.offset, config.parallel_config, config.num_layers, select_recompute=config.parallel_config.recompute.select_recompute) self.layers.append(layer) self.norm_out = LlamaRMSNorm(config.hidden_size, config.rms_norm_eps, compute_type=config.layernorm_compute_type) dp = config.parallel_config.data_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.tok_embeddings.pipeline_stage = 0 if config.parallel_config.pipeline_stage > 1: self.norm_out.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.tok_embeddings.set_comm_fusion(2) self.norm_out.set_comm_fusion(config.num_layers + 3) self.tok_embeddings.shard(config.parallel_config) self.tile.shard(((1, 1, 1, 1), ())) self.sub.shard(((1,), (dp, 1, 1))) self.mul_mask.shard(((dp, 1, 1, 1), (1,))) self.mul_embed.shard(((dp, 1, 1), (1,))) self.expand_dims.shard(((dp, 1, 1),)) self.not_equal.shard(((dp, 1), ())) self.gather.shard(((dp, 1), (1,))) if config.compute_in_2d: self.norm_out.shard((dp, 1)) else: self.norm_out.shard((dp, 1, 1)) if self.use_past: seq_range = np.arange(config.seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (config.batch_size, 1, 1)), mstype.int32) self.gather_past = P.Gather() self.expand_dims = P.ExpandDims() self.le_past = P.LessEqual() ones = np.ones(shape=(config.seq_length, config.seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((dp, 1, 1), (1, 1, 1))) self.equal = P.Equal().shard(((dp, 1), ())) self.cumsum = P.CumSum(reverse=True).shard(((dp, 1),)) self.pow = P.Pow().shard(((), (dp, 1))) self.pow_up_value = 2.0 self.pow_down_value = 0.5 self.mask_reshape = P.Reshape() self.mask_mul = P.BatchMatMul().shard( ((dp, 1, 1), (dp, 1, 1))) self.up_threshold = 1.2 self.down_threshold = 0.8 self.greater = P.Greater().shard(((dp, 1, 1), ())) self.less = P.Less().shard(((dp, 1, 1), ())) self.mask_multiply = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mask_expand = P.ExpandDims().shard(((1, 1),)) self.mask_multiply2 = P.Mul().shard(((dp, 1, 1), (1, 1, 1))) # pylint: disable=W0613 def construct(self, tokens: Tensor, input_position=None, init_reset=True, batch_valid_length=None): """Forward of llama model.""" # preprocess bs, seq_len = tokens.shape if self.is_first_iteration: freqs_cis = (self.tile(self.reshape(self.freqs_cos, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.tile(self.reshape(self.freqs_sin, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.swap_mask) input_mask = self.cast(self.not_equal(tokens, self.pad_token_id), self.dtype) seq_index = self.cast(self.equal(tokens, self.eos_token_id), mstype.float32) seq_mask_rev = self.cumsum(seq_index, 1) seq_mask_exp_up = self.mask_reshape(self.pow(self.pow_up_value, seq_mask_rev), (bs, seq_len, 1)) seq_mask_exp_down = self.mask_reshape(self.pow(self.pow_down_value, seq_mask_rev), (bs, 1, seq_len)) full_mask = self.mask_mul(seq_mask_exp_up, seq_mask_exp_down) full_mask_small = self.cast(self.less(full_mask, self.up_threshold), mstype.float32) full_mask_great = self.cast(self.greater(full_mask, self.down_threshold), mstype.float32) mask = self.mask_multiply(full_mask_small, full_mask_great) lower_triangle = self.mask_expand(self.lower_triangle_mask, 0) mask = self.mask_multiply2(full_mask, lower_triangle) # mask: [bs, seq, seq] # else: # cur_pos = batch_valid_length - 1 # valid_length = self.reshape(cur_pos, (-1, 1, 1)) # freqs_cis = (self.reshape(self.gather_past(self.freqs_cos, cur_pos, 0), (bs, 1, seq_len, -1)), # self.reshape(self.gather_past(self.freqs_sin, cur_pos, 0), (bs, 1, seq_len, -1)), # self.swap_mask) # mask = self.cast(self.le_past(self.range, valid_length), self.dtype) # mask: [bs, 1, 1] mask = self.sub(self.one, self.cast(mask, self.dtype)) if not self.use_flash_attention: mask = self.expand_dims(mask, 1) mask = self.mul_mask(mask, self.multiply_data) # tokens: [bs, seq/1] h = self.tok_embeddings(tokens) h = self.mul_embed(h, self.embed_scale) # h: [bs, seq/1, hidden_dim] for i in range(self.num_layers): h, _ = self.layers[i](h, freqs_cis, mask, init_reset=init_reset, batch_valid_length=batch_valid_length) output = self.norm_out(h) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class LlamaForCausalLM(BaseModel): r""" Provide llama training loss or logits through network. Args: config (LlamaConfig): The config of llama model. Inputs: input_ids(Tensor): the tokenized inputs with datatype int32, Tensor of shape :math:`(batch, seq\_length)`. labels(Tensor): the tokenized labels with datatype int32, Tensor of shape :math:`(batch, seq\_length)`. input_position(Tensor): current position, used by model.predict. position_ids(Tensor): Reserved param, not used. attention_mask(Tensor): Reserved param, not used. input_embeds(Tensor): Reserved param, not used. init_reset(bool, optional): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Default True. batch_valid_length(Tensor): the past calculated the index with datatype int32, used for incremental prediction. Tensor of shape :math:`(batch_size,)`. Default None. Returns: Tensor, the loss or logits of the network. Examples: >>> from mindformers.models.llama import LlamaConfig, LlamaForCausalLM >>> config = LlamaConfig(batch_size=2) >>> network = LlamaForCausalLM(config=config) """ _support_list = MindFormerBook.get_model_support_list()['llama'] #@lazy_inline def __init__(self, config: LlamaConfig = None): super(LlamaForCausalLM, self).__init__(config, auto_prefix=True) _check_config(config.parallel_config) self.ignore_token_id = config.ignore_token_id self.pad_token_id = config.pad_token_id self.eos_token_id = config.eos_token_id self.reshape = P.Reshape() self.cast = P.Cast() self.slice = P.StridedSlice() self.not_equal = P.NotEqual() self.mul = P.Mul() self.add = P.Add() self.model = LlamaModel(config=config) sigma = config.init_std if config.mup_width_base > 0: ratio = config.hidden_size / config.mup_width_base sigma = sigma / ratio self.lm_head = Linear(in_channels=config.hidden_size, out_channels=config.vocab_size, has_bias=False, compute_dtype=config.compute_dtype, param_init_type=config.param_init_type, weight_init=Normal(sigma=sigma, mean=0)) # meta default: xavier_normal self.loss = CrossEntropyLoss(parallel_config=config.parallel_config) dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.slice.shard(((dp, 1),)) self.not_equal.shard(((dp, 1), ())) self.mul.shard(((dp, 1), (dp, 1))) self.add.shard(((dp, 1), ())) if config.parallel_config.vocab_emb_dp: self.lm_head.shard(strategy_matmul=((dp, 1), (1, 1))) else: self.lm_head.shard(strategy_matmul=((dp, 1), (mp, 1))) if config.parallel_config.pipeline_stage > 1: self.lm_head.pipeline_stage = config.parallel_config.pipeline_stage - 1 #self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 def construct(self, input_ids, labels=None, input_position=None, position_ids=None, attention_mask=None, input_embeds=None, init_reset=True, batch_valid_length=None): """LlamaForCausalLM forward.""" bsz, seqlen = input_ids.shape #if self.training: # tokens = self.slice(input_ids, (0, 0), (bsz, seqlen - 1), (1, 1)) #else: # tokens = input_ids tokens = self.slice(input_ids, (0, 0), (bsz, seqlen - 1), (1, 1)) output = self.model(tokens, input_position, init_reset, batch_valid_length) logits = self.lm_head(output) input_mask = self.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) if labels is None: labels = self.slice(input_ids, (0, 1), (bsz, seqlen), (1, 1)) else: if labels.ndim > 1: if self.training: labels = self.slice(labels, (0, 1), (bsz, seqlen), (1, 1)) label_mask = self.cast(self.not_equal(labels, self.ignore_token_id), mstype.float32) input_mask = self.mul(input_mask, label_mask) logits = self.cast(logits, mstype.float32) #if not self.training: # logits = self.reshape(logits, (bsz, seqlen, -1)) # # makes cast effective to avoid allgather issue in Mindspore1.10 # input_mask = self.add(input_mask, 1) # return logits, tokens, input_mask if logits.ndim > 2: logits = self.reshape(logits, (-1, logits.shape[-1])) labels = self.reshape(labels, (-1,)) input_mask = self.reshape(input_mask, (-1,)) loss = self.loss(logits, labels, input_mask) return loss pengcheng.mind-test/mindformers/models/llama/.ipynb_checkpoints/llama_config-checkpoint.py000066400000000000000000000114721471400134000325170ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Llama Config API.""" from typing import Optional from mindformers.modules.transformer.transformer import default_transformer_config, TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['LlamaConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class LlamaConfig(BaseConfig): """ LLaMA config class which defines the model size. """ _support_list = MindFormerBook.get_config_support_list()['llama'] def __init__(self, batch_size: int = 1, seq_length: int = 2048, hidden_size: int = 4096, num_layers: int = 32, num_heads: int = 32, vocab_size: int = 32000, # defined later by tokenizer multiple_of: int = 256, # make SwiGLU hidden layer size multiple of large power of 2 n_kv_heads: Optional[int] = None, ffn_dim_multiplier: Optional[int] = None, mla_multiplier: Optional[int] = None, rms_norm_eps: float = 1e-5, bos_token_id: int = 1, eos_token_id: int = 2, pad_token_id: int = 32000, ignore_token_id: int = -100, compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", rotary_dtype: str = "float32", param_init_type: str = "float16", parallel_config: TransformerOpParallelConfig = default_transformer_config, use_past: bool = False, pretrain_seqlen: int = 2048, extend_method: str = "None", compute_in_2d: bool = False, use_flash_attention: bool = False, offset: int = 0, use_past_shard: bool = False, checkpoint_name_or_path: str = "", repetition_penalty: float = 1.0, max_decode_length: int = 1024, final_logit_softcapping: float = 30.0, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, mup_width_base: int = -1, embed_scale: float = 1.0, # depth_scale: float = 1.0, init_std: float = 0.01, **kwargs): super(LlamaConfig, self).__init__(**kwargs) self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.multiple_of = multiple_of self.n_kv_heads = n_kv_heads self.ffn_dim_multiplier = ffn_dim_multiplier self.mla_multiplier = mla_multiplier self.rms_norm_eps = rms_norm_eps self.param_init_type = convert_mstype(param_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.rotary_dtype = convert_mstype(rotary_dtype) self.compute_dtype = convert_mstype(compute_dtype) self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path self.bos_token_id = bos_token_id self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.ignore_token_id = ignore_token_id self.use_past = use_past self.pretrain_seqlen = pretrain_seqlen self.extend_method = extend_method self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention self.offset = offset self.use_past_shard = use_past_shard self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample self.mup_width_base = mup_width_base self.embed_scale = embed_scale pengcheng.mind-test/mindformers/models/llama/.ipynb_checkpoints/llama_layer-checkpoint.py000066400000000000000000000364631471400134000323750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA Model Layers' APIs.""" from enum import Enum import math import numpy as np from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore import nn import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.nn.cell import Cell from mindspore.common.initializer import Normal try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import log as logger from mindspore.common.initializer import initializer from mindspore.parallel._utils import _get_parallel_mode from mindspore.context import ParallelMode from mindformers.modules.layers import Linear, _check_input_dtype, _args_type_validator_check, _valid_value_checks from mindformers.tools.logger import _LogActionOnce class SeqExtendMethod(Enum): """Stores the acceptable string identifiers for seq length extend method""" PI = "PI" NTK = "NTK" NONE = "None" class LlamaSiLU(Cell): r""" A self-defined SwiGlu. Inputs: - **x** (Tensor) - Tensor. Outputs: Tensor. x = x * sigmod(x). """ def __init__(self): super().__init__() self.sigmoid = nn.Sigmoid() self.mul = P.Mul() def shard(self, strategy): self.sigmoid.sigmoid.shard(strategy) self.mul.shard((strategy[0], strategy[0])) def construct(self, x): return self.mul(x, self.sigmoid(x)) def get_swap_mask(head_dim): """Swap matrix""" zero_block = np.zeros((head_dim // 2, head_dim // 2), dtype=np.float32) id_block = np.identity(head_dim // 2, dtype=np.float32) return np.block([[zero_block, id_block], [-id_block, zero_block]]) def precompute_freqs_cis( dim: int, end: int, theta: float = 10000.0, dtype=mstype.float32, pretrain_seqlen=2048, extend_method=SeqExtendMethod.NONE.value): """ Precompute of freqs and mask for rotary embedding. """ ratio = 1. if extend_method != SeqExtendMethod.NONE.value and end > pretrain_seqlen: ratio = end / pretrain_seqlen if extend_method == SeqExtendMethod.NTK.value: theta *= ratio freqs_base = np.arange(0, dim, 2)[: (dim // 2)].astype(np.float32) # (head_dim // 2, ) freqs = 1.0 / (theta ** (freqs_base / dim)) # (head_dim // 2, ) if extend_method == SeqExtendMethod.PI.value: t = np.arange(0, end / ratio, 1 / ratio).astype(np.float32) else: t = np.arange(0, end, 1).astype(np.float32) # type: ignore # (seq_len,) freqs = np.outer(t, freqs) # type: ignore (seq_len, head_dim // 2) emb = np.concatenate((freqs, freqs), axis=-1) freqs_cos = np.cos(emb) # (seq_len, head_dim) freqs_sin = np.sin(emb) # (seq_len, head_dim) freqs_cos = Tensor(freqs_cos, dtype=dtype) freqs_sin = Tensor(freqs_sin, dtype=dtype) swap_mask = get_swap_mask(dim) swap_mask = Tensor(swap_mask, dtype=dtype) return freqs_cos, freqs_sin, swap_mask class LlamaRotaryEmbedding(Cell): r""" Rotary Position Embedding. Args: - **head_dim** (int): The dim of multi head attention. - **compute_dtype** (mstype): The compute type, default mstype.float16. - **parallel_config** (dict): - Parallel Config. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, head_dim=128, compute_dtype=mstype.float32): super().__init__(auto_prefix=False) self.head_dim = head_dim self.dtype = compute_dtype self.add = P.Add() self.bmm_swap = P.BatchMatMul() self.mul = P.Mul() self.cast = P.Cast() def rotate_half(self, x, swap_mask): # [bs, n_head/n_kv_head, seq/1, head_dim], [head_dim, head_dim] x = self.bmm_swap(x, swap_mask) return x def construct(self, xq: Tensor, xk: Tensor, freqs_cis): """Forward of rotary position embedding.""" original_type = xq.dtype xq = self.cast(xq, self.dtype) xk = self.cast(xk, self.dtype) # xq, xk: [bs, n_head/n_kv_head, seq/1, head_dim] freqs_cos, freqs_sin, swap_mask = freqs_cis xq_out = self.add(self.mul(xq, freqs_cos), self.mul(self.rotate_half(xq, swap_mask), freqs_sin)) xk_out = self.add(self.mul(xk, freqs_cos), self.mul(self.rotate_half(xk, swap_mask), freqs_sin)) xq_out = self.cast(xq_out, original_type) xk_out = self.cast(xk_out, original_type) return xq_out, xk_out def shard(self, strategy_in): self.add.shard((strategy_in, strategy_in)) self.bmm_swap.shard((strategy_in, (1, 1))) self.mul.shard((strategy_in, (strategy_in[0], 1, 1, 1))) class LlamaEmbedding(Cell): """ Embedding Layer. Args: - **vocab_size** (int): Size of the dictionary of embeddings. - **embedding_size** (int): The size of each embedding vector. - **param_init_type** (mstype): The param init type, default mstype.float32. - **parallel_config** (TransformerOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. - **param_init** (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). """ @_LogActionOnce(m_logger=logger, key='Embedding', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(vocab_table_size=Validator.check_positive_int, embedding_size=Validator.check_positive_int) def __init__(self, vocab_table_size, embedding_size, param_init_type=mstype.float32, param_init='normal'): super().__init__() self.vocab_table_size = vocab_table_size self.embedding_size = embedding_size self.embedding_weight = Parameter( initializer(param_init, [self.vocab_table_size, self.embedding_size], dtype=param_init_type), name='embedding_weight', parallel_optimizer=False) self.gather = P.Gather() def construct(self, input_ids): """Forward of vocab embedding.""" _check_input_dtype(F.dtype(input_ids), "input_ids", [mstype.int32, mstype.int64], self.cls_name) output = self.gather(self.embedding_weight, input_ids, 0) return output def shard(self, parallel_config): """sharding for embedding""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel if parallel_config.vocab_emb_dp: self.gather.shard(((1, 1), (dp, 1))) logger.info(f"Using {dp} data parallel for the embedding lookup.") else: if self.vocab_table_size % mp != 0: raise ValueError(f"The vocab size of the embedding {self.vocab_table_size} must be a " f"multiple of parallel_config.model_parallel {mp}.") self.gather.shard(((mp, 1), (dp, 1))) logger.info(f"Using {dp} data parallel and {mp} " f"model parallel for the embedding lookup.") class LlamaRMSNorm(nn.Cell): r""" A self-defined RMSNorm operation using reduce mean. Args: dim (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. compute_type: The compute type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, dim, eps=1e-6, compute_type=mstype.float32): super(LlamaRMSNorm, self).__init__() self.eps = eps self.weight = Parameter(initializer('ones', (dim,), dtype=mstype.float32), parallel_optimizer=False) self.square = P.Square() self.mean = P.ReduceMean(keep_dims=True) self.add = P.Add() self.rsqrt = P.Rsqrt() self.mul = P.Mul() self.mul2 = P.Mul() self.cast = P.Cast() self.compute_type = compute_type def _norm(self, x): norm_factor = self.square(x) norm_factor = self.mean(norm_factor, -1) norm_factor = self.add(norm_factor, self.eps) norm_factor = self.rsqrt(norm_factor) x = self.cast(x, mstype.float16) norm_factor = self.cast(norm_factor, mstype.float16) return self.mul(x, norm_factor) def construct(self, x): """Forward of RMSNorm.""" original_type = x.dtype x = self.cast(x, self.compute_type) output = self._norm(x) output = self.cast(output, mstype.float16) weight = self.cast(self.weight, mstype.float16) output = self.mul2(output, weight) output = self.cast(output, original_type) return output def shard(self, strategy_in): """Parallel strategy configuratiuon interface.""" self.square.shard((strategy_in,)) self.mean.shard((strategy_in,)) self.rsqrt.shard((strategy_in,)) self.add.shard((strategy_in, ())) self.mul.shard((strategy_in, strategy_in)) self.mul2.shard((strategy_in, (1,))) class LlamaFeedForward(Cell): r""" LLaMA FeedForward. .. math:: (xW_1 * xW_3)W_2 Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: ValueError: `hidden_dim` is not a multiple of the model parallel way. ValueError: `dim` is not a multiple of the model parallel way. """ @_LogActionOnce(m_logger=logger, key='FeedForward', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(dim=Validator.check_positive_int, hidden_dim=Validator.check_positive_int, multiple_of=Validator.check_positive_int, compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward")) def __init__(self, dim, hidden_dim, multiple_of, hidden_act=LlamaSiLU, ffn_dim_multiplier=None, mup_width_base=-1, compute_dtype=mstype.float16, param_init_type=mstype.float32): super().__init__() if hidden_act is None or not (isinstance(hidden_act, str) or issubclass(hidden_act, nn.Cell)): raise TypeError(f"For FeedForward cell, the hidden_act should str type or nn.Cell type, " f"but got {hidden_act}.") if ffn_dim_multiplier is not None: hidden_dim = int((ffn_dim_multiplier + 0.01) * hidden_dim) hidden_dim = int(2 * hidden_dim / 3) hidden_dim = multiple_of * \ ((hidden_dim + multiple_of - 1) // multiple_of) self.dtype = compute_dtype self.hidden_act = hidden_act self.dim = dim self.hidden_dim = hidden_dim self.mul = P.Mul() self.cast = P.Cast() sigma = 0.01 if mup_width_base > 0: ratio = dim / mup_width_base sigma = sigma / math.sqrt(ratio) self.w1 = Linear(in_channels=dim, out_channels=hidden_dim, activation=hidden_act, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) self.w2 = Linear(in_channels=hidden_dim, out_channels=dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) self.w3 = Linear(in_channels=dim, out_channels=hidden_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) def construct(self, x): """Forward process of the FeedForward""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, self.dtype) # [bs, seq, hidden_dim] or [bs * seq, hidden_dim] gate = self.w1(x) # dp,1 -> dp, mp hidden = self.w3(x) # dp,1 -> dp, mp hidden = self.mul(hidden, gate) # dp,mp -> dp, mp output = self.w2(hidden) # dp,mp -> dp, 1 return output def shard(self, parallel_config): """sharding for feedforward""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel if self.hidden_dim % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_dim' must be a multiple of the" "num of model parallel, but got the hidden_dim is {} and the num of model " "parallel is {}.".format(self.hidden_dim, mp)) if self.dim % mp != 0: raise ValueError("For 'FeedForward', the class variable 'dim' must be a multiple of the num of " "model parallel, but got the dim is {} and the num of model parallel is {}." .format(self.dim, mp)) self.w1.shard(((dp, 1), (mp, 1)), strategy_activation=((dp, mp),)) self.w1.activation.shard(((dp, mp),)) self.w2.shard(((dp, mp), (1, mp))) self.w3.shard(((dp, 1), (mp, 1))) self.mul.shard(((dp, mp), (dp, mp))) pengcheng.mind-test/mindformers/models/llama/.ipynb_checkpoints/llama_tokenizer-checkpoint.py000066400000000000000000000263521471400134000332670ustar00rootroot00000000000000# coding=utf-8 # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA tokenizer APIs.""" import os from shutil import copyfile from typing import Any, Dict, List, Optional import sentencepiece as spm from mindformers.tools import logger from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import Tokenizer, AddedToken from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['LlamaTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "tokenizer.model"} @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class LlamaTokenizer(Tokenizer): r""" Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Tokenizer of llama will default add bos at the beginning of tokens and add eos token on the tail of tokens. Args: model_path(str): The spiece.model file path. add_bos(bool): The flag defines whether add bos token, Default True. eos_token(str): The token that represents the end-of-sentence. Default "". unk_token(str): The token that represents the unknown. Default "". pad_token(str): The token that represents the pad. Default "". sp_model_kwargs(str): Other kwargs for sp_model`. add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" clean_up_tokenization_spaces (bool): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import LlamaTokenizer >>> tokenizer = LlamaTokenizer.from_pretrained("llama_7b") >>> res = tokenizer("hello world") >>> print(res) {'input_ids': [1, 22172, 3186, 2]} >>> res = tokenizer("hello world", padding='max_length', max_length=10) >>> print(res) {'input_ids': [1, 22172, 3186, 2, 0, 0, 0, 0, 0, 0]} >>> res = tokenizer("hello world", return_tensors='ms') >>> print(res) {'input_ids': Tensor(shape=[3], dtype=Int32, value= [1, 22172, 3186, 2])} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['llama'] def __init__( self, vocab_file, unk_token="", bos_token="", eos_token="", pad_token="", sp_model_kwargs: Optional[Dict[str, Any]] = None, add_bos_token=True, add_eos_token=False, clean_up_tokenization_spaces=False, **kwargs, ): self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token super().__init__( bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, pad_token=pad_token, sp_model_kwargs=self.sp_model_kwargs, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs, ) self.vocab_file = vocab_file self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(vocab_file) def __getstate__(self): state = self.__dict__.copy() state["sp_model"] = None return state def __setstate__(self, d): self.__dict__ = d self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(self.vocab_file) @property def vocab_size(self): """Returns vocab size""" return self.sp_model.get_piece_size() def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def _tokenize(self, text, **kwargs): """Returns a tokenized string.""" return self.sp_model.encode(text, out_type=str) def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.sp_model.piece_to_id(token) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" token = self.sp_model.IdToPiece(index) return token def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" current_sub_tokens = [] out_string = "" prev_is_special = False for i, token in enumerate(tokens): # make sure that special tokens are not decoded using sentencepiece model if token in self.all_special_tokens: if not prev_is_special and i != 0: out_string += " " out_string += self.sp_model.decode(current_sub_tokens) + token prev_is_special = True current_sub_tokens = [] else: current_sub_tokens.append(token) prev_is_special = False out_string += self.sp_model.decode(current_sub_tokens) return out_string def save_vocabulary(self, save_directory, filename_prefix=None): """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. """ if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None out_vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file): copyfile(self.vocab_file, out_vocab_file) elif not os.path.isfile(self.vocab_file): with open(out_vocab_file, "wb") as fi: content_spiece_model = self.sp_model.serialized_model_proto() fi.write(content_spiece_model) return out_vocab_file def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): r"""Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) bos_token_id = [1] if self.add_bos_token else [] eos_token_id = [1] if self.add_eos_token else [] if token_ids_1 is None: return bos_token_id + ([0] * len(token_ids_0)) + eos_token_id return ( bos_token_id + ([0] * len(token_ids_0)) + eos_token_id + bos_token_id + ([0] * len(token_ids_1)) + eos_token_id ) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output pengcheng.mind-test/mindformers/models/llama/.ipynb_checkpoints/llama_transformer-checkpoint.py000066400000000000000000001334451471400134000336210ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA transformer Layer's APIs.""" from typing import Tuple, Optional import math import numpy as np try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import nn, ops import mindspore.common.dtype as mstype from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore.context import ParallelMode from mindspore.ops import operations as P from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindspore.common.initializer import Normal try: from mindspore.nn.layer.flash_attention import FlashAttention FLASHATTENTION_VALID = True except ImportError: FLASHATTENTION_VALID = False from mindformers.models.llama.llama_layer import LlamaFeedForward, LlamaRMSNorm, LlamaRotaryEmbedding from mindformers.modules.layers import _check_input_dtype, Linear from mindformers.modules.transformer import TransformerOpParallelConfig class LLamaAttention(nn.Cell): r""" This is an implementation of multihead attention in LLaMA. Args: - **batch_size** (int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. - **src_seq_length** (int): The sequence length of the query vector. - **tgt_seq_length** (int): The sequence length of the key and value vector. - **dim** (int): The hidden size of the input. - **head_dim** (int): The dim of head. - **n_heads** (int): The number of the heads. - **compute_dtype** (dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. - **softmax_compute_type** (dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. - **param_init_type** (dtype.Number): The parameter initialization type of the module. Default mstype. float32. Should be mstype.float32 or mstype.float16. - **use_past** (bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. - **parallel_config** (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - The input tokens with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, head_dim, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, head_dim). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, head_dim)). """ def __init__(self, batch_size, seq_length, dim: int = 512, n_heads: int = 8, n_kv_heads: Optional[int] = None, mup_width_base=-1, compute_dtype=mstype.float16, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, parallel_config=TransformerOpParallelConfig()): super().__init__() self.seq_length = seq_length self.hidden_size = dim self.n_head = n_heads self.head_dim = dim // n_heads self.n_kv_head = n_heads if n_kv_heads is None else n_kv_heads self.n_rep = self.n_head // self.n_kv_head self.dtype = compute_dtype self.softmax_dtype = softmax_compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention and FLASHATTENTION_VALID if self.hidden_size % self.n_head != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'n_head', but got the hidden_size is {} and the n_head is {}." .format(self.hidden_size, self.n_head)) if self.n_kv_head % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'n_kv_head' must be a multiple of " "'parallel_config.model_parallel', but got the n_kv_head is {} " "and the parallel_config.model_parallel is {}." .format(self.n_kv_head, parallel_config.model_parallel)) self.inv_norm_factor = Tensor(1.0 / math.sqrt(self.head_dim), dtype=compute_dtype) self.reshape = P.Reshape() self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.batch_matmul = P.BatchMatMul() self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.mul = P.Mul() self.add = P.Add() self.softmax = nn.Softmax().to_float(softmax_compute_dtype) self.cast = P.Cast() self.cast_attn = P.Cast() self.tile_kv = P.Tile() sigma = 0.01 if mup_width_base > 0: ratio = self.hidden_size / mup_width_base sigma = sigma / math.sqrt(ratio) self.apply_rotary_emb = LlamaRotaryEmbedding(self.head_dim, rotary_dtype) self.wo = Linear(in_channels=self.hidden_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wq = Linear(self.hidden_size, self.hidden_size, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wk = Linear(self.hidden_size, self.n_kv_head * self.head_dim, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wv = Linear(self.hidden_size, self.n_kv_head * self.head_dim, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.transpose.shard(((dp, 1, mp, 1),)) self.merger_head_transpose.shard(((dp, mp, 1, 1),)) self.batch_matmul_q_k.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.batch_matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul.shard(((dp, mp, 1, 1), ())) self.add.shard(((dp, 1, 1, 1), (dp, mp, 1, 1))) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.tile_kv.shard(((dp * mp, 1, 1, 1),)) self.apply_rotary_emb.shard((dp, mp, 1, 1)) self.wq.shard(((dp, 1), (mp, 1))) self.wk.shard(((dp, 1), (mp, 1))) self.wv.shard(((dp, 1), (mp, 1))) self.wo.shard(((dp, mp), (1, mp))) if parallel_config.use_seq_parallel and self.is_first_iteration: self.wo.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if parallel_config.recompute.select_recompute: self.apply_rotary_emb.recompute() self.tile_kv.recompute() self.batch_matmul_q_k.recompute() self.mul.recompute() self.add.recompute() self.cast_attn.recompute() self.softmax.softmax.recompute() self.batch_matmul.recompute() if self.use_flash_attention: self.flash_attention = FlashAttention(self.head_dim, dp=dp, mp=mp, next_block_num=0, high_precision=(softmax_compute_dtype == mstype.float32)) if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.flash_attention.shard(((dp, mp, 1, 1), (dp, mp, 1, 1), (dp, mp, 1, 1), (dp, 1, 1), ())) if parallel_config.recompute.select_recompute: self.flash_attention.recompute() if self.use_past: # operators used for state reuse seq_range = np.arange(seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.add_past = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.equal = P.Equal().shard(((dp, 1, 1), (dp, 1, 1))) self.less = P.Less().shard(((dp, 1, 1), (dp, 1, 1))) self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.add_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul_past.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def construct(self, x: Tensor, freqs_cis: Tuple[Tensor, Tensor], mask=None, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" ori_dtype = x.dtype # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] x = self.reshape(x, (-1, x.shape[-1])) # [bs * seq/1, hidden_dim] query = self.cast(self.wq(x), self.dtype) # dp, 1 -> dp, mp key = self.cast(self.wk(x), self.dtype) # dp, 1 -> dp, mp value = self.cast(self.wv(x), self.dtype) # dp, 1 -> dp, mp query = self.reshape(query, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) key = self.reshape(key, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_kv_head, self.head_dim)) value = self.reshape(value, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_kv_head, self.head_dim)) # [bs, seq/1, n_head/n_kv_head, head_dim] query = self.transpose(query, (0, 2, 1, 3)) key = self.transpose(key, (0, 2, 1, 3)) value = self.transpose(value, (0, 2, 1, 3)) # [bs, n_head/n_kv_head, seq/1, head_dim] query, key = self.apply_rotary_emb(query, key, freqs_cis) # dp, mp, 1, 1 # kv cache: [bs, n_kv_head, 1, head_dim] -> [bs, n_kv_head, seq, head_dim] key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = ( self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = (self.equal(self.range, valid_length)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) current_value = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add_past(key_past, current_key) value = self.add_past(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # kv share: [bs, n_kv_head, seq, head_dim] -> [bs, n_head, seq, head_dim] key = self._repeat_kv(key, self.n_rep) value = self._repeat_kv(value, self.n_rep) # q, k, v: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim], [bs, n_head, seq, head_dim] if self.use_flash_attention: attention = self.flash_attention(query, key, value, mask) attention = self._merge_heads(attention) else: attention = self._attn(query, key, value, mask) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] output = self.wo(attention) # dp, mp -> dp, 1 / dp * mp, 1 output = self.cast(output, ori_dtype) return output, layer_present def _repeat_kv(self, x, rep): if rep == 1: return x bs, n_kv_head, seqlen, head_dim = x.shape x = self.reshape(x, (bs * n_kv_head, 1, seqlen, head_dim)) x = self.tile_kv(x, (1, rep, 1, 1)) x = self.reshape(x, (bs, n_kv_head * rep, seqlen, head_dim)) return x def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _merge_heads(self, x): """ convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output """ # [bs, n_head, seq/1, head_dim] x = self.merger_head_transpose(x, (0, 2, 1, 3)) # dp,mp,1,1 -> dp,1,mp,1 # [bs, seq/1, n_head, head_dim] x_shape = x.shape if self.compute_in_2d: # [bs * seq/1, hidden_dim] new_shape = (-1, x_shape[-2] * x_shape[-1]) else: # [bs, seq/1, hidden_dim] new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge def _attn(self, query, key, value, mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # q, k: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim] score = self.batch_matmul_q_k(query, key) # score: [bs, n_head, seq/1, seq] score = self.mul(score, self.inv_norm_factor) score = self.add(mask, score) attention_probs = self.softmax(self.cast_attn(score, self.softmax_dtype)) # score, v: [bs, n_head, seq/1, seq], [bs, n_head, seq, head_dim] weighted_values = self.batch_matmul(self.cast(attention_probs, self.dtype), value) # [bs, n_head, seq/1, head_dim] attention_merge = self._merge_heads(weighted_values) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] return attention_merge class LLamaMLAAttention(nn.Cell): def __init__(self, batch_size, seq_length, dim: int = 512, n_heads: int = 8, mla_multiplier: int = 4, compute_dtype=mstype.float16, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, parallel_config=TransformerOpParallelConfig()): super().__init__() self.seq_length = seq_length self.hidden_size = dim self.n_head = n_heads self.head_dim = dim // n_heads self.lora_size = self.head_dim * mla_multiplier self.dtype = compute_dtype self.softmax_dtype = softmax_compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention and FLASHATTENTION_VALID if self.hidden_size % self.n_head != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'n_head', but got the hidden_size is {} and the n_head is {}." .format(self.hidden_size, self.n_head)) self.inv_norm_factor = Tensor(1.0 / math.sqrt(self.head_dim), dtype=compute_dtype) self.reshape = P.Reshape() self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.batch_matmul = P.BatchMatMul() self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.mul = P.Mul() self.add = P.Add() self.softmax = nn.Softmax().to_float(softmax_compute_dtype) self.cast = P.Cast() self.cast_attn = P.Cast() self.tile_kv = P.Tile() self.apply_rotary_emb = LlamaRotaryEmbedding(self.head_dim, rotary_dtype) self.q_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.q_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.wo = Linear(in_channels=self.hidden_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.transpose.shard(((dp, 1, mp, 1),)) self.merger_head_transpose.shard(((dp, mp, 1, 1),)) self.batch_matmul_q_k.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.batch_matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul.shard(((dp, mp, 1, 1), ())) self.add.shard(((dp, 1, 1, 1), (dp, mp, 1, 1))) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.tile_kv.shard(((dp * mp, 1, 1, 1),)) self.apply_rotary_emb.shard((dp, mp, 1, 1)) self.q_proj_down.shard(((dp, 1), (mp, 1))) self.q_proj_up.shard(((dp, 1), (mp, 1))) self.k_proj_down.shard(((dp, 1), (mp, 1))) self.k_proj_up.shard(((dp, 1), (mp, 1))) self.v_proj_down.shard(((dp, 1), (mp, 1))) self.v_proj_up.shard(((dp, 1), (mp, 1))) if parallel_config.use_seq_parallel and self.is_first_iteration: self.wo.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if parallel_config.recompute.select_recompute: self.apply_rotary_emb.recompute() self.tile_kv.recompute() self.batch_matmul_q_k.recompute() self.mul.recompute() self.add.recompute() self.cast_attn.recompute() self.softmax.softmax.recompute() self.batch_matmul.recompute() if self.use_flash_attention: self.flash_attention = FlashAttention(self.head_dim, dp=dp, mp=mp, next_block_num=0, high_precision=(softmax_compute_dtype == mstype.float32)) if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.flash_attention.shard(((dp, mp, 1, 1), (dp, mp, 1, 1), (dp, mp, 1, 1), (dp, 1, 1), ())) if parallel_config.recompute.select_recompute: self.flash_attention.recompute() if self.use_past: # operators used for state reuse seq_range = np.arange(seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.add_past = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.equal = P.Equal().shard(((dp, 1, 1), (dp, 1, 1))) self.less = P.Less().shard(((dp, 1, 1), (dp, 1, 1))) self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.add_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul_past.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def construct(self, x: Tensor, freqs_cis: Tuple[Tensor, Tensor], mask=None, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" ori_dtype = x.dtype # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] x = self.reshape(x, (-1, x.shape[-1])) # [bs * seq/1, hidden_dim] query = self.cast(self.q_proj_up(self.q_proj_down(x)), self.dtype) # dp, 1 -> dp, mp key = self.cast(self.k_proj_up(self.k_proj_down(x)), self.dtype) # dp, 1 -> dp, mp value = self.cast(self.v_proj_up(self.v_proj_down(x)), self.dtype) # dp, 1 -> dp, mp query = self.reshape(query, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) key = self.reshape(key, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) value = self.reshape(value, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) # [bs, seq/1, n_head/n_kv_head, head_dim] query = self.transpose(query, (0, 2, 1, 3)) key = self.transpose(key, (0, 2, 1, 3)) value = self.transpose(value, (0, 2, 1, 3)) # [bs, n_head/n_kv_head, seq/1, head_dim] query, key = self.apply_rotary_emb(query, key, freqs_cis) # dp, mp, 1, 1 # kv cache: [bs, n_kv_head, 1, head_dim] -> [bs, n_kv_head, seq, head_dim] key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = ( self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = (self.equal(self.range, valid_length)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) current_value = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add_past(key_past, current_key) value = self.add_past(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # kv share: [bs, n_kv_head, seq, head_dim] -> [bs, n_head, seq, head_dim] # q, k, v: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim], [bs, n_head, seq, head_dim] if self.use_flash_attention: attention = self.flash_attention(query, key, value, mask) attention = self._merge_heads(attention) else: attention = self._attn(query, key, value, mask) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] output = self.wo(attention) # dp, mp -> dp, 1 / dp * mp, 1 output = self.cast(output, ori_dtype) return output, layer_present def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _merge_heads(self, x): """ convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output """ # [bs, n_head, seq/1, head_dim] x = self.merger_head_transpose(x, (0, 2, 1, 3)) # dp,mp,1,1 -> dp,1,mp,1 # [bs, seq/1, n_head, head_dim] x_shape = x.shape if self.compute_in_2d: # [bs * seq/1, hidden_dim] new_shape = (-1, x_shape[-2] * x_shape[-1]) else: # [bs, seq/1, hidden_dim] new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge def _attn(self, query, key, value, mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # q, k: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim] score = self.batch_matmul_q_k(query, key) # score: [bs, n_head, seq/1, seq] score = self.mul(score, self.inv_norm_factor) score = self.add(mask, score) attention_probs = self.softmax(self.cast_attn(score, self.softmax_dtype)) # score, v: [bs, n_head, seq/1, seq], [bs, n_head, seq, head_dim] weighted_values = self.batch_matmul(self.cast(attention_probs, self.dtype), value) # [bs, n_head, seq/1, head_dim] attention_merge = self._merge_heads(weighted_values) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] return attention_merge class LLamaDecodeLayer(nn.Cell): r""" Transformer Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. seq_length(int): The input sequence length. layer_id(int): The layer id of current transformer block layer. dim(int): The hidden size of the input. num_heads(int): The number of the heads. multiple_of(int): The SwiGLU hidden layer size multiple of large power of 2. norm_eps (float): The epsilon value of the denominator. Default 1e-5. compute_dtype(dtype.Number): The computation type of the layer. Should be mstype.float32 or mstype.float16. Default mstype.float32. layernorm_compute_type(dtype.Number): The computation type of the norm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, seq_length), (batch_size, num_heads, seq_length, head_dim)). """ def __init__(self, batch_size, seq_length, layer_id, dim: int = 512, n_heads: int = 8, multiple_of: int = 256, n_kv_heads: Optional[int] = None, ffn_dim_multiplier: Optional[int] = None, mla_multiplier: Optional[int] = None, norm_eps: float = 1e-5, compute_dtype=mstype.float16, layernorm_compute_dtype=mstype.float32, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, mup_width_base=-1, parallel_config=TransformerOpParallelConfig()): super().__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) self.batch_size = batch_size self.compute_in_2d = compute_in_2d self.seq_length = seq_length self.layer_id = layer_id self.hidden_size = dim self.n_head = n_heads self.head_dim = self.hidden_size // self.n_head self.n_kv_head = n_heads if n_kv_heads is None else n_kv_heads self.dtype = compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.key_past = None self.value_past = None self.reshape = P.Reshape() self.add = P.Add() self.attention_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) self.ffn_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) if mla_multiplier is None: self.attention = LLamaAttention(batch_size=batch_size, seq_length=seq_length, dim=dim, n_heads=n_heads, n_kv_heads=n_kv_heads, mup_width_base=mup_width_base, compute_dtype=compute_dtype, softmax_compute_dtype=softmax_compute_dtype, rotary_dtype=rotary_dtype, param_init_type=param_init_type, use_past=use_past, use_flash_attention=use_flash_attention, compute_in_2d=compute_in_2d, use_past_shard=use_past_shard, parallel_config=parallel_config) else: self.attention = LLamaMLAAttention(batch_size=batch_size, seq_length=seq_length, dim=dim, n_heads=n_heads, compute_dtype=compute_dtype, mla_multiplier=mla_multiplier, softmax_compute_dtype=softmax_compute_dtype, rotary_dtype=rotary_dtype, param_init_type=param_init_type, use_past=use_past, use_flash_attention=use_flash_attention, compute_in_2d=compute_in_2d, use_past_shard=use_past_shard, parallel_config=parallel_config) self.feed_forward = LlamaFeedForward(dim=self.hidden_size, hidden_dim=4 * self.hidden_size, multiple_of=multiple_of, ffn_dim_multiplier=ffn_dim_multiplier, mup_width_base=mup_width_base, compute_dtype=compute_dtype, param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.feed_forward.shard(parallel_config) if self.compute_in_2d: self.add.shard(((dp, 1), (dp, 1))) self.attention_norm.shard((dp, 1)) self.ffn_norm.shard((dp, 1)) else: self.add.shard(((dp, 1, 1), (dp, 1, 1))) self.attention_norm.shard((dp, 1, 1)) self.ffn_norm.shard((dp, 1, 1)) self.feed_forward.mul.shard(((dp, 1, mp), (dp, 1, mp))) if parallel_config.use_seq_parallel and self.is_first_iteration: if self.compute_in_2d: self.add.shard(((dp * mp, 1), (dp * mp, 1))) self.attention_norm.shard((dp * mp, 1)) self.ffn_norm.shard((dp * mp, 1)) else: self.add.shard(((dp, mp, 1), (dp, mp, 1))) self.attention_norm.shard((dp, mp, 1)) self.ffn_norm.shard((dp, mp, 1)) self.feed_forward.w2.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if self.use_past: kv_shape = (batch_size, self.n_kv_head, seq_length, self.head_dim) self.key_past = Parameter(Tensor(np.zeros(kv_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(kv_shape), self.dtype), name="value_past") self.ones = P.Ones() self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (1,))) self.assign_past = P.Assign().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.mul_past.shard(((dp, mp, 1, 1), (1,))) self.assign_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) def construct(self, x, freqs_cis, mask=None, init_reset=True, batch_valid_length=None): """ Forward of transformer block. """ self._check_input(x, freqs_cis, mask, init_reset, batch_valid_length) # [bs, seq/1, hidden_dim] (first) [bs * seq/1, hidden_dim] (others) if self.compute_in_2d and x.ndim != 2: x = self.reshape(x, (-1, x.shape[-1])) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] input_x = self.attention_norm(x) key_reset = None value_reset = None if self.use_past and self.is_first_iteration: # reset states, init_reset True for reuse and False for reset self.assign_past(self.key_past, self.mul_past(self.key_past, self.cast(init_reset, self.dtype))) self.assign_past(self.value_past, self.mul_past(self.value_past, self.cast(init_reset, self.dtype))) key_reset = self.key_past value_reset = self.value_past # add dependency for desired execution order input_x = ops.depend(input_x, key_reset) input_x = ops.depend(input_x, value_reset) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] h, layer_present = self.attention(input_x, freqs_cis, mask, self.key_past, self.value_past, batch_valid_length) h = self.add(x, h) ffn_norm = self.ffn_norm(h) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] ffn_out = self.feed_forward(ffn_norm) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign_past(self.key_past, key_present) self.assign_past(self.value_past, value_present) key_update = self.key_past value_update = self.value_past # add dependency for desired execution order key_update = ops.depend(key_update, key_reset) value_update = ops.depend(value_update, value_reset) # add dependency for desired execution order ffn_out = ops.depend(ffn_out, value_update) ffn_out = ops.depend(ffn_out, key_update) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] out = self.add(h, ffn_out) return out, layer_present def _check_input(self, x, freqs_cis, mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype( x.dtype, "x", [mstype.float32, mstype.float16], self.cls_name) freqs_cos, freqs_sin, swap_mask = freqs_cis _check_input_dtype(freqs_cos.dtype, "freqs_cos", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(freqs_sin.dtype, "freqs_sin", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(swap_mask.dtype, "swap_mask", [mstype.float32, mstype.float16], self.cls_name) if mask is not None: _check_input_dtype(mask.dtype, "input_mask", [mstype.float32, mstype.float16], self.cls_name) if self.use_past: if not isinstance(init_reset, Tensor): init_reset = Tensor([init_reset], mstype.bool_) if not isinstance(batch_valid_length, Tensor): bs = freqs_cis[0].shape[0] batch_valid_length = self.ones((bs, 1), mstype.int32) _check_input_dtype(init_reset.dtype, "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(batch_valid_length.dtype, "batch_valid_length", [mstype.int32], self.cls_name) return True pengcheng.mind-test/mindformers/models/llama/__init__.py000066400000000000000000000017321471400134000237230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for llama""" from .llama_config import LlamaConfig from .llama import LlamaForCausalLM, LlamaModel from .llama_tokenizer import LlamaTokenizer from .llama_processor import LlamaProcessor __all__ = ['LlamaConfig', 'LlamaModel', 'LlamaForCausalLM', 'LlamaTokenizer', 'LlamaProcessor'] pengcheng.mind-test/mindformers/models/llama/__pycache__/000077500000000000000000000000001471400134000240175ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/llama/__pycache__/__init__.cpython-39.pyc000066400000000000000000000007421471400134000302120ustar00rootroot00000000000000a ?cf@sDdZddlmZddlmZmZddlmZddlm Z gdZ dS)zThe export function for llama) LlamaConfig)LlamaForCausalLM LlamaModel)LlamaTokenizer)LlamaProcessor)rrrrrN) __doc__Z llama_configrllamarrZllama_tokenizerrZllama_processorr__all__r r L/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/__init__.pys    pengcheng.mind-test/mindformers/models/llama/__pycache__/llama.cpython-39.pyc000066400000000000000000000251551471400134000275460ustar00rootroot00000000000000a rfB@sdZddlZddlZddlmmZzddlm Z Wne yTddlm Z Yn0ddl m Z mZddlmZddlmZddlmZmZzddlmZd ZWne yd ZYn0dd lmZdd l mZdd lmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddgZ8d"ddZ9Gd dde"Z:e+;e*j<Gd!dde"Z=dS)#zLLaMA models' APIs.N) Validator)Tensornn) ParallelMode) operations)_get_parallel_mode_is_sharding_propagation)FlashAttentionTF)Normal) lazy_inline)CrossEntropyLoss)MindFormerBook) BaseModel)Linear) _check_config) AttentionMask)MindFormerModuleTypeMindFormerRegister) LlamaConfig)LlamaEmbedding LlamaRMSNormprecompute_freqs_cis)LLamaDecodeLayer) cell_reuse)logger LlamaModelLlamaForCausalLMc Cstt|d|jd}t|trbt||jkrDtdt|dt|||jd}||}n$t|trr|}ntdt |dt||||jd} | |_tt|d|j d} t|j t r|j r|s| n|j j r|s|j |j j ddS)a Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: layer(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(Union[int, List[int]]) - Means the layer_index needs a offset, if there are other modules in the net. n_layers(int) - The total layers used for the model. rzThe length of `offset` z@ do not match `pipeline stage` {parallel_config.pipeline_stage}.z1`offset` must be `int` of list of `int`, but got .)recompute_slice_activationN)maxintpipeline_stage isinstancelistlen ValueErrormin TypeErrortypegradient_aggregation_group recomputeboolr!) layerlayer_idoffsetparallel_configZn_layersselect_recomputepp_disiZ offset_layerpp_iddisr8I/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama.pylayer_compute_dtype8s&        r:csDeZdZdZedZd edfdd Zd e dd d Z Z S) raA Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config(LlamaConfig): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 Returns: output: Tensor, the output of llama decoderlayer llamaNconfigcstj|ddt|j|js&|jr2t|j|j|_ |j |_ |j |_ d|_ |j|_|j o`t|_ |j rvtdn|j rtdt|j|j|j|j|j|jd\|_|_|_t|j|jjd|j|_tdg|jd|_ td g|jd|_!t"#|_$t"%|_&t"'|_(t")|_*t"+|_,t"-|_.t"/|_0t"1|_2t3|j4|j|j5d |_6t|j7g|jd|_7t")|_8t9:|_;t<|j D]}t=|j|j||j |j|j|j>|j?|j@|jA|jB|j|jC|jD|j|jE|j5|j|j |jF|jG|jH|jI|jd }tJ|||jK|j|j |jjLjMd |j;N|qtO|j|jB|jCd |_P|jjQ}tRtSjTfvr\tUsHd|j6_V|jjVdkr|jjVd|jP_V|j6W|j|j(Wd|j,Wd|ddff|j*W|dddfdf|j8W|ddfdf|j.W|ddff|j0W|dfdf|j2W|dfdf|jFr6|jPW|dfn|jPW|ddf|jrtXY|j$ddd}ttX(||jddftZj[|_>> from mindformers.models.llama import LlamaConfig, LlamaForCausalLM >>> config = LlamaConfig(batch_size=2) >>> network = LlamaForCausalLM(config=config) r;Nr<c stt|j|ddt|j|j|_|j|_t|_ t |_ t |_ t|_t|_t|_t|d|_|j}|jdkr|j|j}||}t|j|jd|j|jt|ddd|_t|jd|_ |jj!}|jj"}t#t$j%fvrt&s|j '|d ff|j'|d fd f|j'|d f|d ff|j'|d fd f|jj(rh|jj'|d fd fd n|jj'|d f|d ffd |jj)d kr|jj)d |j_)|*|dS) NTr>r<rF)sigmamean) in_channels out_channelshas_biasrJrD weight_initrCrr8rr)strategy_matmul)+rUrrVrr2ignore_token_idrZrirjrkrlrm StridedSliceslicervrwrpmulAddaddrmodelrRrLr^rrzrJrDr lm_headr lossrmodel_parallelrrrrr vocab_emb_dpr$load_checkpoint)rr=rratiormprr8r9rVsH            zLlamaForCausalLM.__init__cKsdt|tjiS)N input_ids)rrr)rrkwargsr8r8r9prepare_inputs_for_generation;s z.LlamaForCausalLM.prepare_inputs_for_generationTc Cs|j\} } ||d| | dfd} || |||} || } ||| |jtj}|durr||d| | fd}nH|j dkr|j r||d| | fd}||||j tj}| ||}|| tj} | j dkr| | d| jdf} | |d}| |d}|| ||}|S) zLlamaForCausalLM forward.)rrrrN)rrrrT)rT)rrrrrmrwrZrfloat32ndimtrainingrrrkr)rrlabelsr position_idsattention_maskZ input_embedsrrbszseqlenrrlogitsrZ label_maskrr8r8r9rAs&       zLlamaForCausalLM.construct)N)NNNNNTN) rrrrr rrr rrVrrrr8r8rr9rs *)F)>rmathnumpyrZmindspore.common.dtypecommonr@rZmindspore._checkparamr ImportError _checkparam mindsporerrZmindspore.contextr mindspore.opsrrimindspore.parallel._utilsrrZ"mindspore.nn.layer.flash_attentionr r\mindspore.common.initializerr r Zmindformers.core.loss.lossr mindformers.mindformer_bookr mindformers.models.base_modelrmindformers.modules.layersr2mindformers.modules.transformer.op_parallel_configrZ+mindformers.modules.transformer.transformerrZ#mindformers.tools.register.registerrr llama_configrZ llama_layerrrrZllama_transformerrutilsrZ tools.loggerr__all__r:rregisterMODELSrr8r8r8r9sH                   % pengcheng.mind-test/mindformers/models/llama/__pycache__/llama_config.cpython-39.pyc000066400000000000000000000055461471400134000310750ustar00rootroot00000000000000a &nfH@szdZddlmZddlmZmZddlmZmZddl m Z ddl m Z dd l mZd gZeejGd d d e Zd S) zLlama Config API.)Optional)default_transformer_configTransformerOpParallelConfig)MindFormerRegisterMindFormerModuleType)convert_mstype) BaseConfig)MindFormerBook LlamaConfigc(seZdZdZedZdddddddd d d d dd dd d dddd edddddddddddddddddf&eeeeeeee ee ee ee eeeee e e e e e e ee e e ee e e eee e ee e e d&fdd ZZS)r z: LLaMA config class which defines the model size. llamaii i}Ngh㈵>rifloat16float32FNonerg?iTg{Gz?)& batch_size seq_length hidden_size num_layers num_heads vocab_size multiple_of n_kv_headsffn_dim_multipliermla_multiplier rms_norm_eps bos_token_id eos_token_id pad_token_idignore_token_id compute_dtypelayernorm_compute_typesoftmax_compute_type rotary_dtypeparam_init_typeparallel_configuse_pastpretrain_seqlen extend_method compute_in_2duse_flash_attentionoffsetuse_past_shardcheckpoint_name_or_pathrepetition_penaltymax_decode_lengthtop_ktop_p do_samplemup_width_base embed_scale depth_scaleinit_stdc'( stt|jfi|'||_||_||_||_||_||_||_ ||_ | |_ | |_ | |_ t||_t||_t||_t||_t||_||_||_| |_| |_||_||_||_||_||_||_||_||_||_ ||_!||_"| |_#|!|_$|"|_%|#|_&|$|_'|%|_(|&|_)dS)N)*superr __init__rrrrrrrrrr r!rr*r'r(r)r&r+r3r"r#r$r%r,r-r.r/r0r1r2r4r5r6r7r8r9r:r;r<)(selfrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<kwargs __class__P/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_config.pyr>$sN(     zLlamaConfig.__init__)__name__ __module__ __qualname____doc__r get_config_support_list _support_listrintrfloatstrrboolr> __classcell__rCrCrArDr s N)rHtypingrZ+mindformers.modules.transformer.transformerrrmindformers.tools.registerrrutilsr base_configr mindformer_bookr __all__registerCONFIGr rCrCrCrDs     pengcheng.mind-test/mindformers/models/llama/__pycache__/llama_layer.cpython-39.pyc000066400000000000000000000313421471400134000307350ustar00rootroot00000000000000a rfV=@sdZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m mZddlmZddlmZdd lmZdd lmZzdd lmZWneyddlmZYn0dd l mZdd lmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*GdddeZ+GdddeZ,ddZ-dej.de+j/j0fe1e1e2dddZ3GdddeZ4Gdd d eZ5Gd!d"d"e jZ6Gd#d$d$eZ7dS)%zLLaMA Model Layers' APIs.)EnumN)Tensor) Parameter)nn) operations) functional)Cell)Normal) Validator)log) initializer)_get_parallel_mode) ParallelMode)Linear_check_input_dtype_args_type_validator_check_valid_value_checks)_LogActionOncec@seZdZdZdZdZdZdS)SeqExtendMethodzEStores the acceptable string identifiers for seq length extend methodPINTKNoneN)__name__ __module__ __qualname____doc__rrNONErrO/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_layer.pyr+srcs0eZdZdZfddZddZddZZS) LlamaSiLUz A self-defined SwiGlu. Inputs: - **x** (Tensor) - Tensor. Outputs: Tensor. x = x * sigmod(x). cs"tt|_t|_dSN)super__init__rSigmoidsigmoidPMulmul)self __class__rrr"<s  zLlamaSiLU.__init__cCs*|jj||j|d|dfdS)Nr)r$shardr')r(strategyrrrr+AszLlamaSiLU.shardcCs||||Sr )r'r$)r(xrrr constructEszLlamaSiLU.construct)rrrrr"r+r. __classcell__rrr)rr2s rcCsHtj|d|dftjd}tj|dtjd}t||g| |ggS)z Swap matrixdtype)npzerosfloat32identityblock)head_dimZ zero_blockZid_blockrrr get_swap_maskIsr9g@i)dimendthetacCsd}|tjjkr ||kr ||}|tjjkr4||9}td|dd|dtj}d|||}|tjjkrtd||d|tj} ntd|dtj} t | |}tj ||fdd} t | } t | } t | |d} t | |d} t|} t | |d} | | | fS) z< Precompute of freqs and mask for rotary embedding. g?rr0N)axisr1)rrvaluerr3arangeastyper5router concatenatecossinrr9)r:r;r<r2pretrain_seqlen extend_methodratioZ freqs_basefreqstZemb freqs_cos freqs_sin swap_maskrrrprecompute_freqs_cisPs&  "        rOcsHeZdZdZdejffdd ZddZeeddd Z d d Z Z S) LlamaRotaryEmbeddinga Rotary Position Embedding. Args: - **head_dim** (int): The dim of multi head attention. - **compute_dtype** (mstype): The compute type, default mstype.float16. - **parallel_config** (dict): - Parallel Config. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. csFtjdd||_||_t|_t|_t |_ t |_ dS)NF) auto_prefix) r!r"r8r2r%Addadd BatchMatMulbmm_swapr&r'Castcast)r(r8 compute_dtyper)rrr"s   zLlamaRotaryEmbedding.__init__cCs|||}|Sr )rV)r(r-rNrrr rotate_halfs z LlamaRotaryEmbedding.rotate_half)xqxkc Cs|j}|||j}|||j}|\}}}|||||||||}|||||||||} |||}|| |} || fS)z%Forward of rotary position embedding.)r2rXrTr'rZ) r(r[r\ freqs_cis original_typerLrMrNZxq_outZxk_outrrrr.s   zLlamaRotaryEmbedding.constructcCs@|j||f|j|df|j||ddddffdS)Nr=r=rr=)rTr+rVr'r(Z strategy_inrrrr+szLlamaRotaryEmbedding.shard) rrrrmstyper5r"rZrr.r+r/rrr)rrPrs  rPcs`eZdZdZeedeejfvde e j e j de j dffdd Zdd Zd d ZZS) LlamaEmbeddinga Embedding Layer. Args: - **vocab_size** (int): Size of the dictionary of embeddings. - **embedding_size** (int): The size of each embedding vector. - **param_init_type** (mstype): The param init type, default mstype.float32. - **parallel_config** (TransformerOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. - **param_init** (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). Embeddingm_loggerkey no_warning)vocab_table_sizeembedding_sizenormalcsFt||_||_tt||j|jg|dddd|_t|_ dS)Nr1embedding_weightT)nameparallel_optimizer) r!r"rhrirr rkr%Gathergather)r(rhriparam_init_type param_initr)rrr"s zLlamaEmbedding.__init__cCs2tt|dtjtjg|j||j|d}|S)zForward of vocab embedding. input_idsr) rFr2raint32int64cls_namerork)r(rroutputrrrr.szLlamaEmbedding.constructcCs|j}|j}|jr:|jd|dfftd|dnV|j|dkr`td|jd|d|j|df|dfftd|d |d d S) zsharding for embeddingr_r=zUsing z( data parallel for the embedding lookup.rz The vocab size of the embedding z6 must be a multiple of parallel_config.model_parallel .z data parallel and z) model parallel for the embedding lookup.N) data_parallelmodel_parallel vocab_emb_dpror+loggerinforh ValueErrorr(parallel_configdpmprrrr+s  zLlamaEmbedding.shard)rrrrrr|r r STAND_ALONErr check_positive_intrar5r"r.r+r/rrr)rrbs  rbcs@eZdZdZdejffdd ZddZddZd d Z Z S) LlamaRMSNorma A self-defined RMSNorm operation using reduce mean. Args: dim (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. compute_type: The compute type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. gư>cstt|||_ttd|ftjddd|_t |_ t j dd|_ t |_t |_t |_t |_t |_||_dS)Nonesr1F)rmT) keep_dims)r!rr"epsrr rar5weightr%Squaresquare ReduceMeanmeanrSrTRsqrtrsqrtr&r'mul2rWrX compute_type)r(r:rrr)rrr"s      zLlamaRMSNorm.__init__cCsV||}||d}|||j}||}||tj}||tj}|||S)Nr>) rrrTrrrXrafloat16r')r(r- norm_factorrrr_norms   zLlamaRMSNorm._normcCsX|j}|||j}||}||tj}||jtj}|||}|||}|S)zForward of RMSNorm.)r2rXrrrarrr)r(r-r^rwrrrrr.s   zLlamaRMSNorm.constructcCs^|j|f|j|f|j|f|j|df|j||f|j|dfdS)z+Parallel strategy configuratiuon interface.r)r=N)rr+rrrTr'rr`rrrr+s zLlamaRMSNorm.shard) rrrrrar5r"rr.r+r/rrr)rrs     rc seZdZdZeedeejfvde e j e j e j e e je jgde e je jgddeddde je jffdd Zd d Zd d ZZS)LlamaFeedForwardaE LLaMA FeedForward. .. math:: (xW_1 * xW_3)W_2 Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: ValueError: `hidden_dim` is not a multiple of the model parallel way. ValueError: `dim` is not a multiple of the model parallel way. FeedForwardrd)r: hidden_dim multiple_ofrYrpNr>{Gz?c s"t|dus(t|ts8t|tjs8td|d|durPt|d|}td|d}|||d|}||_ ||_ ||_ ||_ t |_t |_|} |dkr||} | t| } t|||d|| t| dd d |_t||d|| t| dd d |_t||d|| t| dd d |_dS) NzNFor FeedForward cell, the hidden_act should str type or nn.Cell type, but got rxrr0r=rF)sigmar) in_channels out_channels activationhas_biasrYrp weight_init)rrrrYrpr)r!r" isinstancestr issubclassrr TypeErrorintr2 hidden_actr:rr%r&r'rWrXmathsqrtrr w1w2w3) r(r:rrrffn_dim_multipliermup_width_baseinit_stdrYrprrIr)rrr",sV       zLlamaFeedForward.__init__cCsZtt|dtjtjg|j|||j}||}| |}| ||}| |}|S)z"Forward process of the FeedForwardr-) rrsr2rar5rrvrXrrr'r)r(r-gatehiddenrwrrrr.ls    zLlamaFeedForward.constructcCs|j}|j}|j|dkr,td|j||j|dkrLtd|j||jj|df|dff||ffd|jj||ff|j ||fd|ff|j |df|dff|j ||f||ffdS)zsharding for feedforwardrzFor 'FeedForward', the class variable 'hidden_dim' must be a multiple of thenum of model parallel, but got the hidden_dim is {} and the num of model parallel is {}.zFor 'FeedForward', the class variable 'dim' must be a multiple of the num of model parallel, but got the dim is {} and the num of model parallel is {}.r=)strategy_activationN) ryrzrr~formatr:rr+rrrr'rrrrr+ws"zLlamaFeedForward.shard)rrrrrr|r rrrr rrrar5rrr"r.r+r/rrr)rrs.    7 r)8renumrrnumpyr3mindspore.common.tensorrmindspore.common.parameterr mindsporermindspore.common.dtypecommonr2ra mindspore.opsrr%rrsmindspore.nn.cellrmindspore.common.initializerr mindspore._checkparamr ImportError _checkparamr r|r mindspore.parallel._utilsr mindspore.contextrmindformers.modules.layersrrrrmindformers.tools.loggerrrrr9r5rr@rfloatrOrPrbrrrrrrsJ                "589pengcheng.mind-test/mindformers/models/llama/__pycache__/llama_processor.cpython-39.pyc000066400000000000000000000031611471400134000316360ustar00rootroot00000000000000a ?cf @s^dZddlmZddlmZddlmZddlmZm Z dgZ e e j Gd ddeZ d S) z LlamaProcessor )MindFormerBook) BaseTokenizer) BaseProcessor)MindFormerRegisterMindFormerModuleTypeLlamaProcessorcs8eZdZdZedZd fdd Zd d d ZZ S) r zV Llama processor, consists of a tokenizer (BaseTokenizer) for text input. llamaN max_lengthmscstt|j||||ddS)N) tokenizerr paddingreturn_tensors)superr __init__)selfrr rr __class__S/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_processor.pyr"s  zLlamaProcessor.__init__cCsli}|durh|jrht|jts4tdt|jdt|trD|g}|j||j|j|jdd}||d<|S)z call functionNz;tokenizer should inherited from the BaseTokenizer, but got .)rr r input_idstext) r isinstancer TypeErrortypestrrr r)r text_input image_inputoutput text_outputrrr__call__+s     zLlamaProcessor.__call__)Nr r r )NN) __name__ __module__ __qualname____doc__rget_processor_support_list _support_listrr# __classcell__rrrrr s   N)r'mindformers.mindformer_bookrbase_tokenizerrbase_processorrZtools.registerrr__all__register PROCESSORr rrrrs    pengcheng.mind-test/mindformers/models/llama/__pycache__/llama_tokenizer.cpython-39.pyc000066400000000000000000000226041471400134000316340ustar00rootroot00000000000000a rf,@sdZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl mZddlmZmZddlmZmZd gZd d iZeejGd d d eZdS) zLLaMA tokenizer APIs.N)copyfile)AnyDictListOptional)logger)MindFormerBook) Tokenizer AddedToken)MindFormerRegisterMindFormerModuleTypeLlamaTokenizer vocab_fileztokenizer.modelcseZdZdZeZddgZdgZe dZ d*e e e efd fdd ZddZddZeddZddZddZddZddZddZd+d d!Zd,d"d#Zd-eee eeeeed$fd%d& Zd.eee eeeed'd(d)ZZ S)/r aj Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Tokenizer of llama will default add bos at the beginning of tokens and add eos token on the tail of tokens. Args: model_path(str): The spiece.model file path. add_bos(bool): The flag defines whether add bos token, Default True. eos_token(str): The token that represents the end-of-sentence. Default "". unk_token(str): The token that represents the unknown. Default "". pad_token(str): The token that represents the pad. Default "". sp_model_kwargs(str): Other kwargs for sp_model`. add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" clean_up_tokenization_spaces (bool): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import LlamaTokenizer >>> tokenizer = LlamaTokenizer.from_pretrained("llama_7b") >>> res = tokenizer("hello world") >>> print(res) {'input_ids': [1, 22172, 3186, 2]} >>> res = tokenizer("hello world", padding='max_length', max_length=10) >>> print(res) {'input_ids': [1, 22172, 3186, 2, 0, 0, 0, 0, 0, 0]} >>> res = tokenizer("hello world", return_tensors='ms') >>> print(res) {'input_ids': Tensor(shape=[3], dtype=Int32, value= [1, 22172, 3186, 2])} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. input_idsattention_maskztokenizer_config.jsonllamaNTF)sp_model_kwargsc s|dur in||_t|tr*t|dddn|}t|trFt|dddn|}t|trbt|dddn|}t|tr~t|dddn|}tjf|||||j| d| ||_||_||_t j fi|j|_ |j |dS)NF)lstriprstrip) bos_token eos_token unk_token pad_tokenrclean_up_tokenization_spaces) r isinstancestrr super__init__r add_bos_token add_eos_tokenspmSentencePieceProcessorsp_modelLoad) selfrrrrrrr"r#rkwargs __class__S/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_tokenizer.pyr!Qs(  zLlamaTokenizer.__init__cCs|j}d|d<|S)Nr&)__dict__copy)r(stater,r,r- __getstate__rs zLlamaTokenizer.__getstate__cCs,||_tjfi|j|_|j|jdS)N)r.r$r%rr&r'r)r(dr,r,r- __setstate__wszLlamaTokenizer.__setstate__cCs |jS)zReturns vocab size)r&Zget_piece_sizer(r,r,r- vocab_size|szLlamaTokenizer.vocab_sizecs(fddtjD}|j|S)zReturns vocab as a dictcsi|]}||qSr,)convert_ids_to_tokens).0ir4r,r- z,LlamaTokenizer.get_vocab..)ranger5updateadded_tokens_encoder)r(vocabr,r4r- get_vocabs zLlamaTokenizer.get_vocabcKs|jj|tdS)zReturns a tokenized string.)out_type)r&encoder)r(textr)r,r,r- _tokenizeszLlamaTokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)r&Z piece_to_id)r(tokenr,r,r-_convert_token_to_idsz#LlamaTokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r&Z IdToPiece)r(indexrDr,r,r-_convert_id_to_tokens z#LlamaTokenizer._convert_id_to_tokencCs|g}d}d}t|D]R\}}||jvrX|s:|dkr:|d7}||j||7}d}g}q||d}q||j|7}|S)z:Converts a sequence of tokens (string) in a single string.Fr T) enumerateall_special_tokensr&decodeappend)r(tokensZcurrent_sub_tokens out_stringZprev_is_specialr8rDr,r,r-convert_tokens_to_strings   z'LlamaTokenizer.convert_tokens_to_stringcCstj|std|dStj||r0|dndtd}tj|jtj|krttj |jrtt |j|nLtj |jst |d$}|j }||Wdn1s0Y|S)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. z*Vocabulary path (%s) should be a directoryN-rHrwb)ospathisdirrerrorjoinVOCAB_FILES_NAMESabspathrisfileropenr&Zserialized_model_protowrite)r(save_directoryfilename_prefixZout_vocab_filefiZcontent_spiece_modelr,r,r-save_vocabularys  (  (zLlamaTokenizer.save_vocabularycCsL|jr|jgng}|jr |jgng}|||}|durH||||}|S)z5Insert the special tokens to the input_ids. CurrentlyN)r" bos_token_idr# eos_token_idr( token_ids_0 token_ids_1rarboutputr,r,r- build_inputs_with_special_tokenss  z/LlamaTokenizer.build_inputs_with_special_tokens)rdrealready_has_special_tokensreturncs|rtj||ddS|jr"dgng}|jr2dgng}|durT|dgt||S|dgt|||dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rdrerhNr)r get_special_tokens_maskr"r#len)r(rdrerhrarbr*r,r-rks(  z&LlamaTokenizer.get_special_tokens_mask)rdrericCs`|jr|jgng}|jr |jgng}dgt|||}|dur\|dgt|||7}|S)a Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rNrj)r"rar#rbrlrcr,r,r-$create_token_type_ids_from_sequencess z3LlamaTokenizer.create_token_type_ids_from_sequences)rrrrNTFF)N)N)NF)N)!__name__ __module__ __qualname____doc__rXvocab_files_namesmodel_input_names FILE_LISTrget_tokenizer_support_list _support_listrrrrr!r1r3propertyr5r?rCrErGrPr`rgrintboolrkrm __classcell__r,r,r*r-r 'sP# !    ))rqrSshutilrtypingrrrrZ sentencepiecer$mindformers.toolsrmindformers.mindformer_bookr!mindformers.models.base_tokenizerr r mindformers.tools.registerr r __all__rXregister TOKENIZERr r,r,r,r-s    pengcheng.mind-test/mindformers/models/llama/__pycache__/llama_transformer.cpython-39.pyc000066400000000000000000000613011471400134000321610ustar00rootroot00000000000000a זnf@sXdZddlmZmZddlZddlZzddlmZWne yRddlm ZYn0ddl m Z m Z ddlmmZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZzdd l m!Z!d Z"Wne ydZ"Yn0ddl#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)ddl*m+Z+Gddde j,Z-Gddde j,Z.Gddde j,Z/dS)zLLaMA transformer Layer's APIs.)TupleOptionalN) Validator)nnops)Tensor) Parameter) ParallelMode) operations)_get_parallel_mode_is_sharding_propagation)Normal)FlashAttentionTF)LlamaFeedForward LlamaRMSNormLlamaRotaryEmbedding)_check_input_dtypeLinear)TransformerOpParallelConfigcseZdZdZdddddejejejejddddefeee edfd d Z de e e e fd d d Z ddZddZddZddZZS)LLamaAttentiona This is an implementation of multihead attention in LLaMA. Args: - **batch_size** (int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. - **src_seq_length** (int): The sequence length of the query vector. - **tgt_seq_length** (int): The sequence length of the key and value vector. - **dim** (int): The hidden size of the input. - **head_dim** (int): The dim of head. - **n_heads** (int): The number of the heads. - **compute_dtype** (dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. - **softmax_compute_type** (dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. - **param_init_type** (dtype.Number): The parameter initialization type of the module. Default mstype. float32. Should be mstype.float32 or mstype.float16. - **use_past** (bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. - **parallel_config** (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - The input tokens with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, head_dim, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, head_dim). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, head_dim)). N{Gz?F)dimn_heads n_kv_headsc st||_||_||_|||_|dur2|n||_|j|j|_||_| |_ d|_ | |_ ||_ | ojt |_|j|jdkrtd|j|j|j|jdkrtd|j|jtdt|j|d|_t|_t|_t|_t|_tjdd|_t|_ t!|_"t#$%| |_&t'|_(t'|_)t*|_+|}|dkrn|j|}|t|}t,|j| |_-t.|j|jd|t/|dd | d |_0t.|j|jd|t/|dd | d |_1t.|j|j|jd|t/|dd | d |_2t.|j|j|jd|t/|dd | d |_3|j4}|j}t5t6j7fvr4t8s|j9|d |d ff|j9||d d ff|j9||d d f||d d ff|j9||d d f||d d ff|j 9||d d fd f|j"9|d d d f||d d ff|j&j&9||d d ff|j+9||d d d ff|j-9||d d f|j19|d f|d ff|j29|d f|d ff|j39|d f|d ff|j09||fd |ff|j:r|j r|j0j9||fd |ff||d ffd|j;j<r|j-;|j+;|j;|j ;|j";|j);|j&j&;|j;|jrt=|j||d| t>j?kd|_@t5t6j7fvrLt8s|j@9||d d f||d d f||d d f|d d fd f|j;j<r|j@;|j rtAB|d d d}ttAC||d d ft>jD|_EtF9|d d ff|_Gt!9|d d d f|d d d ff|_HtI9|d d f|d d ff|_JtK9|d d f|d d ff|_Lt9|d d d f|d d d ff|_M|r|jH9||d d f||d d ff|jM9||d d f|d d d ffdS)NTrFor 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple of 'n_head', but got the hidden_size is {} and the n_head is {}.zFor 'MultiHeadAttention', the class variable 'n_kv_head' must be a multiple of 'parallel_config.model_parallel', but got the n_kv_head is {} and the parallel_config.model_parallel is {}.?dtype transpose_bF)sigmamean) in_channels out_channelshas_bias compute_dtype weight_initparam_init_type)r'r(r)r*out_strategy_matmuldpmpnext_block_numhigh_precisionr)Nsuper__init__ seq_length hidden_sizen_headhead_dim n_kv_headn_repr softmax_dtypeis_first_iterationuse_past compute_in_2dFLASHATTENTION_VALIDuse_flash_attention ValueErrorformatmodel_parallelrmathsqrtinv_norm_factorPReshapereshape Transpose transposemerger_head_transpose BatchMatMul batch_matmulbatch_matmul_q_kMulmulAddaddrSoftmaxto_floatsoftmaxCastcast cast_attnTiletile_kvrapply_rotary_embrr wowqwkwv data_parallelr r AUTO_PARALLELr sharduse_seq_parallel recomputeselect_recomputermstypefloat32flash_attentionnparangetileint32range ExpandDims expand_dimsadd_pastEqualequalLesslessmul_past)self batch_sizer6rrrmup_width_baseinit_stdr(softmax_compute_dtype rotary_dtyper*r>rAr?use_past_shardparallel_configr#ratior0r1 seq_range __class__r,U/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_transformer.pyr5is                       &          4  $  $ zLLamaAttention.__init__x freqs_ciscCs8|j}||d|jdf}||||j}||||j} ||||j} ||d||j|j |j f}|| d||j|j |j f} || d||j|j |j f} | |d}| | d} | | d} | || |\}} | } | } |jr|jrP||j|ddd|j} || || d} || || d} nr|d}||d}||j||j} || || d}|| || d}|||} |||} | } | } | | f}|| |j} || |j} |jr ||| | |}||}n||| | |}||}|||}||fSz)Forward process of the MultiHeadAttentionrrr+r+r)rr+r+)r rJshaperYr_r`ra!_get_seq_length_under_incrementalr6r8r9r:rLr]r>r=rvroviewastyperwrqrtrr _repeat_kvr;rArj _merge_heads_attnr^rxrrmaskkey_past value_pastbatch_valid_length ori_dtypequerykeyvalue key_present value_presentvalid_length_vector valid_length current_key current_value layer_present attentionoutputr,r,r constructs^         zLLamaAttention.constructcCsb|dkr |S|j\}}}}||||d||f}||d|ddf}|||||||f}|S)Nr+)rrJr\)rxrrepbsr:seqlenr9r,r,rr4szLLamaAttention._repeat_kvcCs|jr|jsdS|SzxReturn the length of the tensor. For the incremental prediction, the seq length for the input is 1. r+r>r=rxlengthr,r,rr=s z0LLamaAttention._get_seq_length_under_incrementalcCsP||d}|j}|jr.d|d|df}n|d|ddf}|||}|Sz convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output rrrr+rMrr?rJrxrx_shape new_shapex_merger,r,rrEs  zLLamaAttention._merge_headsc Cs^|||}|||j}|||}||||j}||||j |}| |}|Sau Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores rPrRrGrTrWrZr<rOrYr r rxrrrrscoreattention_probsweighted_valuesattention_merger,r,rr\s   zLLamaAttention._attn)NNNN)__name__ __module__ __qualname____doc__rhfloat16ririntrr5rrrrrrr __classcell__r,r,rrr-s8> B rc s|eZdZdddejejejejddddef eeedfdd Zde e e e fd d d Z d d Z ddZ ddZZS)LLamaMLAAttentionrrF)rrmla_multipliercst||_||_||_|||_|j||_||_||_d|_ | |_ | |_ | oVt |_ |j|jdkr~td|j|jtdt|j|d|_t|_t|_t|_t|_tjdd|_t|_t|_ t!"#||_$t%|_&t%|_'t(|_)t*|j||_+t,|j|jd|| d|_-t,|j|jd|| d|_.t,|j|jd|| d|_/t,|j|jd|| d|_0t,|j|jd|| d|_1t,|j|jd|| d|_2t,|j|jd|| d|_3|j4}|j5}t6t7j8fvrt9s|j:|d |d ff|j:||d d ff|j:||d d f||d d ff|j:||d d f||d d ff|j:||d d fd f|j :|d d d f||d d ff|j$j$:||d d ff|j):||d d d ff|j+:||d d f|j-:|d f|d ff|j.:|d f|d ff|j/:|d f|d ff|j0:|d f|d ff|j1:|d f|d ff|j2:|d f|d ff|j;r|j r|j3j:||fd |ff||d ffd |j|j||d|t?j@kd |_At6t7j8fvr2t9sf|jA:||d d f||d d f||d d f|d d fd f|jr?r@rArBrCrrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrrUrVrWrXrYrZr[r\rr]r q_proj_down q_proj_up k_proj_down k_proj_up v_proj_down v_proj_upr^rbrDr r rcr rdrerfrgrrhrirjrkrlrmrnrorprqrrrsrtrurvrw)rxryr6rrrr(r|r}r*r>rAr?r~rr0r1rrr,rr5xs                &          4  $  $ zLLamaMLAAttention.__init__NrcCs.|j}||d|jdf}|||||j}|||||j} ||| ||j} ||d| |j |j |j f}|| d| |j |j |j f} || d| |j |j |j f} ||d}|| d} || d} ||| |\}} | } | } |jr|jrb||j|ddd|j} || || d} || || d} nr|d}||d}||j||j} || || d}|| || d}|||} |||} | } | } | | f}|jr||| | |}||}n||| | |}||}|||}||fSr)r rJrrYrrrrrrrr6r8r9rLr]r>r=rvrorrrwrqrtrrrArjrrr^rr,r,rrsZ         zLLamaMLAAttention.constructcCs|jr|jsdS|Srrrr,r,rrCs z3LLamaMLAAttention._get_seq_length_under_incrementalcCsP||d}|j}|jr.d|d|df}n|d|ddf}|||}|Srrrr,r,rrKs  zLLamaMLAAttention._merge_headsc Cs^|||}|||j}|||}||||j}||||j |}| |}|Srrrr,r,rrbs   zLLamaMLAAttention._attn)NNNN)rrrrhrrirrr5rrrrrrrr,r,rrrws0 @rcseZdZdZdddddddejejejejejdddddd d efeeee ee ee ee d fd d Z dddZ ddZ ZS)LLamaDecodeLayera Transformer Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. seq_length(int): The input sequence length. layer_id(int): The layer id of current transformer block layer. dim(int): The hidden size of the input. num_heads(int): The number of the heads. multiple_of(int): The SwiGLU hidden layer size multiple of large power of 2. norm_eps (float): The epsilon value of the denominator. Default 1e-5. compute_dtype(dtype.Number): The computation type of the layer. Should be mstype.float32 or mstype.float16. Default mstype.float32. layernorm_compute_type(dtype.Number): The computation type of the norm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, seq_length), (batch_size, num_heads, seq_length, head_dim)). rrNgh㈵>Frrr)rr multiple_ofrffn_dim_multiplierrnorm_epscs.t|s|rt|||_||_||_||_||_||_ |j|j |_ |durZ|n||_ | |_ d|_ ||_||_d|_d|_t|t|gtjd|_t|_t|_t|_t|j| | d|_t|j| | d|_ | durt!|||||||| ||||||||d|_"n&t#||||| | ||||||||d|_"t$|jd|j|| ||| |d|_%|j&}|j'}t(t)j*fvrt+sZ|j%,||jr|j,|df|dff|j,|df|j ,|df|j,|dfd fnt|j,|ddf|ddff|j,|ddf|j ,|ddf|j%j-,|d|f|d|ff|j,|ddfd f|j.rR|j rR|jr|j,||df||dff|j,||df|j ,||df|j,||dfd fnV|j,||df||dff|j,||df|j ,||df|j,||dfd f|j%j/j,||fd|ff||dffd |jr*||j ||j f}t0tt12||j d d |_t0tt12||j d d |_t3|_4t,|dddfd f|_5t6,|dddf|dddff|_7|r*|j5,||ddfd f|j7,||ddf||ddffdS)NTr) compute_type)ryr6rrrrzr{r(r|r}r*r>rAr?r~r)ryr6rrr(rr|r}r*r>rAr?r~rr)r hidden_dimrrrzr{r(r*r+)r+r-r)namer)8r4r5rcheck_positive_intryr?r6layer_idr7r8r9r:r r=r>rrrrErFrhr scale_depthrHrQ depth_mulrIrJrSrTrattention_normffn_normrrrr feed_forwardrbrDr r rcr rdrRrew2rrkzerosOnesonesrwAssign assign_past)rxryr6r layer_numrrrrrrrr(layernorm_compute_dtyper|r}r*r>rAr?r~rz depth_scaler{rr0r1Zkv_shaperr,rr5s          ( $zLLamaDecodeLayer.__init__Tc Cs|||||||jr8|jdkr8||d|jdf}||}d}d}|jr|jr||j | |j | ||j ||j | |j | ||j |j }|j }t||}t||}|||||j |j |\} } || |j} ||| } || } || } d} d}|jr`| \}}||j |||j ||j }|j } t||}t| |} t| | } t| |} || |j} || | }|| fS)z Forward of transformer block. rrN) _check_inputr?ndimrJrrr>r=rrrwrYr rrdependrrrrTrr)rxrrr init_resetrinput_x key_reset value_resethrrZffn_out value_update key_updaterroutr,r,rr=sF  ""            zLLamaDecodeLayer.constructc Cst|jdtjtjg|j|\}}}t|jdtjtjg|jt|jdtjtjg|jt|jdtjtjg|j|durt|jdtjtjg|j|jr t|tst|gtj }t|ts|dj d} | | dftj }t|jd tj g|jt|jd tj g|jd S) z Check inputsr freqs_cos freqs_sin swap_maskN input_maskrr+rrT) rr rhrircls_namer> isinstancerbool_rrrn) rxrrrrrrrrrr,r,rrps$   zLLamaDecodeLayer._check_input)NTN)rrrrrhrrirrrfloatr5rrrr,r,rrr}sB? 3r)0rtypingrrrEnumpyrkmindspore._checkparamr ImportError _checkparam mindsporerrmindspore.common.dtypecommonr rhmindspore.common.tensorrmindspore.common.parameterrmindspore.contextr mindspore.opsr rHmindspore.parallel._utilsr r mindspore.common.initializerr "mindspore.nn.layer.flash_attentionrr@Z$mindformers.models.llama.llama_layerrrrmindformers.modules.layersrrmindformers.modules.transformerrCellrrrr,r,r,rs<          Lpengcheng.mind-test/mindformers/models/llama/convert_weight.py000066400000000000000000000155271471400134000252220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Convert llama weight. Support huggingface format and Meta format. """ import os import json import argparse from pathlib import Path import numpy as np import mindspore as ms from mindspore import ops def convert_meta_torch_ckpt(ckpt_dir, output_name, dtype=ms.float16): """Support convert meta weight splited.""" print(f"Trying to convert pytorch checkpoint in '{ckpt_dir}'.", flush=True) try: from torch import load except: raise ImportError(f"Failed to load pytorch checkpoint. Please make sure pytorch is available.") dic = { 'tok_embeddings.weight': 1, 'norm.weight': None, 'output.weight': 0, 'attention.wq.weight': 0, 'attention.wk.weight': 0, 'attention.wv.weight': 0, 'attention.wo.weight': 1, 'feed_forward.w1.weight': 0, 'feed_forward.w2.weight': 1, 'feed_forward.w3.weight': 0, 'attention_norm.weight': None, 'ffn_norm.weight': None, 'rope.freqs': None, } ckpt_paths = sorted(Path(ckpt_dir).glob("*.pth")) if not ckpt_paths: print(f"Do not find pytorch checkpoint in '{ckpt_dir}'.", flush=True) return False with open(Path(ckpt_dir) / "params.json", "r") as f: model_args = json.loads(f.read()) n_heads = model_args["n_heads"] dim = model_args["dim"] def permute(w): return w.view(n_heads, dim // n_heads // 2, 2, dim).transpose(1, 2).reshape(dim, dim) checkpoints = [] for i in range(len(ckpt_paths)): checkpoints.append(load(ckpt_paths[i], map_location="cpu")) ckpt_list = [] for name in checkpoints[0].keys(): for k, v in dic.items(): if k in name: if v is not None: value = np.concatenate( [checkpoints[i][name].numpy() for i in range(len(checkpoints))], v) else: value = checkpoints[0][name].numpy() if name == 'norm.weight': name = 'norm_out.weight' if name == 'output.weight': name = 'lm_head.weight' else: name = 'model.' + name if 'rope.freqs' in name: continue if 'wq' in name or 'wk' in name: value = permute(value) print(f'\rprocessing parameter: {name} {value.shape} ', end='', flush=True) ckpt_list.append({'name': name, 'data': ms.Tensor(value, dtype=dtype)}) ckpt_file = os.path.join(ckpt_dir, output_name) ms.save_checkpoint(ckpt_list, ckpt_file) print(f"\rConvert pytorch checkpoint finished, the mindspore checkpoint is saved in '{ckpt_file}'.", flush=True) return True def read_json(path): with open(path, "r") as f: return json.load(f) def name_replace(name: str): """replace hf param name to ms.""" name = name.replace('embed_tokens.weight', 'tok_embeddings.embedding_weight') name = name.replace('.self_attn.q_proj.', '.attention.wq.') name = name.replace('.self_attn.k_proj.', '.attention.wk.') name = name.replace('.self_attn.v_proj.', '.attention.wv.') name = name.replace('.self_attn.o_proj.', '.attention.wo.') name = name.replace('.mlp.gate_proj.', '.feed_forward.w1.') name = name.replace('.mlp.down_proj.', '.feed_forward.w2.') name = name.replace('.mlp.up_proj.', '.feed_forward.w3.') name = name.replace('.input_layernorm.', '.attention_norm.') name = name.replace('.post_attention_layernorm.', '.ffn_norm.') name = name.replace('.norm.', '.norm_out.') return name def convert_hf_ckpt(ckpt_dir, output_name, dtype=ms.float16): """convert hf weight to ms.""" print(f"Trying to convert huggingface checkpoint in '{ckpt_dir}'.", flush=True) try: from transformers import LlamaForCausalLM except: raise ImportError(f"Failed to load huggingface checkpoint. Please make sure transformers is available.") try: model_hf = LlamaForCausalLM.from_pretrained(ckpt_dir) # pylint: disable=W0703 except Exception as e: print(f"Do not find huggingface checkpoint in '{ckpt_dir}', Error {e.message}.", flush=True) return False ckpt_list = [] for name, value in model_hf.named_parameters(): name = name_replace(name) if name == 'norm.weight': name = 'norm_out.weight' if name[:7] == 'layers.': name = name[7:] value = value.detach().numpy() print(f'\rprocessing parameter: {name} {value.shape} ', end='', flush=True) ckpt_list.append({'name': name, 'data': ms.Tensor(value, dtype=dtype)}) ckpt_file = os.path.join(ckpt_dir, output_name) ms.save_checkpoint(ckpt_list, os.path.join(ckpt_file)) print(f"\rConvert huggingface checkpoint finished, the mindspore checkpoint is saved in '{ckpt_file}'.", flush=True) return True def convert_to_new_ckpt(ckpt_path, config_path): """convert previous ckpt to new ckpt""" load_path = ckpt_path.split('.ckpt')[0] save_path = load_path + "_hf" params = ms.load_checkpoint(load_path.split('.ckpt')[0] + '.ckpt') with open(config_path, "r") as f: model_args = json.loads(f.read()) n_heads = model_args["n_heads"] dim = model_args["dim"] def permute(w): return ops.transpose(w.reshape(n_heads, dim // n_heads // 2, 2, dim), (0, 2, 1, 3)).reshape(dim, dim) ckpt_list = [] for name in params.keys(): value = params[name].value() if '.wq' in name or '.wk' in name: value = permute(value) ckpt_list.append({'name': name, 'data': value}) print("\r", name, value.shape, end=" ") ms.save_checkpoint(ckpt_list, save_path) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--torch_ckpt_dir', default='./llama_model/llama-13b-hf/') parser.add_argument('--mindspore_ckpt_path', default='transform.ckpt') parser.add_argument('--pre_ckpt_path', default=None) parser.add_argument('--config_path', default=None) args = parser.parse_args() if args.pre_ckpt_path is not None and args.config_path is not None: convert_to_new_ckpt(args.pre_ckpt_path, args.config_path) else: convert_hf_ckpt(ckpt_dir=args.torch_ckpt_dir, output_name=args.mindspore_ckpt_path) pengcheng.mind-test/mindformers/models/llama/llama.py000066400000000000000000000472641471400134000232640ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA models' APIs.""" import math import numpy as np import mindspore.common.dtype as mstype try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import Tensor, nn from mindspore.context import ParallelMode from mindspore.ops import operations as P from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation try: # pylint: disable=W0611 from mindspore.nn.layer.flash_attention import FlashAttention FLASHATTENTION_VALID = True except ImportError: FLASHATTENTION_VALID = False from mindspore.common.initializer import Normal from mindspore import lazy_inline from mindformers.core.loss.loss import CrossEntropyLoss from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from mindformers.modules.layers import Linear from mindformers.modules.transformer.op_parallel_config import _check_config from mindformers.modules.transformer.transformer import AttentionMask from mindformers.tools.register.register import MindFormerModuleType, MindFormerRegister #from mindformers.pet.tuners.pet_adapter import PetAdapter #from mindformers.pet.tuners.lora_adapter import LoraAdapter from .llama_config import LlamaConfig from .llama_layer import LlamaEmbedding, LlamaRMSNorm, precompute_freqs_cis from .llama_transformer import LLamaDecodeLayer from ..utils import cell_reuse from ...tools.logger import logger __all__ = ['LlamaModel', 'LlamaForCausalLM'] def layer_compute_dtype(layer, layer_id, offset, parallel_config, n_layers, select_recompute=False): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: layer(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(Union[int, List[int]]) - Means the layer_index needs a offset, if there are other modules in the net. n_layers(int) - The total layers used for the model. """ pp_dis = max(int((n_layers + 1) / parallel_config.pipeline_stage), 1) if isinstance(offset, list): if len(offset) != parallel_config.pipeline_stage: raise ValueError(f"The length of `offset` {len(offset)} do not match " "`pipeline stage` {parallel_config.pipeline_stage}.") i = min(layer_id // pp_dis, parallel_config.pipeline_stage - 1) offset_layer = offset[i] elif isinstance(offset, int): offset_layer = offset else: raise TypeError(f"`offset` must be `int` of list of `int`, but got {type(offset)}.") if parallel_config.pipeline_stage == 4: pp_id_list = [9, 11, 11, 9] pp_id = 0 tmp = layer_id while tmp >= pp_id_list[pp_id]: tmp -= pp_id_list[pp_id] pp_id += 1 else: pp_id = min((layer_id + offset_layer) // pp_dis, parallel_config.pipeline_stage - 1) layer.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((n_layers + 1) / parallel_config.gradient_aggregation_group), 1) layer.set_comm_fusion(3 + layer_id) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not select_recompute: layer.recompute() else: if parallel_config.recompute.recompute and not select_recompute: layer.recompute( recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) #layer.attention.flash_attention.flash_attention.recompute(mode=False) class LlamaModel(BaseModel): r""" Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config(LlamaConfig): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 Returns: output: Tensor, the output of llama decoderlayer """ _support_list = MindFormerBook.get_model_support_list()['llama'] def __init__(self, config: LlamaConfig = None): super().__init__(config, auto_prefix=True) _check_config(config.parallel_config) if config.batch_size or config.use_past: Validator.check_positive_int(config.batch_size) self.dtype = config.compute_dtype self.num_layers = config.num_layers self.pad_token_id = config.pad_token_id self.eos_token_id = config.eos_token_id self.is_first_iteration = True self.use_past = config.use_past self.use_flash_attention = config.use_flash_attention and FLASHATTENTION_VALID if self.use_flash_attention: logger.info("Enable flash attention.") elif config.use_flash_attention: logger.info("Current MindSpore do not support flash attention.") self.freqs_cos, self.freqs_sin, self.swap_mask = precompute_freqs_cis( config.hidden_size // config.num_heads, config.seq_length, dtype=config.rotary_dtype, pretrain_seqlen=config.pretrain_seqlen, extend_method=config.extend_method) self.get_attention_mask = AttentionMask( config.seq_length, parallel_config=config.parallel_config.dp_mp_config).to_float(config.compute_dtype) self.multiply_data = Tensor([-10000.0], dtype=config.compute_dtype) self.one = Tensor([1.0], dtype=config.compute_dtype) self.reshape = P.Reshape() self.cast = P.Cast() self.tile = P.Tile() self.mul_mask = P.Mul() self.sub = P.Sub() self.expand_dims = P.ExpandDims() self.not_equal = P.NotEqual() self.gather = P.Gather() self.tok_embeddings = LlamaEmbedding( config.vocab_size, config.hidden_size, param_init_type=config.param_init_type) self.embed_scale = Tensor([config.embed_scale], dtype=config.compute_dtype) self.mul_embed = P.Mul() self.layers = nn.CellList() for layer_id in range(config.num_layers): layer = LLamaDecodeLayer(config.batch_size, config.seq_length, layer_id, config.num_layers, dim=config.hidden_size, n_heads=config.num_heads, multiple_of=config.multiple_of, n_kv_heads=config.n_kv_heads, ffn_dim_multiplier=config.ffn_dim_multiplier, mla_multiplier=config.mla_multiplier, norm_eps=config.rms_norm_eps, compute_dtype=config.compute_dtype, layernorm_compute_dtype=config.layernorm_compute_type, softmax_compute_dtype=config.softmax_compute_type, rotary_dtype=config.rotary_dtype, mup_width_base=config.mup_width_base, param_init_type=config.param_init_type, use_past=config.use_past, use_flash_attention=config.use_flash_attention, compute_in_2d=config.compute_in_2d, use_past_shard=config.use_past_shard, depth_scale=config.depth_scale, init_std=config.init_std, parallel_config=config.parallel_config) layer_compute_dtype(layer, layer_id, config.offset, config.parallel_config, config.num_layers, select_recompute=config.parallel_config.recompute.select_recompute) self.layers.append(layer) self.norm_out = LlamaRMSNorm(config.hidden_size, config.rms_norm_eps, compute_type=config.layernorm_compute_type) dp = config.parallel_config.data_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.tok_embeddings.pipeline_stage = 0 if config.parallel_config.pipeline_stage > 1: self.norm_out.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.tok_embeddings.set_comm_fusion(2) self.norm_out.set_comm_fusion(config.num_layers + 3) self.tok_embeddings.shard(config.parallel_config) self.tile.shard(((1, 1, 1, 1), ())) self.sub.shard(((1,), (dp, 1, 1))) self.mul_mask.shard(((dp, 1, 1, 1), (1,))) self.mul_embed.shard(((dp, 1, 1), (1,))) self.expand_dims.shard(((dp, 1, 1),)) self.not_equal.shard(((dp, 1), ())) self.gather.shard(((dp, 1), (1,))) if config.compute_in_2d: self.norm_out.shard((dp, 1)) else: self.norm_out.shard((dp, 1, 1)) if self.use_past: seq_range = np.arange(config.seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (config.batch_size, 1, 1)), mstype.int32) self.gather_past = P.Gather() self.expand_dims = P.ExpandDims() self.le_past = P.LessEqual() ones = np.ones(shape=(config.seq_length, config.seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((dp, 1, 1), (1, 1, 1))) self.equal = P.Equal().shard(((dp, 1), ())) self.cumsum = P.CumSum(reverse=True).shard(((dp, 1),)) self.pow = P.Pow().shard(((), (dp, 1))) self.pow_up_value = 2.0 self.pow_down_value = 0.5 self.mask_reshape = P.Reshape() self.mask_mul = P.BatchMatMul().shard( ((dp, 1, 1), (dp, 1, 1))) self.up_threshold = 1.2 self.down_threshold = 0.8 self.greater = P.Greater().shard(((dp, 1, 1), ())) self.less = P.Less().shard(((dp, 1, 1), ())) self.mask_multiply = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mask_expand = P.ExpandDims().shard(((1, 1),)) self.mask_multiply2 = P.Mul().shard(((dp, 1, 1), (1, 1, 1))) # pylint: disable=W0613 def construct(self, tokens: Tensor, input_position=None, init_reset=True, batch_valid_length=None): """Forward of llama model.""" # preprocess bs, seq_len = tokens.shape if self.is_first_iteration: freqs_cis = (self.tile(self.reshape(self.freqs_cos, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.tile(self.reshape(self.freqs_sin, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.swap_mask) input_mask = self.cast(self.not_equal(tokens, self.pad_token_id), self.dtype) seq_index = self.cast(self.equal(tokens, self.eos_token_id), mstype.float32) seq_mask_rev = self.cumsum(seq_index, 1) seq_mask_exp_up = self.mask_reshape(self.pow(self.pow_up_value, seq_mask_rev), (bs, seq_len, 1)) seq_mask_exp_down = self.mask_reshape(self.pow(self.pow_down_value, seq_mask_rev), (bs, 1, seq_len)) full_mask = self.mask_mul(seq_mask_exp_up, seq_mask_exp_down) full_mask_small = self.cast(self.less(full_mask, self.up_threshold), mstype.float32) full_mask_great = self.cast(self.greater(full_mask, self.down_threshold), mstype.float32) mask = self.mask_multiply(full_mask_small, full_mask_great) lower_triangle = self.mask_expand(self.lower_triangle_mask, 0) mask = self.mask_multiply2(full_mask, lower_triangle) # mask: [bs, seq, seq] # else: # cur_pos = batch_valid_length - 1 # valid_length = self.reshape(cur_pos, (-1, 1, 1)) # freqs_cis = (self.reshape(self.gather_past(self.freqs_cos, cur_pos, 0), (bs, 1, seq_len, -1)), # self.reshape(self.gather_past(self.freqs_sin, cur_pos, 0), (bs, 1, seq_len, -1)), # self.swap_mask) # mask = self.cast(self.le_past(self.range, valid_length), self.dtype) # mask: [bs, 1, 1] mask = self.sub(self.one, self.cast(mask, self.dtype)) if not self.use_flash_attention: mask = self.expand_dims(mask, 1) mask = self.mul_mask(mask, self.multiply_data) # tokens: [bs, seq/1] h = self.tok_embeddings(tokens) h = self.mul_embed(h, self.embed_scale) # h: [bs, seq/1, hidden_dim] for i in range(self.num_layers): h, _ = self.layers[i](h, freqs_cis, mask, init_reset=init_reset, batch_valid_length=batch_valid_length) output = self.norm_out(h) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class LlamaForCausalLM(BaseModel): r""" Provide llama training loss or logits through network. Args: config (LlamaConfig): The config of llama model. Inputs: input_ids(Tensor): the tokenized inputs with datatype int32, Tensor of shape :math:`(batch, seq\_length)`. labels(Tensor): the tokenized labels with datatype int32, Tensor of shape :math:`(batch, seq\_length)`. input_position(Tensor): current position, used by model.predict. position_ids(Tensor): Reserved param, not used. attention_mask(Tensor): Reserved param, not used. input_embeds(Tensor): Reserved param, not used. init_reset(bool, optional): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Default True. batch_valid_length(Tensor): the past calculated the index with datatype int32, used for incremental prediction. Tensor of shape :math:`(batch_size,)`. Default None. Returns: Tensor, the loss or logits of the network. Examples: >>> from mindformers.models.llama import LlamaConfig, LlamaForCausalLM >>> config = LlamaConfig(batch_size=2) >>> network = LlamaForCausalLM(config=config) """ _support_list = MindFormerBook.get_model_support_list()['llama'] #@lazy_inline def __init__(self, config: LlamaConfig = None): super(LlamaForCausalLM, self).__init__(config, auto_prefix=True) _check_config(config.parallel_config) self.ignore_token_id = config.ignore_token_id self.pad_token_id = config.pad_token_id self.eos_token_id = config.eos_token_id self.reshape = P.Reshape() self.cast = P.Cast() self.slice = P.StridedSlice() self.not_equal = P.NotEqual() self.mul = P.Mul() self.add = P.Add() self.model = LlamaModel(config=config) self.attn_logit_softcapping = Tensor(config.final_logit_softcapping, dtype=mstype.float32) self.inv_attn_logit_softcapping = Tensor(1.0 / config.final_logit_softcapping, dtype=mstype.float32) self.tanh = P.Tanh() self.mul_logits = P.Mul() sigma = config.init_std if config.mup_width_base > 0: ratio = config.hidden_size / config.mup_width_base sigma = sigma / ratio self.lm_head = Linear(in_channels=config.hidden_size, out_channels=config.vocab_size, has_bias=False, compute_dtype=config.compute_dtype, param_init_type=config.param_init_type, weight_init=Normal(sigma=sigma, mean=0)) # meta default: xavier_normal self.loss = CrossEntropyLoss(parallel_config=config.parallel_config) dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.slice.shard(((dp, 1),)) self.not_equal.shard(((dp, 1), ())) self.mul.shard(((dp, 1), (dp, 1))) self.add.shard(((dp, 1), ())) # if config.parallel_config.vocab_emb_dp: # self.lm_head.shard(strategy_matmul=((dp, 1), (1, 1))) # else: # self.lm_head.shard(strategy_matmul=((dp, 1), (mp, 1))) self.lm_head.shard(((dp * mp, 1), (1, 1)), out_strategy_matmul=((dp * mp, 1),)) self.mul_logits.shard(((dp, mp, 1), ())) self.tanh.shard(((dp, mp, 1),)) if config.parallel_config.pipeline_stage > 1: self.lm_head.pipeline_stage = config.parallel_config.pipeline_stage - 1 #self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 def construct(self, input_ids, labels=None, input_position=None, position_ids=None, attention_mask=None, input_embeds=None, init_reset=True, batch_valid_length=None): """LlamaForCausalLM forward.""" bsz, seqlen = input_ids.shape #if self.training: # tokens = self.slice(input_ids, (0, 0), (bsz, seqlen - 1), (1, 1)) #else: # tokens = input_ids tokens = self.slice(input_ids, (0, 0), (bsz, seqlen - 1), (1, 1)) output = self.model(tokens, input_position, init_reset, batch_valid_length) logits = self.lm_head(output) input_mask = self.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) if labels is None: labels = self.slice(input_ids, (0, 1), (bsz, seqlen), (1, 1)) else: if labels.ndim > 1: if self.training: labels = self.slice(labels, (0, 1), (bsz, seqlen), (1, 1)) label_mask = self.cast(self.not_equal(labels, self.ignore_token_id), mstype.float32) input_mask = self.mul(input_mask, label_mask) logits = self.cast(logits, mstype.float32) ################################################################# logits = self.mul_logits(logits, self.inv_attn_logit_softcapping) logits = self.tanh(logits) logits = self.mul_logits(logits, self.attn_logit_softcapping) ################################################################# #if not self.training: # logits = self.reshape(logits, (bsz, seqlen, -1)) # # makes cast effective to avoid allgather issue in Mindspore1.10 # input_mask = self.add(input_mask, 1) # return logits, tokens, input_mask if logits.ndim > 2: logits = self.reshape(logits, (-1, logits.shape[-1])) labels = self.reshape(labels, (-1,)) input_mask = self.reshape(input_mask, (-1,)) loss = self.loss(logits, labels, input_mask) return loss pengcheng.mind-test/mindformers/models/llama/llama_config.py000066400000000000000000000116771471400134000246100ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Llama Config API.""" from typing import Optional from mindformers.modules.transformer.transformer import default_transformer_config, TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['LlamaConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class LlamaConfig(BaseConfig): """ LLaMA config class which defines the model size. """ _support_list = MindFormerBook.get_config_support_list()['llama'] def __init__(self, batch_size: int = 1, seq_length: int = 2048, hidden_size: int = 4096, num_layers: int = 32, num_heads: int = 32, vocab_size: int = 32000, # defined later by tokenizer multiple_of: int = 256, # make SwiGLU hidden layer size multiple of large power of 2 n_kv_heads: Optional[int] = None, ffn_dim_multiplier: Optional[int] = None, mla_multiplier: Optional[int] = None, rms_norm_eps: float = 1e-5, bos_token_id: int = 1, eos_token_id: int = 2, pad_token_id: int = 32000, ignore_token_id: int = -100, compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", rotary_dtype: str = "float32", param_init_type: str = "float16", parallel_config: TransformerOpParallelConfig = default_transformer_config, use_past: bool = False, pretrain_seqlen: int = 2048, extend_method: str = "None", compute_in_2d: bool = False, use_flash_attention: bool = False, offset: int = 0, use_past_shard: bool = False, checkpoint_name_or_path: str = "", repetition_penalty: float = 1.0, max_decode_length: int = 1024, final_logit_softcapping: float = 30.0, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, mup_width_base: int = -1, embed_scale: float = 1.0, depth_scale: float = 1.0, init_std: float = 0.01, **kwargs): super(LlamaConfig, self).__init__(**kwargs) self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.multiple_of = multiple_of self.n_kv_heads = n_kv_heads self.ffn_dim_multiplier = ffn_dim_multiplier self.mla_multiplier = mla_multiplier self.rms_norm_eps = rms_norm_eps self.param_init_type = convert_mstype(param_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.rotary_dtype = convert_mstype(rotary_dtype) self.compute_dtype = convert_mstype(compute_dtype) self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path self.bos_token_id = bos_token_id self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.ignore_token_id = ignore_token_id self.use_past = use_past self.pretrain_seqlen = pretrain_seqlen self.extend_method = extend_method self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention self.offset = offset self.use_past_shard = use_past_shard self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample self.mup_width_base = mup_width_base self.embed_scale = embed_scale self.depth_scale = depth_scale self.init_std = init_std self.final_logit_softcapping = final_logit_softcapping pengcheng.mind-test/mindformers/models/llama/llama_layer.py000066400000000000000000000365261471400134000244570ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA Model Layers' APIs.""" from enum import Enum import math import numpy as np from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore import nn import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.nn.cell import Cell from mindspore.common.initializer import Normal try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import log as logger from mindspore.common.initializer import initializer from mindspore.parallel._utils import _get_parallel_mode from mindspore.context import ParallelMode from mindformers.modules.layers import Linear, _check_input_dtype, _args_type_validator_check, _valid_value_checks from mindformers.tools.logger import _LogActionOnce class SeqExtendMethod(Enum): """Stores the acceptable string identifiers for seq length extend method""" PI = "PI" NTK = "NTK" NONE = "None" class LlamaSiLU(Cell): r""" A self-defined SwiGlu. Inputs: - **x** (Tensor) - Tensor. Outputs: Tensor. x = x * sigmod(x). """ def __init__(self): super().__init__() self.sigmoid = nn.Sigmoid() self.mul = P.Mul() def shard(self, strategy): self.sigmoid.sigmoid.shard(strategy) self.mul.shard((strategy[0], strategy[0])) def construct(self, x): return self.mul(x, self.sigmoid(x)) def get_swap_mask(head_dim): """Swap matrix""" zero_block = np.zeros((head_dim // 2, head_dim // 2), dtype=np.float32) id_block = np.identity(head_dim // 2, dtype=np.float32) return np.block([[zero_block, id_block], [-id_block, zero_block]]) def precompute_freqs_cis( dim: int, end: int, theta: float = 10000.0, dtype=mstype.float32, pretrain_seqlen=2048, extend_method=SeqExtendMethod.NONE.value): """ Precompute of freqs and mask for rotary embedding. """ ratio = 1. if extend_method != SeqExtendMethod.NONE.value and end > pretrain_seqlen: ratio = end / pretrain_seqlen if extend_method == SeqExtendMethod.NTK.value: theta *= ratio freqs_base = np.arange(0, dim, 2)[: (dim // 2)].astype(np.float32) # (head_dim // 2, ) freqs = 1.0 / (theta ** (freqs_base / dim)) # (head_dim // 2, ) if extend_method == SeqExtendMethod.PI.value: t = np.arange(0, end / ratio, 1 / ratio).astype(np.float32) else: t = np.arange(0, end, 1).astype(np.float32) # type: ignore # (seq_len,) freqs = np.outer(t, freqs) # type: ignore (seq_len, head_dim // 2) emb = np.concatenate((freqs, freqs), axis=-1) freqs_cos = np.cos(emb) # (seq_len, head_dim) freqs_sin = np.sin(emb) # (seq_len, head_dim) freqs_cos = Tensor(freqs_cos, dtype=dtype) freqs_sin = Tensor(freqs_sin, dtype=dtype) swap_mask = get_swap_mask(dim) swap_mask = Tensor(swap_mask, dtype=dtype) return freqs_cos, freqs_sin, swap_mask class LlamaRotaryEmbedding(Cell): r""" Rotary Position Embedding. Args: - **head_dim** (int): The dim of multi head attention. - **compute_dtype** (mstype): The compute type, default mstype.float16. - **parallel_config** (dict): - Parallel Config. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, head_dim=128, compute_dtype=mstype.float32): super().__init__(auto_prefix=False) self.head_dim = head_dim self.dtype = compute_dtype self.add = P.Add() self.bmm_swap = P.BatchMatMul() self.mul = P.Mul() self.cast = P.Cast() def rotate_half(self, x, swap_mask): # [bs, n_head/n_kv_head, seq/1, head_dim], [head_dim, head_dim] x = self.bmm_swap(x, swap_mask) return x def construct(self, xq: Tensor, xk: Tensor, freqs_cis): """Forward of rotary position embedding.""" original_type = xq.dtype xq = self.cast(xq, self.dtype) xk = self.cast(xk, self.dtype) # xq, xk: [bs, n_head/n_kv_head, seq/1, head_dim] freqs_cos, freqs_sin, swap_mask = freqs_cis xq_out = self.add(self.mul(xq, freqs_cos), self.mul(self.rotate_half(xq, swap_mask), freqs_sin)) xk_out = self.add(self.mul(xk, freqs_cos), self.mul(self.rotate_half(xk, swap_mask), freqs_sin)) xq_out = self.cast(xq_out, original_type) xk_out = self.cast(xk_out, original_type) return xq_out, xk_out def shard(self, strategy_in): self.add.shard((strategy_in, strategy_in)) self.bmm_swap.shard((strategy_in, (1, 1))) self.mul.shard((strategy_in, (strategy_in[0], 1, 1, 1))) class LlamaEmbedding(Cell): """ Embedding Layer. Args: - **vocab_size** (int): Size of the dictionary of embeddings. - **embedding_size** (int): The size of each embedding vector. - **param_init_type** (mstype): The param init type, default mstype.float32. - **parallel_config** (TransformerOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. - **param_init** (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). """ @_LogActionOnce(m_logger=logger, key='Embedding', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(vocab_table_size=Validator.check_positive_int, embedding_size=Validator.check_positive_int) def __init__(self, vocab_table_size, embedding_size, param_init_type=mstype.float32, param_init='normal'): super().__init__() self.vocab_table_size = vocab_table_size self.embedding_size = embedding_size self.embedding_weight = Parameter( initializer(param_init, [self.vocab_table_size, self.embedding_size], dtype=param_init_type), name='embedding_weight', parallel_optimizer=True) self.gather = P.Gather() def construct(self, input_ids): """Forward of vocab embedding.""" _check_input_dtype(F.dtype(input_ids), "input_ids", [mstype.int32, mstype.int64], self.cls_name) output = self.gather(self.embedding_weight, input_ids, 0) return output def shard(self, parallel_config): """sharding for embedding""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel if parallel_config.vocab_emb_dp: self.gather.shard(((1, 1), (dp, 1))) logger.info(f"Using {dp} data parallel for the embedding lookup.") else: if self.vocab_table_size % mp != 0: raise ValueError(f"The vocab size of the embedding {self.vocab_table_size} must be a " f"multiple of parallel_config.model_parallel {mp}.") self.gather.shard(((mp, 1), (dp, 1))) logger.info(f"Using {dp} data parallel and {mp} " f"model parallel for the embedding lookup.") class LlamaRMSNorm(nn.Cell): r""" A self-defined RMSNorm operation using reduce mean. Args: dim (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. compute_type: The compute type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, dim, eps=1e-6, compute_type=mstype.float32): super(LlamaRMSNorm, self).__init__() self.eps = eps self.weight = Parameter(initializer('ones', (dim,), dtype=mstype.float32), parallel_optimizer=False) self.square = P.Square() self.mean = P.ReduceMean(keep_dims=True) self.add = P.Add() self.rsqrt = P.Rsqrt() self.mul = P.Mul() self.mul2 = P.Mul() self.cast = P.Cast() self.compute_type = compute_type def _norm(self, x): norm_factor = self.square(x) norm_factor = self.mean(norm_factor, -1) norm_factor = self.add(norm_factor, self.eps) norm_factor = self.rsqrt(norm_factor) x = self.cast(x, mstype.float16) norm_factor = self.cast(norm_factor, mstype.float16) return self.mul(x, norm_factor) def construct(self, x): """Forward of RMSNorm.""" original_type = x.dtype x = self.cast(x, self.compute_type) output = self._norm(x) output = self.cast(output, mstype.float16) weight = self.cast(self.weight, mstype.float16) output = self.mul2(output, weight) output = self.cast(output, original_type) return output def shard(self, strategy_in): """Parallel strategy configuratiuon interface.""" self.square.shard((strategy_in,)) self.mean.shard((strategy_in,)) self.rsqrt.shard((strategy_in,)) self.add.shard((strategy_in, ())) self.mul.shard((strategy_in, strategy_in)) self.mul2.shard((strategy_in, (1,))) class LlamaFeedForward(Cell): r""" LLaMA FeedForward. .. math:: (xW_1 * xW_3)W_2 Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: ValueError: `hidden_dim` is not a multiple of the model parallel way. ValueError: `dim` is not a multiple of the model parallel way. """ @_LogActionOnce(m_logger=logger, key='FeedForward', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(dim=Validator.check_positive_int, hidden_dim=Validator.check_positive_int, multiple_of=Validator.check_positive_int, compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward")) def __init__(self, dim, hidden_dim, multiple_of, hidden_act=LlamaSiLU, ffn_dim_multiplier=None, mup_width_base=-1, init_std=0.01, compute_dtype=mstype.float16, param_init_type=mstype.float32): super().__init__() if hidden_act is None or not (isinstance(hidden_act, str) or issubclass(hidden_act, nn.Cell)): raise TypeError(f"For FeedForward cell, the hidden_act should str type or nn.Cell type, " f"but got {hidden_act}.") if ffn_dim_multiplier is not None: hidden_dim = int((ffn_dim_multiplier + 0.01) * hidden_dim) hidden_dim = int(2 * hidden_dim / 3) hidden_dim = multiple_of * \ ((hidden_dim + multiple_of - 1) // multiple_of) self.dtype = compute_dtype self.hidden_act = hidden_act self.dim = dim self.hidden_dim = hidden_dim self.mul = P.Mul() self.cast = P.Cast() sigma = init_std if mup_width_base > 0: ratio = dim / mup_width_base sigma = sigma / math.sqrt(ratio) self.w1 = Linear(in_channels=dim, out_channels=hidden_dim, activation=hidden_act, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) self.w2 = Linear(in_channels=hidden_dim, out_channels=dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) self.w3 = Linear(in_channels=dim, out_channels=hidden_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) def construct(self, x): """Forward process of the FeedForward""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, self.dtype) # [bs, seq, hidden_dim] or [bs * seq, hidden_dim] gate = self.w1(x) # dp,1 -> dp, mp hidden = self.w3(x) # dp,1 -> dp, mp hidden = self.mul(hidden, gate) # dp,mp -> dp, mp output = self.w2(hidden) # dp,mp -> dp, 1 return output def shard(self, parallel_config): """sharding for feedforward""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel if self.hidden_dim % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_dim' must be a multiple of the" "num of model parallel, but got the hidden_dim is {} and the num of model " "parallel is {}.".format(self.hidden_dim, mp)) if self.dim % mp != 0: raise ValueError("For 'FeedForward', the class variable 'dim' must be a multiple of the num of " "model parallel, but got the dim is {} and the num of model parallel is {}." .format(self.dim, mp)) self.w1.shard(((dp, 1), (mp, 1)), strategy_activation=((dp, mp),)) self.w1.activation.shard(((dp, mp),)) self.w2.shard(((dp, mp), (1, mp))) self.w3.shard(((dp, 1), (mp, 1))) self.mul.shard(((dp, mp), (dp, mp))) pengcheng.mind-test/mindformers/models/llama/llama_processor.py000066400000000000000000000044201471400134000253460ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ LlamaProcessor """ from mindformers.mindformer_book import MindFormerBook from ..base_tokenizer import BaseTokenizer from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['LlamaProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class LlamaProcessor(BaseProcessor): """ Llama processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['llama'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(LlamaProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-test/mindformers/models/llama/llama_tokenizer.py000066400000000000000000000263521471400134000253510ustar00rootroot00000000000000# coding=utf-8 # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA tokenizer APIs.""" import os from shutil import copyfile from typing import Any, Dict, List, Optional import sentencepiece as spm from mindformers.tools import logger from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import Tokenizer, AddedToken from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['LlamaTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "tokenizer.model"} @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class LlamaTokenizer(Tokenizer): r""" Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Tokenizer of llama will default add bos at the beginning of tokens and add eos token on the tail of tokens. Args: model_path(str): The spiece.model file path. add_bos(bool): The flag defines whether add bos token, Default True. eos_token(str): The token that represents the end-of-sentence. Default "". unk_token(str): The token that represents the unknown. Default "". pad_token(str): The token that represents the pad. Default "". sp_model_kwargs(str): Other kwargs for sp_model`. add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" clean_up_tokenization_spaces (bool): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import LlamaTokenizer >>> tokenizer = LlamaTokenizer.from_pretrained("llama_7b") >>> res = tokenizer("hello world") >>> print(res) {'input_ids': [1, 22172, 3186, 2]} >>> res = tokenizer("hello world", padding='max_length', max_length=10) >>> print(res) {'input_ids': [1, 22172, 3186, 2, 0, 0, 0, 0, 0, 0]} >>> res = tokenizer("hello world", return_tensors='ms') >>> print(res) {'input_ids': Tensor(shape=[3], dtype=Int32, value= [1, 22172, 3186, 2])} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['llama'] def __init__( self, vocab_file, unk_token="", bos_token="", eos_token="", pad_token="", sp_model_kwargs: Optional[Dict[str, Any]] = None, add_bos_token=True, add_eos_token=False, clean_up_tokenization_spaces=False, **kwargs, ): self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token super().__init__( bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, pad_token=pad_token, sp_model_kwargs=self.sp_model_kwargs, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs, ) self.vocab_file = vocab_file self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(vocab_file) def __getstate__(self): state = self.__dict__.copy() state["sp_model"] = None return state def __setstate__(self, d): self.__dict__ = d self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(self.vocab_file) @property def vocab_size(self): """Returns vocab size""" return self.sp_model.get_piece_size() def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def _tokenize(self, text, **kwargs): """Returns a tokenized string.""" return self.sp_model.encode(text, out_type=str) def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.sp_model.piece_to_id(token) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" token = self.sp_model.IdToPiece(index) return token def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" current_sub_tokens = [] out_string = "" prev_is_special = False for i, token in enumerate(tokens): # make sure that special tokens are not decoded using sentencepiece model if token in self.all_special_tokens: if not prev_is_special and i != 0: out_string += " " out_string += self.sp_model.decode(current_sub_tokens) + token prev_is_special = True current_sub_tokens = [] else: current_sub_tokens.append(token) prev_is_special = False out_string += self.sp_model.decode(current_sub_tokens) return out_string def save_vocabulary(self, save_directory, filename_prefix=None): """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. """ if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None out_vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file): copyfile(self.vocab_file, out_vocab_file) elif not os.path.isfile(self.vocab_file): with open(out_vocab_file, "wb") as fi: content_spiece_model = self.sp_model.serialized_model_proto() fi.write(content_spiece_model) return out_vocab_file def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): r"""Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) bos_token_id = [1] if self.add_bos_token else [] eos_token_id = [1] if self.add_eos_token else [] if token_ids_1 is None: return bos_token_id + ([0] * len(token_ids_0)) + eos_token_id return ( bos_token_id + ([0] * len(token_ids_0)) + eos_token_id + bos_token_id + ([0] * len(token_ids_1)) + eos_token_id ) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output pengcheng.mind-test/mindformers/models/llama/llama_transformer.py000066400000000000000000001566641471400134000257130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA transformer Layer's APIs.""" from typing import Tuple, Optional import math import numpy as np try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import nn, ops import mindspore.common.dtype as mstype from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore.context import ParallelMode from mindspore.ops import operations as P from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindspore.common.initializer import Normal from flash_attention.flash_attention_impl import get_flash_attention from mindformers.models.llama.llama_layer import LlamaFeedForward, LlamaRMSNorm, LlamaRotaryEmbedding from mindformers.modules.layers import _check_input_dtype, Linear from mindformers.modules.transformer import TransformerOpParallelConfig HEAD_DIM_MAX_LIMIT = 256 class FlashAttention(nn.Cell): """Flash Attention Layer. This function contains the flash attention primitives used in FlashAttention (see paper) `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ` Specifically, it includes the following: 1. An interface for calling flashattention operation. 2. Two configuration parameters for enabling local block sparse of flashattention. Args: head_dim(int): The hidden size of input. dropout_rate(float): The dropout rate of the attention score. Default 0.0. prev_block_num(int): A integer to define the number of blocks to look ahead for local block sparse attention. Default 65536. next_block_num(int): A integer to define the number of blocks to look behind for local block sparse attention. Default 65536. tiling_stgy_name(str): A str to define tiling strategy of flash attention. dp(int): data parallel. Default 1. mp(int): model parallel. Default 1. have_attention_mask_batch(bool): indicates whether attention_mask contains the batch dimension. Default True alibi(bool): This parameter indicates whether the flashattention supports the Alibi. Default: False Inputs: - **query** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **key** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **value** (Tensor) - Tensor value (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp16` [batch_size, seq_length, seq_length]): A matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, head_num, seq_length, head_dim] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from acctransformer.flash_attention.nn.layer.flash_attention import FlashAttention >>> from mindspore import Tensor >>> model = FlashAttention(head_dim=128, ... dropout_rate=0.1, ... prev_block_num=7, ... next_block_num=0 ... ) >>> query = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> key = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> value = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> attention_mask = Tensor(np.triu(np.ones((1, 128, 128)), k=1), mstype.float16) >>> output = model(query, key, value, attention_mask) >>> print(output.shape) (2, 16, 4096, 128) """ def __init__(self, head_dim, dropout_rate=0.0, prev_block_num=65536, next_block_num=65536, tiling_stgy_name="sparse", dp=1, mp=1, have_attention_mask_batch=True, alibi=False ): super(FlashAttention, self).__init__() if head_dim > HEAD_DIM_MAX_LIMIT: raise ValueError("head_dim too large, reduce head_dim and try again.") if alibi: raise ValueError("alibi not supported in the current version.") self.flash_attention = get_flash_attention( prev_block_num=prev_block_num, next_block_num=next_block_num, tiling_stgy_name=tiling_stgy_name ) self.flash_attention.add_prim_attr("primitive_target", "Ascend") self.scale_factor = Tensor([1. / math.sqrt(math.sqrt(head_dim))], dtype=mstype.float16) self.scale_mul = ops.Mul().shard(((dp, mp, 1, 1), (1,))) self.dropout_rate = dropout_rate self.enable_dropout = self.dropout_rate > 0.0 self.have_attention_mask_batch = have_attention_mask_batch self.alibi = alibi if self.enable_dropout: self.keep_prob = Tensor(1 - self.dropout_rate, dtype=mstype.float16) self.fill_v2 = ops.FillV2().shard(((dp, mp, 1, 1), ())) self.tensor_one = Tensor(1.0, mstype.float16) self.drop_gen_mask = ops.DropoutGenMask() self.do_dropout = ops.DropoutDoMask().shard(((dp, mp, 1, 1),)) self.depend = ops.Depend() def shard(self, in_strategy=None, out_strategy=None): """Distributed configuration of FlashAttention :param in_strategy: Describe the split strategy of operator input. Default: None. :param out_strategy: Describe the split strategy of operator output, it is only for certain operators, such as MatMul. Default: None. :return: """ if in_strategy is None: # default: dp=1, mp=1, construct inputs only contain q, k, v in_strategy = ( (1, 1, 1, 1), (1, 1, 1, 1), (1, 1, 1, 1), ) self.flash_attention.shard(in_strategy) dp = in_strategy[0][0] mp = in_strategy[0][1] self.flash_attention.add_prim_attr("dev_matrix_shape", [dp, mp, 1, 1]) inputs_tensor_map = [ [3, 2, 1, 0], [3, 2, 1, 0], [3, 2, 1, 0], ] if self.have_attention_mask_batch: inputs_tensor_map.append([3, 1, 0]) else: inputs_tensor_map.append([-1, 1, 0]) input_empty_args_num = 2 # dropout_mask if self.enable_dropout: input_empty_args_num -= 1 inputs_tensor_map.append([3, 2, 1, 0]) if self.alibi: input_empty_args_num -= 1 inputs_tensor_map.append([3, 2, 1, 0]) self.flash_attention.add_prim_attr("inputs_tensor_map", inputs_tensor_map) self.flash_attention.add_prim_attr("outputs_tensor_map", [ [3, 2, 1, 0], # O [3, 2, 1], # L ]) self.flash_attention.add_prim_attr("as_loss_divisor", 0) self.flash_attention.add_prim_attr("empty_mirror_ops", input_empty_args_num) def construct(self, q, k, v, attn_mask=None, alibi_mask=None): """FlashAttention forward :param q: [bsz, head_num, seq_len, head_dim] :param k: [bsz, head_num, seq_len, head_dim] :param v: [bsz, head_num, seq_len, head_dim] :param attn_mask: [1 or bsz, seq_len, seq_len], if not None :param alibi_mask: [bsz, head_num, 1, seq_len], if not None :return: o [bsz, head_num, seq_len, head_dim] """ q = self.scale_mul(q, self.scale_factor) k = self.scale_mul(k, self.scale_factor) bsz, head_num, seq_len, head_dim = q.shape _, k_head_num, _, _ = k.shape _, v_head_num, _, _ = v.shape if head_num != k_head_num or head_num != v_head_num: raise ValueError( "the head_num of query, key and value must be the same, " "If different head_num are used, users need to change themselves to be same by tile.") if self.enable_dropout: drop_mask_bits = self.drop_gen_mask((bsz, head_num, seq_len, seq_len), self.keep_prob) tensor_shape = Tensor((bsz, head_num, seq_len, seq_len), mstype.int32) ones = self.fill_v2(tensor_shape, self.tensor_one) ones = self.depend(ones, q) drop_mask = self.do_dropout(ones, drop_mask_bits, self.keep_prob) else: drop_mask = None o, _ = self.flash_attention(q, k, v, attn_mask, drop_mask, alibi_mask) return o class LLamaAttention(nn.Cell): r""" This is an implementation of multihead attention in LLaMA. Args: - **batch_size** (int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. - **src_seq_length** (int): The sequence length of the query vector. - **tgt_seq_length** (int): The sequence length of the key and value vector. - **dim** (int): The hidden size of the input. - **head_dim** (int): The dim of head. - **n_heads** (int): The number of the heads. - **compute_dtype** (dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. - **softmax_compute_type** (dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. - **param_init_type** (dtype.Number): The parameter initialization type of the module. Default mstype. float32. Should be mstype.float32 or mstype.float16. - **use_past** (bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. - **parallel_config** (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - The input tokens with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, head_dim, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, head_dim). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, head_dim)). """ def __init__(self, batch_size, seq_length, dim: int = 512, n_heads: int = 8, n_kv_heads: Optional[int] = None, mup_width_base=-1, init_std=0.01, compute_dtype=mstype.float16, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, attn_logit_softcapping=50.0, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, parallel_config=TransformerOpParallelConfig()): super().__init__() self.seq_length = seq_length self.hidden_size = dim self.n_head = n_heads self.head_dim = dim // n_heads self.n_kv_head = n_heads if n_kv_heads is None else n_kv_heads self.n_rep = self.n_head // self.n_kv_head self.dtype = compute_dtype self.softmax_dtype = softmax_compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention if self.hidden_size % self.n_head != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'n_head', but got the hidden_size is {} and the n_head is {}." .format(self.hidden_size, self.n_head)) if self.n_kv_head % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'n_kv_head' must be a multiple of " "'parallel_config.model_parallel', but got the n_kv_head is {} " "and the parallel_config.model_parallel is {}." .format(self.n_kv_head, parallel_config.model_parallel)) self.inv_norm_factor = Tensor(1.0 / math.sqrt(self.head_dim), dtype=compute_dtype) self.reshape = P.Reshape() self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.batch_matmul = P.BatchMatMul() self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.mul = P.Mul() self.add = P.Add() self.softmax = nn.Softmax().to_float(softmax_compute_dtype) self.cast = P.Cast() self.cast_attn = P.Cast() self.tile_kv = P.Tile() sigma = init_std if mup_width_base > 0: ratio = self.hidden_size / mup_width_base sigma = sigma / math.sqrt(ratio) self.apply_rotary_emb = LlamaRotaryEmbedding(self.head_dim, rotary_dtype) self.wo = Linear(in_channels=self.hidden_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wq = Linear(self.hidden_size, self.hidden_size, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wk = Linear(self.hidden_size, self.n_kv_head * self.head_dim, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wv = Linear(self.hidden_size, self.n_kv_head * self.head_dim, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.transpose.shard(((dp, 1, mp, 1),)) self.merger_head_transpose.shard(((dp, mp, 1, 1),)) self.batch_matmul_q_k.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.batch_matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul.shard(((dp, mp, 1, 1), ())) # self.tanh.shard(((dp, mp, 1, 1),)) self.add.shard(((dp, 1, 1, 1), (dp, mp, 1, 1))) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.tile_kv.shard(((dp * mp, 1, 1, 1),)) self.apply_rotary_emb.shard((dp, mp, 1, 1)) self.wq.shard(((dp, 1), (mp, 1))) self.wk.shard(((dp, 1), (mp, 1))) self.wv.shard(((dp, 1), (mp, 1))) self.wo.shard(((dp, mp), (1, mp))) if parallel_config.use_seq_parallel and self.is_first_iteration: self.wo.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if parallel_config.recompute.select_recompute: self.apply_rotary_emb.recompute() self.tile_kv.recompute() self.batch_matmul_q_k.recompute() self.mul.recompute() # self.tanh.recompute() self.add.recompute() self.cast_attn.recompute() self.softmax.softmax.recompute() self.batch_matmul.recompute() if self.use_flash_attention: self.flash_attention = FlashAttention(head_dim=self.head_dim, prev_block_num=65536, next_block_num=0, dp=parallel_config.data_parallel, mp=parallel_config.model_parallel) q_shard_stgy = (dp, mp, 1, 1) k_shard_stgy = (dp, mp, 1, 1) v_shard_stgy = (dp, mp, 1, 1) attn_mask_shard_stgy = (dp, 1, 1) in_stgy = (q_shard_stgy, k_shard_stgy, v_shard_stgy, attn_mask_shard_stgy) self.flash_attention.shard(in_stgy) if parallel_config.recompute.select_recompute: self.flash_attention.recompute() if self.use_past: # operators used for state reuse seq_range = np.arange(seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.add_past = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.equal = P.Equal().shard(((dp, 1, 1), (dp, 1, 1))) self.less = P.Less().shard(((dp, 1, 1), (dp, 1, 1))) self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.add_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul_past.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def construct(self, x: Tensor, freqs_cis: Tuple[Tensor, Tensor], mask=None, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" ori_dtype = x.dtype # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] x = self.reshape(x, (-1, x.shape[-1])) # [bs * seq/1, hidden_dim] query = self.cast(self.wq(x), self.dtype) # dp, 1 -> dp, mp key = self.cast(self.wk(x), self.dtype) # dp, 1 -> dp, mp value = self.cast(self.wv(x), self.dtype) # dp, 1 -> dp, mp query = self.reshape(query, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) key = self.reshape(key, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_kv_head, self.head_dim)) value = self.reshape(value, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_kv_head, self.head_dim)) # [bs, seq/1, n_head/n_kv_head, head_dim] query = self.transpose(query, (0, 2, 1, 3)) key = self.transpose(key, (0, 2, 1, 3)) value = self.transpose(value, (0, 2, 1, 3)) # [bs, n_head/n_kv_head, seq/1, head_dim] query, key = self.apply_rotary_emb(query, key, freqs_cis) # dp, mp, 1, 1 # kv cache: [bs, n_kv_head, 1, head_dim] -> [bs, n_kv_head, seq, head_dim] key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = ( self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = (self.equal(self.range, valid_length)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) current_value = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add_past(key_past, current_key) value = self.add_past(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # kv share: [bs, n_kv_head, seq, head_dim] -> [bs, n_head, seq, head_dim] key = self._repeat_kv(key, self.n_rep) value = self._repeat_kv(value, self.n_rep) # q, k, v: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim], [bs, n_head, seq, head_dim] if self.use_flash_attention: attention = self.flash_attention(query, key, value, mask) attention = self._merge_heads(attention) else: attention = self._attn(query, key, value, mask) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] output = self.wo(attention) # dp, mp -> dp, 1 / dp * mp, 1 output = self.cast(output, ori_dtype) return output, layer_present def _repeat_kv(self, x, rep): if rep == 1: return x bs, n_kv_head, seqlen, head_dim = x.shape x = self.reshape(x, (bs * n_kv_head, 1, seqlen, head_dim)) x = self.tile_kv(x, (1, rep, 1, 1)) x = self.reshape(x, (bs, n_kv_head * rep, seqlen, head_dim)) return x def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _merge_heads(self, x): """ convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output """ # [bs, n_head, seq/1, head_dim] x = self.merger_head_transpose(x, (0, 2, 1, 3)) # dp,mp,1,1 -> dp,1,mp,1 # [bs, seq/1, n_head, head_dim] x_shape = x.shape if self.compute_in_2d: # [bs * seq/1, hidden_dim] new_shape = (-1, x_shape[-2] * x_shape[-1]) else: # [bs, seq/1, hidden_dim] new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge def _attn(self, query, key, value, mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # q, k: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim] score = self.batch_matmul_q_k(query, key) score = self.cast_attn(score, self.softmax_dtype) # score: [bs, n_head, seq/1, seq] score = self.mul(score, self.inv_norm_factor) ######################################################## # score = self.mul(score, self.inv_attn_logit_softcapping) # score = self.tanh(score) # score = self.mul(score, self.attn_logit_softcapping) ######################################################## score = self.add(mask, score) attention_probs = self.softmax(score) # score, v: [bs, n_head, seq/1, seq], [bs, n_head, seq, head_dim] weighted_values = self.batch_matmul(self.cast(attention_probs, self.dtype), value) # [bs, n_head, seq/1, head_dim] attention_merge = self._merge_heads(weighted_values) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] return attention_merge class LLamaMLAAttention(nn.Cell): def __init__(self, batch_size, seq_length, dim: int = 512, n_heads: int = 8, mla_multiplier: int = 4, compute_dtype=mstype.float16, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, parallel_config=TransformerOpParallelConfig()): super().__init__() self.seq_length = seq_length self.hidden_size = dim self.n_head = n_heads self.head_dim = dim // n_heads self.lora_size = self.head_dim * mla_multiplier self.dtype = compute_dtype self.softmax_dtype = softmax_compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention and FLASHATTENTION_VALID if self.hidden_size % self.n_head != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'n_head', but got the hidden_size is {} and the n_head is {}." .format(self.hidden_size, self.n_head)) self.inv_norm_factor = Tensor(1.0 / math.sqrt(self.head_dim), dtype=compute_dtype) self.reshape = P.Reshape() self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.batch_matmul = P.BatchMatMul() self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.mul = P.Mul() self.add = P.Add() self.softmax = nn.Softmax().to_float(softmax_compute_dtype) self.cast = P.Cast() self.cast_attn = P.Cast() self.tile_kv = P.Tile() self.apply_rotary_emb = LlamaRotaryEmbedding(self.head_dim, rotary_dtype) self.q_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.q_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.wo = Linear(in_channels=self.hidden_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.transpose.shard(((dp, 1, mp, 1),)) self.merger_head_transpose.shard(((dp, mp, 1, 1),)) self.batch_matmul_q_k.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.batch_matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul.shard(((dp, mp, 1, 1), ())) self.add.shard(((dp, 1, 1, 1), (dp, mp, 1, 1))) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.tile_kv.shard(((dp * mp, 1, 1, 1),)) self.apply_rotary_emb.shard((dp, mp, 1, 1)) self.q_proj_down.shard(((dp, 1), (mp, 1))) self.q_proj_up.shard(((dp, 1), (mp, 1))) self.k_proj_down.shard(((dp, 1), (mp, 1))) self.k_proj_up.shard(((dp, 1), (mp, 1))) self.v_proj_down.shard(((dp, 1), (mp, 1))) self.v_proj_up.shard(((dp, 1), (mp, 1))) if parallel_config.use_seq_parallel and self.is_first_iteration: self.wo.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if parallel_config.recompute.select_recompute: self.apply_rotary_emb.recompute() self.tile_kv.recompute() self.batch_matmul_q_k.recompute() self.mul.recompute() self.add.recompute() self.cast_attn.recompute() self.softmax.softmax.recompute() self.batch_matmul.recompute() if self.use_flash_attention: self.flash_attention = FlashAttention(self.head_dim, dp=dp, mp=mp, next_block_num=0, high_precision=(softmax_compute_dtype == mstype.float32)) if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.flash_attention.shard(((dp, mp, 1, 1), (dp, mp, 1, 1), (dp, mp, 1, 1), (dp, 1, 1), ())) if parallel_config.recompute.select_recompute: self.flash_attention.recompute() if self.use_past: # operators used for state reuse seq_range = np.arange(seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.add_past = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.equal = P.Equal().shard(((dp, 1, 1), (dp, 1, 1))) self.less = P.Less().shard(((dp, 1, 1), (dp, 1, 1))) self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.add_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul_past.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def construct(self, x: Tensor, freqs_cis: Tuple[Tensor, Tensor], mask=None, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" ori_dtype = x.dtype # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] x = self.reshape(x, (-1, x.shape[-1])) # [bs * seq/1, hidden_dim] query = self.cast(self.q_proj_up(self.q_proj_down(x)), self.dtype) # dp, 1 -> dp, mp key = self.cast(self.k_proj_up(self.k_proj_down(x)), self.dtype) # dp, 1 -> dp, mp value = self.cast(self.v_proj_up(self.v_proj_down(x)), self.dtype) # dp, 1 -> dp, mp query = self.reshape(query, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) key = self.reshape(key, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) value = self.reshape(value, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) # [bs, seq/1, n_head/n_kv_head, head_dim] query = self.transpose(query, (0, 2, 1, 3)) key = self.transpose(key, (0, 2, 1, 3)) value = self.transpose(value, (0, 2, 1, 3)) # [bs, n_head/n_kv_head, seq/1, head_dim] query, key = self.apply_rotary_emb(query, key, freqs_cis) # dp, mp, 1, 1 # kv cache: [bs, n_kv_head, 1, head_dim] -> [bs, n_kv_head, seq, head_dim] key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = ( self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = (self.equal(self.range, valid_length)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) current_value = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add_past(key_past, current_key) value = self.add_past(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # kv share: [bs, n_kv_head, seq, head_dim] -> [bs, n_head, seq, head_dim] # q, k, v: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim], [bs, n_head, seq, head_dim] if self.use_flash_attention: attention = self.flash_attention(query, key, value, mask) attention = self._merge_heads(attention) else: attention = self._attn(query, key, value, mask) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] output = self.wo(attention) # dp, mp -> dp, 1 / dp * mp, 1 output = self.cast(output, ori_dtype) return output, layer_present def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _merge_heads(self, x): """ convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output """ # [bs, n_head, seq/1, head_dim] x = self.merger_head_transpose(x, (0, 2, 1, 3)) # dp,mp,1,1 -> dp,1,mp,1 # [bs, seq/1, n_head, head_dim] x_shape = x.shape if self.compute_in_2d: # [bs * seq/1, hidden_dim] new_shape = (-1, x_shape[-2] * x_shape[-1]) else: # [bs, seq/1, hidden_dim] new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge def _attn(self, query, key, value, mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # q, k: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim] score = self.batch_matmul_q_k(query, key) # score: [bs, n_head, seq/1, seq] score = self.mul(score, self.inv_norm_factor) score = self.add(mask, score) attention_probs = self.softmax(self.cast_attn(score, self.softmax_dtype)) # score, v: [bs, n_head, seq/1, seq], [bs, n_head, seq, head_dim] weighted_values = self.batch_matmul(self.cast(attention_probs, self.dtype), value) # [bs, n_head, seq/1, head_dim] attention_merge = self._merge_heads(weighted_values) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] return attention_merge class LLamaDecodeLayer(nn.Cell): r""" Transformer Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. seq_length(int): The input sequence length. layer_id(int): The layer id of current transformer block layer. dim(int): The hidden size of the input. num_heads(int): The number of the heads. multiple_of(int): The SwiGLU hidden layer size multiple of large power of 2. norm_eps (float): The epsilon value of the denominator. Default 1e-5. compute_dtype(dtype.Number): The computation type of the layer. Should be mstype.float32 or mstype.float16. Default mstype.float32. layernorm_compute_type(dtype.Number): The computation type of the norm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, seq_length), (batch_size, num_heads, seq_length, head_dim)). """ def __init__(self, batch_size, seq_length, layer_id, layer_num, dim: int = 512, n_heads: int = 8, multiple_of: int = 256, n_kv_heads: Optional[int] = None, ffn_dim_multiplier: Optional[int] = None, mla_multiplier: Optional[int] = None, norm_eps: float = 1e-5, compute_dtype=mstype.float16, layernorm_compute_dtype=mstype.float32, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, mup_width_base=-1, depth_scale=1.0, init_std=0.01, parallel_config=TransformerOpParallelConfig()): super().__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) self.batch_size = batch_size self.compute_in_2d = compute_in_2d self.seq_length = seq_length self.layer_id = layer_id self.hidden_size = dim self.n_head = n_heads self.head_dim = self.hidden_size // self.n_head self.n_kv_head = n_heads if n_kv_heads is None else n_kv_heads self.dtype = compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.key_past = None self.value_past = None self.scale_depth = Tensor([depth_scale / math.sqrt(layer_num)], dtype=mstype.float16) self.depth_mul = P.Mul() self.reshape = P.Reshape() self.add = P.Add() self.attention_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) self.ffn_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) self.post_attention_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) self.post_ffn_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) if mla_multiplier is None: self.attention = LLamaAttention(batch_size=batch_size, seq_length=seq_length, dim=dim, n_heads=n_heads, n_kv_heads=n_kv_heads, mup_width_base=mup_width_base, init_std=init_std, compute_dtype=compute_dtype, softmax_compute_dtype=softmax_compute_dtype, rotary_dtype=rotary_dtype, param_init_type=param_init_type, use_past=use_past, use_flash_attention=use_flash_attention, compute_in_2d=compute_in_2d, use_past_shard=use_past_shard, parallel_config=parallel_config) else: self.attention = LLamaMLAAttention(batch_size=batch_size, seq_length=seq_length, dim=dim, n_heads=n_heads, compute_dtype=compute_dtype, mla_multiplier=mla_multiplier, softmax_compute_dtype=softmax_compute_dtype, rotary_dtype=rotary_dtype, param_init_type=param_init_type, use_past=use_past, use_flash_attention=use_flash_attention, compute_in_2d=compute_in_2d, use_past_shard=use_past_shard, parallel_config=parallel_config) self.feed_forward = LlamaFeedForward(dim=self.hidden_size, hidden_dim=4 * self.hidden_size, multiple_of=multiple_of, ffn_dim_multiplier=ffn_dim_multiplier, mup_width_base=mup_width_base, init_std=init_std, compute_dtype=compute_dtype, param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.feed_forward.shard(parallel_config) if self.compute_in_2d: self.add.shard(((dp, 1), (dp, 1))) self.attention_norm.shard((dp, 1)) self.ffn_norm.shard((dp, 1)) self.post_attention_norm.shard((dp, 1)) self.post_ffn_norm.shard((dp, 1)) self.depth_mul.shard(((dp, 1), (1,))) else: self.add.shard(((dp, 1, 1), (dp, 1, 1))) self.attention_norm.shard((dp, 1, 1)) self.ffn_norm.shard((dp, 1, 1)) self.post_attention_norm.shard((dp, 1, 1)) self.post_ffn_norm.shard((dp, 1, 1)) self.feed_forward.mul.shard(((dp, 1, mp), (dp, 1, mp))) self.depth_mul.shard(((dp, 1, 1), (1,))) if parallel_config.use_seq_parallel and self.is_first_iteration: if self.compute_in_2d: self.add.shard(((dp * mp, 1), (dp * mp, 1))) self.attention_norm.shard((dp * mp, 1)) self.ffn_norm.shard((dp * mp, 1)) self.post_attention_norm.shard((dp * mp, 1)) self.post_ffn_norm.shard((dp * mp, 1)) self.depth_mul.shard(((dp * mp, 1), (1,))) else: self.add.shard(((dp, mp, 1), (dp, mp, 1))) self.attention_norm.shard((dp, mp, 1)) self.ffn_norm.shard((dp, mp, 1)) self.post_attention_norm.shard((dp, mp, 1)) self.post_ffn_norm.shard((dp, mp, 1)) self.depth_mul.shard(((dp, mp, 1), (1,))) self.feed_forward.w2.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if self.use_past: kv_shape = (batch_size, self.n_kv_head, seq_length, self.head_dim) self.key_past = Parameter(Tensor(np.zeros(kv_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(kv_shape), self.dtype), name="value_past") self.ones = P.Ones() self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (1,))) self.assign_past = P.Assign().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.mul_past.shard(((dp, mp, 1, 1), (1,))) self.assign_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) def construct(self, x, freqs_cis, mask=None, init_reset=True, batch_valid_length=None): """ Forward of transformer block. """ self._check_input(x, freqs_cis, mask, init_reset, batch_valid_length) # [bs, seq/1, hidden_dim] (first) [bs * seq/1, hidden_dim] (others) if self.compute_in_2d and x.ndim != 2: x = self.reshape(x, (-1, x.shape[-1])) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] input_x = self.attention_norm(x) key_reset = None value_reset = None if self.use_past and self.is_first_iteration: # reset states, init_reset True for reuse and False for reset self.assign_past(self.key_past, self.mul_past(self.key_past, self.cast(init_reset, self.dtype))) self.assign_past(self.value_past, self.mul_past(self.value_past, self.cast(init_reset, self.dtype))) key_reset = self.key_past value_reset = self.value_past # add dependency for desired execution order input_x = ops.depend(input_x, key_reset) input_x = ops.depend(input_x, value_reset) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] h, layer_present = self.attention(input_x, freqs_cis, mask, self.key_past, self.value_past, batch_valid_length) h = self.post_attention_norm(h) h = self.add(x, h) ffn_norm = self.ffn_norm(h) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] ffn_out = self.feed_forward(ffn_norm) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign_past(self.key_past, key_present) self.assign_past(self.value_past, value_present) key_update = self.key_past value_update = self.value_past # add dependency for desired execution order key_update = ops.depend(key_update, key_reset) value_update = ops.depend(value_update, value_reset) # add dependency for desired execution order ffn_out = ops.depend(ffn_out, value_update) ffn_out = ops.depend(ffn_out, key_update) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] ffn_out = self.post_ffn_norm(ffn_out) out = self.add(h, ffn_out) return out, layer_present def _check_input(self, x, freqs_cis, mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype( x.dtype, "x", [mstype.float32, mstype.float16], self.cls_name) freqs_cos, freqs_sin, swap_mask = freqs_cis _check_input_dtype(freqs_cos.dtype, "freqs_cos", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(freqs_sin.dtype, "freqs_sin", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(swap_mask.dtype, "swap_mask", [mstype.float32, mstype.float16], self.cls_name) if mask is not None: _check_input_dtype(mask.dtype, "input_mask", [mstype.float32, mstype.float16], self.cls_name) if self.use_past: if not isinstance(init_reset, Tensor): init_reset = Tensor([init_reset], mstype.bool_) if not isinstance(batch_valid_length, Tensor): bs = freqs_cis[0].shape[0] batch_valid_length = self.ones((bs, 1), mstype.int32) _check_input_dtype(init_reset.dtype, "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(batch_valid_length.dtype, "batch_valid_length", [mstype.int32], self.cls_name) return True pengcheng.mind-test/mindformers/models/mae/000077500000000000000000000000001471400134000212635ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/mae/__init__.py000066400000000000000000000015561471400134000234030ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Mae Model API.""" from .mae import * from .mae_config import * from .mae_processor import * __all__ = [] __all__.extend(mae.__all__) __all__.extend(mae_config.__all__) __all__.extend(mae_processor.__all__) pengcheng.mind-test/mindformers/models/mae/convert_weight.py000066400000000000000000000065301471400134000246700ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/facebook""" import argparse import torch import mindspore as ms def convert_weight(torch_pth_path="mae_pretrain_vit_base.pth", ms_ckpt_path="mae_pretrain_vit_base.ckpt"): """ convert mae_vit_base_p16 weights from pytorch to mindspore pytorch and GPU required. """ param_dict = torch.load(torch_pth_path, map_location=torch.device("cpu")) new_dict = [] for k, v in param_dict["model"].items(): if k in ("cls_token", "mask_token"): k += "s" if "head" not in k: k = "vit." + k if "norm" in k: if "fc_norm" not in k and "vit.norm" not in k and "decoder_norm" not in k: k = k.replace("norm", "layernorm") if "weight" in k: k = k.replace("weight", "gamma") elif "bias" in k: k = k.replace("bias", "beta") if "mlp" in k: k = k.replace("mlp", "output") if "fc1" in k: k = k.replace("fc1", "mapping") if "weight" in k: v = v.transpose(-1, 0) elif "fc2" in k: k = k.replace("fc2", "projection") if "weight" in k: v = v.transpose(-1, 0) if "attn" in k: k = k.replace("attn", "attention") if "proj" in k: k = k.replace("proj", "projection") if "weight" in k: v = v.transpose(-1, 0) if "qkv" not in k: new_dict.append({"name": k, "data": ms.Tensor(v.numpy())}) else: data = ms.Tensor(v.numpy()) length = data.shape[0] // 3 new_dict.append({"name": k.replace(".qkv", ".dense1"), "data": data[:length]}) new_dict.append({"name": k.replace(".qkv", ".dense2"), "data": data[length:length*2]}) new_dict.append({"name": k.replace(".qkv", ".dense3"), "data": data[length*2:]}) ms.save_checkpoint(new_dict, ms_ckpt_path) print("Weights conversion completes. ") if __name__ == "__main__": parser = argparse.ArgumentParser(description="mae vit weight convert script") parser.add_argument("--torch_pth_path", type=str, default="mae_pretrain_vit_base.pth", required=True, help="The torch checkpoint path.") parser.add_argument("--ms_ckpt_path", type=str, required=True, default="mae_pretrain_vit_base.ckpt", help="The output mindspore checkpoint path.") opt = parser.parse_args() convert_weight(opt.torch_pth_path, opt.ms_ckpt_path) pengcheng.mind-test/mindformers/models/mae/mae.py000066400000000000000000000430041471400134000224000ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/facebookresearch/mae # ============================================================================ """Mae Model.""" import numpy as np from mindspore import Tensor, Parameter, nn from mindspore import dtype as mstype from mindspore import ops as P import mindspore.common.initializer as weight_init from mindformers.mindformer_book import MindFormerBook from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.models.mae.mae_modules import Block, LayerNorm, Linear from mindformers.models.mae.mae_modules import PatchEmbed, Patchify, UnPatchify from mindformers.models.mae.mae_modules import get_2d_sincos_pos_embed from mindformers.core.loss import MSELoss from mindformers.models.mae.mae_config import ViTMAEConfig __all__ = ['ViTMAEModel', 'ViTMAEForPreTraining'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTMAEModel(BaseModel): """ Pretrain MAE Module. The supported model name could be selected from ViTMAEConfig.show_support_list(). Args: config (ViTMAEConfig): the config of Mae model. Examples: >>> # input model name >>> model_a = ViTMAEModel.from_pretrained('mae_vit_base_p16') >>> # input config >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('mae_vit_base_p16') >>> model_b = ViTMAEModel(config) """ _support_list = MindFormerBook.get_model_support_list()['mae'] def __init__(self, config=None): config = config if config else ViTMAEConfig() super().__init__(config) self.use_moe = (config.moe_config.expert_num > 1) parallel_config = config.parallel_config dp = parallel_config.data_parallel # -------------------------------------------------------------------------- # MAE encoder specifics self.patch_embed = PatchEmbed(img_size=config.image_size, patch_size=config.patch_size, in_features=config.in_chans, out_features=config.embed_dim, parallel_config=config.parallel_config) num_patches = self.patch_embed.num_patches seq_length = int((1 - config.mask_ratio) * num_patches) + 1 self.seq_length = seq_length self.num_masked = num_patches - seq_length + 1 self.cls_tokens = Parameter( weight_init.initializer(weight_init.Normal(sigma=config.initializer_range), (1, 1, config.embed_dim)), requires_grad=True) self.num_patches = num_patches self.pos_embed = Parameter( weight_init.initializer(weight_init.Zero(), (1, num_patches + 1, config.embed_dim)), name='pos_embed', requires_grad=True) # stochastic depth decay rule hdr = [x.item() for x in np.linspace(0, config.drop_path_rate, config.depth)] parallel_config_args = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config self.blocks = nn.CellList([ Block(hidden_size=config.embed_dim, ffn_hidden_size=config.intermediate_size, seq_length=seq_length, drop_rate=config.drop_rate, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=hdr[i], layer_norm_eps=config.layer_norm_eps, qkv_bias=config.qkv_bias, init_values=config.init_values, weight_init='XavierUniform', layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, window_size=config.window_size, num_heads=config.num_heads, hidden_act=config.hidden_act, moe_config=config.moe_config, post_layernorm_residual=config.post_layernorm_residual, param_init_type=config.param_init_type, parallel_config=parallel_config_args) for i in range(config.depth)]) self.norm = LayerNorm((config.embed_dim,), eps=config.layer_norm_eps).shard(((dp, 1, 1),)) # -------------------------------------------------------------------------- self.stride_slice = P.StridedSlice().shard(((1, 1, 1),)) self.add = P.Add().shard(((dp, 1, 1), (1, 1, 1))) self.expand_dim = P.ExpandDims().shard(((dp, 1),)) self.tile = P.Tile().shard(((dp, 1, 1),)) self.gather = P.GatherD().shard(((dp, 1, 1), (dp, 1, 1))) self.cat = P.Concat(axis=1).shard(((dp, 1, 1), (dp, 1, 1))) self.init_weights() self.init_pos_emd() def init_pos_emd(self): """init values of pos_embed""" encoder_pos_emd = Tensor( get_2d_sincos_pos_embed(self.pos_embed.shape[-1], int(self.num_patches ** .5), cls_token=True), mstype.float32 ) encoder_pos_emd = P.ExpandDims()(encoder_pos_emd, 0) self.pos_embed = Parameter(encoder_pos_emd, name='sincos_pos_embedding', requires_grad=False) def init_weights(self): """ ViT weight initialization.""" for name, cell in self.cells_and_names(): if isinstance(cell, Linear): cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) if isinstance(cell, Linear) and cell.bias is not None: cell.bias.set_data(weight_init.initializer(weight_init.Zero(), cell.bias.shape, cell.bias.dtype)) elif isinstance(cell, (LayerNorm, nn.LayerNorm)): cell.gamma.set_data(weight_init.initializer(weight_init.One(), cell.gamma.shape, cell.gamma.dtype)) cell.beta.set_data(weight_init.initializer(weight_init.Zero(), cell.beta.shape, cell.beta.dtype)) if name == "patch_embed.proj": cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) def construct(self, image, unmask_index): """construct of VisionTransformerForMae Encoder""" # patch to encoder tokens and add positions tokens = self.patch_embed(image) encoder_pos_embedding = self.stride_slice( self.pos_embed, (0, 1, 0), (1, self.pos_embed.shape[1], self.pos_embed.shape[2]), (1, 1, 1)) tokens = self.add(tokens, encoder_pos_embedding) # get the unmasked tokens to be encoded unmask_index_ = self.expand_dim(unmask_index, -1) unmask_index = self.tile(unmask_index_, (1, 1, tokens.shape[2])) unmask_tokens = self.gather(tokens, 1, unmask_index) # cls_tokens add pos_embedding cls_pos_embedding = self.stride_slice( self.pos_embed, (0, 0, 0), (1, 1, self.pos_embed.shape[2]), (1, 1, 1)) batch_size = image.shape[0] if batch_size == 1: cls_tokens = self.cls_tokens else: cls_tokens = self.tile(self.cls_tokens, (batch_size, 1, 1)) cls_tokens = self.add(cls_tokens, cls_pos_embedding) # concat cls_tokens encoded_tokens = self.cat((cls_tokens, unmask_tokens)) # attend with vision transformer encoder_input_mask = P.Ones()((batch_size, self.seq_length, self.seq_length), mstype.float32) for block in self.blocks: encoded_tokens = block(encoded_tokens, encoder_input_mask) encoded_tokens = self.norm(encoded_tokens) return encoded_tokens @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTMAEForPreTraining(BaseModel): """ Pretrain MAE Module. The supported model name could be selected from ViTMAEConfig.show_support_list(). Args: config (ViTMAEConfig): the config of Mae model. Examples: >>> # input model name >>> model_a = ViTMAEForPreTraining.from_pretrained('mae_vit_base_p16') >>> # input config >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('mae_vit_base_p16') >>> model_b = ViTMAEForPreTraining(config) """ _support_list = MindFormerBook.get_model_support_list()['mae'] def __init__(self, config=None): config = config if config else ViTMAEConfig() super().__init__(config) self.use_moe = (config.moe_config.expert_num > 1) parallel_config = config.parallel_config dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.vit = ViTMAEModel(config) self.num_patches = num_patches = self.vit.patch_embed.num_patches self.num_masked = num_patches - self.vit.seq_length + 1 # -------------------------------------------------------------------------- # MAE decoder specifics self.decoder_embed = Linear( config.embed_dim, config.decoder_embed_dim, weight_init="xavier_uniform", compute_dtype=mstype.float16).to_float(mstype.float16) self.decoder_embed.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.mask_tokens = Parameter( weight_init.initializer(weight_init.Normal(sigma=config.initializer_range), (1, 1, config.decoder_embed_dim)), name='mask_tokens', requires_grad=True) self.decoder_pos_embed = Parameter( weight_init.initializer(weight_init.Zero(), (1, num_patches + 1, config.decoder_embed_dim)), name='pos_embedding', requires_grad=False) hdr = [x.item() for x in np.linspace(0, config.drop_path_rate, config.decoder_depth)] self.decoder_blocks = nn.CellList([ Block(hidden_size=config.decoder_embed_dim, ffn_hidden_size=config.decoder_intermediate_size, seq_length=num_patches + 1, drop_rate=config.drop_rate, attention_dropout_rate=config.attention_dropout_rate, layer_norm_eps=config.layer_norm_eps, hidden_dropout_rate=hdr[i], qkv_bias=config.qkv_bias, init_values=config.init_values, weight_init='XavierUniform', layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, window_size=config.window_size, num_heads=config.decoder_num_heads, hidden_act=config.hidden_act, post_layernorm_residual=config.post_layernorm_residual, param_init_type=config.param_init_type, parallel_config=parallel_config.dp_mp_config) for i in range(config.decoder_depth)]) self.decoder_norm = LayerNorm((config.decoder_embed_dim,), eps=config.layer_norm_eps).shard(((dp, 1, 1),)) patch_dim = config.in_chans * config.patch_size ** 2 self.decoder_pred = Linear( config.decoder_embed_dim, patch_dim, weight_init="xavier_uniform", compute_dtype=mstype.float16).to_float(mstype.float16) self.decoder_pred.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) # -------------------------------------------------------------------------- self.patchify = Patchify(patch_size=config.patch_size, parallel_config=parallel_config) self.unpatchify = UnPatchify( patch_size=config.patch_size, seq_length=num_patches, parallel_config=parallel_config) self.stride_slice = P.StridedSlice().shard(((1, 1, 1),)) self.add = P.Add().shard(((dp, 1, 1), (1, 1, 1))) self.expand_dim = P.ExpandDims().shard(((dp, 1),)) self.tile = P.Tile().shard(((dp, 1, 1),)) self.gather = P.GatherD().shard(((dp, 1, 1), (dp, 1, 1))) self.cat = P.Concat(axis=1).shard(((dp, 1, 1), (dp, 1, 1))) self.gather1 = P.GatherD().shard(((dp, 1), (dp, 1))) self.gather2 = P.GatherD().shard(((dp, 1, 1), (dp, 1, 1))) self.mse_loss = MSELoss(config.norm_pixel_loss, parallel_config) self.images_summary = P.ImageSummary().shard(((dp, 1, 1, 1),)) self.init_weights() self.init_pos_emd() self.load_checkpoint(config) def init_pos_emd(self): """init values of pos_embed""" decoder_pos_embed = Tensor( get_2d_sincos_pos_embed(self.decoder_pos_embed.shape[-1], int(self.num_patches ** .5), cls_token=True), mstype.float32 ) decoder_pos_embed = P.ExpandDims()(decoder_pos_embed, 0) self.decoder_pos_embed = Parameter(decoder_pos_embed, name='decoder_pos_embed', requires_grad=False) def init_weights(self): """ ViT weight initialization.""" for _, cell in self.cells_and_names(): if isinstance(cell, Linear): cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) if isinstance(cell, Linear) and cell.bias is not None: cell.bias.set_data(weight_init.initializer(weight_init.Zero(), cell.bias.shape, cell.bias.dtype)) elif isinstance(cell, (LayerNorm, nn.LayerNorm)): cell.gamma.set_data(weight_init.initializer(weight_init.One(), cell.gamma.shape, cell.gamma.dtype)) cell.beta.set_data(weight_init.initializer(weight_init.Zero(), cell.beta.shape, cell.beta.dtype)) def construct(self, image, mask, ids_restore, unmask_index): """construct of VisionTransformerForMae""" self.images_summary("input images", image) encoder_tokens = self.vit(image, unmask_index) unmask_tokens = self.decoder_embed(encoder_tokens) unmask_tokens = self.cast(unmask_tokens, mstype.float32) # mask tokens add the positions using the masked indices derived above batch_size = encoder_tokens.shape[0] mask_tokens = self.tile(self.mask_tokens, (batch_size, self.num_masked, 1)) # concat the masked tokens to the decoder tokens and attend with decoder img_tokens = self.stride_slice( unmask_tokens, (0, 1, 0), (unmask_tokens.shape[0], unmask_tokens.shape[1], unmask_tokens.shape[2]), (1, 1, 1)) full_tokens_ = self.cat((img_tokens, mask_tokens)) ids_restore_copy = ids_restore ids_restore_ = self.expand_dim(ids_restore_copy, -1) ids_restore_ = self.tile(ids_restore_, (1, 1, unmask_tokens.shape[2])) full_tokens_ = self.gather2(full_tokens_, 1, ids_restore_) cls_tokens = self.stride_slice( unmask_tokens, (0, 0, 0), (unmask_tokens.shape[0], 1, unmask_tokens.shape[2]), (1, 1, 1)) decoder_tokens = self.cat((cls_tokens, full_tokens_)) # add position embendding for decoder tokens decoder_tokens = self.add(decoder_tokens, self.decoder_pos_embed) # decoder attention_mask = Tensor(np.ones((batch_size, self.num_patches + 1, self.num_patches + 1)), mstype.float32) for block in self.decoder_blocks: decoder_tokens = block(decoder_tokens, attention_mask) # normalize decoder tokens decoder_tokens = self.decoder_norm(decoder_tokens) # splice out the mask tokens and project to pixel values pred = self.decoder_pred(decoder_tokens) pred = self.cast(pred, mstype.float32) pred = self.stride_slice(pred, (0, 1, 0), (pred.shape[0], pred.shape[1], pred.shape[2]), (1, 1, 1)) reconstruct_images = self.unpatchify(pred) self.images_summary("reconstruct image", reconstruct_images) if not self.training: return reconstruct_images patches = self.patchify(image) mask = self.gather1(mask, 1, ids_restore) mae_loss = self.mse_loss(pred, patches, mask) return mae_loss pengcheng.mind-test/mindformers/models/mae/mae_config.py000066400000000000000000000114651471400134000237330ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Mae Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.modules.transformer.moe import MoEConfig, default_moe_config from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_config import BaseConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) __all__ = ['ViTMAEConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class ViTMAEConfig(BaseConfig): """ Config for Mae model Examples: >>> # init a config with a model name >>> config_a = ViTMAEConfig.from_pretrained('mae_vit_base_p16') >>> # init a config with a config path >>> import os >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), >>> 'configs', 'mae', 'run_mae_vit_base_p16_224_800ep.yaml') >>> config_b = ViTMAEConfig.from_pretrained(config_path) >>> # init a config with args >>> config_c = ViTMAEConfig( >>> patch_size=16, >>> in_chans=3, >>> ... >>> ) """ _support_list = MindFormerBook.get_config_support_list()['mae'] def __init__(self, mask_ratio: float = 0.75, image_size: int = 224, patch_size: int = 16, num_channels: int = 3, initializer_range: float = 0.02, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, qkv_bias: bool = True, hidden_act: str = "gelu", post_layernorm_residual: bool = False, layer_norm_eps: float = 1e-6, attention_probs_dropout_prob: float = 0.0, hidden_dropout_prob: float = 0.0, drop_path_rate: float = 0., decoder_hidden_size: int = 512, decoder_num_hidden_layers: int = 8, decoder_num_attention_heads: int = 16, decoder_intermediate_size: int = 2048, norm_pix_loss: bool = True, checkpoint_name_or_path: str = '', layernorm_compute_type: mstype = mstype.float32, softmax_compute_type: mstype = mstype.float32, param_init_type: mstype = mstype.float32, parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs): super().__init__(**kwargs) self.mask_ratio = mask_ratio self.image_size = image_size self.patch_size = patch_size self.in_chans = num_channels self.initializer_range = initializer_range self.embed_dim = hidden_size self.depth = num_hidden_layers self.num_heads = num_attention_heads self.intermediate_size = intermediate_size self.qkv_bias = qkv_bias self.hidden_act = hidden_act self.post_layernorm_residual = post_layernorm_residual self.layer_norm_eps = layer_norm_eps self.attention_dropout_rate = attention_probs_dropout_prob self.drop_rate = hidden_dropout_prob self.drop_path_rate = drop_path_rate self.decoder_embed_dim = decoder_hidden_size self.decoder_depth = decoder_num_hidden_layers self.decoder_num_heads = decoder_num_attention_heads self.decoder_intermediate_size = decoder_intermediate_size self.norm_pixel_loss = norm_pix_loss self.checkpoint_name_or_path = checkpoint_name_or_path self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.param_init_type = param_init_type self.parallel_config = parallel_config self.moe_config = moe_config pengcheng.mind-test/mindformers/models/mae/mae_modules.py000066400000000000000000001066711471400134000241420ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Modules of ViTMAEForPreTraining, including Linear, Block, MLP, Attention, PatchEmbed, etc. """ import math import numpy as np from mindspore import nn, Parameter, Tensor, context, ParallelMode import mindspore.common.dtype as mstype from mindspore.ops import operations as P, constexpr from mindspore.ops import functional as F import mindspore.common.initializer as init from mindspore.parallel._utils import _get_parallel_mode from mindformers.modules import layers, FeedForward from mindformers.modules.transformer.moe import default_moe_config, _check_moe_config from mindformers.modules.transformer.transformer import default_dpmp_config, _check_config @constexpr def gen_shape(x_shape, ndim): return (x_shape,) + (1,) * (ndim + 1) def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False): """ grid_size: int of the grid height and width return: pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token) """ grid_h = np.arange(grid_size, dtype=np.float32) grid_w = np.arange(grid_size, dtype=np.float32) grid = np.meshgrid(grid_w, grid_h) # here w goes first grid = np.stack(grid, axis=0) grid = grid.reshape([2, 1, grid_size, grid_size]) pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid) if cls_token: pos_embed = np.concatenate([np.zeros([1, embed_dim]), pos_embed], axis=0) return pos_embed def get_2d_sincos_pos_embed_from_grid(embed_dim, grid): assert embed_dim % 2 == 0 # use half of dimensions to encode grid_h emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2) emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2) emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D) return emb def get_1d_sincos_pos_embed_from_grid(embed_dim, pos): """ embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D) """ assert embed_dim % 2 == 0 omega = np.arange(embed_dim // 2, dtype=np.float32) omega /= embed_dim / 2. omega = 1. / 10000 ** omega # (D/2,) pos = pos.reshape(-1) # (M,) out = np.einsum('m,d->md', pos, omega) # (M, D/2), outer product emb_sin = np.sin(out) # (M, D/2) emb_cos = np.cos(out) # (M, D/2) emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D) return emb class LayerNorm(layers.LayerNorm): # pylint: disable=W0212 r""" A self-defined layer norm operation using reduce sum and reduce mean. """ def __init__(self, normalized_shape, eps=1e-6, param_init_type=mstype.float32): super(LayerNorm, self).__init__( normalized_shape, eps=eps, param_init_type=param_init_type) class Identity(nn.Cell): def construct(self, x): return x class RelativePositionBias(nn.Cell): """relative position bias""" def __init__(self, window_size, num_heads): super(RelativePositionBias, self).__init__() self.window_size = window_size # cls to token & token to cls & cls to cls self.num_relative_distance = (2 * window_size[0] - 1) * (2 * window_size[1] - 1) + 3 self.relative_position_bias_table = Parameter( init.initializer( init.TruncatedNormal(sigma=.02), (self.num_relative_distance, num_heads)), name='relative_position_bias_table') # get pair-wise relative position index for each token inside the window coords_h = Tensor(np.arange(window_size[0]), mstype.int32) coords_w = Tensor(np.arange(window_size[1]), mstype.int32) coords = P.Stack(axis=0)(P.Meshgrid(indexing='ij')((coords_h, coords_w))) # 2, Wh, Ww coords_flatten = P.Flatten()(coords) # 2, Wh*Ww relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww relative_coords = P.Transpose()(relative_coords, (1, 2, 0)).asnumpy() # Wh*Ww, Wh*Ww, 2 relative_coords[:, :, 0] += window_size[0] - 1 relative_coords[:, :, 1] += window_size[1] - 1 relative_coords[:, :, 0] *= 2 * window_size[1] - 1 relative_position_index = \ np.zeros(((window_size[0] * window_size[1] + 1),) * 2, dtype=int) relative_position_index[1:, 1:] = np.sum(relative_coords, axis=-1) # Wh*Ww, Wh*Ww relative_position_index[0, 0:] = self.num_relative_distance - 3 relative_position_index[0:, 0] = self.num_relative_distance - 2 relative_position_index[0, 0] = self.num_relative_distance - 1 relative_position_index = Tensor(relative_position_index, mstype.int32) self.relative_position_index = Parameter( relative_position_index, requires_grad=False, name="relative_position_index") self.reshape = P.Reshape() self.transpose = P.Transpose() self.gather = P.Gather() def construct(self): relative_position_index = self.relative_position_index.view(-1) relative_position_bias = self.gather(self.relative_position_bias_table, relative_position_index, 0) relative_position_bias = self.reshape( relative_position_bias, (self.window_size[0] * self.window_size[1] + 1, self.window_size[0] * self.window_size[1] + 1, -1)) relative_position_bias = self.transpose(relative_position_bias, (2, 0, 1)) return relative_position_bias class Linear(layers.Linear): # pylint: disable=W0212 r""" Linear function for RingMo. """ def __init__(self, in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, transpose_b=True, expert_num=1, outer_batch=1, activation_compute_type=mstype.float16, param_init_type=mstype.float32, compute_dtype=mstype.float16): super(Linear, self).__init__( in_channels, out_channels, weight_init=weight_init, bias_init=bias_init, has_bias=has_bias, activation=activation, transpose_b=transpose_b, expert_num=expert_num, outer_batch=outer_batch, param_init_type=param_init_type, compute_dtype=compute_dtype) self.activation_compute_type = activation_compute_type def construct(self, x): """construct of layer""" out_shape = P.Shape()(x)[:-1] + (self.out_channels,) x = P.Reshape()(x, (-1, self.in_channels)) if self.expert_flag: x = P.Reshape()(x, (self.outer_batch, self.expert_num, -1, self.in_channels)) weight = self.cast(self.weight, self.dtype) x = self.matmul(x, weight) if self.has_bias: x = self.bias_add(x, self.cast(self.bias, self.dtype)) if self.activation_flag: x = self.activation(self.cast(x, self.activation_compute_type)) x = self.cast(x, self.dtype) output = P.Reshape()(x, out_shape) return output class Dropout(layers.Dropout): # pylint: disable=W0212 r""" A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for context training. """ def __init__(self, keep_prob=0.5, dtype=mstype.float32): super(Dropout, self).__init__(keep_prob=keep_prob, dtype=dtype) class DropPath(nn.Cell): """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" def __init__(self, drop_prob): super(DropPath, self).__init__() self.drop = Dropout(keep_prob=1 - drop_prob) self.mask = Tensor(np.ones(1,), dtype=mstype.float32) self.tile = P.Tile() self.mul = P.Mul() def construct(self, x): if not self.training: return x mask = self.tile(self.mask, (x.shape[0],) + (1,) * (x.ndim-1)) out = self.drop(mask) out = self.mul(out, x) return out def shard(self, strategy): self.mul.shard(strategy) class Block(nn.Cell): """Block of ringmo""" def __init__(self, hidden_size, ffn_hidden_size, num_heads, seq_length, drop_rate=0., attention_dropout_rate=0., hidden_dropout_rate=0., layer_norm_eps=1e-6, qkv_bias=True, window_size=None, post_layernorm_residual=False, init_values=None, weight_init='XavierUniform', layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', moe_config=default_moe_config, parallel_config=default_dpmp_config): super(Block, self).__init__() _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.seq_length = seq_length self.hidden_size = hidden_size self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None self.layernorm1 = LayerNorm((hidden_size,), eps=layer_norm_eps).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,), eps=layer_norm_eps).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) parallel_config_args = parallel_config.dpmp if self.use_moe else parallel_config self.attention = Attention(src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, window_size=window_size, num_heads=num_heads, weight_init=weight_init, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, qkv_bias=qkv_bias, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=parallel_config_args) self.output = MLP(hidden_size=hidden_size, dropout_rate=drop_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, weight_init=weight_init, hidden_act=hidden_act, use_dropout=False, parallel_config=parallel_config) if init_values is not None: self.gamma_1 = Parameter( Tensor(init_values * np.ones((hidden_size,)), mstype.float32), name="gamma1", requires_grad=True) self.gamma_2 = Parameter( Tensor(init_values * np.ones((hidden_size,)), mstype.float32), name="gamma2", requires_grad=True) else: self.gamma_1, self.gamma_2 = None, None self.mul_gamma = P.Mul().shard(((parallel_config.data_parallel, 1), (1,))) self.drop_path = DropPath(hidden_dropout_rate) self.drop_path.shard(((parallel_config.data_parallel, 1),)) self.drop_path3d = DropPath(hidden_dropout_rate) self.drop_path3d.shard(((parallel_config.data_parallel, 1, 1),)) self.mul = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.reshape = P.Reshape() def construct(self, x, input_mask, rel_pos_bias=None): """construct of Block""" x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) attention = self.attention( input_x, input_x, input_x, input_mask, rel_pos_bias) if self.gamma_1 is not None: attention = self.mul_gamma(attention, self.gamma_1) if len(x_shape) == 3: attention = P.Reshape()(attention, x_shape) attention = self.drop_path3d(attention) attention = P.Reshape()(attention, (-1, x_shape[-1])) else: attention = self.drop_path(attention) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) mlp_logit = self.output(output_x) if self.gamma_2 is not None: mlp_logit = self.mul_gamma(mlp_logit, self.gamma_2) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) mlp_logit = self.drop_path3d(mlp_logit) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: mlp_logit = self.drop_path(mlp_logit) if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) return output class MLP(FeedForward): r"""MLP for ring-mo.""" def __init__(self, hidden_size, ffn_hidden_size=None, out_features=None, dropout_rate=0., hidden_act='gelu', weight_init='XavierUniform', use_dropout=True, param_init_type=mstype.float32, parallel_config=default_dpmp_config): ffn_hidden_size = ffn_hidden_size or hidden_size super(MLP, self).__init__( hidden_size, ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, param_init_type=mstype.float32, parallel_config=parallel_config) mp = parallel_config.model_parallel dp = parallel_config.data_parallel out_features = out_features or hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=hidden_size, out_channels=ffn_hidden_size, activation=hidden_act, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=ffn_hidden_size, out_channels=out_features, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - dropout_rate) self.dropout.shard(((dp, 1),)) self.dropout_3d = Dropout(1 - dropout_rate) self.dropout_3d.shard(((dp, 1, 1),)) self.use_dropout = use_dropout def construct(self, x): """construct of mlp""" x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) if self.use_dropout: if len(F.shape(hidden)) == 3: hidden = self.dropout_3d(hidden) elif len(F.shape(hidden)) == 2: hidden = self.dropout(hidden) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) return output class Attention(nn.Cell): r""" This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following """ def __init__(self, src_seq_length, tgt_seq_length, hidden_size, num_heads, window_size=None, hidden_dropout_rate=0., attention_dropout_rate=0., qkv_bias=True, weight_init='XavierUniform', compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(Attention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] _check_config(parallel_config) self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - hidden_dropout_rate) self.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout = Dropout(1 - attention_dropout_rate) self.prob_dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) if window_size: self.relative_position_bias = RelativePositionBias(window_size, num_heads) else: self.window_size = None self.relative_position_bias_table = None self.relative_position_index = None self.relative_position_bias = None self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1, 1), (1, 1, 1))) self.expand_dims_rpb = P.ExpandDims().shard(((1, 1, 1),)) self.add_rpb = P.Add().shard(((parallel_config.data_parallel, 1, 1, 1), (1, 1, 1, 1))) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, rel_pos_bias=None): """construct of attention""" ori_shape = F.shape(query_tensor) batch_size = F.shape(attention_mask)[0] query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, attention_mask, rel_pos_bias) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) return output def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( F.reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = F.reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask, rel_pos_bias): r"""Get the weighted score along the seq_length.""" # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] factor = P.Cast()(self.scale_factor, P.DType()(query)) query = self.real_div(query, factor) key = self.real_div(key, factor) score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) score = P.Cast()(score, self.softmax_dtype) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(score)), P.Cast()(attention_mask, P.DType()(score))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, score) # add window cross module if self.relative_position_bias is not None: relative_position_bias = self.expand_dims_rpb(self.relative_position_bias(), 0) attention_scores = self.add_rpb(attention_scores, relative_position_bias) if rel_pos_bias is not None: attention_scores = self.add_3d(attention_scores, rel_pos_bias) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = P.Cast()(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge def _merge_heads(self, x): """ convert a 4d input to a 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor class PatchEmbed(nn.Cell): """Construct the embeddings from patch, position embeddings.""" def __init__(self, img_size=224, patch_size=16, in_features=3, out_features=768, norm_layer=False, parallel_config=None): super(PatchEmbed, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 img_size = (img_size, img_size) patch_size = (patch_size, patch_size) self.grid_size = (img_size[0] // patch_size[0], img_size[1] // patch_size[1]) self.num_patches = self.grid_size[0] * self.grid_size[1] self.proj = nn.Conv2d( in_channels=in_features, out_channels=out_features, kernel_size=patch_size, stride=patch_size, weight_init=init.TruncatedNormal(sigma=0.02), has_bias=True, pad_mode='pad') self.proj.conv2d.shard(((dp, 1, 1, 1), (1, 1, 1, 1))) self.proj.bias_add.shard(((dp, 1, 1, 1), (1,))) self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1),)) # usually not use norm self.norm = LayerNorm((out_features,), eps=1e-6).shard(((dp, 1, 1),)) if norm_layer else Identity() def construct(self, x): """construct""" # True x: bs False x: bs * dp x = self.proj(x) b, c, h, w = x.shape x = self.reshape(x, (b, c, h * w)) x = self.transpose(x, (0, 2, 1)) x = self.norm(x) return x def get_kernel_size(patch_size): """ input: 2^i & i <= 5 | 14 output: a list of kernel size """ x = None y = None z = None ans = False for i in range(1, patch_size + 1): if patch_size % i == 0: x = i mul_y_z = patch_size // i for j in range(1, mul_y_z + 1): if mul_y_z % j == 0: y = j z = mul_y_z // j if x >= y >= z: ans = True break if ans: break if not ans: raise ValueError(patch_size) return [x, y, z] class Patchify(nn.Cell): """Patchify""" def __init__(self, patch_size, parallel_config=None): super(Patchify, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.patch_size = patch_size self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, img): p = self.patch_size bs, channels, h, w = img.shape x = self.reshape(img, (bs, channels, h // p, p, w // p, p)) x = self.transpose(x, (0, 2, 4, 3, 5, 1)) patches = self.reshape(x, (bs, (h // p) * (w // p), channels * p * p)) return patches class UnPatchify(nn.Cell): """UnPatchify""" def __init__(self, patch_size, seq_length, parallel_config=None): super(UnPatchify, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.p = patch_size self.h = self.w = int(seq_length ** .5) assert self.h * self.w == seq_length self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, x): bs = x.shape[0] x = self.reshape(x, (bs, self.h, self.w, self.p, self.p, 3)) x = self.transpose(x, (0, 5, 1, 3, 2, 4)) images = self.reshape(x, (bs, 3, self.h * self.p, self.w * self.p)) return images pengcheng.mind-test/mindformers/models/mae/mae_processor.py000066400000000000000000000137271471400134000245100ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ ViTMAEProcessor """ import numpy as np from PIL import Image from mindspore import Tensor from mindspore.dataset.vision.transforms import ToTensor, Normalize from mindformers.mindformer_book import MindFormerBook from mindformers.dataset import Resize from mindformers.dataset.base_dataset import BaseDataset from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['ViTMAEProcessor', 'ViTMAEImageProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class ViTMAEImageProcessor(BaseImageProcessor): """ ViTMAEImageProcessor. Args: image_resolution (int): the target size. """ def __init__(self, size=224, patch_size=16, mask_ratio=0.75): super(ViTMAEImageProcessor, self).__init__(image_resolution=size) self.resize = Resize((size, size), interpolation='cubic') self.to_tensor = ToTensor() self.normalize = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], is_hwc=False) if not 0 < mask_ratio < 1: raise ValueError('masking ratio must be kept between 0 and 1, but get mask_ratio {mask_ratio}.') # seq_length self.num_patches = (size // patch_size) ** 2 # seq masked number self.keep_num = int((1 - mask_ratio) * self.num_patches) def preprocess(self, images, **kwargs): """ Preprocess required by base processor. Args: images (tensor, PIL.Image, numpy.array, list): a batch of images. Return: A 4-rank tensor for a batch of images. """ images = self._format_inputs(images) res = [] ids_restores = [] masks = [] unmask_indexes = [] for image in images: image = self.resize(image) image = self.to_tensor(image) image = self.normalize(image) res.append(image) rand_indices = np.argsort( np.random.uniform(size=(self.num_patches,)), axis=0).astype(np.int32) ids_restore = np.argsort(rand_indices, axis=0).astype(np.int32) ids_restores.append(ids_restore) mask = np.ones((self.num_patches,)).astype(np.int32) mask[:self.keep_num] = 0 masks.append(mask) unmask_index = rand_indices[:self.keep_num] unmask_indexes.append(unmask_index) return Tensor(res), Tensor(masks), Tensor(ids_restores), Tensor(unmask_indexes) def _format_inputs(self, inputs): """ Transform image classification inputs into (bz, h, w, c) or (h, w, c) numpy array. Args: inputs (tensor, numpy.array, PIL.Image, list, BaseDataset): for numpy or tensor input, the channel could be (bz, c, h, w), (c, h, w) or (bz, h, w, c), (h, w, c); for list, the item could be PIL.Image, numpy.array, Tensor; for BaseDataset, return without any operations. Return: transformed images: for PIL.Image, numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for list, return a numpy array for each element; for BaseDataset, it is returned directly. """ if not isinstance(inputs, (list, Image.Image, Tensor, np.ndarray, BaseDataset)): raise TypeError("input type is not Tensor, numpy, Image, list of Image or MindFormer BaseDataset") if isinstance(inputs, list): return [self._format_inputs(item) for item in inputs] if isinstance(inputs, Image.Image): inputs = np.array(inputs) if isinstance(inputs, Tensor): inputs = inputs.asnumpy() if isinstance(inputs, np.ndarray): if len(inputs.shape) == 3: inputs = np.expand_dims(inputs, 0) inputs = self._chw2hwc(inputs) elif len(inputs.shape) == 4: inputs = self._chw2hwc(inputs) else: raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(inputs.shape)}") return inputs @staticmethod def _chw2hwc(inputs): if inputs.shape[-1] != 3: inputs = inputs.transpose(0, 2, 3, 1) return inputs @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class ViTMAEProcessor(BaseProcessor): """ ViTMAEProcessor, consists of a feature extractor (BaseFeatureEXtractor) for image input. Examples: >>> from mindformers import MindFormerBook >>> from mindformers.models import ViTMAEProcessor >>> yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", ... "mae", "model_config", "mae_vit_base_p16.yaml") >>> # build ViTMAEProcessor from pretrained >>> pro_a = ViTMAEProcessor.from_pretrained('mae_vit_base_p16') >>> # build ViTMAEProcessor from config >>> pro_b = ViTMAEProcessor.from_pretrained(yaml_path) """ _support_list = MindFormerBook.get_processor_support_list()['mae'] def __init__(self, image_processor=None, return_tensors='ms'): super(ViTMAEProcessor, self).__init__( image_processor=image_processor, return_tensors=return_tensors ) pengcheng.mind-test/mindformers/models/pangualpha/000077500000000000000000000000001471400134000226415ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/pangualpha/__init__.py000066400000000000000000000017761471400134000247650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for PanGu Alpha""" from .pangualpha import * from .pangualpha_config import * from .pangualpha_tokenizer import * from .pangualpha_processor import * __all__ = [] __all__.extend(pangualpha.__all__) __all__.extend(pangualpha_config.__all__) __all__.extend(pangualpha_tokenizer.__all__) __all__.extend(pangualpha_processor.__all__) pengcheng.mind-test/mindformers/models/pangualpha/pangualpha.py000066400000000000000000000673371471400134000253530ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """PanGuAlpha model""" import copy import numpy as np import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.models.base_model import BaseModel from mindformers.core.loss import CrossEntropyLoss from mindformers.modules.layers import LayerNorm from mindformers.modules.transformer import VocabEmbedding, TransformerEncoder, TransformerEncoderLayer, \ AttentionMask, MoEConfig from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.mindformer_book import MindFormerBook from .pangualpha_config import PanguAlphaConfig __all__ = ['PanguAlphaHeadModel', 'PanguAlphaModel', 'PanguAlphaPromptTextClassificationModel'] class EmbeddingLayer(nn.Cell): r"""Embedding layer of the PanGuAlpha Model""" def __init__(self, config): super(EmbeddingLayer, self).__init__() # Only for the pipeline mode, the embedding needs to be row sliced. self.word_embedding = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, param_init=initializer("normal", [config.vocab_size, config.hidden_size], dtype=mstype.float32), parallel_config=config.parallel_config.embedding_dp_mp_config) copied_parallel_config = copy.deepcopy(config.parallel_config) copied_parallel_config.vocab_emb_dp = True self.position_embedding = VocabEmbedding(vocab_size=config.seq_length, embedding_size=config.hidden_size, param_init=initializer("normal", [config.seq_length, config.hidden_size], dtype=mstype.float32), parallel_config=copied_parallel_config.embedding_dp_mp_config) self.add = P.Add().shard( ((config.parallel_config.data_parallel, 1, 1), (config.parallel_config.data_parallel, 1, 1))) self.dropout = nn.Dropout(keep_prob=1-config.embedding_dropout_prob) self.dropout.dropout.shard(((config.parallel_config.data_parallel, 1, 1),)) self.is_first_iteration = True self.use_past = config.use_past self.batch_size = config.batch_size def construct(self, input_ids, input_position, batch_valid_length): r"""forward pass of the layer""" word_embedding, word_table = self.word_embedding(input_ids) if self.use_past and not self.is_first_iteration: _, seq_length = F.shape(input_ids) input_position = batch_valid_length.view(self.batch_size, seq_length) position_embedding, _ = self.position_embedding(input_position) embed = self.add(word_embedding, position_embedding) embed = self.dropout(embed) return embed, word_table def get_word_embedding_weight(self): r"""get word embedding weight""" return self.word_embedding.embedding_table class QueryLayer(TransformerEncoderLayer): r"""Query Layer at the final layer.""" def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, param_init_type=mstype.float32, hidden_act='fast_gelu', use_past=False, parallel_config=None, softmax_compute_type=mstype.float32): super(QueryLayer, self).__init__(batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, hidden_act=hidden_act, use_past=use_past, parallel_config=parallel_config.dp_mp_config, softmax_compute_type=softmax_compute_type) # pylint: disable=W0221 def construct(self, x, query_vector, input_mask, init_reset=True, batch_valid_length=None): r""" The forward process of the block. """ # [bs * seq_length, embedding_size] input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) value_reset = self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(query_vector, input_x, input_x, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step key_update = self.assign(self.key_past, key_present) value_update = self.assign(self.value_past, value_present) # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) return output, layer_present class PanguAlphaHead(nn.Cell): """ Head to get the logits of each token in the vocab Args: config(): the config of network Inputs: state: the output of the backbone embedding_table: the embedding table of the vocabulary Returns: logits: Tensor, the logits of the corresponding inputs """ def __init__(self, hidden_size, compute_type=mstype.float16, parallel_config=None): super(PanguAlphaHead, self).__init__() if parallel_config.vocab_emb_dp: self.matmul = P.MatMul(transpose_b=True).shard(((parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = P.MatMul(transpose_b=True).shard(((parallel_config.data_parallel, 1), ( parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = compute_type self.cast = P.Cast() def construct(self, state, embed): r"""forward pass of the head""" state = P.Reshape()(state, (-1, self.hidden_size)) # output logits over vocabulary [bs*seq_length, vocab_size] logits = self.matmul(self.cast(state, self.dtype), self.cast(embed, self.dtype)) return logits def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # Used for the pipeline's stages setting # As the final layer is not included here, so we need to manually add here. # original: if set two stages, layers on two stages will be [15, 16+1] # with 1 added, the layers on two stages will be [16, 15 +1] pp_dis = max(int((layers + 1)/ parallel_config.pipeline_stage), 1) # the pipeline stage must be in [0, parallel_config.pipeline_stage - 1] pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id logger.info("pipeline stage id is %s", pp_id) # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: # we give the fusion in pipeline mode a fixed value, otherwise the performance may become worse. network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class PanguAlphaModel(nn.Cell): r"""The base backbone of the PanGuAlpha model""" def __init__(self, config): super(PanguAlphaModel, self).__init__() self.is_pipeline = config.parallel_config.pipeline_stage > 1 self.embedding = EmbeddingLayer(config) self.config = config self.layernorm = LayerNorm((config.hidden_size,)).to_float(mstype.float32) if config.parallel_config.pipeline_stage > 1: self.layernorm.set_comm_fusion(2) else: self.layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.layernorm.pipeline_stage = config.parallel_config.pipeline_stage - 1 # Configure the shard configure of the Embedding layer self.embedding.pipeline_stage = 0 self.num_layers = config.num_layers if config.use_moe: moe_config = MoEConfig(expert_num=config.expert_num, num_experts_chosen=config.per_token_num_experts_chosen) else: moe_config = MoEConfig(expert_num=1) # The shard setting of Transformer is set within the class StackedTransformer self.blocks = TransformerEncoder(num_layers=config.num_layers - 1, batch_size=config.batch_size, hidden_size=config.hidden_size, ffn_hidden_size=config.ffn_hidden_size, num_heads=config.num_heads, seq_length=config.seq_length, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, lambda_func=set_parallel_configure_for_layer, hidden_act=config.hidden_act, param_init_type=config.param_init_type, use_past=config.use_past, parallel_config=config.parallel_config, moe_config=moe_config, softmax_compute_type=config.softmax_compute_type).blocks copied_parallel_config = copy.deepcopy(config.parallel_config) copied_parallel_config.vocab_emb_dp = True self.top_query_embedding = VocabEmbedding(vocab_size=config.seq_length, embedding_size=config.hidden_size, param_init=initializer("normal", [config.seq_length, config.hidden_size], dtype=mstype.float32), parallel_config=copied_parallel_config.embedding_dp_mp_config) self.top_query_embedding.pipeline_stage = config.parallel_config.pipeline_stage - 1 if config.parallel_config.pipeline_stage > 1: self.top_query_embedding.set_comm_fusion(2) else: self.top_query_embedding.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.top_query_layer = QueryLayer(batch_size=config.batch_size, hidden_size=config.hidden_size, ffn_hidden_size=config.ffn_hidden_size, num_heads=config.num_heads, seq_length=config.seq_length, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, hidden_act=config.hidden_act, param_init_type=config.param_init_type, use_past=config.use_past, softmax_compute_type=config.softmax_compute_type, parallel_config=config.parallel_config) if isinstance(config.parallel_config.recompute, bool): if config.parallel_config.recompute: self.top_query_layer.recompute() else: if config.parallel_config.recompute.recompute: self.top_query_layer.recompute(recompute_slice_activation= config.parallel_config.recompute.recompute_slice_activation) self.top_query_layer.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.top_query_layer.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.dtype = mstype.float16 self.use_past = config.use_past self.add = P.TensorAdd() def construct(self, input_ids, input_position, encoder_masks, init_reset=True, batch_valid_length=None): r"""forward pass of the model""" embed, word_table = self.embedding(input_ids, input_position, batch_valid_length) hidden_state = P.Cast()(embed, self.dtype) hidden_shape = F.shape(hidden_state) hidden_state = F.reshape(hidden_state, (-1, hidden_shape[-1])) # the input of the incremental prediction is 3d if self.blocks is not None: for i in range(self.num_layers - 1): hidden_state, _ = self.blocks[i](hidden_state, encoder_masks, init_reset, batch_valid_length) if self.use_past and not self.is_first_iteration: input_position = self.add(input_position, F.expand_dims(batch_valid_length, -1)) if self.is_pipeline: top_query_hidden_states, _ = self.top_query_embedding(input_position) top_query_hidden_states = self.reshape_to_2d(top_query_hidden_states) encoder_output, _ = self.top_query_layer(hidden_state, top_query_hidden_states, encoder_masks, init_reset, batch_valid_length) encoder_output = self.layernorm(encoder_output) else: hidden_state = self.reshape_to_2d(hidden_state) encoder_output = self.layernorm(hidden_state) encoder_output = P.Cast()(encoder_output, self.dtype) top_query_hidden_states, _ = self.top_query_embedding(input_position) top_query_hidden_states = self.reshape_to_2d(top_query_hidden_states) encoder_output, _ = self.top_query_layer(encoder_output, top_query_hidden_states, encoder_masks, init_reset, batch_valid_length) return encoder_output, word_table def reshape_to_2d(self, x): r"""reshape nd tensor to 2d, if n <= 2, keep original shape.""" shape = F.shape(x) if len(shape) <= 2: return x x = F.reshape(x, (-1, shape[-1])) return x @MindFormerRegister.register(MindFormerModuleType.MODELS) class PanguAlphaHeadModel(BaseModel): """ The PanguAlpha network consisting of two parts the backbone and the head Args: config(PanguAlphaConfig): the config of network Inputs: input_ids: the tokenized inputs input_position(Tensor): current position, used by model.predict. attention_mask(Tensor): input sentences padding mask, where 0 indicates padding position. position_ids(Tensor): used to identify each token's position in the list of tokens. input_embeds(Tensor): reserved param, not used. labels(Tensor): the labels of corresponding input sequences. init_reset(bool, optional): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Default True. batch_valid_length(Tensor): the past calculated the index with datatype int32, used for incremental prediction. Tensor of shape :math:`(batch_size,)`. Default None. Returns: logits: Tensor: the logits of the corresponding inputs with shape (batch_size, seq_length, vocab_size) Examples: >>> # input model name, load model and weights >>> model_a = PanguAlphaHeadModel.from_pretrained('pangualpha_2_6b') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('pangualpha_2_6b') >>> model_b = PanguAlphaHeadModel(config) """ _support_list = MindFormerBook.get_model_support_list()['pangualpha'] def __init__(self, config: PanguAlphaConfig = None): config = config if config is not None else PanguAlphaConfig() super(PanguAlphaHeadModel, self).__init__(config) logger.info(config.parallel_config) # Network head to get logits over vocabulary copied_parallel_config = copy.deepcopy(config.parallel_config) logger.info(copied_parallel_config) if copied_parallel_config.pipeline_stage > 1: copied_parallel_config.vocab_emb_dp = False dp = config.parallel_config.data_parallel self.pad_token_id = config.pad_token_id self.get_attention_mask = AttentionMask(seq_length=config.seq_length, parallel_config=config.parallel_config.dp_mp_config) self.head = PanguAlphaHead(hidden_size=config.hidden_size, parallel_config=copied_parallel_config) self.head.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.backbone = PanguAlphaModel(config) self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) loss_parallel_config = copy.deepcopy(config.parallel_config) self.loss = CrossEntropyLoss(parallel_config=loss_parallel_config) self.reshape = P.Reshape() self.cast = P.Cast() self.tile = P.Tile().shard(((dp,),)) self.slice = P.StridedSlice().shard(((dp, 1),)) self.slice2 = P.StridedSlice().shard(((dp, 1, 1),)) self.not_equal = P.NotEqual().shard(((dp, 1), ())) self.input_position = Tensor(np.arange(config.seq_length), mstype.int32) self.expand = P.ExpandDims() self.add = P.Add().shard(((dp, 1), ())) self.gather = P.Gather() self.use_past = config.use_past self.is_first_iteration = True self.position_ids = Tensor(np.arange(config.seq_length), mstype.int32) self.all_ones_attention_mask = P.Ones()((1, 1, 1), mstype.float32) self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): input_position = kwargs.get("current_index", None) if input_position is not None: input_position = Tensor(input_position, mstype.int32) return { "input_ids": Tensor(input_ids, mstype.int32), "input_position": input_position } # pylint: disable=W0613 def construct(self, input_ids, input_position=None, attention_mask=None, position_ids=None, input_embeds=None, labels=None, init_reset=True, batch_valid_length=None): r"""forward pass of the model""" batch_size, seq_length = input_ids.shape if self.training: seq_length = seq_length - 1 tokens = self.slice(input_ids, (0, 0), (batch_size, seq_length), (1, 1)) input_mask = F.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) input_position = self.slice(input_position, (0, 0), (batch_size, seq_length), (1, 1)) attention_mask = self.cast(attention_mask, mstype.float32) attention_mask = self.slice2(attention_mask, (0, 0, 0), (batch_size, seq_length, seq_length), (1, 1, 1)) else: tokens = input_ids input_mask = F.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) if attention_mask is None: attention_mask = self.get_attention_mask(input_mask) if not self.is_first_iteration: attention_mask = self.tile(self.all_ones_attention_mask, (batch_size, 1, 1)) if input_position is None or self.is_first_iteration: if batch_size == 1: input_position = F.reshape(self.position_ids, (1, seq_length)) else: input_position = self.tile(self.position_ids, (batch_size, 1)) # when incremental reasoning is not the first iteration, it goes into the following logic. else: bias = Tensor(np.arange(batch_size) * self.config.seq_length, mstype.int32) input_position = F.sub(input_position, bias) input_position = F.reshape(input_position, (batch_size, 1)) # [batch_size, seq_length, vocab_size] output_states, word_table = self.backbone(tokens, input_position, attention_mask, init_reset, batch_valid_length) logits = self.head(output_states, word_table) if not self.training: logits = logits.reshape((batch_size, seq_length, -1)) logits = logits.reshape((-1, logits.shape[-1])) if (not self.use_past or self.is_first_iteration) and input_position is not None: logits = self.gather(logits, input_position, 0) return logits, tokens, input_mask labels = self.slice(input_ids, (0, 1), (batch_size, seq_length + 1), (1, 1)) labels = self.reshape(labels, (-1,)) input_mask = self.reshape(input_mask, (-1,)) loss = self.loss(logits, labels, input_mask) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class PanguAlphaPromptTextClassificationModel(PanguAlphaHeadModel): """ The PanguAlpha network for prompt text classification consisting of two parts the backbone and the head Args: config(PanguAlphaConfig): the config of network Inputs: input_ids: the tokenized inputs, shape is [num_labels, seq_length] labels: the index of the true label, shape is [1,] attention_mask: the mask indicating whether each position is a valid input and is not the added prompt, shape is [num_labels, seq_length] Returns: logits: Tensor: corresponding outputs for calculating metrics Examples: >>> # input model name, load model and weights >>> model_a = PanguAlphaHeadModel.from_pretrained('pangualpha_2_6b') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('pangualpha_2_6b') >>> model_b = PanguAlphaPromptTextClassificationModel(config) """ def __init__(self, config: PanguAlphaConfig = None): config = config if config is not None else PanguAlphaConfig() super().__init__(config) self.num_labels = config.num_labels # pylint: disable=arguments-differ def construct(self, input_ids=None, labels=None, attention_mask=None, position_ids=None, input_embeds=None, input_position=None, init_reset=True, batch_valid_length=None): r"""forward pass of the model""" if self.training: raise ValueError("PanguAlphaPromptTextClassificationModel just supports evaluate mode, " "please set run_mode to eval") if input_ids is None and input_embeds is None: raise ValueError("input_ids and input_embeds can not be all None") batch_size, num_labels_mul_seq_length = input_ids.shape seq_length = num_labels_mul_seq_length // self.num_labels input_ids = self.reshape(input_ids, (-1, seq_length)) input_mask = self.cast(self.reshape(attention_mask, (-1, seq_length)), mstype.float32) attention_mask = self.cast(self.not_equal(input_ids, self.pad_token_id), mstype.float32) attention_mask = self.get_attention_mask(attention_mask) if position_ids is None: position_ids = F.tuple_to_array(F.make_range(seq_length)) position_ids = self.expand(position_ids, 0) position_ids = self.tile(position_ids, (self.num_labels * batch_size, 1)) logits, vocab_table = self.backbone(input_ids, position_ids, attention_mask, init_reset, batch_valid_length) logits = self.head(logits, vocab_table) logits = self.reshape(logits, (batch_size, self.num_labels, seq_length, -1)) return logits, input_ids, input_mask, labels pengcheng.mind-test/mindformers/models/pangualpha/pangualpha_config.py000066400000000000000000000104241471400134000266610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """PanGuAlpha Config API.""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config from mindformers.modules.transformer.transformer import default_moe_config from mindformers.modules.transformer.transformer import TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['PanguAlphaConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class PanguAlphaConfig(BaseConfig): """ PanGuAlpha config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['pangualpha'] def __init__(self, batch_size: int = 1, seq_length: int = 1024, vocab_size: int = 40000, hidden_size: int = 2560, ffn_hidden_size: int = 2560 * 4, num_layers: int = 32, num_heads: int = 32, pad_token_id: int = 6, eos_token_id: int = 8, post_layernorm_residual: bool = False, param_init_type: str = 'float32', compute_dtype: str = 'float16', softmax_compute_type: str = 'float16', embedding_dropout_prob: float = 0.1, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, hidden_act: str = 'fast_gelu', use_past: bool = False, parallel_config: TransformerOpParallelConfig = default_transformer_config, moe_config: MoEConfig = default_moe_config, use_moe: bool = False, expert_num: int = 1, per_token_num_experts_chosen: int = 1, checkpoint_name_or_path: str = '', repetition_penalty: float = 1.0, max_decode_length: int = 1024, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, **kwargs): super(PanguAlphaConfig, self).__init__(**kwargs) self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.ffn_hidden_size = ffn_hidden_size self.num_layers = num_layers self.num_heads = num_heads self.pad_token_id = pad_token_id self.eos_token_id = eos_token_id # Use post-layernorm or pre-layernrom, default:pre-layernorm self.post_layernorm_residual = post_layernorm_residual self.param_init_type = convert_mstype(param_init_type) self.compute_dtype = convert_mstype(compute_dtype) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.embedding_dropout_prob = embedding_dropout_prob self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.hidden_act = hidden_act self.use_past = use_past self.parallel_config = parallel_config self.moe_config = moe_config self.use_moe = bool(use_moe) self.expert_num = expert_num self.per_token_num_experts_chosen = per_token_num_experts_chosen self.checkpoint_name_or_path = checkpoint_name_or_path self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample pengcheng.mind-test/mindformers/models/pangualpha/pangualpha_processor.py000066400000000000000000000045321471400134000274360ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ PanguAlphaProcessor """ from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import BaseTokenizer from mindformers.models.base_processor import BaseProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['PanguAlphaProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class PanguAlphaProcessor(BaseProcessor): """ PanguAlpha processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['pangualpha'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(PanguAlphaProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-test/mindformers/models/pangualpha/pangualpha_tokenizer.py000066400000000000000000000175451471400134000274410ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """PanguAlpha Tokenzier""" import os from typing import List, Optional import jieba import sentencepiece as spm from mindformers.tools import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_tokenizer import Tokenizer from ...mindformer_book import MindFormerBook __all__ = ['PanguAlphaTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "vocab.model"} @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class PanguAlphaTokenizer(Tokenizer): r""" Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Args: vocab_file(str): The vocabulary file path. eos_token(str): The token that represents the end-of-sentence. Default "". bos_token(str): The token that represents the begin-of-sentence. Default """. unk_token(str): The token that represents the unknown. Default "". pad_token(str): The token that represents the pad. Default "". add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "False" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import PanguAlphaTokenizer >>> tokenizer = PanguAlphaTokenizer.from_pretrained("pangualpha_2_6b") >>> res = tokenizer("你好，今天天气不错。") >>> print(res) {'input_ids': [5772, 10, 465, 235, 464, 1123, 12], \ 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0], \ 'attention_mask': [1, 1, 1, 1, 1, 1, 1]} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['pangualpha'] def __init__(self, vocab_file, eos_token='', bos_token='', unk_token='', pad_token='', add_bos_token=False, add_eos_token=False, **kwargs): super(PanguAlphaTokenizer, self).__init__( eos_token=eos_token, bos_token=bos_token, unk_token=unk_token, pad_token=pad_token, **kwargs ) self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token self.encoder = {} self.sp = spm.SentencePieceProcessor(model_file=vocab_file) for i in range(self.sp.get_piece_size()): self.encoder[self.sp.id_to_piece(i)] = i self.translator = str.maketrans(" \n", "\u2582\u2583") def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): if not isinstance(text, str): raise ValueError("Text should be type str, but found type", type(text)) return self._tokenize(text) def _tokenize(self, text, **kwargs): """ Tokenize a string using bpe encode. """ seg_list = [x.translate(self.translator) for x in jieba.cut(text, cut_all=False)] return seg_list def convert_tokens_to_ids(self, tokens): """ the index of the tokens in the vocabulary. """ if isinstance(tokens, str): return self._convert_token_to_id(tokens) if isinstance(tokens, list): tokens = " ".join(tokens) return self.sp.encode(tokens) return None def _convert_token_to_id(self, token): """ the index of the tokens in the vocabulary. """ return self.sp.piece_to_id(token) def _convert_id_to_token(self, index): """ return the origin bpe tokens according to ids """ text = self.sp.decode(index) text = text.replace(' ', '').replace('\u2582', ' ').replace('\u2583', '\n') return text def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" text = "".join(tokens) return text def process_tokens(self, text): r"""replace special tokens with space and \n""" text = text.replace(' ', '').replace('\u2582', ' ').replace('\u2583', '\n') return text def pangu_encode(self, text): """pangu encode""" res = self.tokenize(text) return res def pangu_decode(self, tokens): """pangu decode""" text = self.sp.decode(tokens) return self.process_tokens(text) def save_vocabulary(self, save_directory, filename_prefix=None): """write the word to the files""" if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None output_file_path = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]) with open(output_file_path, 'w', encoding="utf8") as fp: for k in self.encoder: fp.write(k + '\n') return output_file_path @property def vocab_size(self): return len(self.encoder) def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): r"""Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab pengcheng.mind-test/mindformers/models/sam/000077500000000000000000000000001471400134000213015ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/sam/__init__.py000066400000000000000000000022341471400134000234130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """models init""" from .sam import Sam from .sam_image_encoder import SAMImageEncoder from .sam_prompt_encoder import SAMPromptEncoder from .sam_mask_decoder import SAMMaskDecoder from .sam_config import SAMConfig, ImageEncoderConfig from .sam_processor import SAMProcessor, SAMImageProcessor from .sam_utils import * __all__ = ["Sam", "SAMImageEncoder", "SAMPromptEncoder", "SAMMaskDecoder", "SAMConfig", "ImageEncoderConfig", 'SAMProcessor', "SAMImageProcessor"] __all__.extend(sam_utils.__all__) pengcheng.mind-test/mindformers/models/sam/__pycache__/000077500000000000000000000000001471400134000235115ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/sam/__pycache__/__init__.cpython-39.pyc000066400000000000000000000012171471400134000277020ustar00rootroot00000000000000a ?cf@stdZddlmZddlmZddlmZddlmZddl m Z m Z ddl m Z mZddlTgd Zeejd S) z models init)Sam)SAMImageEncoder)SAMPromptEncoder)SAMMaskDecoder) SAMConfigImageEncoderConfig) SAMProcessorSAMImageProcessor)*)rrrrrrrr N)__doc__samrZsam_image_encoderrZsam_prompt_encoderrZsam_mask_decoderrZ sam_configrrZ sam_processorrr Z sam_utils__all__extendrrJ/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/__init__.pys    pengcheng.mind-test/mindformers/models/sam/__pycache__/sam.cpython-39.pyc000066400000000000000000000105361471400134000267270ustar00rootroot00000000000000a ?cfO@sxdZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZe e jGdd d eZdS) z SAM Model)TupleN) build_model)MindFormerRegisterMindFormerModuleType)MindFormerBook) BaseModelc sxeZdZUdZdZeed<dZeed<e dZ ddfd d Z dd dZ ejejeedfejdddZZS)Samz SAM predicts object masks from an image and input prompts. If prompts are not known in advance, using Sam is recommended over calling the model directly. gmask_thresholdRGB image_formatsamN)returncs>t|t|j|_t|j|_t|j|_||dS)N) super__init__r image_encoderZ prompt_configprompt_encoderZdecoder_config mask_decoderload_checkpoint)selfconfig __class__E/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam.pyr$s     z Sam.__init__TFc Cs|dus|dusJd|dur0|dus0Jd|durB||}|j||||d\} } |j||j| | | d\} }|j| ||d}| s||jk}|| |fS)a Args: 'image': The image as a mindspore tensor in 3xHxW format, already transformed for input to the model. 'features': The features of an image, extracted by image encoder. 'input_size': (tuple(int, int)) The input size of the image after transformation, as (H, W). 'original_size': (tuple(int, int)) The original size of the image before transformation, as (H, W). 'point_coords': (ms.Tensor) Batched point prompts for this image, with shape BxNx2. Already transformed to the input frame of the model. 'point_labels': (ms.Tensor) Batched labels for point prompts, with shape BxN. 'boxes': (ms.Tensor) Batched box inputs, with shape Bx4. Already transformed to the input frame of the model. 'mask_inputs': (ms.Tensor) Batched mask inputs to the model, in the form Bx1xHxW. multimask_output (bool): Whether the model should predict multiple disambiguating masks, or return a single mask. return_logits (bool): Whether the mask return logits. Returns: 'masks': (ms.Tensor) Batched binary mask predictions, with shape BxCxHxW, where B is the number of input prompts, C is determined by multimask_output, and (H, W) is the original size of the image. 'iou_predictions': (ms.Tensor) The model's predictions of mask quality, in shape BxC. 'low_res_logits': (ms.Tensor) Low resolution logits with shape BxCxHxW, where H=W=256. Can be passed as mask input to subsequent iterations of prediction. Nz%image and feature can't be both None.z:point labels can't be None, when point coords is not None.) point_coords point_labelsboxes mask_inputs)Zimage_embeddingsZimage_peZsparse_prompt_embeddingsZdense_prompt_embeddingsmultimask_output) input_size original_size)rrrZ get_dense_pepostprocess_masksr )rimagefeaturesrr rrrrr return_logitsZsparse_embeddingsZdense_embeddings low_res_masksiou_predictionsmasksrrr construct,s6,     z Sam.construct.)r'rr r cCsRtj||jj|jjfddd}|dd|dd|df}tj||ddd}|S)zpostprocess masksbilinearT)sizemode align_corners.Nr)ops interpolaterimg_size)rr'rr rrrr!vszSam.postprocess_masks) NNNNNNNNTF)__name__ __module__ __qualname____doc__r float__annotations__r strrget_model_support_list _support_listrr(msTensorrintr! __classcell__rrrrrs(      Kr)r4typingr mindsporer: mindspore.opsr.Zmindformers.models.build_modelrmindformers.tools.registerrrmindformers.mindformer_bookrmindformers.models.base_modelrregisterMODELSrrrrrs      pengcheng.mind-test/mindformers/models/sam/__pycache__/sam_config.cpython-39.pyc000066400000000000000000000117511471400134000302540ustar00rootroot00000000000000a ?cfN@sdZddlmZmZddlmmZddlm Z m Z ddl m Z m Z ddlmZddlmZddlmZmZe Ze ed ZeejGd d d eZeejGd d d eZeejGdddeZeejGdddeZdS)zSAM Config API.)UnionTupleN)TransformerOpParallelConfigTransformerRecomputeConfig) MoEConfigdefault_moe_config)MindFormerBook) BaseConfig)MindFormerRegisterMindFormerModuleType) recomputecs6eZdZdZedZdeddfdd ZZ S) SAMConfigz0 Configuration class for the SAM model. samN)checkpoint_name_or_pathreturnc s.tjfi|||_||_||_||_dS)N)super__init__ image_encoder prompt_configdecoder_configr)selfrrrrkwargs __class__L/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_config.pyr%s zSAMConfig.__init__)r) __name__ __module__ __qualname____doc__rget_config_support_list _support_liststrr __classcell__rrrrr s r cseZdZdZedZdddddddd d d d d d d d ddddeefe e e e e e e e e e e e e e e e e e e e eddfdd ZZS)ImageEncoderConfigzE Configuration class for the image encoder of the SAM model. ri T) rfloat16float32N)img_size patch_sizein_chans embed_dimdepth num_heads mlp_ratio out_chansqkv_bias use_abs_pos use_rel_posrel_pos_zero_init window_sizeglobal_attn_indexesr compute_dtypelayernorm_compute_typesoftmax_compute_typeparam_init_typeparallel_config moe_configrc stjfi|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_||_|dkrttjntj|_|dkrtjntj|_|dkrtjntj|_|dkrtjntj|_||_||_||_dSNr2)rrr3r4r5r6r7r8r9r:r;r<r=r>r?r@mstyper2r1rArBrCrDrErFr)rr3r4r5r6r7r8r9r:r;r<r=r>r?r@rrArBrCrDrErFrrrrr9s,zImageEncoderConfig.__init__)rrrr rr!r"default_parallel_configrintbooltupler#rrrr$rrrrr%2s^ r%c sleZdZdZedZddddddeefe e e e e fe e e e fe e e e ed d fd d ZZS) PromptEncoderConfigz@ Configuration for the prompt encoder of the SAM model. rr+)@rN)r&r&r'rr1N) prompt_embed_dimimage_embedding_sizeinput_image_size mask_in_chansrrArErFrc  sVtjfi| ||_||_||_||_|dkr8tjntj|_ ||_ ||_ ||_ dSrG) rrrOrPrQrRrHr2r1rArErFr) rrOrPrQrRrrArErFrrrrrqs zPromptEncoderConfig.__init__)rrrr rr!r"rIrrJrrr#rrrr$rrrrrMjs* rMcspeZdZdZedZdddddddddd d d d eefe e e e e e e e e e e e e e e d d fd d Z ZS)MaskDecoderConfigz> Configuration for the mask decoder of the SAM model. rr(r-r+ir/rr1r2N)num_multimask_outputs decoder_depthdecoder_embed_dimdecoder_mlp_dimdecoder_num_headstransformer_dimiou_head_depthiou_head_hidden_dimrrArBrCrDrErFrc stjfi|||_||_||_||_||_||_||_||_ | dkrPt j nt j |_ | dkrft j nt j |_| dkr|t j nt j |_| dkrt j nt j |_||_||_| |_dSrG)rrrTrUrVrWrXrYrZr[rHr2r1rArBrCrDrErFr)rrTrUrVrWrXrYrZr[rrArBrCrDrErFrrrrrs zMaskDecoderConfig.__init__)rrrr rr!r"rIrrJr#rrrr$rrrrrSsF rS)r typingrrZmindspore.common.dtypecommondtyperHmindformers.modules.transformerrr#mindformers.modules.transformer.moerrmindformers.mindformer_bookrZmindformers.models.base_configr mindformers.tools.registerr r default_recompute_configrIregisterCONFIGr r%rMrSrrrrs"     7  pengcheng.mind-test/mindformers/models/sam/__pycache__/sam_image_encoder.cpython-39.pyc000066400000000000000000000277241471400134000315770ustar00rootroot00000000000000a ?cf3D @sdZddlmZmZddlZddlmZddlm Z ddl m Z ddl mmZddlmZddlmZddlmZmZddlmZddlmZmZd d lmZmZeej Gd d d eZ!Gd ddej"Z#Gdddej"Z$dej%e&ee&e&feej%ee&e&ffdddZ'ej%e&ee&e&fee&e&fej%dddZ(e&e&ej%ej%dddZ)ej%ej%ej%ej%ee&e&fee&e&fej%dddZ*Gdddej"Z+dS) zSAM Image Encoder Model)OptionalTupleN) Parameter) operations)MindFormerRegisterMindFormerModuleType) BaseModel)Linear LayerNorm) LayerNorm2dMLPBlockcs:eZdZdZddfdd ZejejdddZZS) SAMImageEncoderz Image encoder N)returnc s`t||j|_|j|_|j|_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |_ |j |_ |j |_ |j|_|j|_|j|_|j|_|j|_|j|_t|jtr|j}|j}n |j\}}||j}||j}||f|_|jdkr"|j||j|j}|j||j|j}||f|_nd|_t|j|jf|j|jf|j|jd|_d|_|j rttd|j|j|j|j|jftj|_t !|_"t#|jD]t}t$|j|j|j|j |j ||jvr|jnd|j|j|j|j|j|jf|j |j|j|j|jd} |j"%| qt &t j'|j|j dddt(|j t j'|j |j dd ddd t(|j |_)dS) Nrrr) kernel_sizestridein_chans embed_dimr )dim num_heads mlp_ratioqkv_bias use_rel_pos window_sizepad_size feat_size input_sizelayer_norm_eps compute_dtypelayernorm_compute_typesoftmax_compute_typeparam_init_typeF)rhas_biaspad)rpad_modepaddingr#)*super__init__img_size patch_sizerrdepthrrZ out_chansrrZ use_abs_posrrZglobal_attn_indexesrr r!r" isinstanceintrr PatchEmbed patch_embed pos_embedrPZerosmstypefloat32nnCellListblocksrangeBlockappendSequentialCellConv2dr neck) selfconfigimg_himg_wZfeat_hZfeat_wpad_hpad_wiblock __class__S/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_image_encoder.pyr)&s           zSAMImageEncoder.__init__xrcCsL||}|jdur||j}|jD] }||}q$||dddd}|S)z Args: x (ms.Tensor): Input image tensor. Returns: ms.Tensor: Encoded image tensor. Nrr$r )r0r1r8r> transpose)r?rLblkrIrIrJ constructs     zSAMImageEncoder.construct) __name__ __module__ __qualname____doc__r)msTensorrP __classcell__rIrIrGrJr!sYrc seZdZdZdddddddd ejejejejf eeee e ee eefe eefe e eefedd fd d Z e je jd ddZZS)r:z] Transformer blocks with support of window attention and residual propagation blocks g@TFrr)@rXNg-q=) rrrrrrrrrrrc stt|| d|_t|||||dkr.| n||f| | | |d |_t|f| d|_t|t||| |d|_ ||_ ||_ ||_ dS)N)epsr)rrrrrr r!r")Z embedding_dimZmlp_dimrr") r(r)r norm1 Attentionattnnorm2r r.mlprrr)r?rrrrrrrrrrrr r!r"rGrIrJr)s,   zBlock.__init__rKcCsp|}||}|jdkr(t||j|j}||}|jdkrPt||j|j|j}||}||||}|S)x Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. r) rZrwindow_partitionrr\window_unpartitionrr^r])r?rLZshortcutrIrIrJrPs    zBlock.construct)rQrRrSrTr4float16r5r.floatboolrrr)rUrVrPrWrIrIrGrJr:s6  'r:c sjeZdZdZddddejejejejfeeeee e eefddfdd Z e j e j d d d ZZS) r[zG Multi-head Attention block with relative position embeddings. TFN)rrrrrrc st||_||_||_||_||} | d|_t||d||| d|_t|||| d|_ ||_ |j r|duszJdt t d|dd| f|j|_t t d|dd| f|j|_t jd d |_t |_t jd d |_t |_t jdd |_dS) Ngr$) in_channels out_channelsr#rr")rfrgrr"zBInput size must be provided if using relative positional encoding.rMrr )axisT) transpose_b)r(r)rr r!rscaler qkvprojrrr2r3 rel_pos_h rel_pos_wSoftmaxsoftmax BatchMatMul batchmatmulbatchmatmul_trans_bCastcastUnstackunstack) r?rrrrrrr r!r"head_dimrGrIrJr)s>  &&  zAttention.__init__rKc Cs,|j\}}}}|j}|||||d|jdddddd}|||j}||d||j||d\}} } | ||j | } |j rt | ||j |j||f||f} || |j} || } || |j} || | }|||j||d}|ddddd}||||d}||}|||}|S)r_r$rhrMrr )shapedtyperlreshaperpermutervrrxrtrkradd_decomposed_rel_posrnror!rqrsviewrm) r?rLbhw_Zori_typerlqkvr\rIrIrJrPs$,&    zAttention.construct)rQrRrSrTr4rbr5r.rdrrr)rUrVrPrWrIrIrGrJr[s$,r[)rLrrrc Cs|j\}}}}|\}}|dks&|dkrJtjdd|fd|fdfd} | |}||||} } ||| ||| |||}|ddddddd |||} | S) a Partition the input tensor into non-overlapping windows with optional padding. Args: x (ms.Tensor): Input tensor with shape [B, H, W, C]. window_size (int): Window size. pad_size (tuple[int, int]): Padding size as (pad_h, pad_w). Returns: windows (ms.Tensor): Windows after partition with shape [B * num_windows, window_size, window_size, C]. rr)paddingsr r$rMrzrh)r{r2PadrrN) rLrrrrrcrCrDr%hpwpwindowsrIrIrJr`(s r`)rrpad_hwhwrc Cs|\}}|\}}||||}} |jd|| ||} || ||| |||d} | dddddd| || d} ||ks| |kr| ddd|d|ddf} | S) a Unpartition windows back into original sequences and remove padding if needed. Args: windows (ms.Tensor): Input windows with shape [B * num_windows, window_size, window_size, C]. window_size (int): Window size. pad_hw (Tuple[int, int]): Padded height and width (Hp, Wp). hw (Tuple[int, int]): Original height and width (H, W) before padding. Returns: x (ms.Tensor): Unpartitioned sequences with shape [B, H, W, C]. rrhr r$rMrzrN)r{rrN) rrrrrCrDrrrrrrLrIrIrJraBs  ra)q_sizek_sizerel_posrcCstdt||d}|jd|kr\tj|d|jddddd|dd}|d|}n|}t|dddft||d}t|dddft||d}|||dt||d}|| t j S) a_ Get relative positional embeddings based on the relative positions of query and key sizes. Args: q_size (int): Size of query q. k_size (int): Size of key k. rel_pos (ms.Tensor): Relative position embeddings (L, C). Returns: ms.Tensor: Extracted positional embeddings according to relative positions. rMr rrhlinear)sizemodeNg?) r.maxr{ops interpolater}rNnparangeastyper4int32)rrrZ max_rel_distZrel_pos_resizedZq_coordsZk_coordsZrelative_coordsrIrIrJ get_rel_pos^s$$r)r\rrnrorrrcCs|\}}|\}} t|||} t|| |} |j\} } }|| |||}t|| ddd| ||| jd}t|dddd| ddd| ||| jd}|dddd}|| |||| |dddddddddf|dddddddddf| |||| }|S)a  Calculate decomposed Relative Positional Embeddings from mvitv2 paper. https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py # noqa B950 Args: attn (ms.Tensor): Attention map. q (ms.Tensor): Query q in the attention layer with shape (B, q_h * q_w, C). rel_pos_h (ms.Tensor): Relative position embeddings (Lh, C) for the height axis. rel_pos_w (ms.Tensor): Relative position embeddings (Lw, C) for the width axis. q_size (Tuple[int, int]): Spatial sequence size of query q with (q_h, q_w). k_size (Tuple[int, int]): Spatial sequence size of key k with (k_h, k_w). Returns: ms.Tensor: Attention map with added relative positional embeddings. rrMr r$N)rr{r}rmatmulrNr)r\rrnrorrZq_hZq_wk_hk_wrhrwrrrZr_qZrel_hZrel_wrIrIrJrs"   (4  rcsbeZdZdZd eeefeeefeeeeefeeddfdd Zejejd d d Z Z S)r/z# Image to Patch Embedding. rrrrrr$N)rrr'rrrcs&ttj|||||dd|_dS)a Initialize the PatchEmbed layer. Args: kernel_size (Tuple[int, int]): Kernel size of the projection layer. stride (Tuple[int, int]): Stride of the projection layer. padding (Tuple[int, int, int, int]): Padding size of the projection layer. in_chans (int): Number of input image channels. embed_dim (int): Patch embedding dimension. T)rrr'r#N)r(r)r6r=rm)r?rrr'rrrGrIrJr)s zPatchEmbed.__init__rKcCs||}|dddd}|S)z Forward pass of the PatchEmbed layer. Args: x (ms.Tensor): Input image tensor. Returns: ms.Tensor: Patch embeddings tensor. rrMr$r )rmrN)r?rLrIrIrJrPs zPatchEmbed.construct)rrrr$r) rQrRrSrTrr.r)rUrVrPrWrIrIrGrJr/s  r/)r),rTtypingrr mindsporerU mindspore.nnr6 mindspore.opsrZmindspore.numpynumpyrZmindspore.common.dtypecommonr|r4rrr2mindformers.tools.registerrrmindformers.models.base_modelrmindformers.modules.layersr r Z sam_layersr r registerMODELSrCellr:r[rVr.r`rarrr/rIrIrIrJsL       qCT    #  'pengcheng.mind-test/mindformers/models/sam/__pycache__/sam_layers.cpython-39.pyc000066400000000000000000000046561471400134000303140ustar00rootroot00000000000000a ?cf @sdZddlmZddlZddlmZddlmm Z ddlm Z ddl m ZddlmZGdddejZGd d d ejZdS) z SAM Layers)TypeN) Parameter) operations)LinearcsTeZdZdZejejejfe e e ej ddfdd Z e je jdddZZS) MLPBlockz Multi-Layer Perceptron (MLP) block. Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. N) embedding_dimmlp_dimactreturncs:tt||||d|_t||||d|_||_dS)N) in_channels out_channels compute_dtypeparam_init_type)super__init__rlin1lin2r )selfrrr r r __class__L/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_layers.pyr$s zMLPBlock.__init__xr cCs||||SN)rr r)rrrrr construct5szMLPBlock.construct)__name__ __module__ __qualname____doc__nnGELUmstypefloat16float32intrCellrmsTensorr __classcell__rrrrrs rcs@eZdZdZd eeddfdd Zejejddd Z Z S) LayerNorm2dz Layer Normalization for 2D data. Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Normalized tensor. ư>N) num_channelsepsr csTttt|tj|_tt|tj|_ ||_ t |_ t |_dSr)rrrPOnesr'r$weightZerosbiasr-PowpowSqrtsqrt)rr,r-rrrrDs   zLayerNorm2d.__init__rcCsp|jddd}|||djddd}|||||j}|jddddf||jddddf}|S)NT) keep_dims)meanr4r6r-r0r2)rrusrrrrLs ,zLayerNorm2d.construct)r+) rrrrr%floatrr'r(rr)rrrrr*:s r*)rtypingr mindsporer' mindspore.nnr mindspore.common.dtypecommondtyper"r mindspore.opsrr.mindformers.modules.layersrr&rr*rrrrs      pengcheng.mind-test/mindformers/models/sam/__pycache__/sam_mask_decoder.cpython-39.pyc000066400000000000000000000335771471400134000314410ustar00rootroot00000000000000a ?cfX@sdZddlZddlmZmZddlZddlmZddl m Z ddl m m Zddl mZddlmZmZddlmZddlmZmZdd lmZmZeejGd d d eZGd d d ejZ GdddejZ!GdddejZ"GdddejZ#dS)zSAM Mask Decoder ModelN)TupleType) operations)MindFormerRegisterMindFormerModuleType) BaseModel)Linear LayerNorm)MLPBlock LayerNorm2dcs|eZdZdZddfdd ZejejejejeeejejfdddZ ejejejejeejejfd d d Z Z S) SAMMaskDecoderzf A class to predict masks given image and prompt embeddings using a transformer architecture. N)returnc st||j_|j_|j_|j_|j_|j_|j_|j _ |j _ t j _ |j_|j_|j_|j_tjjjjj jjjjd _t dj_jd_t jj_t t jjjdddddtjd t jjdjddddd _t fdd tjD_tjj jjjjd _ t!j"d d _#t!j"dd _$t!%_&t!j'dd _(dS) N) depth embedding_dimmlp_dim num_headslayer_norm_eps compute_dtypelayernorm_compute_typesoftmax_compute_typeparam_init_typer T) kernel_sizestridehas_biasc s.g|]&}tjjjddjjdqS)rrr)MLPtransformer_dimrr).0iselfR/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_mask_decoder.py Xsz+SAMMaskDecoder.__init__..rraxis))super__init__r!Z decoder_depthZdecoder_embed_dimZdecoder_mlp_dimZdecoder_num_headsZnum_multimask_outputsZiou_head_depthZiou_head_hidden_dimrnnGELU activationrrrrTwoWayTransformer transformer Embedding iou_tokennum_mask_tokens mask_tokensSequentialCellConv2dTransposer output_upscalingCellListrangeoutput_hypernetworks_mlpsr iou_prediction_headPConcatconcat_0concat_1 ExpandDims expand_dimsStackstack)r%config __class__r$r'r,&sv       zSAMMaskDecoder.__init__)image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsmultimask_outputrc Csd|j||||d\}}|r&tdd}n tdd}|dd|ddddf}|dd|f}||fS)a8 Args: image_embeddings (ms.Tensor): Embeddings from the image encoder. image_pe (ms.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (ms.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (ms.Tensor): Embeddings of the mask inputs. multimask_output (bool): Whether to return multiple masks or a single mask. Returns: Tuple[ms.Tensor, ms.Tensor]: Batched predicted masks and batched predictions of mask quality. )rHrIrJrKr Nr) predict_masksslice) r%rHrIrJrKrLmasksiou_predZ mask_slicer&r&r' constructis   zSAMMaskDecoder.construct)rHrIrJrKrc Cs|jd}||jj|jjg}t||d|ddf}t|rT| ||g}n|}| |jdd}||}| |jdd} |j\} } } } | || |\}}|dddddf}|dddd|j ddf}| ddd| | | | }||}g}t|j D],}||j||dd|ddfq ||}|j\} } } } t||| | | | | d| | }||}||fS)a Args: image_embeddings (ms.Tensor): Embeddings from the image encoder. image_pe (ms.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (ms.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (ms.Tensor): Embeddings of the mask inputs. Returns: Tuple[ms.Tensor, ms.Tensor]: Batched predicted masks and batched predictions of mask quality. rNr r)shaper?r3embedding_tabler5ops broadcast_torBanyr@repeat_interleaver1r4 transposeviewr8r:appendr;rDmatmulr<)r%rHrIrJrKbs output_tokenstokenssrcZpos_srcbchwhsZ iou_token_outZmask_tokens_outZupscaled_embeddingZ hyper_in_listr#Zhyper_inrOrPr&r&r'rMs.    * & zSAMMaskDecoder.predict_masks) __name__ __module__ __qualname____doc__r,msTensorboolrrQrM __classcell__r&r&rFr'r sD $r c szeZdZdZejddejejejejfe e e e e ej e e ddfdd Z ejejejeejejfdd d ZZS) r0z Two-way transformer decoder that attends to an input image using queries whose positional embedding is supplied. r-q=N)rrrrr/attention_downsample_raterrc st||_||_||_||_||_| |_| |_| |_ t |_ t |D]6} |j t|||||| dk||j|j|j|j d qLt||||j|j|j|j d|_t|f|d|_dS)Nr) rrrr/roskip_first_layer_perrrrrdownsample_raterrrreps)r+r,rZ embeding_dimrZmpl_dimrrrrr-r9layersr:r[TwoWayAttentionBlock Attentionfinal_attn_token_to_imager norm_final_attn) r%rrrrr/rorrrrrr#rFr&r'r,sH    zTwoWayTransformer.__init__)image_embeddingrIpoint_embeddingrcCs|j\}}}}|||||ddd}|||||ddd}|}|} |jD]} | || ||d\}} qT||} | |} |j| | | d} || }||}|| fS)au Apply the two-way transformer. Args: image_embedding (ms.Tensor): Image to attend to. Should have shape B x embedding_dim x h x w for any h and w. image_pe (ms.Tensor): The positional encoding to add to the image. Must have the same shape as image_embedding. point_embedding (ms.Tensor): The embedding to add to the query points. Must have shape B x N_points x embedding_dim for any N_points. Returns: ms.Tensor: The processed point_embedding. ms.Tensor: The processed image_embedding. rrr )querieskeysquery_pekey_peqkv)rSreshapepermuterurxry)r%rzrIr{r]rbrcrdr|r}layerrrattn_outr&r&r'rQs$   zTwoWayTransformer.construct)rfrgrhrir-ReLUmstypefloat16float32intrCellfloatr,rjrkrrQrmr&r&rFr'r0s, 5r0c seZdZdZdejdddejejejejf e e e e ej e e e ddfdd Zejejejejeejejfd d d ZZS) rvz^ Two-way attention block consisting of self-attention and cross-attention layers. irFrnN)rrrr/rorprrc stt|||| | | d|_t|f|d|_t||||| | | d|_t|f|d|_t||||| d|_ t|f|d|_ t|f|d|_ t||||| | | d|_ ||_ dS)N)rrrrrsrqr)r+r,rw self_attnr norm1cross_attn_token_to_imagenorm2r mlpnorm3norm4cross_attn_image_to_tokenrp) r%rrrr/rorprrrrrrFr&r'r,+sF zTwoWayAttentionBlock.__init__)r|r}r~rrc Cs|jr|j|||d}n ||}|j|||d}||}||}||}||}|j|||d}||}||}||}||}||}||}||}|j|||d}||}||}||fS)a Apply the two-way attention block. Args: queries (ms.Tensor): Queries tensor. keys (ms.Tensor): Keys tensor. query_pe (ms.Tensor): Query positional encoding. key_pe (ms.Tensor): Key positional encoding. Returns: ms.Tensor: Processed queries tensor. ms.Tensor: Processed keys tensor. r) rprrrrrrrr) r%r|r}r~rrrrZmlp_outr&r&r'rQ[s(     zTwoWayAttentionBlock.construct)rfrgrhrir-rrrrrrrrlrr,rjrkrrQrmr&r&rFr'rv's21rvcseZdZdZdejejejejfeeeddfdd Ze j ee j ddd Z e j e j d d d Z e j e j e j e j d ddZ ZS)rwzu Attention layer that supports downscaling the embedding size after projection to queries, keys, and values. r N)rrrrrcst||_|||_||_||_||_||_||_|j|dksNJdt dt |j|j|_ t||j||d|_t||j||d|_t||j||d|_t|j|||d|_tjdd|_tjdd|_t|_t|_d S) a Initialize the Attention layer. Args: embedding_dim (int): The channel dimension of the embeddings. num_heads (int): The number of heads in the attention layer. downsample_rate (int): Downsample rate for the attention. rz$num_heads must divide embedding_dim.g? in_channels out_channelsrrrRr)T) transpose_bN)r+r,rZ internal_dimrrrrrrjrkmathsqrtinv_norm_factorrq_projk_projv_projout_projr=Softmaxsoftmax BatchMatMulbatchmatmul_trans_b batchmatmulMulmul)r%rrrrrrrrrFr&r'r,sD   zAttention.__init__)xrrcCs0|j\}}}||||||}|ddddS)a Separate the input tensor into multiple heads. Args: x (ms.Tensor): Input tensor. num_heads (int): Number of heads to separate into. Returns: ms.Tensor: Tensor with dimensions reshaped for separate heads. rrr r)rSrrY)r%rrranrbr&r&r'_separate_headss zAttention._separate_heads)rrcCs0|j\}}}}|dddd}|||||S)z Recombine the tensor with multiple heads into a single tensor. Args: x (ms.Tensor): Input tensor with separate heads. Returns: ms.Tensor: Recombined tensor. rrr r)rSrYr)r%rran_headsZn_tokensZ c_per_headr&r&r'_recombine_headss zAttention._recombine_heads)rrrrcCs|j}|||j}|||j}|||j}|||j}|||j}|||j}|||}| ||j }| ||j }| ||j|}||}|||}|S)a Apply the attention mechanism. Args: q (ms.Tensor): Queries tensor. k (ms.Tensor): Keys tensor. v (ms.Tensor): Values tensor. Returns: ms.Tensor: Output tensor after attention. )dtyperastyperrrrrrrrrrrrr)r%rrrori_typeattnoutr&r&r'rQs  zAttention.construct)rfrgrhrirrrrr,rjrkrrrQrmr&r&rFr'rws$3rwcsDeZdZdZdejejfeeeeeddfdd Z ddZ Z S) r z< Multi-Layer Perceptron (MLP) class definition. FN) input_dim hidden_dim output_dim num_layerssigmoid_outputrc sjt||_|g|d}tfddt|g|||gD|_||_t |_ t |_ dS)Nr cs g|]\}}t||dqS)r)r)r"rrrr&r'r(s z MLP.__init__..) r+r,rr-r9ziprurr=rreluSigmoidsigmoid) r%rrrrrrrrcrFrr'r,s   z MLP.__init__cCsLt|jD],\}}||jdkr.|||n||}q |jrH||}|S)z} Args: x (Tensor): Input tensor. Returns: Tensor: Output tensor of the MLP. r ) enumeraterurrrr)r%rr#rr&r&r'rQ%s & z MLP.construct) rfrgrhrirrrrrlr,rQrmr&r&rFr'r sr )$rirtypingrr mindsporerj mindspore.nnr- mindspore.opsrUZmindspore.common.dtypecommonrrrr=mindformers.tools.registerrrmindformers.models.base_modelrmindformers.modules.layersrr sam_layersr r registerMODELSr rr0rvrwr r&r&r&r's$     he~pengcheng.mind-test/mindformers/models/sam/__pycache__/sam_processor.cpython-39.pyc000066400000000000000000000200761471400134000310260ustar00rootroot00000000000000a ?cfl!@sdZddlmZddlmZddlZddlmZddl Z ddl m Z ddl mZddlmZdd lmZdd lmZmZdd lmZmZeejGd d d eZGdddZeejGdddeZdS)z SAM Processor)deepcopy)TupleN)Image)vision)Inter) operations)MindFormerBook) BaseProcessorBaseImageProcessor)MindFormerRegisterMindFormerModuleTypecs*eZdZdZd fdd ZddZZS) SAMImageProcessora" Subclass of BaseImageProcessor implementing image preprocessing for SAM model. Args: img_size (int): The target size for image processing. mean (list): Mean values for image normalization. std (list): Standard deviation values for image normalization. g33333^@gR]@gRY@g(\2M@g(\L@gL@csJt||_t|ddd|_t|ddd|_t||_ dS)N) super__init__img_sizemsTensorview pixel_mean pixel_stdResizeLongestSide transform)selfrmeanstd __class__O/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_processor.pyr*s  zSAMImageProcessor.__init__c Ks|j|}|ddddddddddf}t|}|jdd}||j|j}|jdd\}}|j|}|j|}|j dkrt j ddd|fd|ffd}n$|j d krt j dd|fd|ffd}||}||fS) a3 Preprocess the input image for SAM model. Args: image (PIL.Image or numpy.array): Input image to be preprocessed. Returns: image (tensor): Preprocessed image as a tensor. input_size (tuple): Size of the input image after preprocessing. rrN)rr)paddings) r apply_image transposerrshaperrrndimPPad) rimageskwargs input_sizehwpadhpadwpadr"r"r# preprocess1s &     zSAMImageProcessor.preprocess)rrr)__name__ __module__ __qualname____doc__rr7 __classcell__r"r"r r#r s r c@seZdZdZeddddZejejdddZeje ed fejd d d Z eje ed fejd ddZ ejejdddZ eje ed fejd ddZ eje ed fejd ddZeeeee eefdddZdS)rz Resizes images to the longest side 'target_length', as well as provides methods for resizing coordinates and boxes. Provides methods for transforming both numpy array and batched mindspore tensors. N) target_lengthreturncCs ||_dS)z Initialize the ResizeLongestSide class. Args: target_length (int): Target length of the longest side. N)r=)rr=r"r"r#rXszResizeLongestSide.__init__)imager>cCsD||jd|jd|j}t|}tj|d}t||}|S)z Apply the resizing transformation to a single image. Args: image (np.ndarray): Input image as HxWxC numpy array. Returns: np.ndarray: Resized image as H'xW'xC numpy array. rr)size) get_preprocess_shaper+r=r fromarrayrResizenparray)rr? target_size pil_imageZ resize_opZ resized_imager"r"r#r)as   zResizeLongestSide.apply_image.)coords original_sizer>cCs^|\}}||d|d|j\}}t|t}|d|||d<|d|||d<|S)aS Apply the resizing transformation to coordinates. Args: coords (np.ndarray): Input coordinates as a numpy array of shape (N, 2). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized coordinates as a numpy array of shape (N, 2). rr.r.r)rAr=rastypefloatrrHrIZold_hZold_wnew_hnew_wr"r"r# apply_coordsqs zResizeLongestSide.apply_coords)boxesrIr>cCs"||ddd|}|ddS)aI Apply the resizing transformation to bounding boxes. Args: boxes (np.ndarray): Input boxes as a numpy array of shape (B, 4). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized boxes as a numpy array of shape (B, 4). rr$r&)rQreshaperrRrIr"r"r# apply_boxess zResizeLongestSide.apply_boxescCs6||jd|jd|j}t|tj}||}|S)a Apply the resizing transformation to a batch of images. Args: image (np.ndarray): Input batched images as BxCxHxW numpy array. Returns: np.ndarray: Resized batched images as BxC'xH'xW' numpy array. r$r()rAr+r=rrCrBILINEAR)rr?rFZresize_r"r"r#apply_image_batchs z#ResizeLongestSide.apply_image_batchcCsX|\}}||d|d|j\}}t|}|d|||d<|d|||d<|S)ap Apply the resizing transformation to a batch of coordinates. Args: coords (np.ndarray): Input batched coordinates as numpy array of shape (B, N, 2). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized batched coordinates as numpy array of shape (B, N, 2). rrrJrK)rAr=rrNr"r"r#apply_coords_batchs z$ResizeLongestSide.apply_coords_batchcCs"||ddd|}|ddS)af Apply the resizing transformation to a batch of bounding boxes. Args: boxes (np.ndarray): Input batched boxes as numpy array of shape (B, N, 4). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized batched boxes as numpy array of shape (B, N, 4). rr$r&)rXrSrTr"r"r#apply_boxes_batchs z#ResizeLongestSide.apply_boxes_batch)oldholdwlong_side_lengthr>cCsD|dt||}||||}}t|d}t|d}||fS)aX Compute the output size given input size and target long side length. Args: oldh (int): Original image height. oldw (int): Original image width. long_side_length (int): Target length of the longest side. Returns: Tuple[int, int]: Output size as (new_h, new_w). g?g?)maxint)rZr[r\scaleZnewhZnewwr"r"r#rAs   z&ResizeLongestSide.get_preprocess_shape)r8r9r:r;r^rrDndarrayr)rrQrUrWrXrY staticmethodrAr"r"r"r#rQs rcs.eZdZdZedZdfdd ZZS) SAMProcessorz Vit processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. samNrcstj||ddS)N)image_processorreturn_tensors)rr)rrdrer r"r#rszSAMProcessor.__init__)Nr) r8r9r:r;rget_processor_support_list _support_listrr<r"r"r r#rbs rb)r;copyrtypingrnumpyrDPILr mindsporermindspore.datasetrZmindspore.dataset.visionr mindspore.opsrr-mindformers.mindformer_bookrZ!mindformers.models.base_processorr r mindformers.tools.registerr r register PROCESSORr rrbr"r"r"r#s"        1 pengcheng.mind-test/mindformers/models/sam/__pycache__/sam_prompt_encoder.cpython-39.pyc000066400000000000000000000207751471400134000320350ustar00rootroot00000000000000a ?cfO)@sdZddlmZmZddlZddlmZddlm Z ddl m Z ddl mmZddlmZddlmZddlmZmZddlmZddlmZd d lmZeejGd d d eZGd ddej Z!dS)zSAM Prompt Encoder)OptionalTupleN) Parameter) operations)MindFormerRegisterMindFormerModuleType)MindFormerBook) BaseModel) LayerNorm2dcseZdZdZedZddfdd Zej dddZ ej ej e ej d d d Z ej ej d d dZ ej ej dddZeej eej eej edddZeej eej eej eej eej ej fdddZZS)SAMPromptEncoderz: Encodes prompts for input to SAM's mask decoder. samN)returncsVt||j_|j_|j_|j_tj_ t jd_ d_ fddt j D}t|_tdj_djddjdf_ttjdjdddddtjd tjjdjddddtj tjjjddd _tdj_t_t_t_tjdd _ t!_"dS) Ncsg|]}tdjqS)r )nn Embedding embed_dim).0iselfT/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_prompt_encoder.py 2z-SAMPromptEncoder.__init__..r rT) kernel_sizestridehas_bias)rraxis)#super__init__Zprompt_embed_dimrinput_image_sizeimage_embedding_sizeZ mask_in_chansrGELU activationPositionEmbeddingRandompe_layerZnum_point_embeddingsrangeCellListpoint_embeddingsrnot_a_point_embedZmask_input_sizeSequentialCellConv2dr mask_downscaling no_mask_embedPZeroszerosOnesones ExpandDims expand_dimsConcatconcatTiletile)rconfigr+ __class__rrr"'s6       zSAMPromptEncoder.__init__cCs|||jdS)a Get the positional encoding used to encode point prompts, applied to a dense set of points the shape of the image encoding. Returns: ms.Tensor: Positional encoding with shape 1x(embed_dim)x(embedding_h)x(embedding_w) r)r7r(r$rrrr get_dense_peHs zSAMPromptEncoder.get_dense_pe)pointslabelspadrc Cs|jd}|d}|j||j}|rp|||jjd|ddf}|||g}||| |dft j  g}|dk |dd}|t ||jdjt |jdj7}|dk |dd}|t ||jdjt |jdj7}|S)zEmbeds point prompts.r?r )shaper(forward_with_coordsr#r;r7r,embedding_tabler9r5mstypeint32reshapenpwherer+ zeros_like) rr@rArBbsZpoint_embeddingr,Zmask_negZmask_posrrr _embed_pointsSs  zSAMPromptEncoder._embed_points)boxesrcCsv|d}|ddd}|j||j}|dddddf|jdj7<|dddddf|jdj7<|S)zEmbeds box prompts.rCrDrNrr )rJr(rFr#r+rG)rrPcoordsZcorner_embeddingrrr _embed_boxesfs &&zSAMPromptEncoder._embed_boxes)masksrcCs||}|S)zEmbeds mask inputs.)r/)rrTZmask_embeddingrrr _embed_masksos zSAMPromptEncoder._embed_masks)r@rPrTrcCs:|dur|jdS|dur$|jdS|dur6|jdSdS)z^ Gets the batch size of the output given the batch size of the input prompts. Nrr )rE)rr@rPrTrrr_get_batch_sizets   z SAMPromptEncoder._get_batch_size) point_coords point_labelsrP mask_inputsrc Cs||||}t|d|jf}|dur@|j|||dud}|}|durV||}|}|durj||} n0t|j j dddd|d|j d|j df} || fS)aD Args: point_coords (ms.Tensor or none): Point coordinates to embed. point_labels (ms.Tensor or none): Point labels to embed. boxes (ms.Tensor or none): Boxes to embed. mask_inputs (ms.Tensor or none): Masks to embed. Returns: Tuple[ms.Tensor, ms.Tensor]: Sparse embeddings for the points and boxes, with shape BxNx(embed_dim), where N is determined by the number of input points and boxes. Dense embeddings for the masks, in the shape Bx(embed_dim)x(embed_H)x(embed_W). r N)rBrDr) rVrKemptyrrOrSrUops broadcast_tor0rGrJr$) rrWrXrPrYrNsparse_embeddingsr+Zbox_embeddingsdense_embeddingsrrr constructs  zSAMPromptEncoder.construct)__name__ __module__ __qualname____doc__rget_model_support_list _support_listr"msTensorr?boolrOrSrUrintrVrr_ __classcell__rrr=rr s  !   r cseZdZdZddejfeeeddfdd Z e j e j ddd Z e eefe j d d d Ze j e eefe j d ddZZS)r'z? Positional encoding using random spatial frequencies. @N) num_pos_featsscalercst||_|dus |dkr$d}|td|f|j}t|dd|_t|_ t |_ t |_ t|_tjdd|_tjdd|_dS) z Initialize the PositionEmbeddingRandom. Args: num_pos_feats (int): Number of positional encoding features. scale (float): Scale factor for the positional encoding Gaussian matrix. Ngg?r#positional_encoding_gaussian_matrix)namerDr)r!r" compute_dtyper1StandardNormalastyperrnCastcastr4r5SinsinCoscosr8r9Stackstack)rrlrmrpZpe_gaussian_matrixr=rrr"s     z PositionEmbeddingRandom.__init__)rRrcCs\d|d}|j}t||j|j}|||}dtj|}| | || |gS)z Positionally encode points that are normalized to [0,1]. Args: coords (ms.Tensor): Coordinates normalized to [0, 1]. Returns: ms.Tensor: Positional encoding for the given coordinates. rr ) dtyper[matmulrrrprnrtrKpir9rvrx)rrRori_typerrr _pe_encodings  z$PositionEmbeddingRandom._pe_encoding)sizercCsl|\}}|||ftj}|jddd}|jddd}||}||}||||g}|dddS)z Generate positional encoding for a grid of the specified size. Args: size (Tuple[int, int]): Grid size (H, W). Returns: ms.Tensor: Positional encoding for the grid. rrrCr r)r5rffloat32cumsumrrz transpose)rrhwgridZy_embedZx_embedperrrr_s z!PositionEmbeddingRandom.construct) coords_input image_sizercCsn|}|dddddf|d|dddddf<|dddddf|d|dddddf<||S)a> Positionally encode points that are not normalized to [0,1]. Args: coords_input (ms.Tensor): Coordinates not normalized to [0, 1]. image_size (Tuple[int, int]): Image size (H, W). Returns: ms.Tensor: Positional encoding for the given coordinates. Nrr )r)rrrrRrrrrFs 00z+PositionEmbeddingRandom.forward_with_coords)r`rarbrcrHfloat16rirfloatr"rfrgrrr_rFrjrrr=rr'sr')"rctypingrr mindsporerf mindspore.nnr mindspore.opsr[Zmindspore.numpynumpyrKZmindspore.common.dtypecommonr{rHrrr1mindformers.tools.registerrrmindformers.mindformer_bookrmindformers.models.base_modelr sam_layersr registerMODELSr Cellr'rrrrs          pengcheng.mind-test/mindformers/models/sam/__pycache__/sam_utils.cpython-39.pyc000066400000000000000000000505211471400134000301450ustar00rootroot00000000000000a ?cfX@sTdZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z ddl Z ddlZgdZGdddZdAeje ee eeejd d d Zejejd ddZee e eddfdddZeje eeefdddZeeefejdddZeeefedddZejeeejdddZeejddd Zeeee ejd!d"d#Ze ed$feee e e ee efd%d&d'Zeje eejd(d)d*Z eje eejd+d,d-Z!eje eeeejd.d/d0Z"ejeee eje#fd1d2d3Z$eeefeeefd4d5d6Z%d7d8Z&d9d:Z'ejejd;dd?d@Z)dS)Bz SAM UtilsNdeepcopy)product)AnyDict Generator ItemsViewListTuple)MaskData area_from_rlebatch_iteratorbatched_mask_to_boxbox_xyxy_to_xywhbuild_all_layer_point_gridscalculate_stability_scorecoco_encode_rlegenerate_crop_boxesis_box_near_crop_edge mask_to_rleremove_small_regions rle_to_maskuncrop_boxes_xyxy uncrop_masks uncrop_pointsbox_areanmsc@seZdZdZddddZeeddddZedd d d Zeed d d Z e eefdddZ e j ddddZdddddZdS)r z A structure for storing masks and their related data in batched format. Implements basic filtering and concatenation. N)returncKs:|D]}t|ttjfsJdqtfi||_dS)z Initialize the MaskData. Args: **kwargs: Keyword arguments representing different mask-related data. z:MaskData only supports list, numpy arrays, and ms tensors.N)values isinstancelistnpndarraydict_stats)selfkwargsvr(K/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_utils.py__init__2s  zMaskData.__init__)keyitemrcCs&t|ttjfsJd||j|<dS)z Set an item in the MaskData. Args: key (str): The key of the item. item (Any): The item to be set. z*MaskData only supports list, numpy arrays.N)rr r!r"r$)r%r+r,r(r(r) __setitem__?s  zMaskData.__setitem__)r+rcCs |j|=dS)z| Delete an item from the MaskData. Args: key (str): The key of the item to be deleted. Nr$r%r+r(r(r) __delitem__LszMaskData.__delitem__cCs |j|S)z Get an item from the MaskData. Args: key (str): The key of the item to be retrieved. Returns: Any: The retrieved item. r.r/r(r(r) __getitem__Us zMaskData.__getitem__cCs |jS)z Get the items stored in the MaskData. Returns: ItemsView[str, Any]: A view of the items in the MaskData. )r$items)r%r(r(r)r2aszMaskData.items)keeprcs|jD]\}dur&d|j|<q ttjrB|j|<q ttrvjtjkrvfddtD|j|<q ttrfddD|j|<q t d|dt dq dS)z Filter the MaskData based on a boolean mask. Args: keep (np.ndarray): Boolean mask indicating which items to keep. Ncsg|]\}}|r|qSr(r().0ia)r3r(r) wz#MaskData.filter..csg|] }|qSr(r(r4r5)r'r(r)r7yr8 MaskData key  has an unsupported type .) r$r2rr!r"r dtypebool enumerate TypeErrortype)r%r3kr()r3r'r)filterjs   zMaskData.filter) new_statsrcCs|D]\}}||jvs(|j|dur8t||j|<qt|tjrdtj|j||gdd|j|<qt|tr|j|t||j|<qtd|dt |dqdS)z Concatenate new data from another MaskData instance. Args: new_stats (MaskData): The MaskData instance to concatenate. Nraxisr:r;r<) r2r$rrr!r" concatenater r@rA)r%rDrBr'r(r(r)cat}s   z MaskData.cat)__name__ __module__ __qualname____doc__r*strrr-r0r1rr2r!r"rCrHr(r(r(r)r ,s     r 4@)boxescrop_boxorig_boxatolrcCstj|tjd}tj|tjd}t||tj}tj|t|dddf|jddf|ddd}tj|t|dddf|jddf|ddd}t||}tj |ddS) a  Filter boxes at the edge of a crop, but not at the edge of the original image. Args: boxes (np.ndarray): Bounding boxes ndarray of shape (N, 4) in format (x1, y1, x2, y2). crop_box (List[int]): Crop box coordinates in format (x1, y1, x2, y2). orig_box (List[int]): Original image box coordinates in format (x1, y1, x2, y2). atol (float): Absolute tolerance for box comparison. Returns: np.ndarray: Boolean ndarray indicating whether boxes are near the crop edge. r=NrgT)rRrtol equal_nanrE) r!arrayfloat32rastypeisclosetileshape logical_andany)rOrPrQrRZnear_crop_edgeZnear_image_edger(r(r)rs"  r)box_xyxyrcCs4t|}|d|d|d<|d|d|d<|S)a6 Convert bounding box coordinates from (x1, y1, x2, y2) format to (x, y, w, h) format. Args: box_xyxy (np.ndarray): Bounding boxes tensor of shape (N, 4) in (x1, y1, x2, y2) format. Returns: box_xywh (np.ndarray): Bounding boxes tensor of shape (N, 4) in (x, y, w, h) format. rrTr)r_Zbox_xywhr(r(r)rs r) batch_sizerc'srrtfddDs"Jdtdttddk}t|D]fddDVqRdS)a Generate batches of data from input arrays. Args: batch_size (int): Size of each batch. *args: Variable number of input arrays to iterate over. Yields: List[Any]: A batch of data containing corresponding elements from each input array. c3s"|]}t|tdkVqdS)rN)len)r4r6)argsr(r) sz!batch_iterator..z8Batched iteration must have inputs of all the same size.rcs$g|]}|dqSrTr()r4arg)brbr(r)r7r8z"batch_iterator..N)allrcintrange)rbrd n_batchesr()rdrhrbr)r s  ( r )tensorrc Cs4|j\}}}|ddd|jdd}t|ddddftj|ddddftj}tt|}g}t |D]}||dddf|kdf}t tj dg|j d|dtj ||g|j dg}|dd|dd} ||dfdkrgndg} | | |||g| dq|S)a+ Encode masks to an uncompressed RLE format, in the format expected by pycoco tools. Args: tensor (np.ndarray): Binary mask tensor of shape (B, H, W) where B is the batch size. Returns: List[Dict[str, Any]]: List of dictionaries representing RLE encoded masks. rr`rTNrS)sizecounts)r\ transposereshaper! bitwise_xorrYint32stacknonzerorkrGrWr=extendtolistappend) rmrhhwdiffZchange_indicesoutr5cur_idxsZbtw_idxsrpr(r(r)rs$ < r)rlercCsh|d\}}tj||td}d}d}|dD]$}|||||<||7}|dN}q.|||}|S)z Compute a binary mask from an uncompressed RLE. Args: rle (Dict[str, Any]): Dictionary containing RLE encoded mask information. Returns: np.ndarray: Binary mask as a NumPy array. rorSrFrpT)r!emptyr>rrrq)rrzr{maskidxparitycountr(r(r)rs    rcCst|ddddS)z Calculate the area of a binary mask from an uncompressed RLE. Args: rle (Dict[str, Any]): Dictionary containing RLE encoded mask information. Returns: int: Calculated area of the mask. rprTNr`)sum)rr(r(r)r s r )masksmask_thresholdthreshold_offsetrcCsP|||kjdtjdjdtjd}|||kjdtjdjdtjd}||S)a Calculate the stability score for a batch of masks. The stability score is the Intersection over Union (IoU) between the binary masks obtained by thresholding the predicted mask logits at high and low values. Args: masks (np.ndarray): Predicted mask logits tensor of shape (B, H, W). mask_threshold (float): Threshold value for binary mask creation. threshold_offset (float): Offset to adjust the threshold. Returns: np.ndarray: Stability scores for each mask in the batch. rnrS)rr!rX)rrr intersectionsunionsr(r(r)rs  r) n_per_sidercCstdd|}t|d||}t|dddf|df}t|dddfd|f}tj||gdddd}|S)z Generate a 2D grid of points evenly spaced in the [0, 1] x [0, 1] range. Args: n_per_side (int): Number of points per side of the grid. Returns: np.ndarray: 2D grid of points as a NumPy array. rTr`NrnrE)r!linspacer[rurr)roffsetZpoints_one_sideZpoints_xZpoints_ypointsr(r(r)build_point_grid<s r)rn_layersscale_per_layerrcCs8g}t|dD]"}t|||}|t|q|S)aj Generate point grids for all crop layers. Args: n_per_side (int): Number of points per side of the original grid. n_layers (int): Number of layers in the crop hierarchy. scale_per_layer (int): Scaling factor applied per layer. Returns: List[np.ndarray]: List of point grids for each crop layer as NumPy arrays. rT)rkrjryr)rrrZpoints_by_layerr5n_pointsr(r(r)rNs r.)im_sizer overlap_ratiorcs gg}}|\}}t||}|dd||g|ddd}t|D]} d| d} t||d| ||| ||| fddt| D} fddt| D} t| | D]@\} }| |t| |t||g}|||| dqqH||fS) a Generate a list of crop boxes of different sizes. Each layer has (2**i)**2 boxes for the ith layer. Args: im_size (Tuple[int, ...]): Size of the original image in (height, width). n_layers (int): Number of layers in the crop hierarchy. overlap_ratio (float): Overlap ratio for calculating crop box size. Returns: Tuple[List[List[int]], List[int]]: Tuple containing a list of crop boxes and a list of layer indices for each crop box. rcSstt||d||S)NrT)rjmathceil)Zorig_lenZn_cropsoverlapr(r(r)crop_len{sz%generate_crop_boxes..crop_lenr`rTcsg|]}t|qSr(rjr9)crop_wrr(r)r7r8z'generate_crop_boxes..csg|]}t|qSr(rr9)crop_hrr(r)r7r8)minryrkrjr)rrr crop_boxes layer_idxsZim_hZim_wZ short_siderZi_layerZn_crops_per_sideZ crop_box_x0Z crop_box_y0x0y0boxr()rrrr)rcs$         r)rOrPrcCsH|\}}}}tj||||ggtjd}t|jdkr@t|d}||S)ar Uncrop the coordinates of bounding boxes from a cropped region to the original image. Args: boxes (np.ndarray): Bounding box coordinates in XYXY format (x0, y0, x1, y1). crop_box (List[int]): Cropped region's bounding box in XYXY format (x0, y0, x1, y1). Returns: np.ndarray: Uncropped bounding box coordinates in XYXY format. rSrarTr!rWrtrcr\ expand_dims)rOrPrr_rr(r(r)rs  r)rrPrcCsD|\}}}}tj||ggtjd}t|jdkr.Frcsg|]}|vr|qSr(r(r9) fill_labelsr(r)r7r8T) rYr!uint8cv2connectedComponentsWithStatsr?rkrjargmaxisin) rrrZ correct_holesZ working_maskn_labelsregionsstatsrsizesZ small_regionsr()rrr)rs   r)uncompressed_rlercCs<ddlm}|d\}}||||}|dd|d<|S)z Encode uncompressed RLE (Run-Length Encoding) to COCO format RLE. Args: uncompressed_rle (Dict[str, Any]): Uncompressed RLE dictionary. Returns: Dict[str, Any]: Encoded RLE in COCO format. r)rrorpzutf-8)Z pycocotoolsrZ frPyObjectsdecode)rZ mask_utilsrzr{rr(r(r)rs  rcCst|jdkr4tjg|jdddRtjdS|j}|dd\}}t|dkrr|d|jd|jd}n t|d}tj| tj dd}|t |dddf}tj|dd}||| tj }tj |dd}tj| tj dd}|t |dddf} tj| dd} | || tj } tj | dd} | | k tj||k tjB tj } tj| || |gdd} | tj| dd} t|dkr| g|dddR} n| d} | tjS) z Convert masks to bounding boxes in XYXY format. Return [0,0,0,0] for an empty mask. Args: masks (np.ndarray): Binary masks of shape C1xC2x...xHxW. Returns: np.ndarray: Bounding boxes in XYXY format of shape C1xC2x...x4. rNrSr`rnrE)r!prodr\zerosrtrcrrrmaxrYrXaranger>rru)rr\rzr{ in_heightZin_height_coordsZ bottom_edgesZ top_edgesin_widthZin_width_coordsZ right_edgesZ left_edgesZ empty_filterr}r(r(r)rs8 $  rcCs@|dddf|dddf|dddf|dddfS)a Compute the area of a set of bounding boxes. Args: boxes (np.ndarray): Bounding boxes for which the area will be computed. Expected shape is [N, 4], where N is the number of boxes. Each box is specified by its coordinates (x1, y1, x2, y2) in the format with ``0 <= x1 < x2`` and ``0 <= y1 < y2``. Returns: np.ndarray: The area for each box in an ndarray of shape [N]. Nr`rrarTr()rOr(r(r)r7s r)box1box2c Cst|}t|}t|ddtjddf|ddddf}t|ddtjddf|ddddf}||}td|}|dddddf|dddddf}||ddtjf||}|S)a Calculate the Intersection over Union (IoU) between two sets of bounding boxes. Args: box1 (np.ndarray): NumPy array representing the first set of bounding boxes in format [x_min, y_min, x_max, y_max]. box2 (np.ndarray): NumPy array representing the second set of bounding boxes in the same format. Returns: iou (np.ndarray): NumPy array containing the IoU values for each pair of boxes. Example: >>> box1 = np.array([[0, 0, 10, 10], [5, 5, 15, 15]]) >>> box2 = np.array([[5, 5, 15, 15], [8, 8, 18, 18]]) >>> iou = box_iou(box1, box2) >>> print(iou) [[0.11111111 0. ] [1. 0.23076923]] Nr`rrT)rr!maximumnewaxisminimum) rrarea1Zarea2ltrbZwhinteriour(r(r)box_iouFs00 ,r)rOscores iou_thresholdc Cs|}g}|jdkrx|d}||dddf}|||jdkrHqx|dd}||}t||}||d|k}q t|}|S)a Apply Non-Maximum Suppression (NMS) algorithm to filter a set of bounding boxes, eliminating overlapping boxes based on the Intersection over Union (IoU) threshold. Args: boxes (np.ndarray): NumPy array containing bounding box coordinates, with each row representing a box [x_min, y_min, x_max, y_max]. scores (np.ndarray): NumPy array containing scores for each corresponding bounding box. iou_threshold (float): IoU (Intersection over Union) threshold to determine if boxes overlap. Returns: keep (np.ndarray): NumPy array containing indices of the retained bounding boxes. Example: >>> boxes = np.array([[0, 0, 10, 10], [5, 5, 15, 15], [8, 8, 18, 18]]) >>> scores = np.array([0.9, 0.75, 0.85]) >>> iou_threshold = 0.5 >>> keep = nms(boxes, scores, iou_threshold) >>> print(keep) [0 2] rrnNrT)argsortroryrr!rW) rOrridxsr3Zmax_score_indexZ max_score_boxZ other_boxesZiousr(r(r)rfs      r)rN)*rLrcopyr itertoolsrtypingrrrrr r rnumpyr!__all__r r"rjfloatrrr rMrrr rrrrrrrr>rrrrrrr(r(r(r)s`   f #%     .  4 pengcheng.mind-test/mindformers/models/sam/conver_weight.py000066400000000000000000000065021471400134000245210ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/facebook""" import argparse import mindspore as ms import torch def convert_weight_sam(pth_file="sam_vit_b_01ec64.pth", ckpt_file=None): """ convert sam weights from pytorch to mindspore. """ ckpt = torch.load(pth_file, map_location=torch.device('cpu')) if 'model_state' in ckpt: state_dict = ckpt['model_state'] elif 'module' in ckpt: state_dict = ckpt['module'] elif 'model' in ckpt: state_dict = ckpt['model'] else: state_dict = ckpt if not ckpt_file: ms_ckpt_path = pth_file.split('.')[0] + '.ckpt' else: ms_ckpt_path = ckpt_file ms_ckpt = [] for k, v in state_dict.items(): new_k = k if new_k.startswith('norm.'): new_k = new_k.replace('norm.', 'fc_norm.') if 'norm' in new_k: if 'weight' in new_k: new_k = new_k.replace('weight', 'gamma') if 'bias' in k: new_k = new_k.replace('bias', 'beta') if 'prompt_encoder' in new_k: if 'embed' in new_k and 'weight' in new_k: new_k = new_k.replace('weight', 'embedding_table') if 'mask_decoder' in new_k: if 'iou_token' in new_k or 'mask_tokens' in new_k: new_k = new_k.replace('weight', 'embedding_table') ms_ckpt.append({'name': new_k, 'data': ms.Tensor(v.numpy())}) ms.save_checkpoint(ms_ckpt, ms_ckpt_path) print("Convert finished!") def show_info(file_path="sam_vit_b_01ec64.pth"): """ show info of pth or ckpt """ if '.ckpt' in file_path: state_dict = ms.load_checkpoint(file_path) elif '.pth' in file_path: state_dict = torch.load(file_path, map_location=torch.device('cpu')) if 'model_state' in state_dict: state_dict = state_dict['model_state'] elif 'module' in state_dict: state_dict = state_dict['module'] elif 'model' in state_dict: state_dict = state_dict['model'] for k, v in state_dict.items(): info = k + ' ' + str(v.shape) + ' ' + str(v.dtype) print(info) if __name__ == '__main__': parser = argparse.ArgumentParser(description="mae vit weight convert script") parser.add_argument("--torch_path", type=str, default="path/to/sam_vit_b_01ec64.pth.pth", help="The torch checkpoint path.") parser.add_argument("--ckpt_path", type=str, default="path/to/sam_vit_b_01ec64.pth.ckpt", help="The torch checkpoint path.") opt = parser.parse_args() convert_weight_sam(opt.torch_path, opt.ckpt_path) pengcheng.mind-test/mindformers/models/sam/sam.py000066400000000000000000000135171471400134000224420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Model""" from typing import Tuple import mindspore as ms import mindspore.ops as ops from mindformers.models.build_model import build_model from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel @MindFormerRegister.register(MindFormerModuleType.MODELS) class Sam(BaseModel): """ SAM predicts object masks from an image and input prompts. If prompts are not known in advance, using Sam is recommended over calling the model directly. """ mask_threshold: float = 0.0 image_format: str = "RGB" _support_list = MindFormerBook.get_model_support_list()['sam'] def __init__(self, config) -> None: super().__init__(config) self.image_encoder = build_model(config.image_encoder) self.prompt_encoder = build_model(config.prompt_config) self.mask_decoder = build_model(config.decoder_config) self.load_checkpoint(config) def construct(self, image=None, features=None, input_size=None, original_size=None, point_coords=None, point_labels=None, boxes=None, mask_inputs=None, multimask_output=True, return_logits=False): """ Args: 'image': The image as a mindspore tensor in 3xHxW format, already transformed for input to the model. 'features': The features of an image, extracted by image encoder. 'input_size': (tuple(int, int)) The input size of the image after transformation, as (H, W). 'original_size': (tuple(int, int)) The original size of the image before transformation, as (H, W). 'point_coords': (ms.Tensor) Batched point prompts for this image, with shape BxNx2. Already transformed to the input frame of the model. 'point_labels': (ms.Tensor) Batched labels for point prompts, with shape BxN. 'boxes': (ms.Tensor) Batched box inputs, with shape Bx4. Already transformed to the input frame of the model. 'mask_inputs': (ms.Tensor) Batched mask inputs to the model, in the form Bx1xHxW. multimask_output (bool): Whether the model should predict multiple disambiguating masks, or return a single mask. return_logits (bool): Whether the mask return logits. Returns: 'masks': (ms.Tensor) Batched binary mask predictions, with shape BxCxHxW, where B is the number of input prompts, C is determined by multimask_output, and (H, W) is the original size of the image. 'iou_predictions': (ms.Tensor) The model's predictions of mask quality, in shape BxC. 'low_res_logits': (ms.Tensor) Low resolution logits with shape BxCxHxW, where H=W=256. Can be passed as mask input to subsequent iterations of prediction. """ assert image is not None or features is not None, \ "image and feature can't be both None." if point_coords is not None: assert point_labels is not None, \ "point labels can't be None, when point coords is not None." if features is None: features = self.image_encoder(image) sparse_embeddings, dense_embeddings = self.prompt_encoder(point_coords=point_coords, point_labels=point_labels, boxes=boxes, mask_inputs=mask_inputs) low_res_masks, iou_predictions = self.mask_decoder(image_embeddings=features, image_pe=self.prompt_encoder.get_dense_pe(), sparse_prompt_embeddings=sparse_embeddings, dense_prompt_embeddings=dense_embeddings, multimask_output=multimask_output) masks = self.postprocess_masks(low_res_masks, input_size=input_size, original_size=original_size) if not return_logits: masks = masks > self.mask_threshold return masks, low_res_masks, iou_predictions def postprocess_masks(self, masks: ms.Tensor, input_size: ms.Tensor, original_size: Tuple[int, ...]) -> ms.Tensor: """postprocess masks""" masks = ops.interpolate( masks, size=(self.image_encoder.img_size, self.image_encoder.img_size), mode="bilinear", align_corners=True ) masks = masks[..., : input_size[0], : input_size[1]] masks = ops.interpolate(masks, size=original_size, mode="bilinear", align_corners=True) return masks pengcheng.mind-test/mindformers/models/sam/sam_config.py000066400000000000000000000171161471400134000237660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Config API.""" from typing import Union, Tuple import mindspore.common.dtype as mstype from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.modules.transformer.moe import MoEConfig, default_moe_config from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_config import BaseConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class SAMConfig(BaseConfig): """ Configuration class for the SAM model. """ _support_list = MindFormerBook.get_config_support_list()['sam'] def __init__(self, image_encoder, prompt_config, decoder_config, checkpoint_name_or_path: str = '', **kwargs) -> None: super().__init__(**kwargs) self.image_encoder = image_encoder self.prompt_config = prompt_config self.decoder_config = decoder_config self.checkpoint_name_or_path = checkpoint_name_or_path @MindFormerRegister.register(MindFormerModuleType.CONFIG) class ImageEncoderConfig(BaseConfig): """ Configuration class for the image encoder of the SAM model. """ _support_list = MindFormerBook.get_config_support_list()['sam'] def __init__(self, img_size: int = 1024, patch_size: int = 16, in_chans: int = 3, embed_dim: int = 768, depth: int = 12, num_heads: int = 12, mlp_ratio: int = 4, out_chans: int = 256, qkv_bias: bool = True, use_abs_pos: bool = True, use_rel_pos: bool = True, # 仅支持默认False rel_pos_zero_init: bool = True, window_size: int = 14, global_attn_indexes: tuple = (2, 5, 8, 11), checkpoint_name_or_path: str = '', compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", param_init_type: str = "float32", parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs) -> None: super().__init__(**kwargs) self.img_size = img_size self.patch_size = patch_size self.in_chans = in_chans self.embed_dim = embed_dim self.depth = depth self.num_heads = num_heads self.mlp_ratio = mlp_ratio self.out_chans = out_chans self.qkv_bias = qkv_bias self.use_abs_pos = use_abs_pos self.use_rel_pos = use_rel_pos # 仅支持默认False self.rel_pos_zero_init = rel_pos_zero_init self.window_size = window_size self.global_attn_indexes = global_attn_indexes self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.layernorm_compute_type = mstype.float32 if layernorm_compute_type == "float32" else mstype.float16 self.softmax_compute_type = mstype.float32 if softmax_compute_type == "float32" else mstype.float16 self.param_init_type = mstype.float32 if param_init_type == "float32" else mstype.float16 self.parallel_config = parallel_config self.moe_config = moe_config self.checkpoint_name_or_path = checkpoint_name_or_path @MindFormerRegister.register(MindFormerModuleType.CONFIG) class PromptEncoderConfig(BaseConfig): """ Configuration for the prompt encoder of the SAM model. """ _support_list = MindFormerBook.get_config_support_list()['sam'] def __init__(self, prompt_embed_dim: int = 256, image_embedding_size: Union[int, Tuple[int]] = (64, 64), input_image_size: Union[int, Tuple[int]] = (1024, 1024), mask_in_chans: int = 16, checkpoint_name_or_path: str = '', compute_dtype: str = "float16", parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs) -> None: super().__init__(**kwargs) self.prompt_embed_dim = prompt_embed_dim self.image_embedding_size = image_embedding_size self.input_image_size = input_image_size self.mask_in_chans = mask_in_chans self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.parallel_config = parallel_config self.moe_config = moe_config self.checkpoint_name_or_path = checkpoint_name_or_path @MindFormerRegister.register(MindFormerModuleType.CONFIG) class MaskDecoderConfig(BaseConfig): """ Configuration for the mask decoder of the SAM model. """ _support_list = MindFormerBook.get_config_support_list()['sam'] def __init__(self, num_multimask_outputs: int = 3, decoder_depth: int = 2, decoder_embed_dim: int = 256, decoder_mlp_dim: int = 2048, decoder_num_heads: int = 8, transformer_dim: int = 256, iou_head_depth: int = 3, iou_head_hidden_dim: int = 256, checkpoint_name_or_path: str = '', compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", param_init_type: str = "float32", parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs) -> None: super().__init__(**kwargs) self.num_multimask_outputs = num_multimask_outputs # transformer: self.decoder_depth = decoder_depth self.decoder_embed_dim = decoder_embed_dim self.decoder_mlp_dim = decoder_mlp_dim self.decoder_num_heads = decoder_num_heads self.transformer_dim = transformer_dim self.iou_head_depth = iou_head_depth self.iou_head_hidden_dim = iou_head_hidden_dim self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.layernorm_compute_type = mstype.float32 if layernorm_compute_type == "float32" else mstype.float16 self.softmax_compute_type = mstype.float32 if softmax_compute_type == "float32" else mstype.float16 self.param_init_type = mstype.float32 if param_init_type == "float32" else mstype.float16 self.parallel_config = parallel_config self.moe_config = moe_config self.checkpoint_name_or_path = checkpoint_name_or_path pengcheng.mind-test/mindformers/models/sam/sam_image_encoder.py000066400000000000000000000420631471400134000253010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Image Encoder Model""" from typing import Optional, Tuple import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops import mindspore.numpy as np import mindspore.common.dtype as mstype from mindspore import Parameter from mindspore.ops import operations as P from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.modules.layers import Linear, LayerNorm from .sam_layers import LayerNorm2d, MLPBlock # This class and its supporting functions below lightly adapted from the ViTDet backbone available at: https://github.com/facebookresearch/detectron2/blob/main/detectron2/modeling/backbone/vit.py # noqa @MindFormerRegister.register(MindFormerModuleType.MODELS) class SAMImageEncoder(BaseModel): """ Image encoder """ def __init__(self, config) -> None: super().__init__(config) self.img_size = config.img_size self.patch_size = config.patch_size self.in_chans = config.in_chans self.embed_dim = config.embed_dim self.depth = config.depth self.num_heads = config.num_heads self.mlp_ratio = config.mlp_ratio self.out_chans = config.out_chans self.qkv_bias = config.qkv_bias self.layer_norm_eps = config.layer_norm_eps self.use_abs_pos = config.use_abs_pos self.use_rel_pos = config.use_rel_pos self.window_size = config.window_size self.global_attn_indexes = config.global_attn_indexes self.compute_dtype = config.compute_dtype self.layernorm_compute_type = config.layernorm_compute_type self.softmax_compute_type = config.softmax_compute_type self.param_init_type = config.param_init_type if isinstance(self.img_size, int): img_h = self.img_size img_w = self.img_size else: img_h, img_w = self.img_size feat_h = img_h // self.patch_size feat_w = img_w // self.patch_size self.feat_size = (feat_h, feat_w) if self.window_size > 0: pad_h = (self.window_size - feat_h % self.window_size) % self.window_size pad_w = (self.window_size - feat_w % self.window_size) % self.window_size self.pad_size = (pad_h, pad_w) else: self.pad_size = (0, 0) self.patch_embed = PatchEmbed( kernel_size=(self.patch_size, self.patch_size), stride=(self.patch_size, self.patch_size), in_chans=self.in_chans, embed_dim=self.embed_dim, ) self.pos_embed: Optional[Parameter] = None if self.use_abs_pos: # Initialize absolute positional embedding with pretrain image size. self.pos_embed = Parameter(P.Zeros()((1, self.img_size // self.patch_size, self.img_size // self.patch_size, self.embed_dim), mstype.float32)) self.blocks = nn.CellList() for i in range(self.depth): block = Block( dim=self.embed_dim, num_heads=self.num_heads, mlp_ratio=self.mlp_ratio, qkv_bias=self.qkv_bias, use_rel_pos=self.use_rel_pos, window_size=self.window_size if i not in self.global_attn_indexes else 0, pad_size=self.pad_size, feat_size=self.feat_size, input_size=(self.img_size // self.patch_size, self.img_size // self.patch_size), layer_norm_eps=self.layer_norm_eps, compute_dtype=self.compute_dtype, layernorm_compute_type=self.layernorm_compute_type, softmax_compute_type=self.softmax_compute_type, param_init_type=self.param_init_type ) self.blocks.append(block) self.neck = nn.SequentialCell( nn.Conv2d( self.embed_dim, self.out_chans, kernel_size=1, has_bias=False ), LayerNorm2d(self.out_chans), nn.Conv2d( self.out_chans, self.out_chans, kernel_size=3, pad_mode='pad', padding=1, has_bias=False, ), LayerNorm2d(self.out_chans), ) def construct(self, x: ms.Tensor) -> ms.Tensor: """ Args: x (ms.Tensor): Input image tensor. Returns: ms.Tensor: Encoded image tensor. """ x = self.patch_embed(x) if self.pos_embed is not None: x = x + self.pos_embed for blk in self.blocks: x = blk(x) x = self.neck(x.transpose(0, 3, 1, 2)) return x class Block(nn.Cell): """ Transformer blocks with support of window attention and residual propagation blocks """ def __init__(self, dim: int, num_heads: int, mlp_ratio: float = 4.0, qkv_bias: bool = True, use_rel_pos: bool = False, window_size: int = 0, pad_size: Tuple[int, int] = (0, 0), feat_size: Tuple[int, int] = (64, 64), input_size: Optional[Tuple[int, int]] = None, layer_norm_eps: float = 1.e-12, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: super().__init__() self.norm1 = LayerNorm((dim), eps=layer_norm_eps) self.attn = Attention( dim, num_heads=num_heads, qkv_bias=qkv_bias, use_rel_pos=use_rel_pos, input_size=input_size if window_size == 0 else (window_size, window_size), compute_dtype=compute_dtype, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type ) self.norm2 = LayerNorm((dim,), eps=layer_norm_eps) self.mlp = MLPBlock(embedding_dim=dim, mlp_dim=int(dim * mlp_ratio), compute_dtype=compute_dtype, param_init_type=param_init_type) self.window_size = window_size self.pad_size = pad_size self.feat_size = feat_size def construct(self, x: ms.Tensor) -> ms.Tensor: """ Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. """ shortcut = x x = self.norm1(x) # Window partition if self.window_size > 0: x = window_partition(x, self.window_size, self.pad_size) x = self.attn(x) # Reverse window partition if self.window_size > 0: x = window_unpartition(x, self.window_size, self.pad_size, self.feat_size) x = shortcut + x x = x + self.mlp(self.norm2(x)) return x class Attention(nn.Cell): """ Multi-head Attention block with relative position embeddings. """ def __init__(self, dim: int, num_heads: int = 8, qkv_bias: bool = True, use_rel_pos: bool = False, input_size: Optional[Tuple[int, int]] = None, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: super().__init__() self.compute_dtype = compute_dtype self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.num_heads = num_heads head_dim = dim // num_heads self.scale = head_dim**-0.5 self.qkv = Linear(in_channels=dim, out_channels=dim * 3, has_bias=qkv_bias, compute_dtype=compute_dtype, param_init_type=param_init_type) self.proj = Linear(in_channels=dim, out_channels=dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.use_rel_pos = use_rel_pos if self.use_rel_pos: assert ( input_size is not None ), "Input size must be provided if using relative positional encoding." # initialize relative positional embeddings self.rel_pos_h = Parameter(P.Zeros()((2 * input_size[0] - 1, head_dim), self.compute_dtype)) self.rel_pos_w = Parameter(P.Zeros()((2 * input_size[1] - 1, head_dim), self.compute_dtype)) self.softmax = P.Softmax(axis=-1) self.batchmatmul = P.BatchMatMul() self.batchmatmul_trans_b = P.BatchMatMul(transpose_b=True) self.cast = P.Cast() self.unstack = P.Unstack(axis=0) def construct(self, x: ms.Tensor) -> ms.Tensor: """ Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. """ b, h, w, _ = x.shape ori_type = x.dtype # qkv with shape (3, B, nHead, H * W, C) qkv = self.qkv(x).reshape(b, h * w, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4) qkv = self.cast(qkv, self.compute_dtype) # q, k, v with shape (B * nHead, H * W, C) q, k, v = self.unstack(qkv.reshape(3, b * self.num_heads, h * w, -1)) attn = self.batchmatmul_trans_b((q * self.scale), k) if self.use_rel_pos: attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (h, w), (h, w)) attn = self.cast(attn, self.softmax_compute_type) attn = self.softmax(attn) attn = self.cast(attn, self.compute_dtype) x = self.batchmatmul(attn, v) x = x.view(b, self.num_heads, h, w, -1) x = x.permute(0, 2, 3, 1, 4) x = x.reshape(b, h, w, -1) x = self.proj(x) x = self.cast(x, ori_type) return x def window_partition(x: ms.Tensor, window_size: int, pad_size: Tuple[int, int] = 0) -> Tuple[ms.Tensor, Tuple[int, int]]: """ Partition the input tensor into non-overlapping windows with optional padding. Args: x (ms.Tensor): Input tensor with shape [B, H, W, C]. window_size (int): Window size. pad_size (tuple[int, int]): Padding size as (pad_h, pad_w). Returns: windows (ms.Tensor): Windows after partition with shape [B * num_windows, window_size, window_size, C]. """ b, h, w, c = x.shape pad_h, pad_w = pad_size if pad_h > 0 or pad_w > 0: pad = P.Pad(paddings=((0, 0), (0, pad_h), (0, pad_w), (0, 0))) x = pad(x) hp, wp = h + pad_h, w + pad_w x = x.view(b, hp // window_size, window_size, wp // window_size, window_size, c) windows = x.transpose(0, 1, 3, 2, 4, 5).view(-1, window_size, window_size, c) return windows def window_unpartition(windows: ms.Tensor, window_size: int, pad_hw: Tuple[int, int], hw: Tuple[int, int]) -> ms.Tensor: """ Unpartition windows back into original sequences and remove padding if needed. Args: windows (ms.Tensor): Input windows with shape [B * num_windows, window_size, window_size, C]. window_size (int): Window size. pad_hw (Tuple[int, int]): Padded height and width (Hp, Wp). hw (Tuple[int, int]): Original height and width (H, W) before padding. Returns: x (ms.Tensor): Unpartitioned sequences with shape [B, H, W, C]. """ pad_h, pad_w = pad_hw h, w = hw hp, wp = h + pad_h, w + pad_w b = windows.shape[0] // (hp * wp // window_size // window_size) x = windows.view(b, hp // window_size, wp // window_size, window_size, window_size, -1) x = x.transpose(0, 1, 3, 2, 4, 5).view(b, hp, wp, -1) if hp > h or wp > w: x = x[:, :h, :w, :] return x def get_rel_pos(q_size: int, k_size: int, rel_pos: ms.Tensor) -> ms.Tensor: """ Get relative positional embeddings based on the relative positions of query and key sizes. Args: q_size (int): Size of query q. k_size (int): Size of key k. rel_pos (ms.Tensor): Relative position embeddings (L, C). Returns: ms.Tensor: Extracted positional embeddings according to relative positions. """ max_rel_dist = int(2 * max(q_size, k_size) - 1) # Interpolate rel pos if needed. if rel_pos.shape[0] != max_rel_dist: # Interpolate rel pos. rel_pos_resized = ops.interpolate( rel_pos.reshape(1, rel_pos.shape[0], -1).transpose(0, 2, 1), size=max_rel_dist, mode='linear', ) rel_pos_resized = rel_pos_resized.reshape(-1, max_rel_dist) else: rel_pos_resized = rel_pos # Scale the coords with short length if shapes for q and k are different. q_coords = np.arange(q_size)[:, None] * max(k_size / q_size, 1.0) k_coords = np.arange(k_size)[None, :] * max(q_size / k_size, 1.0) relative_coords = (q_coords - k_coords) + (k_size - 1) * max(q_size / k_size, 1.0) return rel_pos_resized[relative_coords.astype(mstype.int32)] def add_decomposed_rel_pos(attn: ms.Tensor, q: ms.Tensor, rel_pos_h: ms.Tensor, rel_pos_w: ms.Tensor, q_size: Tuple[int, int], k_size: Tuple[int, int]) -> ms.Tensor: """ Calculate decomposed Relative Positional Embeddings from mvitv2 paper. https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py # noqa B950 Args: attn (ms.Tensor): Attention map. q (ms.Tensor): Query q in the attention layer with shape (B, q_h * q_w, C). rel_pos_h (ms.Tensor): Relative position embeddings (Lh, C) for the height axis. rel_pos_w (ms.Tensor): Relative position embeddings (Lw, C) for the width axis. q_size (Tuple[int, int]): Spatial sequence size of query q with (q_h, q_w). k_size (Tuple[int, int]): Spatial sequence size of key k with (k_h, k_w). Returns: ms.Tensor: Attention map with added relative positional embeddings. """ q_h, q_w = q_size k_h, k_w = k_size rh = get_rel_pos(q_h, k_h, rel_pos_h) rw = get_rel_pos(q_w, k_w, rel_pos_w) b, _, dim = q.shape r_q = q.reshape(b, q_h, q_w, dim) rel_h = ops.matmul(r_q, rh.transpose(0, 2, 1)).reshape(b, q_h, q_w, rh.shape[1]) rel_w = ops.matmul(r_q.transpose(0, 2, 1, 3), rw.transpose(0, 2, 1)).reshape(b, q_h, q_w, rw.shape[1]) rel_w = rel_w.transpose(0, 2, 1, 3) attn = (attn.view(b, q_h, q_w, k_h, k_w) + \ rel_h[:, :, :, :, None] + \ rel_w[:, :, :, None, :]).view(b, q_h * q_w, k_h * k_w) return attn class PatchEmbed(nn.Cell): """ Image to Patch Embedding. """ def __init__(self, kernel_size: Tuple[int, int] = (16, 16), stride: Tuple[int, int] = (16, 16), padding: Tuple[int, int, int, int] = (0, 0, 0, 0), in_chans: int = 3, embed_dim: int = 768) -> None: """ Initialize the PatchEmbed layer. Args: kernel_size (Tuple[int, int]): Kernel size of the projection layer. stride (Tuple[int, int]): Stride of the projection layer. padding (Tuple[int, int, int, int]): Padding size of the projection layer. in_chans (int): Number of input image channels. embed_dim (int): Patch embedding dimension. """ super().__init__() self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=kernel_size, stride=stride, padding=padding, has_bias=True) def construct(self, x: ms.Tensor) -> ms.Tensor: """ Forward pass of the PatchEmbed layer. Args: x (ms.Tensor): Input image tensor. Returns: ms.Tensor: Patch embeddings tensor. """ x = self.proj(x) # B C H W -> B H W C x = x.transpose(0, 2, 3, 1) return x pengcheng.mind-test/mindformers/models/sam/sam_layers.py000066400000000000000000000056001471400134000240130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Layers""" from typing import Type import mindspore as ms import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore import Parameter from mindspore.ops import operations as P from mindformers.modules.layers import Linear class MLPBlock(nn.Cell): """ Multi-Layer Perceptron (MLP) block. Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. """ def __init__(self, embedding_dim: int, mlp_dim: int, act: Type[nn.Cell] = nn.GELU, compute_dtype=mstype.float16, param_init_type=mstype.float32) -> None: super().__init__() self.lin1 = Linear(in_channels=embedding_dim, out_channels=mlp_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.lin2 = Linear(in_channels=mlp_dim, out_channels=embedding_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.act = act() def construct(self, x: ms.Tensor) -> ms.Tensor: return self.lin2(self.act(self.lin1(x))) # From https://github.com/facebookresearch/detectron2/blob/main/detectron2/layers/batch_norm.py # noqa # Itself from https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 # noqa class LayerNorm2d(nn.Cell): """ Layer Normalization for 2D data. Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Normalized tensor. """ def __init__(self, num_channels: int, eps: float = 1e-6) -> None: super().__init__() self.weight = Parameter(P.Ones()(num_channels, ms.float32)) self.bias = Parameter(P.Zeros()(num_channels, ms.float32)) self.eps = eps self.pow = P.Pow() self.sqrt = P.Sqrt() def construct(self, x: ms.Tensor) -> ms.Tensor: u = x.mean(1, keep_dims=True) s = self.pow(x - u, 2).mean(1, keep_dims=True) x = (x - u) / self.sqrt(s + self.eps) x = self.weight[:, None, None] * x + self.bias[:, None, None] return x pengcheng.mind-test/mindformers/models/sam/sam_mask_decoder.py000066400000000000000000000542571471400134000251500ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Mask Decoder Model""" import math from typing import Tuple, Type import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.modules.layers import Linear, LayerNorm from .sam_layers import MLPBlock, LayerNorm2d @MindFormerRegister.register(MindFormerModuleType.MODELS) class SAMMaskDecoder(BaseModel): """ A class to predict masks given image and prompt embeddings using a transformer architecture. """ def __init__(self, config) -> None: super().__init__(config) self.transformer_dim = config.transformer_dim self.decoder_depth = config.decoder_depth self.decoder_embed_dim = config.decoder_embed_dim self.decoder_mlp_dim = config.decoder_mlp_dim self.decoder_num_heads = config.decoder_num_heads self.num_multimask_outputs = config.num_multimask_outputs self.iou_head_depth = config.iou_head_depth self.iou_head_hidden_dim = config.iou_head_hidden_dim self.layer_norm_eps = config.layer_norm_eps self.activation = nn.GELU self.compute_dtype = config.compute_dtype self.layernorm_compute_type = config.layernorm_compute_type self.softmax_compute_type = config.softmax_compute_type self.param_init_type = config.param_init_type self.transformer = TwoWayTransformer( depth=self.decoder_depth, embedding_dim=self.decoder_embed_dim, mlp_dim=self.decoder_mlp_dim, num_heads=self.decoder_num_heads, layer_norm_eps=self.layer_norm_eps, compute_dtype=self.compute_dtype, layernorm_compute_type=self.layernorm_compute_type, softmax_compute_type=self.softmax_compute_type, param_init_type=self.param_init_type ) self.iou_token = nn.Embedding(1, self.transformer_dim) self.num_mask_tokens = self.num_multimask_outputs + 1 self.mask_tokens = nn.Embedding(self.num_mask_tokens, self.transformer_dim) self.output_upscaling = nn.SequentialCell( nn.Conv2dTranspose(self.transformer_dim, self.transformer_dim // 4, kernel_size=2, stride=2, has_bias=True), LayerNorm2d(self.transformer_dim // 4), self.activation(), nn.Conv2dTranspose(self.transformer_dim // 4, self.transformer_dim // 8, kernel_size=2, stride=2, has_bias=True), self.activation(), ) self.output_hypernetworks_mlps = nn.CellList( [ MLP(self.transformer_dim, self.transformer_dim, self.transformer_dim // 8, 3, compute_dtype=self.compute_dtype, param_init_type=self.param_init_type) for i in range(self.num_mask_tokens) ] ) self.iou_prediction_head = MLP( self.transformer_dim, self.iou_head_hidden_dim, self.num_mask_tokens, self.iou_head_depth, compute_dtype=self.compute_dtype, param_init_type=self.param_init_type ) self.concat_0 = P.Concat(axis=0) self.concat_1 = P.Concat(axis=1) self.expand_dims = P.ExpandDims() self.stack = P.Stack(axis=1) def construct(self, image_embeddings: ms.Tensor, image_pe: ms.Tensor, sparse_prompt_embeddings: ms.Tensor, dense_prompt_embeddings: ms.Tensor, multimask_output: bool) -> Tuple[ms.Tensor, ms.Tensor]: """ Args: image_embeddings (ms.Tensor): Embeddings from the image encoder. image_pe (ms.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (ms.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (ms.Tensor): Embeddings of the mask inputs. multimask_output (bool): Whether to return multiple masks or a single mask. Returns: Tuple[ms.Tensor, ms.Tensor]: Batched predicted masks and batched predictions of mask quality. """ masks, iou_pred = self.predict_masks( image_embeddings=image_embeddings, image_pe=image_pe, sparse_prompt_embeddings=sparse_prompt_embeddings, dense_prompt_embeddings=dense_prompt_embeddings, ) # Select the correct mask or masks for output if multimask_output: mask_slice = slice(1, None) else: mask_slice = slice(0, 1) masks = masks[:, mask_slice, :, :] iou_pred = iou_pred[:, mask_slice] # Prepare output return masks, iou_pred def predict_masks(self, image_embeddings: ms.Tensor, image_pe: ms.Tensor, sparse_prompt_embeddings: ms.Tensor, dense_prompt_embeddings: ms.Tensor) -> Tuple[ms.Tensor, ms.Tensor]: """ Args: image_embeddings (ms.Tensor): Embeddings from the image encoder. image_pe (ms.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (ms.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (ms.Tensor): Embeddings of the mask inputs. Returns: Tuple[ms.Tensor, ms.Tensor]: Batched predicted masks and batched predictions of mask quality. """ bs = sparse_prompt_embeddings.shape[0] # Concatenate output tokens output_tokens = self.concat_0([self.iou_token.embedding_table, self.mask_tokens.embedding_table]) output_tokens = ops.broadcast_to(self.expand_dims(output_tokens, 0), (bs, -1, -1)) if ops.any(sparse_prompt_embeddings): tokens = self.concat_1([output_tokens, sparse_prompt_embeddings]) else: tokens = output_tokens # Expand per-image data in batch direction to be per-mask src = image_embeddings.repeat_interleave(tokens.shape[0], 0) src = src + dense_prompt_embeddings pos_src = image_pe.repeat_interleave(tokens.shape[0], 0) b, c, h, w = src.shape # Run the transformer hs, src = self.transformer(src, pos_src, tokens) iou_token_out = hs[:, 0, :] mask_tokens_out = hs[:, 1 : (1 + self.num_mask_tokens), :] # Upscale mask embeddings and predict masks using the mask tokens src = src.transpose(0, 2, 1).view(b, c, h, w) upscaled_embedding = self.output_upscaling(src) hyper_in_list = [] for i in range(self.num_mask_tokens): hyper_in_list.append(self.output_hypernetworks_mlps[i](mask_tokens_out[:, i, :])) hyper_in = self.stack(hyper_in_list) b, c, h, w = upscaled_embedding.shape masks = ops.matmul(hyper_in, upscaled_embedding.view(b, c, h * w)).view(b, -1, h, w) # Generate mask quality predictions iou_pred = self.iou_prediction_head(iou_token_out) return masks, iou_pred class TwoWayTransformer(nn.Cell): """ Two-way transformer decoder that attends to an input image using queries whose positional embedding is supplied. """ def __init__(self, depth: int, embedding_dim: int, num_heads: int, mlp_dim: int, activation: Type[nn.Cell] = nn.ReLU, attention_downsample_rate: int = 2, layer_norm_eps: float = 1.e-12, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: super().__init__() self.depth = depth self.embeding_dim = embedding_dim self.num_heads = num_heads self.mpl_dim = mlp_dim self.compute_dtype = compute_dtype self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.param_init_type = param_init_type self.layers = nn.CellList() for i in range(depth): self.layers.append( TwoWayAttentionBlock( embedding_dim=embedding_dim, num_heads=num_heads, mlp_dim=mlp_dim, activation=activation, attention_downsample_rate=attention_downsample_rate, skip_first_layer_pe=(i == 0), layer_norm_eps=layer_norm_eps, compute_dtype=self.compute_dtype, layernorm_compute_type=self.layernorm_compute_type, softmax_compute_type=self.softmax_compute_type, param_init_type=self.param_init_type ) ) self.final_attn_token_to_image = Attention( embedding_dim, num_heads, downsample_rate=attention_downsample_rate, compute_dtype=self.compute_dtype, layernorm_compute_type=self.layernorm_compute_type, softmax_compute_type=self.softmax_compute_type, param_init_type=self.param_init_type ) self.norm_final_attn = LayerNorm((embedding_dim,), eps=layer_norm_eps) def construct(self, image_embedding: ms.Tensor, image_pe: ms.Tensor, point_embedding: ms.Tensor) -> Tuple[ms.Tensor, ms.Tensor]: """ Apply the two-way transformer. Args: image_embedding (ms.Tensor): Image to attend to. Should have shape B x embedding_dim x h x w for any h and w. image_pe (ms.Tensor): The positional encoding to add to the image. Must have the same shape as image_embedding. point_embedding (ms.Tensor): The embedding to add to the query points. Must have shape B x N_points x embedding_dim for any N_points. Returns: ms.Tensor: The processed point_embedding. ms.Tensor: The processed image_embedding. """ # BxCxHxW -> BxHWxC == B x N_image_tokens x C bs, c, h, w = image_embedding.shape image_embedding = image_embedding.reshape(bs, c, h*w).permute(0, 2, 1) image_pe = image_pe.reshape(bs, c, h*w).permute(0, 2, 1) # Prepare queries queries = point_embedding keys = image_embedding # Apply transformer blocks and final layernorm for layer in self.layers: queries, keys = layer( queries=queries, keys=keys, query_pe=point_embedding, key_pe=image_pe, ) # Apply the final attention layer from the points to the image q = queries + point_embedding k = keys + image_pe attn_out = self.final_attn_token_to_image(q=q, k=k, v=keys) queries = queries + attn_out queries = self.norm_final_attn(queries) return queries, keys class TwoWayAttentionBlock(nn.Cell): """ Two-way attention block consisting of self-attention and cross-attention layers. """ def __init__(self, embedding_dim: int, num_heads: int, mlp_dim: int = 2048, activation: Type[nn.Cell] = nn.ReLU, attention_downsample_rate: int = 2, skip_first_layer_pe: bool = False, layer_norm_eps: float = 1.e-12, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: super().__init__() self.self_attn = Attention(embedding_dim, num_heads, compute_dtype=compute_dtype, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type) self.norm1 = LayerNorm((embedding_dim,), eps=layer_norm_eps) self.cross_attn_token_to_image = Attention(embedding_dim, num_heads, downsample_rate=attention_downsample_rate, compute_dtype=compute_dtype, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type) self.norm2 = LayerNorm((embedding_dim,), eps=layer_norm_eps) self.mlp = MLPBlock(embedding_dim, mlp_dim, activation, compute_dtype=compute_dtype, param_init_type=param_init_type) self.norm3 = LayerNorm((embedding_dim,), eps=layer_norm_eps) self.norm4 = LayerNorm((embedding_dim,), eps=layer_norm_eps) self.cross_attn_image_to_token = Attention(embedding_dim, num_heads, downsample_rate=attention_downsample_rate, compute_dtype=compute_dtype, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type) self.skip_first_layer_pe = skip_first_layer_pe def construct(self, queries: ms.Tensor, keys: ms.Tensor, query_pe: ms.Tensor, key_pe: ms.Tensor) -> Tuple[ms.Tensor, ms.Tensor]: """ Apply the two-way attention block. Args: queries (ms.Tensor): Queries tensor. keys (ms.Tensor): Keys tensor. query_pe (ms.Tensor): Query positional encoding. key_pe (ms.Tensor): Key positional encoding. Returns: ms.Tensor: Processed queries tensor. ms.Tensor: Processed keys tensor. """ # Self attention block if self.skip_first_layer_pe: queries = self.self_attn(q=queries, k=queries, v=queries) else: q = queries + query_pe attn_out = self.self_attn(q=q, k=q, v=queries) queries = queries + attn_out queries = self.norm1(queries) # Cross attention block, tokens attending to image embedding q = queries + query_pe k = keys + key_pe attn_out = self.cross_attn_token_to_image(q=q, k=k, v=keys) queries = queries + attn_out queries = self.norm2(queries) # MLP block mlp_out = self.mlp(queries) queries = queries + mlp_out queries = self.norm3(queries) # Cross attention block, image embedding attending to tokens q = queries + query_pe k = keys + key_pe attn_out = self.cross_attn_image_to_token(q=k, k=q, v=queries) keys = keys + attn_out keys = self.norm4(keys) return queries, keys class Attention(nn.Cell): """ Attention layer that supports downscaling the embedding size after projection to queries, keys, and values. """ def __init__(self, embedding_dim: int, num_heads: int, downsample_rate: int = 1, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: """ Initialize the Attention layer. Args: embedding_dim (int): The channel dimension of the embeddings. num_heads (int): The number of heads in the attention layer. downsample_rate (int): Downsample rate for the attention. """ super().__init__() self.embedding_dim = embedding_dim self.internal_dim = embedding_dim // downsample_rate self.num_heads = num_heads self.compute_dtype = compute_dtype self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.param_init_type = param_init_type assert self.internal_dim % num_heads == 0, "num_heads must divide embedding_dim." self.inv_norm_factor = ms.Tensor(1.0 / math.sqrt(self.internal_dim // self.num_heads)) self.q_proj = Linear(in_channels=embedding_dim, out_channels=self.internal_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj = Linear(in_channels=embedding_dim, out_channels=self.internal_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj = Linear(in_channels=embedding_dim, out_channels=self.internal_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.out_proj = Linear(in_channels=self.internal_dim, out_channels=embedding_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.softmax = P.Softmax(axis=-1) self.batchmatmul_trans_b = P.BatchMatMul(transpose_b=True) self.batchmatmul = P.BatchMatMul() self.mul = P.Mul() def _separate_heads(self, x: ms.Tensor, num_heads: int) -> ms.Tensor: """ Separate the input tensor into multiple heads. Args: x (ms.Tensor): Input tensor. num_heads (int): Number of heads to separate into. Returns: ms.Tensor: Tensor with dimensions reshaped for separate heads. """ b, n, c = x.shape x = x.reshape(b, n, num_heads, c // num_heads) return x.transpose(0, 2, 1, 3) def _recombine_heads(self, x: ms.Tensor) -> ms.Tensor: """ Recombine the tensor with multiple heads into a single tensor. Args: x (ms.Tensor): Input tensor with separate heads. Returns: ms.Tensor: Recombined tensor. """ b, n_heads, n_tokens, c_per_head = x.shape x = x.transpose(0, 2, 1, 3) return x.reshape(b, n_tokens, n_heads * c_per_head) def construct(self, q: ms.Tensor, k: ms.Tensor, v: ms.Tensor) -> ms.Tensor: """ Apply the attention mechanism. Args: q (ms.Tensor): Queries tensor. k (ms.Tensor): Keys tensor. v (ms.Tensor): Values tensor. Returns: ms.Tensor: Output tensor after attention. """ ori_type = k.dtype # Input projections q = self.q_proj(q).astype(self.compute_dtype) k = self.k_proj(k).astype(self.compute_dtype) v = self.v_proj(v).astype(self.compute_dtype) # Separate into heads q = self._separate_heads(q, self.num_heads) k = self._separate_heads(k, self.num_heads) v = self._separate_heads(v, self.num_heads) # Attention attn = self.batchmatmul_trans_b(q, k) attn = self.mul(attn, self.inv_norm_factor) attn = self.softmax(attn.astype(self.softmax_compute_type)) # Get output out = self.batchmatmul(attn.astype(self.compute_dtype), v) out = self._recombine_heads(out) out = self.out_proj(out).astype(ori_type) return out # Lightly adapted from # https://github.com/facebookresearch/MaskFormer/blob/main/mask_former/modeling/transformer/transformer_predictor.py # noqa class MLP(nn.Cell): """ Multi-Layer Perceptron (MLP) class definition. """ def __init__(self, input_dim: int, hidden_dim: int, output_dim: int, num_layers: int, sigmoid_output: bool = False, compute_dtype=mstype.float16, param_init_type=mstype.float32) -> None: super().__init__() self.num_layers = num_layers h = [hidden_dim] * (num_layers - 1) self.layers = nn.CellList([ Linear(in_channels=n, out_channels=k, compute_dtype=compute_dtype, param_init_type=param_init_type) for n, k in zip([input_dim] + h, h + [output_dim]) ]) self.sigmoid_output = sigmoid_output self.relu = P.ReLU() self.sigmoid = P.Sigmoid() def construct(self, x): """ Args: x (Tensor): Input tensor. Returns: Tensor: Output tensor of the MLP. """ for i, layer in enumerate(self.layers): x = self.relu(layer(x)) if i < self.num_layers - 1 else layer(x) if self.sigmoid_output: x = self.sigmoid(x) return x pengcheng.mind-test/mindformers/models/sam/sam_processor.py000066400000000000000000000205541471400134000245400ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Processor""" from copy import deepcopy from typing import Tuple import numpy as np from PIL import Image import mindspore as ms from mindspore.dataset import vision from mindspore.dataset.vision import Inter from mindspore.ops import operations as P from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class SAMImageProcessor(BaseImageProcessor): """ Subclass of BaseImageProcessor implementing image preprocessing for SAM model. Args: img_size (int): The target size for image processing. mean (list): Mean values for image normalization. std (list): Standard deviation values for image normalization. """ def __init__(self, img_size=1024, mean=(123.675, 116.28, 103.53), std=(58.395, 57.12, 57.375)): super().__init__() self.img_size = img_size self.pixel_mean = ms.Tensor(mean).view(-1, 1, 1) self.pixel_std = ms.Tensor(std).view(-1, 1, 1) self.transform = ResizeLongestSide(img_size) def preprocess(self, images, **kwargs): """ Preprocess the input image for SAM model. Args: image (PIL.Image or numpy.array): Input image to be preprocessed. Returns: image (tensor): Preprocessed image as a tensor. input_size (tuple): Size of the input image after preprocessing. """ images = self.transform.apply_image(images) images = images.transpose(2, 0, 1)[None, :, :, :] images = ms.Tensor(images) input_size = images.shape[-2:] # Normalize colors images = (images - self.pixel_mean) / self.pixel_std # Pad h, w = images.shape[-2:] padh = self.img_size - h padw = self.img_size - w if images.ndim == 4: pad = P.Pad(paddings=((0, 0), (0, 0), (0, padh), (0, padw))) elif images.ndim == 3: pad = P.Pad(paddings=((0, 0), (0, padh), (0, padw))) images = pad(images) return images, input_size class ResizeLongestSide: """ Resizes images to the longest side 'target_length', as well as provides methods for resizing coordinates and boxes. Provides methods for transforming both numpy array and batched mindspore tensors. """ def __init__(self, target_length: int) -> None: """ Initialize the ResizeLongestSide class. Args: target_length (int): Target length of the longest side. """ self.target_length = target_length def apply_image(self, image: np.ndarray) -> np.ndarray: """ Apply the resizing transformation to a single image. Args: image (np.ndarray): Input image as HxWxC numpy array. Returns: np.ndarray: Resized image as H'xW'xC numpy array. """ target_size = self.get_preprocess_shape(image.shape[0], image.shape[1], self.target_length) pil_image = Image.fromarray(image) resize_op = vision.Resize(size=target_size) resized_image = np.array(resize_op(pil_image)) return resized_image def apply_coords(self, coords: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray: """ Apply the resizing transformation to coordinates. Args: coords (np.ndarray): Input coordinates as a numpy array of shape (N, 2). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized coordinates as a numpy array of shape (N, 2). """ old_h, old_w = original_size new_h, new_w = self.get_preprocess_shape( original_size[0], original_size[1], self.target_length ) coords = deepcopy(coords).astype(float) coords[..., 0] = coords[..., 0] * (new_w / old_w) coords[..., 1] = coords[..., 1] * (new_h / old_h) return coords def apply_boxes(self, boxes: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray: """ Apply the resizing transformation to bounding boxes. Args: boxes (np.ndarray): Input boxes as a numpy array of shape (B, 4). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized boxes as a numpy array of shape (B, 4). """ boxes = self.apply_coords(boxes.reshape(-1, 2, 2), original_size) return boxes.reshape(-1, 4) def apply_image_batch(self, image: np.ndarray) -> np.ndarray: """ Apply the resizing transformation to a batch of images. Args: image (np.ndarray): Input batched images as BxCxHxW numpy array. Returns: np.ndarray: Resized batched images as BxC'xH'xW' numpy array. """ # Expects an image in BCHW format. May not exactly match apply_image. target_size = self.get_preprocess_shape(image.shape[2], image.shape[3], self.target_length) resize_ = vision.Resize(target_size, Inter.BILINEAR) image = resize_(image) return image def apply_coords_batch(self, coords: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray: """ Apply the resizing transformation to a batch of coordinates. Args: coords (np.ndarray): Input batched coordinates as numpy array of shape (B, N, 2). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized batched coordinates as numpy array of shape (B, N, 2). """ old_h, old_w = original_size new_h, new_w = self.get_preprocess_shape( original_size[0], original_size[1], self.target_length ) coords = deepcopy(coords) coords[..., 0] = coords[..., 0] * (new_w / old_w) coords[..., 1] = coords[..., 1] * (new_h / old_h) return coords def apply_boxes_batch(self, boxes: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray: """ Apply the resizing transformation to a batch of bounding boxes. Args: boxes (np.ndarray): Input batched boxes as numpy array of shape (B, N, 4). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized batched boxes as numpy array of shape (B, N, 4). """ boxes = self.apply_coords_batch(boxes.reshape(-1, 2, 2), original_size) return boxes.reshape(-1, 4) @staticmethod def get_preprocess_shape(oldh: int, oldw: int, long_side_length: int) -> Tuple[int, int]: """ Compute the output size given input size and target long side length. Args: oldh (int): Original image height. oldw (int): Original image width. long_side_length (int): Target length of the longest side. Returns: Tuple[int, int]: Output size as (new_h, new_w). """ scale = long_side_length * 1.0 / max(oldh, oldw) newh, neww = oldh * scale, oldw * scale neww = int(neww + 0.5) newh = int(newh + 0.5) return (newh, neww) @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class SAMProcessor(BaseProcessor): """ Vit processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['sam'] def __init__(self, image_processor=None, return_tensors='ms'): super().__init__( image_processor=image_processor, return_tensors=return_tensors ) pengcheng.mind-test/mindformers/models/sam/sam_prompt_encoder.py000066400000000000000000000245171471400134000255440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Prompt Encoder""" from typing import Optional, Tuple import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops import mindspore.numpy as np import mindspore.common.dtype as mstype from mindspore import Parameter from mindspore.ops import operations as P from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from .sam_layers import LayerNorm2d @MindFormerRegister.register(MindFormerModuleType.MODELS) class SAMPromptEncoder(BaseModel): """ Encodes prompts for input to SAM's mask decoder. """ _support_list = MindFormerBook.get_model_support_list()['sam'] def __init__(self, config) -> None: super().__init__(config) self.embed_dim = config.prompt_embed_dim self.input_image_size = config.input_image_size self.image_embedding_size = config.image_embedding_size self.mask_in_chans = config.mask_in_chans self.activation = nn.GELU self.pe_layer = PositionEmbeddingRandom(self.embed_dim // 2) self.num_point_embeddings = 4 # pos/neg point + 2 box corners point_embeddings = [nn.Embedding(1, self.embed_dim) for i in range(self.num_point_embeddings)] self.point_embeddings = nn.CellList(point_embeddings) self.not_a_point_embed = nn.Embedding(1, self.embed_dim) self.mask_input_size = (4 * self.image_embedding_size[0], 4 * self.image_embedding_size[1]) self.mask_downscaling = nn.SequentialCell( nn.Conv2d(1, self.mask_in_chans // 4, kernel_size=2, stride=2, has_bias=True), LayerNorm2d(self.mask_in_chans // 4), self.activation(), nn.Conv2d(self.mask_in_chans // 4, self.mask_in_chans, kernel_size=2, stride=2, has_bias=True), LayerNorm2d(self.mask_in_chans), self.activation(), nn.Conv2d(self.mask_in_chans, self.embed_dim, kernel_size=1, has_bias=True), ) self.no_mask_embed = nn.Embedding(1, self.embed_dim) self.zeros = P.Zeros() self.ones = P.Ones() self.expand_dims = P.ExpandDims() self.concat = P.Concat(axis=1) self.tile = P.Tile() def get_dense_pe(self) -> ms.Tensor: """ Get the positional encoding used to encode point prompts, applied to a dense set of points the shape of the image encoding. Returns: ms.Tensor: Positional encoding with shape 1x(embed_dim)x(embedding_h)x(embedding_w) """ return self.expand_dims(self.pe_layer(self.image_embedding_size), 0) def _embed_points(self, points: ms.Tensor, labels: ms.Tensor, pad: bool) -> ms.Tensor: """Embeds point prompts.""" bs = points.shape[0] points = points + 0.5 # Shift to center of pixel point_embedding = self.pe_layer.forward_with_coords(points, self.input_image_size) if pad: not_a_point_embed = self.tile(self.expand_dims(self.not_a_point_embed.embedding_table, 0), (bs, 1, 1)) point_embedding = self.concat([point_embedding, not_a_point_embed]) labels = self.concat([labels, -self.ones((bs, 1), mstype.int32)]) mask_neg = (labels == 0).reshape(bs, -1, 1) point_embedding += np.where(mask_neg, self.point_embeddings[0].embedding_table,\ np.zeros_like(self.point_embeddings[0].embedding_table)) mask_pos = (labels == 1).reshape(bs, -1, 1) point_embedding += np.where(mask_pos, self.point_embeddings[1].embedding_table,\ np.zeros_like(self.point_embeddings[1].embedding_table)) return point_embedding def _embed_boxes(self, boxes: ms.Tensor) -> ms.Tensor: """Embeds box prompts.""" boxes = boxes + 0.5 # Shift to center of pixel coords = boxes.reshape(-1, 2, 2) corner_embedding = self.pe_layer.forward_with_coords(coords, self.input_image_size) corner_embedding[:, 0, :] += self.point_embeddings[2].embedding_table corner_embedding[:, 1, :] += self.point_embeddings[3].embedding_table return corner_embedding def _embed_masks(self, masks: ms.Tensor) -> ms.Tensor: """Embeds mask inputs.""" mask_embedding = self.mask_downscaling(masks) return mask_embedding def _get_batch_size(self, points: Optional[ms.Tensor], boxes: Optional[ms.Tensor], masks: Optional[ms.Tensor]) -> int: """ Gets the batch size of the output given the batch size of the input prompts. """ if points is not None: return points.shape[0] if boxes is not None: return boxes.shape[0] if masks is not None: return masks.shape[0] return 1 def construct(self, point_coords: Optional[ms.Tensor], point_labels: Optional[ms.Tensor], boxes: Optional[ms.Tensor], mask_inputs: Optional[ms.Tensor]) -> Tuple[ms.Tensor, ms.Tensor]: """ Args: point_coords (ms.Tensor or none): Point coordinates to embed. point_labels (ms.Tensor or none): Point labels to embed. boxes (ms.Tensor or none): Boxes to embed. mask_inputs (ms.Tensor or none): Masks to embed. Returns: Tuple[ms.Tensor, ms.Tensor]: Sparse embeddings for the points and boxes, with shape BxNx(embed_dim), where N is determined by the number of input points and boxes. Dense embeddings for the masks, in the shape Bx(embed_dim)x(embed_H)x(embed_W). """ bs = self._get_batch_size(point_coords, boxes, mask_inputs) sparse_embeddings = np.empty((bs, 1, self.embed_dim)) # sparse_embeddings = None if point_coords is not None: point_embeddings = self._embed_points(point_coords, point_labels, pad=(boxes is None)) sparse_embeddings = point_embeddings if boxes is not None: box_embeddings = self._embed_boxes(boxes) sparse_embeddings = box_embeddings if mask_inputs is not None: dense_embeddings = self._embed_masks(mask_inputs) else: dense_embeddings = ops.broadcast_to(self.no_mask_embed.embedding_table.reshape(1, -1, 1, 1),\ (bs, -1, self.image_embedding_size[0], self.image_embedding_size[1])) return sparse_embeddings, dense_embeddings class PositionEmbeddingRandom(nn.Cell): """ Positional encoding using random spatial frequencies. """ def __init__(self, num_pos_feats: int = 64, scale: Optional[float] = None, compute_dtype=mstype.float16) -> None: """ Initialize the PositionEmbeddingRandom. Args: num_pos_feats (int): Number of positional encoding features. scale (float): Scale factor for the positional encoding Gaussian matrix. """ super().__init__() self.compute_dtype = compute_dtype if scale is None or scale <= 0.0: scale = 1.0 pe_gaussian_matrix = scale * P.StandardNormal()((2, num_pos_feats)).astype(self.compute_dtype) self.positional_encoding_gaussian_matrix = Parameter(pe_gaussian_matrix,\ name="positional_encoding_gaussian_matrix") self.cast = P.Cast() self.ones = P.Ones() self.sin = P.Sin() self.cos = P.Cos() self.concat = P.Concat(axis=-1) self.stack = P.Stack(axis=-1) def _pe_encoding(self, coords: ms.Tensor) -> ms.Tensor: """ Positionally encode points that are normalized to [0,1]. Args: coords (ms.Tensor): Coordinates normalized to [0, 1]. Returns: ms.Tensor: Positional encoding for the given coordinates. """ # assuming coords are in [0, 1]^2 square and have d_1 x ... x d_n x 2 shape coords = 2 * coords - 1 ori_type = coords.dtype # here must use ops.matmul, because 'x', 'y' have different dimension coords = ops.matmul(coords.astype(self.compute_dtype), self.positional_encoding_gaussian_matrix) coords = self.cast(coords, ori_type) coords = 2 * np.pi * coords # outputs d_1 x ... x d_n x C shape return self.concat([self.sin(coords), self.cos(coords)]) def construct(self, size: Tuple[int, int]) -> ms.Tensor: """ Generate positional encoding for a grid of the specified size. Args: size (Tuple[int, int]): Grid size (H, W). Returns: ms.Tensor: Positional encoding for the grid. """ h, w = size grid = self.ones((h, w), ms.float32) y_embed = grid.cumsum(axis=0) - 0.5 x_embed = grid.cumsum(axis=1) - 0.5 y_embed = y_embed / h x_embed = x_embed / w pe = self._pe_encoding(self.stack([x_embed, y_embed])) return pe.transpose(2, 0, 1) # C x H x W def forward_with_coords(self, coords_input: ms.Tensor, image_size: Tuple[int, int]) -> ms.Tensor: """ Positionally encode points that are not normalized to [0,1]. Args: coords_input (ms.Tensor): Coordinates not normalized to [0, 1]. image_size (Tuple[int, int]): Image size (H, W). Returns: ms.Tensor: Positional encoding for the given coordinates. """ coords = coords_input coords[:, :, 0] = coords[:, :, 0] / image_size[1] coords[:, :, 1] = coords[:, :, 1] / image_size[0] return self._pe_encoding(coords) pengcheng.mind-test/mindformers/models/sam/sam_utils.py000066400000000000000000000543451471400134000236660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Utils""" import math from copy import deepcopy from itertools import product from typing import Any, Dict, Generator, ItemsView, List, Tuple import cv2 import numpy as np __all__ = [ "MaskData", "area_from_rle", "batch_iterator", "batched_mask_to_box", "box_xyxy_to_xywh", "build_all_layer_point_grids", "calculate_stability_score", "coco_encode_rle", "generate_crop_boxes", "is_box_near_crop_edge", "mask_to_rle", "remove_small_regions", "rle_to_mask", "uncrop_boxes_xyxy", "uncrop_masks", "uncrop_points", "box_area", "nms" ] class MaskData: """ A structure for storing masks and their related data in batched format. Implements basic filtering and concatenation. """ def __init__(self, **kwargs) -> None: """ Initialize the MaskData. Args: **kwargs: Keyword arguments representing different mask-related data. """ for v in kwargs.values(): assert isinstance( v, (list, np.ndarray) ), "MaskData only supports list, numpy arrays, and ms tensors." self._stats = dict(**kwargs) def __setitem__(self, key: str, item: Any) -> None: """ Set an item in the MaskData. Args: key (str): The key of the item. item (Any): The item to be set. """ assert isinstance( item, (list, np.ndarray) ), "MaskData only supports list, numpy arrays." self._stats[key] = item def __delitem__(self, key: str) -> None: """ Delete an item from the MaskData. Args: key (str): The key of the item to be deleted. """ del self._stats[key] def __getitem__(self, key: str) -> Any: """ Get an item from the MaskData. Args: key (str): The key of the item to be retrieved. Returns: Any: The retrieved item. """ return self._stats[key] def items(self) -> ItemsView[str, Any]: """ Get the items stored in the MaskData. Returns: ItemsView[str, Any]: A view of the items in the MaskData. """ return self._stats.items() def filter(self, keep: np.ndarray) -> None: """ Filter the MaskData based on a boolean mask. Args: keep (np.ndarray): Boolean mask indicating which items to keep. """ for k, v in self._stats.items(): if v is None: self._stats[k] = None elif isinstance(v, np.ndarray): self._stats[k] = v[keep] elif isinstance(v, list) and keep.dtype == np.bool: self._stats[k] = [a for i, a in enumerate(v) if keep[i]] elif isinstance(v, list): self._stats[k] = [v[i] for i in keep] else: raise TypeError(f"MaskData key {k} has an unsupported type {type(v)}.") def cat(self, new_stats: "MaskData") -> None: """ Concatenate new data from another MaskData instance. Args: new_stats (MaskData): The MaskData instance to concatenate. """ for k, v in new_stats.items(): if k not in self._stats or self._stats[k] is None: self._stats[k] = deepcopy(v) elif isinstance(v, np.ndarray): self._stats[k] = np.concatenate([self._stats[k], v], axis=0) elif isinstance(v, list): self._stats[k] = self._stats[k] + deepcopy(v) else: raise TypeError(f"MaskData key {k} has an unsupported type {type(v)}.") def is_box_near_crop_edge(boxes: np.ndarray, crop_box: List[int], orig_box: List[int], atol: float = 20.0) -> np.ndarray: """ Filter boxes at the edge of a crop, but not at the edge of the original image. Args: boxes (np.ndarray): Bounding boxes ndarray of shape (N, 4) in format (x1, y1, x2, y2). crop_box (List[int]): Crop box coordinates in format (x1, y1, x2, y2). orig_box (List[int]): Original image box coordinates in format (x1, y1, x2, y2). atol (float): Absolute tolerance for box comparison. Returns: np.ndarray: Boolean ndarray indicating whether boxes are near the crop edge. """ crop_box = np.array(crop_box, dtype=np.float32) orig_box = np.array(orig_box, dtype=np.float32) boxes = uncrop_boxes_xyxy(boxes, crop_box).astype(np.float32) near_crop_edge = np.isclose(boxes, np.tile(crop_box[None, :], (boxes.shape[0], 1)), atol=atol, rtol=0., equal_nan=True) near_image_edge = np.isclose(boxes, np.tile(orig_box[None, :], (boxes.shape[0], 1)), atol=atol, rtol=0., equal_nan=True) near_crop_edge = np.logical_and(near_crop_edge, ~near_image_edge) return np.any(near_crop_edge, axis=1) def box_xyxy_to_xywh(box_xyxy: np.ndarray) -> np.ndarray: """ Convert bounding box coordinates from (x1, y1, x2, y2) format to (x, y, w, h) format. Args: box_xyxy (np.ndarray): Bounding boxes tensor of shape (N, 4) in (x1, y1, x2, y2) format. Returns: box_xywh (np.ndarray): Bounding boxes tensor of shape (N, 4) in (x, y, w, h) format. """ box_xywh = deepcopy(box_xyxy) box_xywh[2] = box_xywh[2] - box_xywh[0] box_xywh[3] = box_xywh[3] - box_xywh[1] return box_xywh def batch_iterator(batch_size: int, *args) -> Generator[List[Any], None, None]: """ Generate batches of data from input arrays. Args: batch_size (int): Size of each batch. *args: Variable number of input arrays to iterate over. Yields: List[Any]: A batch of data containing corresponding elements from each input array. """ assert args and all( len(a) == len(args[0]) for a in args ), "Batched iteration must have inputs of all the same size." n_batches = len(args[0]) // batch_size + int(len(args[0]) % batch_size != 0) for b in range(n_batches): yield [arg[b * batch_size : (b + 1) * batch_size] for arg in args] def mask_to_rle(tensor: np.ndarray) -> List[Dict[str, Any]]: """ Encode masks to an uncompressed RLE format, in the format expected by pycoco tools. Args: tensor (np.ndarray): Binary mask tensor of shape (B, H, W) where B is the batch size. Returns: List[Dict[str, Any]]: List of dictionaries representing RLE encoded masks. """ # Put in fortran order and flatten h,w b, h, w = tensor.shape tensor = tensor.transpose(0, 2, 1).reshape(tensor.shape[0], -1) # Compute change indices diff = np.bitwise_xor(tensor[:, 1:].astype(np.int32), tensor[:, :-1].astype(np.int32)) change_indices = np.stack(np.nonzero(diff)).transpose() # Encode run length out = [] for i in range(b): cur_idxs = change_indices[change_indices[:, 0] == i, 1] cur_idxs = np.concatenate( [ np.array([0], dtype=cur_idxs.dtype), cur_idxs + 1, np.array([h * w], dtype=cur_idxs.dtype) ] ) btw_idxs = cur_idxs[1:] - cur_idxs[:-1] counts = [] if tensor[i, 0] == 0 else [0] counts.extend(btw_idxs.tolist()) out.append({"size": [h, w], "counts": counts}) return out def rle_to_mask(rle: Dict[str, Any]) -> np.ndarray: """ Compute a binary mask from an uncompressed RLE. Args: rle (Dict[str, Any]): Dictionary containing RLE encoded mask information. Returns: np.ndarray: Binary mask as a NumPy array. """ h, w = rle["size"] mask = np.empty(h * w, dtype=bool) idx = 0 parity = False for count in rle["counts"]: mask[idx : idx + count] = parity idx += count parity ^= True mask = mask.reshape(w, h) return mask.transpose() # Put in C order def area_from_rle(rle: Dict[str, Any]) -> int: """ Calculate the area of a binary mask from an uncompressed RLE. Args: rle (Dict[str, Any]): Dictionary containing RLE encoded mask information. Returns: int: Calculated area of the mask. """ return sum(rle["counts"][1::2]) def calculate_stability_score(masks: np.ndarray, mask_threshold: float, threshold_offset: float) -> np.ndarray: """ Calculate the stability score for a batch of masks. The stability score is the Intersection over Union (IoU) between the binary masks obtained by thresholding the predicted mask logits at high and low values. Args: masks (np.ndarray): Predicted mask logits tensor of shape (B, H, W). mask_threshold (float): Threshold value for binary mask creation. threshold_offset (float): Offset to adjust the threshold. Returns: np.ndarray: Stability scores for each mask in the batch. """ # One mask is always contained inside the other. # Save memory by preventing unnecessary cast to torch.int64 intersections = ( (masks > (mask_threshold + threshold_offset)) .sum(-1, dtype=np.float32) .sum(-1, dtype=np.float32) ) unions = ( (masks > (mask_threshold - threshold_offset)) .sum(-1, dtype=np.float32) .sum(-1, dtype=np.float32) ) return intersections / unions def build_point_grid(n_per_side: int) -> np.ndarray: """ Generate a 2D grid of points evenly spaced in the [0, 1] x [0, 1] range. Args: n_per_side (int): Number of points per side of the grid. Returns: np.ndarray: 2D grid of points as a NumPy array. """ offset = 1 / (2 * n_per_side) points_one_side = np.linspace(offset, 1 - offset, n_per_side) points_x = np.tile(points_one_side[None, :], (n_per_side, 1)) points_y = np.tile(points_one_side[:, None], (1, n_per_side)) points = np.stack([points_x, points_y], axis=-1).reshape(-1, 2) return points def build_all_layer_point_grids(n_per_side: int, n_layers: int, scale_per_layer: int) -> List[np.ndarray]: """ Generate point grids for all crop layers. Args: n_per_side (int): Number of points per side of the original grid. n_layers (int): Number of layers in the crop hierarchy. scale_per_layer (int): Scaling factor applied per layer. Returns: List[np.ndarray]: List of point grids for each crop layer as NumPy arrays. """ points_by_layer = [] for i in range(n_layers + 1): n_points = int(n_per_side / (scale_per_layer**i)) points_by_layer.append(build_point_grid(n_points)) return points_by_layer def generate_crop_boxes(im_size: Tuple[int, ...], n_layers: int, overlap_ratio: float) -> Tuple[List[List[int]], List[int]]: """ Generate a list of crop boxes of different sizes. Each layer has (2**i)**2 boxes for the ith layer. Args: im_size (Tuple[int, ...]): Size of the original image in (height, width). n_layers (int): Number of layers in the crop hierarchy. overlap_ratio (float): Overlap ratio for calculating crop box size. Returns: Tuple[List[List[int]], List[int]]: Tuple containing a list of crop boxes and a list of layer indices for each crop box. """ crop_boxes, layer_idxs = [], [] im_h, im_w = im_size short_side = min(im_h, im_w) # Original image crop_boxes.append([0, 0, im_w, im_h]) layer_idxs.append(0) def crop_len(orig_len, n_crops, overlap): return int(math.ceil((overlap * (n_crops - 1) + orig_len) / n_crops)) for i_layer in range(n_layers): n_crops_per_side = 2 ** (i_layer + 1) overlap = int(overlap_ratio * short_side * (2 / n_crops_per_side)) crop_w = crop_len(im_w, n_crops_per_side, overlap) crop_h = crop_len(im_h, n_crops_per_side, overlap) crop_box_x0 = [int((crop_w - overlap) * i) for i in range(n_crops_per_side)] crop_box_y0 = [int((crop_h - overlap) * i) for i in range(n_crops_per_side)] # Crops in XYWH format for x0, y0 in product(crop_box_x0, crop_box_y0): box = [x0, y0, min(x0 + crop_w, im_w), min(y0 + crop_h, im_h)] crop_boxes.append(box) layer_idxs.append(i_layer + 1) return crop_boxes, layer_idxs def uncrop_boxes_xyxy(boxes: np.ndarray, crop_box: List[int]) -> np.ndarray: """ Uncrop the coordinates of bounding boxes from a cropped region to the original image. Args: boxes (np.ndarray): Bounding box coordinates in XYXY format (x0, y0, x1, y1). crop_box (List[int]): Cropped region's bounding box in XYXY format (x0, y0, x1, y1). Returns: np.ndarray: Uncropped bounding box coordinates in XYXY format. """ x0, y0, _, _ = crop_box offset = np.array([[x0, y0, x0, y0]], dtype=np.int32) # Check if boxes has a channel dimension if len(boxes.shape) == 3: offset = np.expand_dims(offset, 1) return boxes + offset def uncrop_points(points: np.ndarray, crop_box: List[int]) -> np.ndarray: """ Uncrop the coordinates of points from a cropped region to the original image. Args: points (np.ndarray): Point coordinates in XY format (x, y). crop_box (List[int]): Cropped region's bounding box in XYXY format (x0, y0, x1, y1). Returns: np.ndarray: Uncropped point coordinates in XY format. """ x0, y0, _, _ = crop_box offset = np.array([[x0, y0]], dtype=np.int32) # Check if points has a channel dimension if len(points.shape) == 3: offset = np.expand_dims(offset, 1) return points + offset def uncrop_masks(masks: np.ndarray, crop_box: List[int], orig_h: int, orig_w: int) -> np.ndarray: """ Uncrop binary masks from a cropped region to the original image size. Args: masks (np.ndarray): Binary masks ndarray. crop_box (List[int]): Cropped region's bounding box in XYXY format (x0, y0, x1, y1). orig_h (int): Original height of the image. orig_w (int): Original width of the image. Returns: np.ndarray: Uncropped binary masks ndarray. """ x0, y0, x1, y1 = crop_box if x0 == 0 and y0 == 0 and x1 == orig_w and y1 == orig_h: return masks # Coordinate transform masks pad_x, pad_y = orig_w - (x1 - x0), orig_h - (y1 - y0) # Calculate pad widths pad_width = [(0, 0), (y0, pad_y - y0), (x0, pad_x - x0)] # Pad the masks return np.pad(masks, pad_width, mode='constant', constant_values=0) def remove_small_regions(mask: np.ndarray, area_thresh: float, mode: str) -> Tuple[np.ndarray, bool]: """ Removes small disconnected regions or holes in a binary mask. Args: mask (np.ndarray): Binary mask. area_thresh (float): Threshold area for removing small regions. mode (str): Either "holes" or "islands" indicating the type of regions to remove. Returns: Tuple[np.ndarray, bool]: A tuple containing the modified mask and a boolean indicating if the mask was modified. """ assert mode in ["holes", "islands"] correct_holes = mode == "holes" working_mask = (correct_holes ^ mask).astype(np.uint8) n_labels, regions, stats, _ = cv2.connectedComponentsWithStats(working_mask, 8) sizes = stats[:, -1][1:] # Row 0 is background label small_regions = [i + 1 for i, s in enumerate(sizes) if s < area_thresh] if not small_regions: return mask, False fill_labels = [0] + small_regions if not correct_holes: fill_labels = [i for i in range(n_labels) if i not in fill_labels] # If every region is below threshold, keep largest if not fill_labels: fill_labels = [int(np.argmax(sizes)) + 1] mask = np.isin(regions, fill_labels) return mask, True def coco_encode_rle(uncompressed_rle: Dict[str, Any]) -> Dict[str, Any]: """ Encode uncompressed RLE (Run-Length Encoding) to COCO format RLE. Args: uncompressed_rle (Dict[str, Any]): Uncompressed RLE dictionary. Returns: Dict[str, Any]: Encoded RLE in COCO format. """ from pycocotools import mask as mask_utils # type: ignore h, w = uncompressed_rle["size"] rle = mask_utils.frPyObjects(uncompressed_rle, h, w) rle["counts"] = rle["counts"].decode("utf-8") # Necessary to serialize with json return rle def batched_mask_to_box(masks): """ Convert masks to bounding boxes in XYXY format. Return [0,0,0,0] for an empty mask. Args: masks (np.ndarray): Binary masks of shape C1xC2x...xHxW. Returns: np.ndarray: Bounding boxes in XYXY format of shape C1xC2x...x4. """ # NumPy doesn't raise an error on empty inputs, so check for empty masks here if np.prod(masks.shape) == 0: return np.zeros((*masks.shape[:-2], 4), dtype=np.int32) # Normalize shape to CxHxW shape = masks.shape h, w = shape[-2:] if len(shape) > 2: masks = masks.reshape(-1, masks.shape[-2], masks.shape[-1]) else: masks = np.expand_dims(masks, 0) # Get top and bottom edges in_height = np.max(masks.astype(np.float32), axis=-1) in_height_coords = in_height * np.arange(h)[None, :] bottom_edges = np.max(in_height_coords, axis=-1) in_height_coords = in_height_coords + h * (~in_height.astype(np.bool)) top_edges = np.min(in_height_coords, axis=-1) # Get left and right edges in_width = np.max(masks.astype(np.float32), axis=-2) in_width_coords = in_width * np.arange(w)[None, :] right_edges = np.max(in_width_coords, axis=-1) in_width_coords = in_width_coords + w * (~in_width.astype(np.bool)) left_edges = np.min(in_width_coords, axis=-1) # If the mask is empty, the right edge will be to the left of the left edge. # Replace these boxes with [0, 0, 0, 0] empty_filter = ((right_edges < left_edges).astype(np.int32) |\ (bottom_edges < top_edges).astype(np.int32)).astype(np.bool) out = np.stack([left_edges, top_edges, right_edges, bottom_edges], axis=-1) out = out * np.expand_dims(~empty_filter, axis=-1) # Return to original shape if len(shape) > 2: out = out.reshape((*shape[:-2], 4)) else: out = out[0] return out.astype(np.int32) def box_area(boxes): """ Compute the area of a set of bounding boxes. Args: boxes (np.ndarray): Bounding boxes for which the area will be computed. Expected shape is [N, 4], where N is the number of boxes. Each box is specified by its coordinates (x1, y1, x2, y2) in the format with ``0 <= x1 < x2`` and ``0 <= y1 < y2``. Returns: np.ndarray: The area for each box in an ndarray of shape [N]. """ return (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1]) def box_iou(box1: np.ndarray, box2: np.ndarray): """ Calculate the Intersection over Union (IoU) between two sets of bounding boxes. Args: box1 (np.ndarray): NumPy array representing the first set of bounding boxes in format [x_min, y_min, x_max, y_max]. box2 (np.ndarray): NumPy array representing the second set of bounding boxes in the same format. Returns: iou (np.ndarray): NumPy array containing the IoU values for each pair of boxes. Example: >>> box1 = np.array([[0, 0, 10, 10], [5, 5, 15, 15]]) >>> box2 = np.array([[5, 5, 15, 15], [8, 8, 18, 18]]) >>> iou = box_iou(box1, box2) >>> print(iou) [[0.11111111 0. ] [1. 0.23076923]] """ area1 = box_area(box1) area2 = box_area(box2) lt = np.maximum(box1[:, np.newaxis, :2], box2[:, :2]) rb = np.minimum(box1[:, np.newaxis, 2:], box2[:, 2:]) wh = rb - lt wh = np.maximum(0, wh) inter = wh[:, :, 0] * wh[:, :, 1] iou = inter / (area1[:, np.newaxis] + area2 - inter) return iou def nms(boxes: np.ndarray, scores: np.ndarray, iou_threshold: float): """ Apply Non-Maximum Suppression (NMS) algorithm to filter a set of bounding boxes, eliminating overlapping boxes based on the Intersection over Union (IoU) threshold. Args: boxes (np.ndarray): NumPy array containing bounding box coordinates, with each row representing a box [x_min, y_min, x_max, y_max]. scores (np.ndarray): NumPy array containing scores for each corresponding bounding box. iou_threshold (float): IoU (Intersection over Union) threshold to determine if boxes overlap. Returns: keep (np.ndarray): NumPy array containing indices of the retained bounding boxes. Example: >>> boxes = np.array([[0, 0, 10, 10], [5, 5, 15, 15], [8, 8, 18, 18]]) >>> scores = np.array([0.9, 0.75, 0.85]) >>> iou_threshold = 0.5 >>> keep = nms(boxes, scores, iou_threshold) >>> print(keep) [0 2] """ idxs = scores.argsort() keep = [] while idxs.size > 0: max_score_index = idxs[-1] max_score_box = boxes[max_score_index][None, :] keep.append(max_score_index) if idxs.size == 1: break idxs = idxs[:-1] other_boxes = boxes[idxs] ious = box_iou(max_score_box, other_boxes) idxs = idxs[ious[0] <= iou_threshold] keep = np.array(keep) return keep pengcheng.mind-test/mindformers/models/swin/000077500000000000000000000000001471400134000215015ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/swin/__init__.py000066400000000000000000000015651471400134000236210ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Swin Model API.""" from .swin import * from .swin_config import * from .swin_processor import * __all__ = [] __all__.extend(swin.__all__) __all__.extend(swin_config.__all__) __all__.extend(swin_processor.__all__) pengcheng.mind-test/mindformers/models/swin/convert_weight.py000066400000000000000000000115251471400134000251060ustar00rootroot00000000000000"""Convert checkpoint from torch/MicroSoft""" import os import argparse import mindspore as ms import torch from mindformers.tools.utils import str2bool def convert_pretrained_weight(pth_file="swin_base_patch4_window7_224.pth", ms_ckpt_path="swin_base_p4w7.ckpt"): """ convert swin_base_p4w7 weights from pytorch to mindspore pytorch required. """ ms_ckpt = [] state_dict = torch.load(pth_file, map_location=torch.device('cpu')) for k, v in state_dict['model'].items(): if 'decoder.0' in k: k = k.replace('decoder.0', 'decoder') if 'patch_embed.' in k: k = k.replace('proj', 'projection') if 'relative_position' in k: k = k.replace('relative_position', 'relative_position_bias.relative_position') if 'norm' in k: if '.weight' in k: k = k.replace('.weight', '.gamma') if '.bias' in k: k = k.replace('.bias', '.beta') if '.mlp' in k: if '.fc1' in k: k = k.replace('.fc1', '.mapping') if "weight" in k: v = v.transpose(-1, 0) if '.fc2' in k: k = k.replace('.fc2', '.projection') if "weight" in k: v = v.transpose(-1, 0) if '.qkv' not in k: ms_ckpt.append({'name': k, 'data': ms.Tensor(v.numpy())}) else: data = ms.Tensor(v.numpy()) length = len(data) ms_ckpt.append({'name': k.replace('.qkv', '.q'), 'data': data[:length // 3]}) ms_ckpt.append({'name': k.replace('.qkv', '.k'), 'data': data[length // 3:length // 3 * 2]}) ms_ckpt.append({'name': k.replace('.qkv', '.v'), 'data': data[length // 3 * 2:]}) if not os.path.exists(ms_ckpt_path): try: ms.save_checkpoint(ms_ckpt, ms_ckpt_path) except: raise RuntimeError(f'Save checkpoint to {ms_ckpt_path} failed, please checkout the path.') def convert_finetuned_weight(pth_file="swin_base_patch4_window7_224.pth", ms_ckpt_path="swin_base_p4w7.ckpt"): """ convert swin_base_p4w7 weights from pytorch to mindspore pytorch required. """ ms_ckpt = [] state_dict = torch.load(pth_file, map_location=torch.device('cpu')) for k, v in state_dict['model'].items(): if 'head' in k: ms_ckpt.append({'name': k, 'data': ms.Tensor(v.numpy())}) continue if 'patch_embed.' in k: k = k.replace('proj', 'projection') if 'relative_position' in k: k = k.replace('relative_position', 'relative_position_bias.relative_position') if 'norm' in k: if '.weight' in k: k = k.replace('.weight', '.gamma') if '.bias' in k: k = k.replace('.bias', '.beta') if '.mlp' in k: if '.fc1' in k: k = k.replace('.fc1', '.mapping') if "weight" in k: v = v.transpose(-1, 0) if '.fc2' in k: k = k.replace('.fc2', '.projection') if "weight" in k: v = v.transpose(-1, 0) if '.qkv' not in k: ms_ckpt.append({'name': 'encoder.' + k, 'data': ms.Tensor(v.numpy())}) else: data = ms.Tensor(v.numpy()) length = len(data) ms_ckpt.append({'name': 'encoder.' + k.replace('.qkv', '.q'), 'data': data[:length // 3]}) ms_ckpt.append( {'name': 'encoder.' + k.replace('.qkv', '.k'), 'data': data[length // 3:length // 3 * 2]}) ms_ckpt.append({'name': 'encoder.' + k.replace('.qkv', '.v'), 'data': data[length // 3 * 2:]}) if not os.path.exists(ms_ckpt_path): try: ms.save_checkpoint(ms_ckpt, ms_ckpt_path) except: raise RuntimeError(f'Save checkpoint to {ms_ckpt_path} failed, please checkout the path.') if __name__ == '__main__': parser = argparse.ArgumentParser(description="swin weight convert script") parser.add_argument("--torch_path", type=str, default="swin_base_patch4_window7_224.pth", required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="swin_base_p4w7.ckpt", help="The output mindspore checkpoint path.") parser.add_argument("--is_pretrain", type=str2bool, default=False, help="whether converting pretrained weights.") opt = parser.parse_args() if opt.is_pretrain: convert_pretrained_weight(opt.torch_path, opt.mindspore_path) else: convert_finetuned_weight(opt.torch_path, opt.mindspore_path) pengcheng.mind-test/mindformers/models/swin/swin.py000066400000000000000000000373131471400134000230420ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/microsoft/Swin-Transformer # ============================================================================ """Swin Model.""" import os import numpy as np from scipy import interpolate from mindspore import nn from mindspore import Tensor from mindspore import Parameter from mindspore import dtype as mstype import mindspore.ops.operations as P import mindspore.common.initializer as weight_init_ from mindspore.train.serialization import load_checkpoint from mindspore.train.serialization import load_param_into_net from mindformers.tools.logger import logger from mindformers.tools.download_tools import download_with_progress_bar from mindformers.core.loss import build_loss from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from mindformers.models.base_config import BaseConfig from mindformers.models.swin.swin_config import SwinConfig from mindformers.models.swin.swin_modules import Linear from mindformers.models.swin.swin_modules import LayerNorm from mindformers.models.swin.swin_modules import Dropout from mindformers.models.swin.swin_modules import SwinPatchEmbeddings from mindformers.models.swin.swin_modules import SwinPatchMerging from mindformers.models.swin.swin_modules import SwinStage from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.utils import try_sync_file __all__ = ['SwinForImageClassification', 'SwinModel'] @MindFormerRegister.register(MindFormerModuleType.ENCODER) class SwinBaseModel(BaseModel): """ An abstract class to handle weights initialization and save weights decay grouping. """ def init_weights(self): """ Swin weight initialization, original timm impl (for reproducibility) """ for _, cell in self.cells_and_names(): if isinstance(cell, Linear): cell.weight.set_data(weight_init_.initializer( weight_init_.TruncatedNormal(sigma=0.02), cell.weight.shape, cell.weight.dtype)) if isinstance(cell, Linear) and cell.bias is not None: cell.bias.set_data(weight_init_.initializer(weight_init_.Zero(), cell.bias.shape, cell.bias.dtype)) elif isinstance(cell, (LayerNorm, nn.LayerNorm)): cell.gamma.set_data(weight_init_.initializer(weight_init_.One(), cell.gamma.shape, cell.gamma.dtype)) cell.beta.set_data(weight_init_.initializer(weight_init_.Zero(), cell.beta.shape, cell.beta.dtype)) @staticmethod def no_weight_decay(): return {'absolute_pos_embed'} @staticmethod def no_weight_decay_keywords(): return {'relative_position_bias_table'} @MindFormerRegister.register(MindFormerModuleType.ENCODER) class SwinModel(SwinBaseModel): """ Swin Transformer. The supported model name could be selected from SwinForImageClassification.show_support_list(). Args: config (SwinConfig): the config of Swin model. """ def __init__(self, config: BaseConfig = None): if config is None: config = SwinConfig() super(SwinModel, self).__init__(config) dp = config.parallel_config.data_parallel self.parallel_config = config.parallel_config self.use_moe = config.moe_config.expert_num > 1 self.num_labels = config.num_labels self.num_layers = len(config.depths) self.embed_dim = config.embed_dim self.ape = config.use_absolute_embeddings self.num_channels = config.num_channels self.patch_size = config.patch_size self.patch_norm = config.patch_norm self.num_features = int(config.embed_dim * 2 ** (self.num_layers - 1)) self.mlp_ratio = config.mlp_ratio self.cast = P.Cast() # split image into non-overlapping patches self.patch_embed = SwinPatchEmbeddings(config) num_patches = self.patch_embed.num_patches self.num_patches = num_patches patches_resolution = self.patch_embed.grid_size self.patches_resolution = patches_resolution # absolute position embedding if self.ape: self.absolute_pos_embed = Parameter( Tensor(np.zeros((1, num_patches, config.embed_dim)), dtype=mstype.float32), name="ape") self.pos_drop = Dropout(keep_prob=1.0 - config.hidden_dropout_prob) self.pos_drop.shard(((dp, 1, 1),)) # stochastic depth dpr = list(np.linspace(0, config.drop_path_rate, sum(config.depths))) # stochastic depth decay rule if self.use_moe: parallel_config_args = config.parallel_config.moe_parallel_config else: parallel_config_args = config.parallel_config.dp_mp_config # build layers self.layers = nn.CellList() self.final_seq = num_patches # downsample seq_length for i_layer in range(self.num_layers): layer = SwinStage( config=config, dim=int(config.embed_dim * 2 ** i_layer), input_resolution=(patches_resolution[0] // (2 ** i_layer), patches_resolution[1] // (2 ** i_layer)), depth=config.depths[i_layer], num_heads=config.num_heads[i_layer], drop_path=dpr[sum(config.depths[:i_layer]):sum(config.depths[:i_layer + 1])], norm_layer=LayerNorm, downsample=SwinPatchMerging if (i_layer < self.num_layers - 1) else None, parallel_config=parallel_config_args) # downsample seq_length if i_layer < self.num_layers - 1: self.final_seq = self.final_seq // 4 self.layers.append(layer) self.norm = LayerNorm([self.num_features,], eps=config.layer_norm_eps).shard(((dp, 1, 1),)) self.transpose = P.Transpose().shard(((dp, 1, 1),)) self.avgpool = P.ReduceMean(keep_dims=False).shard(((dp, 1, 1),)) self.init_weights() def construct(self, x): x = self.patch_embed(x) if self.ape: x = x + self.absolute_pos_embed x = self.pos_drop(x) for layer in self.layers: x = layer(x) x = self.norm(x) # B L C x = self.avgpool(self.transpose(x, (0, 2, 1)), 2) # B C 1 return x @MindFormerRegister.register(MindFormerModuleType.MODELS) class SwinForImageClassification(SwinBaseModel): """ Swin Transformer Model. The supported model name could be selected from SwinForImageClassification.show_support_list(). Args: config (SwinConfig): the config of Swin model. Examples: >>> # input model name, load model and weights >>> model_a = SwinForImageClassification.from_pretrained('swin_base_p4w7') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('swin_base_p4w7') >>> model_b = SwinForImageClassification(config) """ _support_list = MindFormerBook.get_model_support_list()['swin'] def __init__(self, config: BaseConfig = None): if config is None: config = SwinConfig() super(SwinForImageClassification, self).__init__(config) self.encoder = SwinModel(config) parallel_config = config.parallel_config dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.head = Linear( self.encoder.num_features, self.encoder.num_labels, weight_init=weight_init_.TruncatedNormal(sigma=2e-5), compute_dtype=mstype.float32).to_float(mstype.float32) self.head.shard(strategy_bias=((dp, mp), (mp,)), strategy_matmul=((dp, 1), (mp, 1))) self.loss = build_loss(class_name=config.loss_type) self.load_checkpoint(config) def construct(self, image, target=None): x = self.encoder(image) out = self.head(x) if not self.training: return out, target loss = self.loss(out, target) return loss def load_checkpoint(self, config): """ load checkpoint for SwinForImageClassification. Args: config (ModelConfig): a model config instance, which could have attribute "checkpoint_name_or_path (str)". set checkpoint_name_or_path to a supported model name or a path to checkpoint, to load model weights. """ checkpoint_name_or_path = config.checkpoint_name_or_path if checkpoint_name_or_path: if not isinstance(checkpoint_name_or_path, str): raise TypeError(f"checkpoint_name_or_path should be a str," f" but got {type(checkpoint_name_or_path)}") if os.path.exists(checkpoint_name_or_path): param = load_checkpoint(checkpoint_name_or_path) ckpt_file = checkpoint_name_or_path param = self.remap_pretrained_keys_swin(param) try: load_param_into_net(self, param) except RuntimeError: logger.error("the given config and weights in %s are" " mismatched, and weights load failed", ckpt_file) logger.info("weights in %s are loaded", ckpt_file) elif checkpoint_name_or_path not in self._support_list: raise ValueError(f"{checkpoint_name_or_path} is not a supported default model" f" or a valid path to checkpoint," f" please select from {self._support_list}.") else: default_checkpoint_download_folder = os.path.join( MindFormerBook.get_default_checkpoint_download_folder(), checkpoint_name_or_path.split("_")[0]) if not os.path.exists(default_checkpoint_download_folder): os.makedirs(default_checkpoint_download_folder, exist_ok=True) ckpt_file = os.path.join(default_checkpoint_download_folder, checkpoint_name_or_path + ".ckpt") if not os.path.exists(ckpt_file): url = MindFormerBook.get_model_ckpt_url_list()[checkpoint_name_or_path][0] succeed = download_with_progress_bar(url, ckpt_file) if not succeed: logger.info("checkpoint download failed, and pretrained weights are unloaded.") return try_sync_file(ckpt_file) logger.info("start to read the ckpt file: %s", os.path.getsize(ckpt_file)) param = load_checkpoint(ckpt_file) try: load_param_into_net(self, param) except RuntimeError: logger.error("the given config and weights in %s are" " mismatched, and weights load failed", ckpt_file) logger.info("weights in %s are loaded", ckpt_file) else: logger.info("model built, but weights is unloaded, since the config has no" " checkpoint_name_or_path attribute or" " checkpoint_name_or_path is None.") def remap_pretrained_keys_swin(self, checkpoint_model): # This class was refer to project: # https://github.com/microsoft/unilm/blob/master/beit/run_class_finetuning.py """remap pretrained keys swin""" # Geometric interpolation when pre-trained patch size mismatch with fine-tuned patch size state_dict = self.parameters_dict() all_keys = list(checkpoint_model.keys()) for key in all_keys: if "relative_position_bias_table" in key and "adam" not in key: relative_position_bias_table_pretrained = checkpoint_model[key] relative_position_bias_table_current = state_dict[key] l1, n_h1 = relative_position_bias_table_pretrained.shape l2, n_h2 = relative_position_bias_table_current.shape if n_h1 != n_h2: logger.info("Error in loading %s, passing......", key) else: if l1 != l2: logger.info("%s: Interpolate relative_position_bias_table using geo.", key) src_size = int(l1 ** 0.5) dst_size = int(l2 ** 0.5) def geometric_progression(a, r, n): return a * (1.0 - r ** n) / (1.0 - r) left, right = 1.01, 1.5 while right - left > 1e-6: q = (left + right) / 2.0 gp = geometric_progression(1, q, src_size // 2) if gp > dst_size // 2: right = q else: left = q dis = [] cur = 1 for i in range(src_size // 2): dis.append(cur) cur += q ** (i + 1) r_ids = [-_ for _ in reversed(dis)] x = r_ids + [0] + dis y = r_ids + [0] + dis t = dst_size // 2.0 dx = np.arange(-t, t + 0.1, 1.0) dy = np.arange(-t, t + 0.1, 1.0) str_x = str(x) str_dx = str(dx) logger.info("Original positions = %s", str_x) logger.info("Target positions = %s", str_dx) all_rel_pos_bias = [] for i in range(n_h1): z = relative_position_bias_table_pretrained[:, i].view( src_size, src_size).asnumpy().astype(np.float32) f_cubic = interpolate.interp2d(x, y, z, kind='cubic') all_rel_pos_bias.append(Tensor(f_cubic(dx, dy), mstype.float32).view(-1, 1)) new_rel_pos_bias = P.Concat(axis=-1)(tuple(all_rel_pos_bias)) new_rel_pos_bias = Parameter(new_rel_pos_bias) checkpoint_model[key] = new_rel_pos_bias # delete relative_coords_table and attn_mask, since we always re-init it checkpoint_model = self._del_unused_keys(checkpoint_model, "relative_position_index") checkpoint_model = self._del_unused_keys(checkpoint_model, "relative_coords_table") checkpoint_model = self._del_unused_keys(checkpoint_model, "attn_mask") return checkpoint_model @staticmethod def _del_unused_keys(checkpoint_model, key_name): unwanted_key = [k for k in checkpoint_model.keys() if key_name in k] for k in unwanted_key: del checkpoint_model[k] return checkpoint_model pengcheng.mind-test/mindformers/models/swin/swin_config.py000066400000000000000000000135421471400134000243650ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writingsoftware # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Swin Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer.moe import MoEConfig, default_moe_config from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_config import BaseConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['SwinConfig'] default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class SwinConfig(BaseConfig): """ Swin config class which defines the model size Args: image_size: The input image size, Default 224. patch_size: patch size, Default 4. num_channels: channels of input images, Default 3. embed_dim: embedding dimension, Default 128. depths: number of transformer blocks for each swin layer, Default (2, 2, 18, 2). num_heads: number of attention heads for each swin layer, Default (4, 8, 16, 32). window_size: window size for swin, Default 7. shift_size: window shift size, Default 0. mlp_ratio: ffn_hidden_size = mlp_ratio * embed_dim, Default 4. qkv_bias: has transformer qkv bias or not, Default True. hidden_dropout_prob: drop rate of MLP, Default 0. attention_probs_dropout_prob: drop rate of Attention, Default 0. drop_path_rate: drop path rate of transformer blocks, Default 0.1. use_absolute_embeddings: if using absolute position embedding, Default False. patch_norm: use norm in SwinPatchEmbeddings, Default True. hidden_act: activation of MLP, Default "gelu". weight_init: weight initialize type, Default "normal". num_labels: number of labels in downstream tasks, Default 1000. loss_type: loss type, Default "SoftTargetCrossEntropy". param_init_type:, Default mstype.float32. moe_config:, Default default_moe_config. parallel_config:, Default default_parallel_config. checkpoint_name_or_path:, Default "swin_base_p4w7". **kwargs Examples: >>> # init a config with a model name >>> config_a = SwinConfig.from_pretrained('swin_base_p4w7') >>> # init a config with a config path >>> import os >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), >>> 'configs', 'swin', 'run_swin_base_p4w7_224_100ep.yaml') >>> config_b = SwinConfig.from_pretrained(config_path) >>> # init a config with args >>> config_c = SwinConfig( >>> patch_size=4, >>> in_chans=3, >>> ... >>> ) """ _support_list = MindFormerBook.get_config_support_list()['swin'] def __init__(self, image_size: int = 224, patch_size: int = 4, num_channels: int = 3, embed_dim: int = 128, depths: list = (2, 2, 18, 2), num_heads: list = (4, 8, 16, 32), window_size: int = 7, shift_size: int = 0, mlp_ratio: float = 4., qkv_bias: bool = True, layer_norm_eps: float = 1e-5, hidden_dropout_prob: float = 0., attention_probs_dropout_prob: float = 0., drop_path_rate: float = 0.1, use_absolute_embeddings: bool = False, patch_norm: bool = True, hidden_act: str = 'gelu', weight_init: str = 'normal', num_labels: int = 1000, loss_type: str = "SoftTargetCrossEntropy", param_init_type: mstype = mstype.float32, moe_config: MoEConfig = default_moe_config, parallel_config: TransformerOpParallelConfig = default_parallel_config, checkpoint_name_or_path: str = '', **kwargs): """Swin Base Config""" self.image_size = image_size self.patch_size = patch_size self.num_channels = num_channels self.embed_dim = embed_dim self.depths = depths self.num_heads = num_heads self.window_size = window_size self.shift_size = shift_size self.mlp_ratio = mlp_ratio self.qkv_bias = qkv_bias self.layer_norm_eps = layer_norm_eps self.hidden_dropout_prob = hidden_dropout_prob self.attention_probs_dropout_prob = attention_probs_dropout_prob self.drop_path_rate = drop_path_rate self.use_absolute_embeddings = use_absolute_embeddings self.patch_norm = patch_norm self.hidden_act = hidden_act self.weight_init = weight_init self.num_labels = num_labels self.loss_type = loss_type self.param_init_type = param_init_type self.moe_config = moe_config self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path super(SwinConfig, self).__init__(**kwargs) pengcheng.mind-test/mindformers/models/swin/swin_modules.py000066400000000000000000000746751471400134000246060ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/microsoft/Swin-Transformer # ============================================================================ """Swin Transformer API.""" import collections.abc from itertools import repeat import numpy as np from mindspore import nn from mindspore import numpy from mindspore import context from mindspore import Tensor from mindspore import Parameter from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.ops.primitive import constexpr import mindspore.common.dtype as mstype import mindspore.common.initializer as weight_init_ from mindformers.modules import layers, FeedForward from mindformers.modules.transformer.op_parallel_config import default_dpmp_config from mindformers.models.base_model import BaseModel # utils def _ntuple(n): def parse(x): if isinstance(x, collections.abc.Iterable): return x return tuple(repeat(x, n)) return parse to_2tuple = _ntuple(2) @constexpr def gen_shape(x_shape, ndim): """generate shape for x.""" return (x_shape,) + (1,) * (ndim + 1) class LayerNorm(layers.LayerNorm): # pylint: disable=W0212 """ A self-defined layer norm operation using reduce sum and reduce mean. """ def __init__(self, normalized_shape, eps=1e-5, param_init_type=mstype.float32): super(LayerNorm, self).__init__( normalized_shape, eps=eps, param_init_type=param_init_type) class Linear(layers.Linear): # pylint: disable=W0212 """ Linear function for Swin. """ def __init__(self, in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, transpose_b=True, expert_num=1, outer_batch=1, activation_compute_type=mstype.float16, param_init_type=mstype.float32, compute_dtype=mstype.float16): super(Linear, self).__init__( in_channels, out_channels, weight_init=weight_init, bias_init=bias_init, has_bias=has_bias, activation=activation, transpose_b=transpose_b, expert_num=expert_num, outer_batch=outer_batch, param_init_type=param_init_type, compute_dtype=compute_dtype) self.activation_compute_type = activation_compute_type def construct(self, x): """construct of linear.""" out_shape = P.Shape()(x)[:-1] + (self.out_channels,) x = P.Reshape()(x, (-1, self.in_channels)) if self.expert_flag: x = P.Reshape()(x, (self.outer_batch, self.expert_num, -1, self.in_channels)) weight = self.cast(self.weight, self.dtype) x = self.matmul(x, weight) if self.has_bias: x = self.bias_add(x, self.cast(self.bias, self.dtype)) if self.activation_flag: x = self.activation(self.cast(x, self.activation_compute_type)) x = self.cast(x, self.dtype) output = P.Reshape()(x, out_shape) return output class Identity(nn.Cell): """No construct for x.""" def __init__(self, auto_prefix=True): super(Identity, self).__init__(auto_prefix=auto_prefix) def construct(self, x): """No construct.""" return x class Dropout(layers.Dropout): # pylint: disable=W0212 """ A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for context training. """ def __init__(self, keep_prob=0.5, dtype=mstype.float32): super(Dropout, self).__init__(keep_prob=keep_prob, dtype=dtype) class DropPath(nn.Cell): """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" def __init__(self, drop_prob, ndim=1, parallel_config=None): # pylint: disable=W0613 super(DropPath, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.drop = Dropout(keep_prob=1 - drop_prob) self.drop.shard(((1, 1, 1),)) shape = (1,) + (1,) * (ndim + 1) self.ndim = ndim self.mask = Tensor(np.ones(shape), dtype=mstype.float32) self.tile = P.Tile().shard(((1, 1, 1),)) self.mul = P.Mul().shard(((dp, 1, 1),)) def construct(self, x): """Dropout.""" if not self.training: return x shape = gen_shape(x.shape[0], self.ndim) mask = self.tile(self.mask, shape) out = self.drop(mask) out = self.mul(out, x) return out class SwinPatchEmbeddings(BaseModel): """Construct the embeddings from patch, position embeddings.""" def __init__(self, config): super(SwinPatchEmbeddings, self).__init__(config) if config.parallel_config: dp = config.parallel_config.data_parallel else: dp = 1 img_size = (config.image_size, config.image_size) patch_size = (config.patch_size, config.patch_size) self.grid_size = (img_size[0] // patch_size[0], img_size[1] // patch_size[1]) self.num_patches = self.grid_size[0] * self.grid_size[1] self.projection = nn.Conv2d(in_channels=config.num_channels, out_channels=config.embed_dim, kernel_size=patch_size[0], stride=patch_size[0], weight_init=weight_init_.TruncatedNormal(sigma=0.02), has_bias=True, pad_mode='pad') self.projection.conv2d.shard(((dp, 1, 1, 1), (1, 1, 1, 1))) self.projection.bias_add.shard(((dp, 1, 1, 1), (1,))) self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1),)) # usually not use norm if config.patch_norm: self.norm = LayerNorm((config.embed_dim,), eps=config.layer_norm_eps).shard(((dp, 1, 1),)) else: self.norm = Identity() def construct(self, x): """construct""" # True x: bs False x: bs * dp x = self.projection(x) b, c, h, w = x.shape x = self.reshape(x, (b, c, h * w)) x = self.transpose(x, (0, 2, 1)) x = self.norm(x) return x class SwinStage(BaseModel): """ Swin Basic Layer """ def __init__(self, config, dim, input_resolution, depth, num_heads, drop_path, norm_layer=LayerNorm, downsample=None, parallel_config=default_dpmp_config): super(SwinStage, self).__init__(config) self.dim = dim self.input_resolution = input_resolution self.depth = depth # build blocks self.blocks = nn.CellList([ SwinTransformerBlock(config=config, dim=dim, input_resolution=input_resolution, num_heads=num_heads, shift_size=0 if (i % 2 == 0) else config.window_size // 2, drop_path=drop_path[i] if isinstance(drop_path, list) else drop_path, norm_layer=norm_layer, parallel_config=parallel_config) for i in range(depth)]) # patch merging layer if downsample is not None: self.downsample = downsample(dim, input_resolution, config.weight_init, norm_layer=norm_layer, parallel_config=parallel_config) else: self.downsample = None def construct(self, x): """construct""" for blk in self.blocks: x = blk(x) if self.downsample is not None: x = self.downsample(x) return x def extra_repr(self) -> str: return f"dim={self.dim}, input_resolution={self.input_resolution}, depth={self.depth}" class SwinTransformerBlock(BaseModel): """ Swin Transformer Block. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input resolution. num_heads (int): Number of attention heads. shift_size (int): Shift size for SW-MSA. drop_path (float, optional): Stochastic depth rate. Default: 0.0 norm_layer (nn.Cell, optional): Normalization layer. Default: nn.LayerNorm/LayerNorm """ def __init__(self, config, dim, input_resolution, num_heads, shift_size, drop_path, norm_layer=LayerNorm, parallel_config=default_dpmp_config): super(SwinTransformerBlock, self).__init__(config) self.use_moe = (config.moe_config.expert_num > 1) if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.dim = dim self.input_resolution = input_resolution self.num_heads = num_heads self.window_size = config.window_size self.shift_size = shift_size self.mlp_ratio = config.mlp_ratio self.reshape = P.Reshape() if min(self.input_resolution) <= self.window_size: # if window size is larger than input resolution, we don't partition windows self.shift_size = 0 self.window_size = min(self.input_resolution) self.norm1 = norm_layer((dim,), eps=config.layer_norm_eps) self.norm1.shard(((dp, 1, 1),)) self.attn = SwinAttention(dim, window_size=to_2tuple(self.window_size), num_heads=num_heads, qkv_bias=config.qkv_bias, attn_drop=config.attention_probs_dropout_prob, proj_drop=config.hidden_dropout_prob) self.drop_path = DropPath(drop_path, ndim=1, parallel_config=parallel_config) \ if drop_path > 0. else Identity() self.norm2 = norm_layer((dim,), eps=config.layer_norm_eps) self.norm2.shard(((dp, 1, 1),)) self.mlp = SwinIntermediate(hidden_size=dim, ffn_hidden_size=int(dim * config.mlp_ratio), dropout_rate=config.hidden_dropout_prob, weight_init=config.weight_init, hidden_act=config.hidden_act, parallel_config=parallel_config) if self.shift_size > 0: # calculate attention mask for SW-MSA in_h, in_w = self.input_resolution img_mask = np.zeros((1, in_h, in_w, 1)) # 1 H W 1 h_slices = (slice(0, -self.window_size), slice(-self.window_size, -self.shift_size), slice(-self.shift_size, None)) w_slices = (slice(0, -self.window_size), slice(-self.window_size, -self.shift_size), slice(-self.shift_size, None)) cnt = 0 for h in h_slices: for w in w_slices: img_mask[:, h, w, :] = cnt cnt += 1 # img_mask: [1, 56, 56, 1] window_size: 7 mask_windows = window_partition(img_mask, self.window_size) # nW, window_size, window_size, 1 mask_windows = mask_windows.reshape(-1, self.window_size * self.window_size) # mask: [64, 49, 49] attn_mask = mask_windows[:, np.newaxis] - mask_windows[:, :, np.newaxis] attn_mask = Tensor(np.where(attn_mask == 0, 0., -100.), dtype=mstype.float32) self.attn_mask = Parameter(attn_mask, requires_grad=False, name="attention_mask") self.roll_pos = Roll(self.shift_size, parallel_config=parallel_config) self.roll_neg = Roll(-self.shift_size, parallel_config=parallel_config) else: self.attn_mask = None self.dtype = P.DType() self.add_3d = P.Add().shard(((dp, 1, 1), (dp, 1, 1))) self.window_partition = WindowPartition(self.window_size) self.window_reverse = WindowReverse() def construct(self, x): """construct function""" h, w = self.input_resolution b, _, c = x.shape ori_type = self.dtype(x) shortcut = x x = self.norm1(x) x = self.reshape(x, (b, h, w, c)) # cyclic shift if self.shift_size > 0: shifted_x = self.roll_neg(x) else: shifted_x = x # partition windows x_windows = self.window_partition(shifted_x) # nW*B, window_size, window_size, C x_windows = self.reshape(x_windows, (-1, self.window_size * self.window_size, c)) # nW*B, window_size*window_size, C # W-MSA/SW-MSA attn_windows = self.attn(x_windows, mask=self.attn_mask) # nW*B, window_size*window_size, C # merge windows attn_windows = self.reshape(attn_windows, (-1, self.window_size, self.window_size, c)) shifted_x = self.window_reverse(attn_windows, self.window_size, h, w) # B H' W' C # reverse cyclic shift if self.shift_size > 0: x = self.roll_pos(shifted_x) else: x = shifted_x x = self.reshape(x, (b, h * w, c)) # FFN x = self.drop_path(x) x = self.cast(x, ori_type) x = self.add_3d(shortcut, x) x_tmp = self.norm2(x) x_tmp = self.mlp(x_tmp) x_tmp = self.drop_path(x_tmp) output = self.add_3d(x, x_tmp) output = self.cast(output, ori_type) return output def extra_repr(self) -> str: return f"dim={self.dim}, input_resolution={self.input_resolution}, num_heads={self.num_heads}, " \ f"window_size={self.window_size}, shift_size={self.shift_size}, mlp_ratio={self.mlp_ratio}" class SwinPatchMerging(nn.Cell): """ Patch Merging Layer. Args: input_resolution (tuple[int]): Resolution of input feature. dim (int): Number of input channels. norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm/LayerNorm """ def __init__(self, dim, input_resolution, weight_init='normal', norm_layer=LayerNorm, parallel_config=default_dpmp_config): super(SwinPatchMerging, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.input_resolution = input_resolution self.dim = dim[0] if isinstance(dim, tuple) and len(dim) == 1 else dim # Default False self.reduction = Linear(in_channels=4 * dim, out_channels=2 * dim, has_bias=False, weight_init=weight_init).to_float(mstype.float16) self.reduction.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.norm = norm_layer([dim * 4,], eps=1e-5) self.norm.shard(((dp, 1, 1),)) self.h, self.w = self.input_resolution self.h_2, self.w_2 = self.h // 2, self.w // 2 self.h2w2 = int(self.h * self.w // 4) self.dim_mul_4 = int(dim * 4) self.cast = P.Cast() self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, x): """ x: B, H*W, C """ b = x.shape[0] x = self.reshape(x, (b, self.h_2, 2, self.w_2, 2, self.dim)) x = self.transpose(x, (0, 1, 3, 4, 2, 5)) x = self.reshape(x, (b, self.h2w2, self.dim_mul_4)) x = self.norm(x) x = self.cast(x, mstype.float16) x = self.reduction(x) x = self.cast(x, mstype.float32) return x def extra_repr(self) -> str: return f"input_resolution={self.input_resolution}, dim={self.dim}" class SwinAttention(nn.Cell): r""" Window based multi-head self attention (W-MSA) Cell with relative position bias. It supports both of shifted and non-shifted window. Args: dim (int): Number of input channels. window_size (tuple[int]): The height and width of the window. num_heads (int): Number of attention heads. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0 proj_drop (float, optional): Dropout ratio of output. Default: 0.0 """ def __init__(self, dim, window_size, num_heads, qkv_bias=True, qk_scale=None, attn_drop=0., proj_drop=0., compute_dtype=mstype.float16, param_init_type=mstype.float32, softmax_compute_type=mstype.float32, parallel_config=default_dpmp_config): super(SwinAttention, self).__init__() if isinstance(dim, tuple) and len(dim) == 1: dim = dim[0] if parallel_config: dp = parallel_config.data_parallel mp = parallel_config.model_parallel else: dp = mp = 1 self._is_ascend = context.get_context('device_target') in ["Ascend"] self.dim = dim self.window_size = window_size # Wh, Ww self.num_heads = num_heads head_dim = dim // num_heads self.scale_factor = Tensor(qk_scale or head_dim ** -0.5, mstype.float32) self.relative_position_bias = SwinRelativePositionBias(self.window_size, num_heads) # get pair-wise relative position index for each token inside the window self.q = Linear( in_channels=dim, out_channels=dim, has_bias=qkv_bias, param_init_type=param_init_type).to_float(compute_dtype) self.q.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.k = Linear( in_channels=dim, out_channels=dim, has_bias=qkv_bias, param_init_type=param_init_type).to_float(compute_dtype) self.k.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.v = Linear( in_channels=dim, out_channels=dim, has_bias=qkv_bias, param_init_type=param_init_type).to_float(compute_dtype) self.v.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.attn_drop = Dropout(keep_prob=1.0 - attn_drop) self.proj = Linear( in_channels=dim, out_channels=dim, has_bias=True, param_init_type=param_init_type).to_float(compute_dtype) self.proj.shard(strategy_bias=((dp, 1), (1,)), strategy_matmul=((dp, mp), (mp, 1))) self.proj_drop = Dropout(keep_prob=1.0 - proj_drop) self.proj_drop.shard(((dp, mp, 1, 1),)) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.cast = P.Cast() self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1),)) self.matmul = P.BatchMatMul().shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.real_div = P.RealDiv().shard(((dp, mp, 1, 1), ())) self.sub = P.Sub().shard(((1,), (dp, 1, 1, 1))) self.mul = P.Mul().shard(((dp, 1, 1, 1), (1,))) self.add_4d = P.Add().shard(((dp, 1, 1, 1), (1, 1, 1, 1))) self.add_5d = P.Add().shard(((dp, 1, 1, 1, 1), (dp, 1, 1, 1, 1))) self.dtype = P.DType() self.shape = P.Shape() self.compute_type = compute_dtype self.softmax_dtype = softmax_compute_type def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ attention_probs = self.softmax(attention_scores) return attention_probs def construct(self, x, mask=None): """ Args: x: input features with shape of (num_windows*B, N, C) mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None """ b, seq, c = x.shape ori_type = self.dtype(x) x = self.cast(x, self.compute_type) q = self.q(x) k = self.k(x) v = self.v(x) q = self.transpose(self.reshape(q, (b, seq, self.num_heads, c // self.num_heads)), (0, 2, 1, 3)) k = self.transpose(self.reshape(k, (b, seq, self.num_heads, c // self.num_heads)), (0, 2, 3, 1)) v = self.transpose(self.reshape(v, (b, seq, self.num_heads, c // self.num_heads)), (0, 2, 1, 3)) factor = self.cast(self.scale_factor, self.compute_type) q = self.mul(q, factor) attn = self.matmul(q, k) attn = self.cast(attn, ori_type) attn = self.add_4d(attn, self.relative_position_bias()) if mask is not None: nw, ws2, _ = mask.shape # mask: [64, 49, 49] ==> [1, 64, 1, 49, 49] mask = self.reshape(mask, (1, nw, 1, ws2, ws2)) attn = self.reshape(attn, (b // nw, nw, self.num_heads, seq, seq)) attn = self.add_5d(attn, mask) attn = self.reshape(attn, (-1, self.num_heads, seq, seq)) attn = self._softmax(attn) else: attn = self._softmax(attn) attn = self.cast(attn, self.compute_type) attn = self.attn_drop(attn) x = self.matmul(attn, v) x = self.transpose(x, (0, 2, 1, 3)) x = self.reshape(x, (b, seq, c)) x = self.proj(x) x = self.proj_drop(x) return x def extra_repr(self) -> str: return f'dim={self.dim}, window_size={self.window_size}, num_heads={self.num_heads}' class SwinIntermediate(FeedForward): """MLP for Swin Transformer Blocks.""" def __init__(self, hidden_size, ffn_hidden_size=None, out_features=None, dropout_rate=0., hidden_act='gelu', weight_init='normal', use_dropout=True, param_init_type=mstype.float32, parallel_config=default_dpmp_config): ffn_hidden_size = ffn_hidden_size or hidden_size super(SwinIntermediate, self).__init__( hidden_size, ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, param_init_type=mstype.float32, parallel_config=parallel_config) mp = parallel_config.model_parallel dp = parallel_config.data_parallel out_features = out_features or hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=hidden_size, out_channels=ffn_hidden_size, activation=hidden_act, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=ffn_hidden_size, out_channels=out_features, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - dropout_rate) self.dropout.shard(((dp, 1),)) self.dropout_3d = Dropout(1 - dropout_rate) self.dropout_3d.shard(((dp, 1, 1),)) self.use_dropout = use_dropout def construct(self, x): """construct of mlp""" x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) if self.use_dropout: if len(F.shape(hidden)) == 3: hidden = self.dropout_3d(hidden) elif len(F.shape(hidden)) == 2: hidden = self.dropout(hidden) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) return output class SwinRelativePositionBias(nn.Cell): """relative position bias for swin""" def __init__(self, window_size, num_heads): super(SwinRelativePositionBias, self).__init__() self.window_size = window_size # cls to token & token to cls & cls to cls self.num_relative_distance = (2 * window_size[0] - 1) * (2 * window_size[1] - 1) self.relative_position_bias_table = Parameter( weight_init_.initializer( weight_init_.TruncatedNormal(sigma=.02), (self.num_relative_distance, num_heads)), name='relative_position_bias_table') # get pair-wise relative position index for each token inside the window coords_h = Tensor(np.arange(window_size[0]), mstype.int32) coords_w = Tensor(np.arange(window_size[1]), mstype.int32) coords = P.Stack(axis=0)(P.Meshgrid(indexing='ij')((coords_h, coords_w))) # 2, Wh, Ww coords_flatten = P.Flatten()(coords) # 2, Wh*Ww relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww relative_coords = P.Transpose()(relative_coords, (1, 2, 0)).asnumpy() # Wh*Ww, Wh*Ww, 2 relative_coords[:, :, 0] += window_size[0] - 1 relative_coords[:, :, 1] += window_size[1] - 1 relative_coords[:, :, 0] *= 2 * window_size[1] - 1 relative_position_index = Tensor(np.sum(relative_coords, axis=-1), mstype.int32) # Wh*Ww, Wh*Ww self.relative_position_index = Parameter( relative_position_index, requires_grad=False, name="relative_position_index") self.reshape = P.Reshape() self.transpose = P.Transpose() self.expand_dim = P.ExpandDims() self.gather = P.Gather() def construct(self): relative_position_index = self.relative_position_index.view(-1) relative_position_bias = self.gather(self.relative_position_bias_table, relative_position_index, 0) relative_position_bias = self.reshape( relative_position_bias, (self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1)) relative_position_bias = self.transpose(relative_position_bias, (2, 0, 1)) relative_position_bias = self.expand_dim(relative_position_bias, 0) return relative_position_bias def window_partition(x, window_size): """ Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C) """ b, h, w, c = x.shape x = np.reshape(x, (b, h // window_size, window_size, w // window_size, window_size, c)) windows = x.transpose(0, 1, 3, 2, 4, 5).reshape(-1, window_size, window_size, c) return windows class Roll(nn.Cell): """Roll Cell""" def __init__(self, shift_size, shift_axis=(1, 2), parallel_config=default_dpmp_config): # pylint: disable=W0613 super(Roll, self).__init__() if parallel_config: dp = parallel_config.data_parallel # pylint: disable=W0612 else: dp = 1 # pylint: disable=W0612 self.shift_size = to_2tuple(shift_size) self.shift_axis = shift_axis def construct(self, x): x = numpy.roll(x, self.shift_size, self.shift_axis) return x class WindowPartition(nn.Cell): """WindowPartitionConstruct Cell""" def __init__(self, window_size, parallel_config=None): super(WindowPartition, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1),)) self.window_size = window_size def construct(self, x): """ Args: x: (B, H, W, C) Returns: windows: (num_windows*B, window_size, window_size, C) """ b, h, w, c = x.shape x = self.reshape(x, (b, h // self.window_size, self.window_size, w // self.window_size, self.window_size, c)) x = self.transpose(x, (0, 1, 3, 2, 4, 5)) x = self.reshape(x, (b * h * w // (self.window_size ** 2), self.window_size, self.window_size, c)) return x class WindowReverse(nn.Cell): """WindowReverseConstruct Cell""" def __init__(self, parallel_config=None): super(WindowReverse, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1),)) def construct(self, windows, window_size, h, w): """ Args: windows: (num_windows*B, window_size, window_size, C) window_size (int): Window size h (int): Height of image w (int): Width of image Returns: x: (B, H, W, C) """ b = windows.shape[0] // (h * w // window_size // window_size) x = self.reshape(windows, (b, h // window_size, w // window_size, window_size, window_size, -1)) x = self.transpose(x, (0, 1, 3, 2, 4, 5)) x = self.reshape(x, (b, h, w, -1)) return x pengcheng.mind-test/mindformers/models/swin/swin_processor.py000066400000000000000000000122141471400134000251320ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ SwinProcessor """ import numpy as np from PIL import Image from mindspore import Tensor from mindspore.dataset.vision.transforms import CenterCrop, ToTensor, Normalize, Rescale from mindformers.mindformer_book import MindFormerBook from mindformers.dataset import Resize from mindformers.dataset.base_dataset import BaseDataset from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['SwinProcessor', 'SwinImageProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class SwinImageProcessor(BaseImageProcessor): """ SwinImageProcessor. Args: image_resolution (int): the target size. """ def __init__(self, size=224): super(SwinImageProcessor, self).__init__(image_resolution=size) self.resize = Resize(256, interpolation='cubic') self.center_crop = CenterCrop(size) self.to_tensor = ToTensor() self.rescale = Rescale(1.0 / 255.0, 0.0) self.normalize = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], is_hwc=False) def preprocess(self, images, **kwargs): """ Preprocess required by base processor. Args: images (tensor, PIL.Image, numpy.array, list): a batch of images. Return: A 4-rank tensor for a batch of images. """ images = self._format_inputs(images) res = [] for image in images: image = self.resize(image) image = self.center_crop(image) image = self.to_tensor(image) image = self.normalize(image) res.append(image) return Tensor(res) def _format_inputs(self, inputs): """ Transform image classification inputs into (bz, h, w, c) or (h, w, c) numpy array. Args: inputs (tensor, numpy.array, PIL.Image, list, BaseDataset): for numpy or tensor input, the channel could be (bz, c, h, w), (c, h, w) or (bz, h, w, c), (h, w, c); for list, the item could be PIL.Image, numpy.array, Tensor; for BaseDataset, return without any operations. Return: transformed images: for PIL.Image, numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for list, return a numpy array for each element; for BaseDataset, it is returned directly. """ if not isinstance(inputs, (list, Image.Image, Tensor, np.ndarray, BaseDataset)): raise TypeError("input type is not Tensor, numpy, Image, list of Image or MindFormer BaseDataset") if isinstance(inputs, list): return [self._format_inputs(item) for item in inputs] if isinstance(inputs, Image.Image): inputs = np.array(inputs) if isinstance(inputs, Tensor): inputs = inputs.asnumpy() if isinstance(inputs, np.ndarray): if len(inputs.shape) == 3: inputs = np.expand_dims(inputs, 0) inputs = self._chw2hwc(inputs) elif len(inputs.shape) == 4: inputs = self._chw2hwc(inputs) else: raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(inputs.shape)}") return inputs @staticmethod def _chw2hwc(inputs): if inputs.shape[-1] != 3: inputs = inputs.transpose(0, 2, 3, 1) return inputs @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class SwinProcessor(BaseProcessor): """ Swin processor, consists of a feature extractor (BaseFeatureEXtractor) for image input. Examples: >>> from mindformers import MindFormerBook >>> from mindformers.models import SwinProcessor >>> yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", ... "swin", "run_swin_base_p4w7_224_100ep.yaml") >>> # build SwinProcessor from pretrained >>> pro_a = SwinProcessor.from_pretrained('swin_base_p4w7') >>> # build SwinProcessor from config >>> pro_b = SwinProcessor.from_pretrained(yaml_path) """ _support_list = MindFormerBook.get_processor_support_list()['swin'] def __init__(self, image_processor=None, return_tensors='ms'): super(SwinProcessor, self).__init__( image_processor=image_processor, return_tensors=return_tensors ) pengcheng.mind-test/mindformers/models/t5/000077500000000000000000000000001471400134000210515ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/t5/__init__.py000066400000000000000000000021171471400134000231630ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """t5 init""" from . import t5, t5_config from .mt5 import MT5ForConditionalGeneration from .t5 import T5ForConditionalGeneration from .t5_config import T5Config from .t5_processor import T5Processor from .t5_tokenizer import T5PegasusTokenizer, T5Tokenizer __all__ = [] __all__.extend(t5.__all__) __all__.extend(mt5.__all__) __all__.extend(t5_config.__all__) __all__.extend(t5_tokenizer.__all__) __all__.extend(t5_processor.__all__) pengcheng.mind-test/mindformers/models/t5/convert_weight.py000066400000000000000000000176151471400134000244640ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/huggingface""" import argparse import numpy as np import torch from mindspore import save_checkpoint, Tensor def generate_params_dict(total_layers, mindspore_params_per_layer, torch_params_per_layer, mindspore_additional_params, torch_additional_params): """ Generate the total parameter mapping of mindspore and pytorch. Args: total_layers(int): The total layers of the net. mindspore_params_per_layer(list): The list of params per layer for the net of mindspore. torch_params_per_layer(list): The list of params per layer for the net of pytorch. mindspore_additional_params(list): The list of params outside the layer for the net of mindspore torch_additional_params(list): The list of params outside the layer for the net of pytorch. Returns: A list of tuple. The first element is the parameter name of mindspore, the another is the parameter name of pytorch. """ mapped_params = list(zip(mindspore_params_per_layer, torch_params_per_layer)) ms_extend_param_list = [] torch_extend_param_list = [] for i in range(total_layers): for ms_para, torch_para in mapped_params: src = ms_para.format(i) tgt = torch_para.format(i) ms_extend_param_list.append(src) torch_extend_param_list.append(tgt) mapped_params = list(zip(mindspore_additional_params, torch_additional_params)) for ms_para, torch_para in mapped_params: ms_extend_param_list.append(ms_para) torch_extend_param_list.append(torch_para) return list(zip(ms_extend_param_list, torch_extend_param_list)) def get_converted_ckpt(mapped_params, weight_dict): """ Print the keys of the loaded checkpoint Args: mapped_params(dict): The loaded checkpoint. The key is parameter name and value is the numpy array. weight_dict(dict): The loaded pytorch checkpoint. Returns: None """ new_ckpt_list = [] # Currently, the ms_extend_param the torch_extend_param is the full parameters. for src, tgt in mapped_params: value = weight_dict[tgt].numpy() is_transpose = "" if '.o.' in tgt or '.wi.' in tgt or '.wo.' in tgt: value = np.transpose(value, [1, 0]) is_transpose = " transposed" print(f"Mapping table Mindspore:{src:<30} \t Torch:{tgt:<30} with shape {value.shape}" + f"---{is_transpose}") new_ckpt_list.append({"data": Tensor(value), "name": src}) return new_ckpt_list if __name__ == '__main__': parser = argparse.ArgumentParser(description="OPT convert script") parser.add_argument('--layers', type=int, default=1, help="The number of layers of the model to be converted.") parser.add_argument("--torch_path", type=str, default=None, required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="The output mindspore checkpoint path.", help="Use device nums, default is 128.") opt = parser.parse_args() state_dict = torch.load(opt.torch_path, map_location='cpu') ms_name = [ "t5_model.tfm_encoder.blocks.{}.layernorm1.gamma", "t5_model.tfm_encoder.blocks.{}.layernorm2.gamma", "t5_model.tfm_encoder.blocks.{}.attention.dense1.weight", "t5_model.tfm_encoder.blocks.{}.attention.dense2.weight", "t5_model.tfm_encoder.blocks.{}.attention.dense3.weight", "t5_model.tfm_encoder.blocks.{}.attention.projection.weight", "t5_model.tfm_encoder.blocks.{}.output.mapping.weight", "t5_model.tfm_encoder.blocks.{}.output.projection.weight", "t5_model.tfm_decoder.blocks.{}.layernorm1.gamma", "t5_model.tfm_decoder.blocks.{}.cross_attention_layernorm.gamma", "t5_model.tfm_decoder.blocks.{}.layernorm2.gamma", "t5_model.tfm_decoder.blocks.{}.attention.dense1.weight", "t5_model.tfm_decoder.blocks.{}.attention.dense2.weight", "t5_model.tfm_decoder.blocks.{}.attention.dense3.weight", "t5_model.tfm_decoder.blocks.{}.attention.projection.weight", "t5_model.tfm_decoder.blocks.{}.cross_attention.dense1.weight", "t5_model.tfm_decoder.blocks.{}.cross_attention.dense2.weight", "t5_model.tfm_decoder.blocks.{}.cross_attention.dense3.weight", "t5_model.tfm_decoder.blocks.{}.cross_attention.projection.weight", "t5_model.tfm_decoder.blocks.{}.output.mapping.weight", "t5_model.tfm_decoder.blocks.{}.output.projection.weight", ] torch_name = [ "encoder.block.{}.layer.0.layer_norm.weight", "encoder.block.{}.layer.1.layer_norm.weight", "encoder.block.{}.layer.0.SelfAttention.q.weight", "encoder.block.{}.layer.0.SelfAttention.k.weight", "encoder.block.{}.layer.0.SelfAttention.v.weight", "encoder.block.{}.layer.0.SelfAttention.o.weight", "encoder.block.{}.layer.1.DenseReluDense.wi.weight", "encoder.block.{}.layer.1.DenseReluDense.wo.weight", "decoder.block.{}.layer.0.layer_norm.weight", "decoder.block.{}.layer.1.layer_norm.weight", "decoder.block.{}.layer.2.layer_norm.weight", "decoder.block.{}.layer.0.SelfAttention.q.weight", "decoder.block.{}.layer.0.SelfAttention.k.weight", "decoder.block.{}.layer.0.SelfAttention.v.weight", "decoder.block.{}.layer.0.SelfAttention.o.weight", "decoder.block.{}.layer.1.EncDecAttention.q.weight", "decoder.block.{}.layer.1.EncDecAttention.k.weight", "decoder.block.{}.layer.1.EncDecAttention.v.weight", "decoder.block.{}.layer.1.EncDecAttention.o.weight", "decoder.block.{}.layer.2.DenseReluDense.wi.weight", "decoder.block.{}.layer.2.DenseReluDense.wo.weight", ] addition_mindspore = [ "t5_model.encoder_layernorm.gamma", "t5_model.decoder_layernorm.gamma", "t5_model.tfm_embedding_lookup.embedding_table", "t5_model.tfm_encoder.blocks.0.attention.bias_generator.embeddings_table", "t5_model.tfm_decoder.blocks.0.attention.bias_generator.embeddings_table", ] addition_torch = [ "encoder.final_layer_norm.weight", "decoder.final_layer_norm.weight", "shared.weight", "encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight", "decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight", ] mapped_param = generate_params_dict(total_layers=opt.layers, mindspore_params_per_layer=ms_name, torch_params_per_layer=torch_name, mindspore_additional_params=addition_mindspore, torch_additional_params=addition_torch) new_ckpt = get_converted_ckpt(mapped_param, state_dict) save_checkpoint(new_ckpt, opt.mindspore_path) print(f"Convert finished, the output is saved to {opt.mindspore_path}") pengcheng.mind-test/mindformers/models/t5/mt5.py000066400000000000000000000253241471400134000221360ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """mt5 model.""" import mindspore.common.dtype as mstype from mindspore import nn, ops from mindspore.common.tensor import Tensor from mindspore.ops import functional as F from mindspore.ops import operations as P from mindformers.core.loss import CrossEntropyLoss from mindformers.modules import FeedForward from mindformers.modules.layers import Linear from mindformers.modules.transformer.transformer import ( OpParallelConfig, default_dpmp_config, ) from ...mindformer_book import MindFormerBook from ...tools.register import MindFormerModuleType, MindFormerRegister from ..base_model import BaseModel from .t5 import T5Config, T5Model __all__ = ["MT5ForConditionalGeneration"] class T5FeedFowardGatedGelu(FeedForward): """ T5 feedfoward cell with gated-gelu as hidden act """ def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act="gelu", expert_num=1, expert_group_size=None, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(T5FeedFowardGatedGelu, self).__init__( hidden_size, ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, expert_num=expert_num, expert_group_size=expert_group_size, param_init_type=param_init_type, parallel_config=parallel_config, ) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = int(parallel_config.data_parallel / ep) del self.mapping # replace origin mapping with 2 mappings self.mapping_0 = Linear( in_channels=hidden_size, out_channels=ffn_hidden_size, has_bias=False, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type, ) self.mapping_0.shard( strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),), ) self.mapping_1 = Linear( in_channels=hidden_size, out_channels=ffn_hidden_size, has_bias=False, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type, ) self.mapping_1.shard( strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), ) self.projection = Linear( in_channels=ffn_hidden_size, out_channels=hidden_size, has_bias=False, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type, ) self.projection.shard( strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,)) ) self.mul = ops.Mul() self.mul.shard(in_strategy=((dp, 1), (dp, 1))) def construct(self, x): """construct of T5 FeedFoward-GatedGelu""" x = self.cast(x, mstype.float16) hidden_gelu = self.mapping_0(x) hidden_linear = self.mapping_1(x) hidden_states = self.mul(hidden_gelu, hidden_linear) if len(F.shape(hidden_states)) == 3: hidden_states = self.dropout_3d(hidden_states) elif len(F.shape(hidden_states)) == 2: hidden_states = self.dropout(hidden_states) hidden_states = self.projection(hidden_states) return hidden_states class MT5Head(nn.Cell): """MT5 model Head. No parameter share between embedding tale and head liner. """ def __init__(self, in_channels, out_channels, compute_dtype=mstype.float16, parallel_config=default_dpmp_config): super(MT5Head, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.linear = Linear( in_channels=in_channels, out_channels=out_channels, has_bias=False ) self.linear.shard( strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),), ) self.dtype = compute_dtype self.cast = ops.Cast() def construct(self, state, emb_table): # to fit t5 head usage _ = emb_table # output logits over vocabulary [bs, seq_length, vocab_size] return self.linear(self.cast(state, self.dtype)) class MT5Model(T5Model): """MT5 Model, a variant of T5 model. Main changes are: use gated-gelu feedforward, no share parameters between embedding tale and head liner. here is the paper: Args: config(T5Config) : The network config of mt5. """ def __init__(self, config): super(MT5Model, self).__init__(config) for block in self.tfm_encoder.blocks: block.output = T5FeedFowardGatedGelu( hidden_size=config.hidden_size, dropout_rate=config.hidden_dropout_rate, ffn_hidden_size=config.d_ff, hidden_act=config.hidden_act, parallel_config=default_dpmp_config, ) for block in self.tfm_decoder.blocks: block.output = T5FeedFowardGatedGelu( hidden_size=config.hidden_size, dropout_rate=config.hidden_dropout_rate, ffn_hidden_size=config.d_ff, hidden_act=config.hidden_act, parallel_config=default_dpmp_config, ) # no parameter sharing between embedding and classifier layer self.projection = MT5Head( in_channels=config.hidden_size, out_channels=config.vocab_size, parallel_config=default_dpmp_config, ) @MindFormerRegister.register(MindFormerModuleType.MODELS) class MT5ForConditionalGeneration(BaseModel): """ An MT5 model with the loss added. Args: config(T5Config) : The network config of mt5. """ _support_list = MindFormerBook.get_model_support_list()["mt5"] def __init__(self, config: T5Config): super(MT5ForConditionalGeneration, self).__init__(config) parallel_config = config.parallel_config self.mt5_model = MT5Model(config=config) self.vocab_size = config.vocab_size self.loss = CrossEntropyLoss( parallel_config=OpParallelConfig( data_parallel=parallel_config.data_parallel, model_parallel=parallel_config.model_parallel, ) ) self.cast = ops.Cast() self.shape = ops.Shape() # The value of start and end should get from the tokenizer start_token_id = config.start_token_id eos_token_id = config.eos_token_id self.start_token_id = Tensor(((start_token_id,),), dtype=mstype.int32) # [CLS] self.eos_token_id = Tensor(((eos_token_id,),), dtype=mstype.int32) # [SEP] self.concat = P.Concat(axis=1) self.tile = P.Tile() self.reshape = ops.Reshape() # disable the bias for param in self.trainable_params(): if "bias" in param.name or "beta" in param.name and "relative" not in param.name: param.requires_grad = False self.load_checkpoint(config) def _add_start_to_inputs(self, target_ids): """concat the start id to the decoder inputs""" start_token = self.tile(self.start_token_id, (F.shape(target_ids)[0], 1)) decoder_inputs = self.concat((start_token, target_ids)) return decoder_inputs def _add_eos_to_inputs(self, target_ids): """concat the eos id to the end of the decoder inputs""" eos_token = self.tile(self.eos_token_id, (F.shape(target_ids)[0], 1)) inputs_with_eos = self.concat((target_ids, eos_token)) return inputs_with_eos def encoder_forward(self, source_ids, source_mask): """Execute the encoder forward process""" return self.mt5_model.encoder_forward(source_ids, source_mask) def construct(self, input_ids, attention_mask, labels=None, decoder_input_ids=None, decoder_attention_mask=None, memory_mask=None, encoder_outputs=None): """t5_model network with loss.""" if decoder_attention_mask is None: decoder_attention_mask = F.cast(labels != 0, mstype.float32) if decoder_input_ids is None: decoder_input_ids = labels # replace start token using self.start_token_id decoder_input_ids = self._add_start_to_inputs(decoder_input_ids[:, 1:]) # shape: [bs, seq_len, hidden_size] logits = self.mt5_model( input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, memory_mask, encoder_cache=encoder_outputs, ) total_loss = None if labels is not None: # on training label_ids = ops.Reshape()(decoder_input_ids[:, 1:], (-1,)) label_weights = ops.Reshape()(decoder_attention_mask[:, 1:], (-1,)) logits = logits[:, :-1] # logits keep shape same with label_ids total_loss = self.loss( self.reshape(logits, (-1, self.vocab_size)), label_ids, self.cast(label_weights, mstype.float32), ) # reshape: [bs, seq_len, hidden_size] -> [bs*seq, hidden_size] logits = self.reshape(logits, (-1, self.vocab_size)) if self.training: return total_loss return logits pengcheng.mind-test/mindformers/models/t5/t5.py000066400000000000000000002602641471400134000217650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """T5 model.""" import math import numpy as np import mindspore.ops as ops import mindspore.nn as nn from mindspore import context from mindspore.ops.primitive import constexpr from mindspore.ops import functional as F from mindspore.ops import operations as P import mindspore.numpy from mindspore.context import ParallelMode from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter import mindspore.common.dtype as mstype from mindspore.common.initializer import TruncatedNormal, initializer from mindspore.parallel._utils import _get_parallel_mode from mindformers.modules.layers import Linear, _check_past_none_input_none, _check_input_dtype from mindformers.modules.transformer.moe import MoE, _check_moe_config from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ default_dpmp_config, \ EmbeddingOpParallelConfig, OpParallelConfig from mindformers.core.loss import CrossEntropyLoss from mindformers.modules import VocabEmbedding from .t5_config import T5Config from ..base_model import BaseModel from ...tools import logger from ...tools.register import MindFormerRegister, MindFormerModuleType from ...mindformer_book import MindFormerBook __all__ = ['T5ForConditionalGeneration'] class LayerNorm(nn.Cell): """ T5 layer norm nn.Cell """ def __init__(self, normalized_shape, eps=1e-5, param_init_type=mstype.float32): super(LayerNorm, self).__init__() if param_init_type not in [mstype.float32, mstype.float16]: raise TypeError("The type of parameter 'param_init_type' should in [float32, float16], " "but got the type : {}.".format(type(param_init_type))) self.gamma = Parameter(initializer('ones', normalized_shape, param_init_type), name="gamma", parallel_optimizer=False) self.mean = P.ReduceMean(keep_dims=True) self.square = P.Square() self.sqrt = P.Sqrt() self.sub1 = P.Sub() self.sub2 = P.Sub() self.add = P.Add() self.eps = eps self.mul = P.Mul() self.add2 = P.Add() self.real_div = P.RealDiv() def construct(self, x): r""" x : batch x seq_length x hidden_size """ variance = self.mean(self.square(x), -1) variance_eps = self.sqrt(self.add(variance, self.eps)) output = self.real_div(x, variance_eps) output = self.mul(output, self.gamma) return output def shard(self, strategy): r""" Set the shard for the layer norm. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy (tuple): The strategy for the dropout. Should be the same shape as the inputs. """ self.mean.shard(strategy) self.square.shard(strategy) self.sqrt.shard(strategy) self.sub1.shard((strategy[0], strategy[0])) self.sub2.shard((strategy[0], strategy[0])) self.add.shard((strategy[0], ())) self.mul.shard((strategy[0], (1,))) self.add2.shard((strategy[0], (1,))) self.real_div.shard((strategy[0], strategy[0])) return self class T5FeedFoward(nn.Cell): """ T5 feedfoward cell with relu as hidden act """ def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act='gelu', expert_num=1, expert_group_size=None, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(T5FeedFoward, self).__init__() mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = int(parallel_config.data_parallel / ep) if ffn_hidden_size % mp != 0: raise ValueError("For 'T5FeedFoward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'T5FeedFoward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'T5FeedFoward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, has_bias=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.mapping.shard(strategy_matmul=((dp, ep, 1, 1), (ep, 1, mp)), strategy_bias=((dp, ep, 1, mp), (1, ep, 1, mp)), strategy_activation=((dp, ep, 1, mp),)) else: self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, has_bias=False, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1)), strategy_bias=((dp, ep, 1, 1), (1, ep, 1, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.dropout = nn.Dropout(1 - dropout_rate) self.dropout.dropout.shard(((dp, 1),)) self.dropout_3d = nn.Dropout(1 - dropout_rate) self.dropout_3d.dropout.shard(((dp, 1, 1),)) self.dropout_4d = nn.Dropout(1 - dropout_rate) self.dropout_4d.dropout.shard(((dp, ep, 1, 1),)) self.cast = P.Cast() def construct(self, x): """The forward function of FFN""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) else: output = self.dropout_4d(output) return output class RelaPosMatrixGenerator(nn.Cell): """ The relative position index generator. The result of the cell should be feed into the bias embedding table. """ def __init__(self, max_relative_position, log_relative_distance): super(RelaPosMatrixGenerator, self).__init__() self._max_relative_position = max_relative_position self._min_relative_position = -max_relative_position self.tile = P.Tile() self.range_mat = P.Reshape() self.sub = P.Sub() self.expanddims = P.ExpandDims() self.cast = P.Cast() self.log_relative_distance = log_relative_distance def construct(self, relative_position, bidirectional=True, num_buckets=32): """The forward of the bias position""" relative_bucket = 0 if bidirectional: num_buckets = num_buckets // 2 relative_bucket = relative_bucket + (relative_position > 0).astype(mstype.int32) * num_buckets relative_position = P.Abs()(relative_position) else: relative_position = -P.Minimum()(relative_position, P.ZerosLike()(relative_position)) max_exact = num_buckets // 2 is_small = relative_position < max_exact relative_position_if_large = max_exact + (P.Log()(relative_position.astype(mstype.float32) / max_exact) / self.log_relative_distance * (num_buckets - max_exact)) relative_position_if_large = relative_position_if_large.astype(mstype.int32) relative_position_if_large = P.Minimum()(relative_position_if_large, mindspore.numpy.full_like(relative_position_if_large, num_buckets - 1)) relative_bucket += mindspore.numpy.where(is_small, relative_position, relative_position_if_large) return relative_bucket class RelaPosEmbeddingsGenerator(nn.Cell): """The relative position embedding generator.""" def __init__(self, depth, max_relative_position, initializer_range, is_decoder): super(RelaPosEmbeddingsGenerator, self).__init__() self.depth = depth self.vocab_size = max_relative_position self.embeddings_table = Parameter(initializer(TruncatedNormal(initializer_range), [self.vocab_size, self.depth])) self.reshape = P.Reshape() self.one_hot = nn.OneHot(depth=self.vocab_size) self.shape = P.Shape() self.gather = P.Gather() self.matmul = P.BatchMatMul() self.relative_attention_num_buckets = 32 self.relative_attention_max_distance = 128 self.is_decoder = is_decoder num_buckets = self.relative_attention_num_buckets max_exact = self.relative_attention_num_buckets // 2 if not self.is_decoder: max_exact = max_exact // 2 num_buckets //= 2 self.log_relative_distance = math.log(self.relative_attention_max_distance / max_exact) self.relative_position_matrix = RelaPosMatrixGenerator(max_relative_position=max_relative_position, log_relative_distance=self.log_relative_distance) def construct(self, query_length, key_length): """The forward function""" context_position = mindspore.numpy.arange(query_length, dtype=mstype.int32).expand_dims(-1) memory_position = mindspore.numpy.arange(key_length, dtype=mstype.int32).expand_dims(0) relative_position = memory_position - context_position relative_position_bucket = self.relative_position_matrix( relative_position, bidirectional=(not self.is_decoder), num_buckets=self.relative_attention_num_buckets) embeddings = self.gather(self.embeddings_table, relative_position_bucket, 0) embeddings = embeddings.transpose((2, 0, 1)).expand_dims(0) return embeddings class T5MultiHeadAttention(nn.Cell): """ T5 multi head attention """ def __init__(self, batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, kv_size=64, hidden_dropout_rate=0.1, attention_dropout_rate=0.1, compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, use_past=False, has_relative_bias=False, is_decoder=False, is_cross_atten=False, parallel_config=default_dpmp_config): super(T5MultiHeadAttention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size self.has_relative_bias = has_relative_bias if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'T5MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'T5MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'T5MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'T5MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) if self.is_parallel_mode and batch_size % parallel_config.data_parallel != 0: raise ValueError("For 'T5MultiHeadAttention', the class variable 'batch_size' must be a multiple of " "'parallel_config.data_parallel', but got the batch_size is {} " "and the parallel_config.data_parallel is {}." .format(batch_size, parallel_config.data_parallel)) self.is_first_iteration = True self.inner_dim = num_heads * kv_size # Output layer self.projection = Linear(in_channels=self.inner_dim, out_channels=hidden_size, transpose_b=False, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = kv_size self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.use_past = use_past self.dropout = nn.Dropout(1 - hidden_dropout_rate) self.dropout.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout = nn.Dropout(1 - attention_dropout_rate) self.prob_dropout.dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, self.inner_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, self.inner_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, self.inner_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type self.is_decoder = is_decoder self.has_relative_bias = has_relative_bias self.is_cross_atten = is_cross_atten self.cross_bias = None if self.has_relative_bias: if not self.is_cross_atten: self.bias_generator = RelaPosEmbeddingsGenerator(depth=num_heads, max_relative_position=32, initializer_range=0.02, is_decoder=self.is_decoder) else: self.cross_bias = Parameter(initializer("zero", [1, self.src_seq_length, self.tgt_seq_length]), name='cross_attention_bias', parallel_optimizer=False) if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, bias=None, key_past=None, value_past=None, batch_valid_length=None): """forward function for attention""" self._check_inputs(query_tensor, key_tensor, value_tensor, attention_mask, key_past, value_past, batch_valid_length) query_tensor, key_tensor, value_tensor, batch_size, ori_shape = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor, attention_mask) ori_dtype = F.dtype(query_tensor) query_tensor = F.cast(query_tensor, self.dtype) key_tensor = F.cast(key_tensor, self.dtype) value_tensor = F.cast(value_tensor, self.dtype) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # key and value for current token(s) key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key, self.expand_dims(valid_length_vector, 2)) value_present = self.mul1(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: # Get the current token position index valid_length = self.reducesum(F.cast(self.not_equal(self.slice(key_past, (0, 0, 0, 0), (F.shape(key_tensor)[0], 1, 1, self.src_seq_length), (1, 1, 1, 1)), 0), mstype.float32), (1, 2, 3)) valid_length = F.reshape(valid_length, (-1, 1, 1)) valid_length_vector = F.cast(self.equal(valid_length, self.range), self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) current_value = self.mul1(self.tile(value, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add(key_past, current_key) value = self.add(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value attention_mask = F.reshape(self.attention_mask, (self.seq_length, self.seq_length, 1, 1)) layer_present = (key_present, value_present) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention, bias = self._attn(query, key, value, attention_mask, bias) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) output = F.cast(output, ori_dtype) return output, layer_present, bias def _check_inputs(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): r"""Check inputs""" _check_input_dtype(F.dtype(query_tensor), "query_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(key_tensor), "key_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_tensor), "value_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) key_is_tensor = isinstance(key_past, Tensor) value_is_tensor = isinstance(value_past, Tensor) batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) key_is_default = key_past is None value_is_default = value_past is None batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "key_past", self.cls_name, None, key_is_tensor, key_is_default) _check_past_none_input_none(self.use_past, "value_past", self.cls_name, None, value_is_tensor, value_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) return True def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor, attention_mask): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor, F.shape(attention_mask)[0], query_shape def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( F.reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = F.reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask, bias): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) score = P.Cast()(score, self.softmax_dtype) # for input size of (bs, 1) namely the second graph, # the shape of attention_mask matrix should be (bs, 1, 1, seq_length) if self.use_past and not self.is_first_iteration: # Calculate the current total token current_index = self.reducesum(F.cast(self.not_equal(self.slice(key, (0, 0, 0, 0), (F.shape(query)[0], 1, 1, self.seq_length), (1, 1, 1, 1)), 0), mstype.float32), (1, 2, 3)) # Get the precise position index index = self.sub1(F.cast(current_index, mstype.int32), 1) index = F.reshape(index, (-1, 1, 1)) # Calculate the attention_mask matrix via the position index attention_mask = F.cast(self.tensor_le(self.range, index), mstype.int32) attention_mask = self.expand_dims(attention_mask, 2) if bias is None and self.has_relative_bias: if not self.is_cross_atten: bias = self.bias_generator(F.shape(score)[-1], F.shape(score)[-1]) else: bias = P.ExpandDims()(self.cross_bias, 0) score = self.add(score, bias) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(score)), P.Cast()(attention_mask, P.DType()(score))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, score) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = P.Cast()(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge, bias class TransformerEncoderLayer(nn.Cell): """ Transformer Encoder Layer """ def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, kv_size=64, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, layer_norm_epsilon=1e-6, hidden_act='gelu', use_past=False, moe_config=default_moe_config, has_bias=False, parallel_config=default_dpmp_config): super(TransformerEncoderLayer, self).__init__() if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}.".format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.batch_size = batch_size self.layernorm1 = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float(layernorm_compute_type) self.attention = T5MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, kv_size=kv_size, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, is_decoder=False, has_relative_bias=has_bias, parallel_config=parallel_config.dpmp if self.use_moe else parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = T5FeedFoward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = int(hidden_size / num_heads) self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): pass elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, x, input_mask, bias, init_reset=True, batch_valid_length=None): """Forward function of the EncoderLayer""" self._check_input(x, input_mask, init_reset, batch_valid_length) x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) value_reset = self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present, bias = self.attention(input_x, input_x, input_x, input_mask, bias, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step key_update = self.assign(self.key_past, key_present) value_update = self.assign(self.value_past, value_present) # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present, bias def _check_input(self, x, input_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(init_reset), "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True class TransformerDecoderLayer(nn.Cell): """ The Transformer Decoder Layer """ def __init__(self, hidden_size, ffn_hidden_size, num_heads, batch_size, src_seq_length, tgt_seq_length, kv_size=64, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, use_past=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, layer_norm_epsilon=1e-6, hidden_act='gelu', has_bias=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerDecoderLayer, self).__init__() if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) self.attention = T5MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, kv_size=kv_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, is_decoder=True, has_relative_bias=has_bias, parallel_config=parallel_config.dpmp if self.use_moe else parallel_config) # Cross attention with the output of encoder as memory tensor self.cross_attention = T5MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, kv_size=kv_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, is_decoder=True, is_cross_atten=True, has_relative_bias=has_bias, param_init_type=param_init_type, parallel_config=parallel_config.dpmp if self.use_moe else parallel_config) self.cross_attention_layernorm = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float( layernorm_compute_type) self.cross_attention_layernorm.shard(((parallel_config.data_parallel, 1),)) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = T5FeedFoward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = int(hidden_size / num_heads) self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) def construct(self, hidden_stats, decoder_mask, encoder_output=None, memory_mask=None, self_bias=None, encoder_attention_bias=None, init_reset=True, batch_valid_length=None): """The forward function of the decoder layer""" self._check_input(hidden_stats, decoder_mask, encoder_output, memory_mask, init_reset, batch_valid_length) # the returned shape is [bs, seq_length, embedding_size] or [bs * seq_length, embedding_size] hidden_shape = F.shape(hidden_stats) hidden_stats = F.reshape(hidden_stats, (-1, hidden_shape[-1])) input_x = self.layernorm1(hidden_stats) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) value_reset = self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present, self_bias = self.attention(input_x, input_x, input_x, decoder_mask, self_bias, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(hidden_stats, attention) middle_output = None if encoder_output is not None: middle_output = self.cross_attention_layernorm(x) middle_output = F.cast(middle_output, self.dtype) encoder_output = F.cast(encoder_output, self.dtype) cross_attn_out, cross_layer_present, encoder_attention_bias = self.cross_attention(middle_output, encoder_output, encoder_output, memory_mask, encoder_attention_bias, self.key_past, self.value_past, batch_valid_length) layer_present += cross_layer_present if self.post_layernorm_residual: x = self.add(middle_output, cross_attn_out) else: x = self.add(x, cross_attn_out) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step key_update = self.assign(self.key_past, key_present) value_update = self.assign(self.value_past, value_present) # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(hidden_shape) == 3: output_x = P.Reshape()(output_x, hidden_shape) mlp_logit = P.Reshape()(mlp_logit, hidden_shape) x = P.Reshape()(x, hidden_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, hidden_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present, self_bias, encoder_attention_bias def _check_input(self, hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(hidden_states), "hidden_states", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) if encoder_output is not None: _check_input_dtype(F.dtype(encoder_output), "encoder_output", [mstype.float32, mstype.float16], self.cls_name) if memory_mask is not None: _check_input_dtype(F.dtype(memory_mask), "memory_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(init_reset), "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _get_lambda_func(total_layer=None): r""" A wrapper function of specifying pipeline stage and gradient aggregation fusion. If the total layer is not None, for example, set in the transformer model, the pipeline stage setting function will be `(layer_id + 0) // (total_layers / parallel_config.pipeline_stage)` for the encoder and, `(layer_id + offset) // (total_layers / parallel_config.pipeline_stage)` for the decoder, where `offset` is the layers in the encoder. """ def _set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs an offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # override the layers if total_layer: layers = total_layer # Used for the pipeline's stages setting if layers < parallel_config.pipeline_stage: raise ValueError(f"layers {layers} must be larger than pipeline stage {parallel_config.pipeline_stage}") pp_dis = max(int(layers / parallel_config.pipeline_stage), 1) # the pipeline stage must be in [0, parallel_config.pipeline_stage - 1] pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int(layers / parallel_config.gradient_aggregation_group), 1) network.set_comm_fusion(int((layer_id + offset) / dis) + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: paralel_op_comm_compute = parallel_config.recompute.parallel_optimizer_comm_recompute network.recompute(parallel_optimizer_comm_recompute=paralel_op_comm_compute, mp_comm_recompute=parallel_config.recompute.mp_comm_recompute, recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) return _set_parallel_configure_for_layer class TransformerEncoder(nn.Cell): """The TransformerEncoder Cell""" def __init__(self, batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, kv_size=64, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', layer_norm_epsilon=1e-6, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, offset=0, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerEncoder, self).__init__() _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, kv_size=kv_size, num_heads=num_heads, hidden_act=hidden_act, has_bias=(i == 0), layer_norm_epsilon=layer_norm_epsilon, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): pass elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, init_reset=True, batch_valid_length=None): """The forward process of the encoder""" present_layer = () attention_bias = None if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss for i in range(self.num_layers): hidden_states, present, attention_bias = self.blocks[i](hidden_states, attention_mask, attention_bias, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class TransformerDecoder(nn.Cell): """The TransformerDecoder cell""" def __init__(self, num_layers, batch_size, hidden_size, ffn_hidden_size, src_seq_length, tgt_seq_length, num_heads, kv_size=64, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, layer_norm_epsilon=1e-6, hidden_act='gelu', lambda_func=None, use_past=False, offset=0, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerDecoder, self).__init__() self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, kv_size=kv_size, use_past=use_past, has_bias=(i == 0), param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, layer_norm_epsilon=layer_norm_epsilon, moe_config=moe_config, parallel_config=parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): pass elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """For forward process of the decoder""" present_layer = () self_bias = None encoder_decoder_bias = None if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss # Loop through each self-attention layer for i in range(self.num_layers): hidden_states, present, self_bias, encoder_decoder_bias = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, self_bias, encoder_decoder_bias, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer def position_encoding(length, depth, min_timescale=1, max_timescale=1e4): """ Create Tensor of sinusoids of different frequencies. Args: length (int): Length of the Tensor to create, i.e. Number of steps. depth (int): Hidden size. min_timescale (float): Default: 1. max_timescale (float): Default: 10000. Returns: Tensor of shape (length, depth) """ depth = depth // 2 positions = np.arange(length, dtype=np.float32) log_timescale_increment = (np.log(max_timescale / min_timescale) / (depth - 1)) inv_timescales = min_timescale * np.exp(np.arange(depth, dtype=np.float32) * -log_timescale_increment) scaled_time = np.expand_dims(positions, 1) * np.expand_dims(inv_timescales, 0) x = np.concatenate([np.sin(scaled_time), np.cos(scaled_time)], axis=1) return x class EmbeddingPostprocessor(nn.Cell): """ Postprocessors apply positional embeddings to word embeddings. Args: embedding_size (int): The size of each embedding vector. initializer_range (float): Initialization value of TruncatedNormal. Default: 0.02. max_position_embeddings (int): Maximum length of sequences used in this model. Default: 128. dropout_prob (float): The dropout probability. Default: 0.1. """ def __init__(self, embedding_size, max_position_embeddings=128, dropout_prob=0.1): super(EmbeddingPostprocessor, self).__init__() self.scores_mul = Tensor([math.sqrt(float(embedding_size))], dtype=mstype.float32) self.multiply = ops.Mul() self.add = ops.Add() self.dropout = nn.Dropout(1 - dropout_prob, dtype=mstype.float32) self.expand_dims = ops.ExpandDims() self.position_embedding_table = Tensor(position_encoding(max_position_embeddings, embedding_size), mstype.float32) self.shape = ops.Shape() self.slice = ops.StridedSlice().shard(((1, 1),)) def construct(self, word_embeddings): """Postprocessors apply positional embeddings to word embeddings.""" output = self.multiply(word_embeddings, self.scores_mul) output = self.dropout(output) return output class CastWrapper(nn.Cell): """ Cast wrapper. """ def __init__(self, dst_type=mstype.float32): super(CastWrapper, self).__init__() self.cast = ops.Cast() self.dst_type = dst_type def construct(self, x): return self.cast(x, self.dst_type) class CreateAttentionMaskFromInputMask(nn.Cell): """ Create attention mask according to input mask. Args: config (:class:`TransformerConfig`): Configuration for Transformer. """ def __init__(self, parallel_config): super(CreateAttentionMaskFromInputMask, self).__init__() self.reshape = ops.Reshape() self.shape = ops.Shape() self.batch_matmul = ops.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) def construct(self, input_mask): """Create attention mask according to input mask.""" input_shape = self.shape(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) input_mask = F.cast(input_mask, mstype.float32) mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.batch_matmul(mask_left, mask_right) return attention_mask @constexpr def convert_np_to_tensor_encoder(seq_length): ones = np.ones(shape=(seq_length, seq_length)) return Tensor(np.tril(ones), dtype=mstype.float32) class T5Head(nn.Cell): """ Head to get the logits of each token in the vocab Args: config(): the config of network Inputs: state: the output of the backbone embedding_table: the embedding table of the vocabulary Returns: logits: Tensor, the logits of the corresponding inputs """ def __init__(self, hidden_size, compute_dtype=mstype.float16, parallel_config=None): super(T5Head, self).__init__() if parallel_config.vocab_emb_dp: self.matmul = ops.MatMul(transpose_b=True).shard(((parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = ops.MatMul(transpose_b=True).shard(((parallel_config.data_parallel, 1), ( parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = compute_dtype self.cast = ops.Cast() def construct(self, state, embed): state = ops.Reshape()(state, (-1, self.hidden_size)) # output logits over vocabulary [bs*seq_length, vocab_size] logits = self.matmul(self.cast(state, self.dtype), self.cast(embed, self.dtype)) return logits class T5Model(BaseModel): """ T5Model with encoder and decoder. Args: config (Class): Configuration for T5Model. """ def __init__(self, config): super(T5Model, self).__init__(config) self.batch_size = config.batch_size self.hidden_size = config.hidden_size self.max_decode_length = config.max_decode_length self.scale_output = config.scale_output embedding_config = EmbeddingOpParallelConfig(data_parallel=config.parallel_config.data_parallel, model_parallel=config.parallel_config.model_parallel) self.tfm_embedding_lookup = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, parallel_config=embedding_config) self.tfm_embedding_postprocessor_for_encoder = EmbeddingPostprocessor(embedding_size= config.hidden_size, max_position_embeddings= config.max_position_embeddings, dropout_prob= config.embedding_dropout_prob) self.tfm_embedding_postprocessor_for_decoder = EmbeddingPostprocessor( embedding_size=config.hidden_size, max_position_embeddings=config.max_position_embeddings, dropout_prob=config.embedding_dropout_prob) self.tfm_encoder = TransformerEncoder( batch_size=self.batch_size, hidden_size=config.hidden_size, num_heads=config.num_heads, num_layers=config.num_layers, seq_length=config.seq_length, ffn_hidden_size=config.d_ff, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, layer_norm_epsilon=config.layer_norm_epsilon, kv_size=config.kv_size, hidden_act=config.hidden_act, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, post_layernorm_residual=config.post_layernorm_residual, offset=config.offset, use_past=config.use_past, moe_config=config.moe_config) self.tfm_decoder = TransformerDecoder( batch_size=self.batch_size, hidden_size=config.hidden_size, src_seq_length=config.seq_length, tgt_seq_length=config.max_decode_length, num_heads=config.num_heads, ffn_hidden_size=config.d_ff, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, layer_norm_epsilon=config.layer_norm_epsilon, kv_size=config.kv_size, num_layers=config.num_decoder_layers if config.num_decoder_layers else config.num_layers, hidden_act=config.hidden_act, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, post_layernorm_residual=config.post_layernorm_residual, offset=config.offset, use_past=config.use_past, moe_config=config.moe_config) self.projection = T5Head(config.hidden_size, compute_dtype=mstype.float16, parallel_config=config.parallel_config) self.cast = ops.Cast() self.dtype = config.dtype self.cast_compute_type = CastWrapper(dst_type=config.compute_dtype) self.expand = ops.ExpandDims() self.multiply = ops.Mul() self.shape = ops.Shape() self.encoder_layernorm = LayerNorm(normalized_shape=(config.hidden_size,), eps=config.layer_norm_epsilon).to_float(mstype.float32) self.decoder_layernorm = LayerNorm(normalized_shape=(config.hidden_size,), eps=config.layer_norm_epsilon).to_float(mstype.float32) self.encoder_layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.decoder_layernorm.shard(((config.parallel_config.data_parallel, 1),)) self._create_attention_mask_from_input_mask = CreateAttentionMaskFromInputMask(config.parallel_config) self.ones_like = P.OnesLike() def construct(self, source_ids=None, source_mask=None, target_ids=None, target_mask=None, memory_mask=None, encoder_cache=None): """T5Model with encoder and decoder.""" if source_mask is None and source_ids is not None: source_mask = self.ones_like(source_ids) source_mask = self._create_attention_mask_from_input_mask(source_mask) if source_ids is not None: encoder_output = self.encoder_forward(source_ids, source_mask) else: encoder_output = encoder_cache if target_ids is None: return encoder_output # process target sentence tgt_embedding_output, embedding_table = self.tfm_embedding_lookup(target_ids) # attention mask [batch_size, seq_length, seq_length] tgt_length = self.shape(target_ids)[1] if memory_mask is None: memory_mask = self.create_memory_mask(source_mask, target_mask) if len(ops.shape(target_mask)) == 2: future_mask = convert_np_to_tensor_encoder(tgt_length) tgt_attention_mask = self._create_attention_mask_from_input_mask(target_mask) tgt_attention_mask = self.multiply(tgt_attention_mask, self.expand(future_mask, 0)) else: tgt_attention_mask = target_mask # transformer decoder decoder_output, _ = self.tfm_decoder(self.cast_compute_type(tgt_embedding_output), self.cast_compute_type(tgt_attention_mask), encoder_output, memory_mask) decoder_output = self.decoder_layernorm(decoder_output) if self.scale_output: decoder_output = decoder_output * (self.hidden_size ** -0.5) # calculate logits and log_probs log_probs = self.projection(decoder_output, embedding_table) return log_probs def encoder_forward(self, source_ids, source_mask): """Execute the forward process""" # process source sentence src_embedding_output, _ = self.tfm_embedding_lookup(source_ids) # attention mask [batch_size, seq_length, seq_length] if len(F.shape(source_mask)) == 2: enc_attention_mask = self._create_attention_mask_from_input_mask(source_mask) else: enc_attention_mask = source_mask # transformer encoder encoder_output, _ = self.tfm_encoder(self.cast_compute_type(src_embedding_output), self.cast_compute_type(enc_attention_mask)) encoder_output = self.encoder_layernorm(encoder_output) return encoder_output def create_memory_mask(self, source_mask, target_mask): memory_mask = P.Ones()((F.shape(source_mask)[0], F.shape(target_mask)[-1], F.shape(source_mask)[-1]), mstype.float32) memory_mask = memory_mask * F.expand_dims(source_mask, 1) memory_mask = memory_mask * F.expand_dims(target_mask, 2) return memory_mask @MindFormerRegister.register(MindFormerModuleType.MODELS) class T5ForConditionalGeneration(BaseModel): """ A T5 model with the loss added. Args: config(T5Config) : The network of the transformer. Examples: >>> from mindformers import T5ForConditionalGeneration, T5Tokenizer >>> model = T5ForConditionalGeneration.from_pretrained('t5_small') >>> tokenizer = T5Tokenizer.from_pretrained('t5_small') >>> src_output = tokenizer(["hello world"], padding='max_length', max_length=model.config.seq_length, ... return_tensors='ms') >>> model_input = tokenizer(["So happy to see you!"], padding='max_length', ... max_length=model.config.max_decode_length, ... return_tensors='ms')["input_ids"] >>> input_ids = src_output['input_ids'] >>> attention_mask = src_output['attention_mask'] >>> output = model(input_ids, attention_mask, model_input) >>> print(output) [5.64458] """ _support_list = MindFormerBook.get_model_support_list()['t5'] def __init__(self, config: T5Config): super(T5ForConditionalGeneration, self).__init__(config) parallel_config = config.parallel_config self.t5_model = T5Model(config=config) self.loss = CrossEntropyLoss(parallel_config=OpParallelConfig(data_parallel=parallel_config.data_parallel, model_parallel=parallel_config.model_parallel)) self.cast = ops.Cast() self.shape = ops.Shape() # The value of start and end should get from the tokenizer self.start_token = Tensor(np.zeros((1, 1)).astype(np.int32)) self.eod_token = Tensor(np.zeros((1, 1)).astype(np.int32)) self.concat = P.Concat(axis=1) self.tile = P.Tile() # disable the bias for param in self.trainable_params(): if ('bias' in param.name or 'beta' in param.name) and 'relative' not in param.name: param.requires_grad = False self.set_train(True) self.load_checkpoint(config) def _add_start_to_inputs(self, target_ids): """concat the start id to the decoder inputs""" start_token = self.tile(self.start_token, (F.shape(target_ids)[0], 1)) decoder_inputs = self.concat((start_token, target_ids)) return decoder_inputs def _add_eos_to_inputs(self, target_ids): """concat the eos id to the end of the decoder inputs""" eod_token = self.tile(self.eod_token, (F.shape(target_ids)[0], 1)) inputs_with_eos = self.concat((target_ids, eod_token)) return inputs_with_eos def encoder_forward(self, source_ids, source_mask): """Execute the encoder forward process""" return self.t5_model.encoder_forward(source_ids, source_mask) def construct(self, input_ids, attention_mask, labels=None, decoder_input_ids=None, decoder_attention_mask=None, memory_mask=None, encoder_outputs=None, return_loss=False): """t5_model network with loss.""" if decoder_attention_mask is None: decoder_attention_mask = F.cast(labels != 0, mstype.float32) if decoder_input_ids is None: decoder_input_ids = self._add_start_to_inputs(labels[:, :-1]) logits = self.t5_model(input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, memory_mask, encoder_cache=encoder_outputs) total_loss = None if labels is not None: label_ids = ops.Reshape()(labels, (-1,)) label_weights = ops.Reshape()(decoder_attention_mask, (-1,)) total_loss = self.loss(logits, label_ids, self.cast(label_weights, mstype.float32)) if self.training: return total_loss if return_loss: return total_loss return logits pengcheng.mind-test/mindformers/models/t5/t5_config.py000066400000000000000000000124141471400134000233020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """T5 Configuration""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['T5Config'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class T5Config(BaseConfig): """ T5 config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['t5'] def __init__(self, vocab_size: int = 32128, hidden_size: int = 512, d_kv: int = 64, d_ff: int = 2048, num_layers: int = 6, num_decoder_layers: int = None, num_heads: int = 8, relative_attention_num_buckets: int = 32, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, embedding_dropout_prob: float = 0.1, layer_norm_epsilon: float = 1e-6, initializer_factor: float = 1.0, is_encoder_decoder: bool = True, use_cache: bool = True, pad_token_id: int = 0, start_token_id: int = 0, eos_token_id: int = 1, batch_size: int = 1, seq_length: int = 1024, max_position_embeddings: int = 1024, initializer_range: float = 0.02, max_decode_length: int = 128, length_penalty_weight: float = 1.0, compute_dtype: str = "float32", has_relative_bias: bool = True, scale_output: bool = True, parallel_config: TransformerOpParallelConfig = default_transformer_config, checkpoint_name_or_path: str = None, top_p: float = 0.95, top_k: int = 1, repetition_penalty: float = 1.0, max_length: int = 20, do_sample: bool = False, param_init_type: str = "float32", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", hidden_act: str = 'relu', post_layernorm_residual: bool = False, offset: int = 0, use_past: bool = False, moe_config: MoEConfig = default_moe_config, **kwargs): super(T5Config, self).__init__(**kwargs) self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.d_ff = d_ff self.hidden_act = hidden_act self.kv_size = d_kv self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.embedding_dropout_prob = embedding_dropout_prob self.initializer_factor = initializer_factor self.max_position_embeddings = max_position_embeddings self.initializer_range = initializer_range self.max_decode_length = max_decode_length self.length_penalty_weight = length_penalty_weight self.has_relative_bias = has_relative_bias self.scale_output = scale_output self.parallel_config = parallel_config self.num_decoder_layers = num_decoder_layers self.relative_attention_num_buckets = relative_attention_num_buckets self.layer_norm_epsilon = layer_norm_epsilon self.use_cache = use_cache self.checkpoint_name_or_path = checkpoint_name_or_path self.pad_token_id = pad_token_id self.top_p = top_p self.top_k = top_k self.repetition_penalty = repetition_penalty self.max_length = max_length self.start_token_id = start_token_id self.eos_token_id = eos_token_id self.is_encoder_decoder = is_encoder_decoder self.do_sample = do_sample self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.compute_dtype = convert_mstype(compute_dtype) self.use_past = use_past self.post_layernorm_residual = post_layernorm_residual self.offset = offset self.moe_config = moe_config self.param_init_type = convert_mstype(param_init_type) pengcheng.mind-test/mindformers/models/t5/t5_processor.py000066400000000000000000000055511471400134000240600ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ T5Processor """ from mindformers.models.base_tokenizer import Tokenizer from mindformers.mindformer_book import MindFormerBook from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['T5Processor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class T5Processor(BaseProcessor): """ T5 processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['t5'] def __init__(self, tokenizer=None, max_length=77, tgt_max_length=128, padding='max_length', return_tensors='ms'): super(T5Processor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) self.tgt_max_length = tgt_max_length def __call__(self, text_input=None, text_pair=None): """call function""" output = {} if not self.tokenizer: raise ValueError(f"For {self.__name__}, the `tokenizer` should not be None.") if not isinstance(self.tokenizer, Tokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") if text_input: # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output if text_pair: # Format the input into a batch if isinstance(text_pair, str): text_input = [text_pair] text_output = self.tokenizer(text_pair, return_tensors=self.return_tensors, max_length=self.tgt_max_length, padding=self.padding)["input_ids"] output['tgt_output'] = text_output return output pengcheng.mind-test/mindformers/models/t5/t5_tokenizer.py000066400000000000000000000376331471400134000240610ustar00rootroot00000000000000# coding=utf-8 # Copyright 2018 T5 Authors and HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """T5 Tokenzier""" import os import re import warnings from shutil import copyfile from typing import Any, Dict, List, Optional import jieba import sentencepiece as spm from mindformers.tools import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_tokenizer import Tokenizer from mindformers.models.bert import BertTokenizer from ...mindformer_book import MindFormerBook __all__ = ["T5Tokenizer", "T5PegasusTokenizer"] VOCAB_FILES_NAMES = {"vocab_file": "spiece.model"} @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class T5Tokenizer(Tokenizer): """ Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 100): Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Attributes: sp_model (`SentencePieceProcessor`): The *SentencePiece* processor that is used for every conversion (string, tokens and IDs). """ vocab_files_names = VOCAB_FILES_NAMES FILE_LIST = ['tokenizer_config.json'] model_input_names = ["input_ids", "attention_mask"] _support_list = MindFormerBook.get_tokenizer_support_list()['t5'] def __init__( self, vocab_file, eos_token="", unk_token="", pad_token="", extra_ids=100, additional_special_tokens=None, sp_model_kwargs: Optional[Dict[str, Any]] = None, **kwargs, ) -> None: # Add extra_ids to the special token list if extra_ids > 0 and additional_special_tokens is None: additional_special_tokens = [f"" for i in range(extra_ids)] elif extra_ids > 0 and additional_special_tokens is not None: # Check that we have the right number of extra_id special tokens extra_tokens = len(set(filter(lambda x: bool("extra_id" in str(x)), additional_special_tokens))) if extra_tokens != extra_ids: raise ValueError( f"Both extra_ids ({extra_ids}) and additional_special_tokens ({additional_special_tokens}) are" " provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids" " tokens" ) self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs super().__init__( eos_token=eos_token, unk_token=unk_token, pad_token=pad_token, extra_ids=extra_ids, additional_special_tokens=additional_special_tokens, sp_model_kwargs=self.sp_model_kwargs, **kwargs, ) self.vocab_file = vocab_file self._extra_ids = extra_ids self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(vocab_file) @staticmethod def _eventually_correct_t5_max_length(pretrained_model_name_or_path, max_model_length, init_max_model_length): """eventually_correct_t5_max_length""" if pretrained_model_name_or_path in T5Tokenizer.max_model_input_sizes: deprecated_max_model_length = T5Tokenizer.max_model_input_sizes[pretrained_model_name_or_path] if init_max_model_length is not None and init_max_model_length != max_model_length: return init_max_model_length if init_max_model_length is None: warnings.warn( "This tokenizer was incorrectly instantiated with a model max length of" f" {deprecated_max_model_length} which will be corrected in Transformers v5.\nFor now, this" " behavior is kept to avoid breaking backwards compatibility when padding/encoding with" " `truncation is True`.\n- Be aware that you SHOULD NOT rely on" f" {pretrained_model_name_or_path} automatically truncating your input to" f" {deprecated_max_model_length} when padding/encoding.\n- If you want to encode/pad to sequences" f" longer than {deprecated_max_model_length} you can either instantiate this tokenizer with" " `model_max_length` or pass `max_length` when encoding/padding.\n- To avoid this warning, please" " instantiate this tokenizer with `model_max_length` set to your preferred value.", FutureWarning, ) return max_model_length @property def vocab_size(self): return self.sp_model.get_piece_size() + self._extra_ids def get_vocab(self): vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) # normal case: some special tokens if token_ids_1 is None: return ([0] * len(token_ids_0)) + [1] return ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1] def get_sentinel_tokens(self): return list( set(filter(lambda x: bool(re.search(r"", x)) is not None, self.additional_special_tokens)) ) def get_sentinel_token_ids(self): return [self._convert_token_to_id(token) for token in self.get_sentinel_tokens()] def _add_eos_if_not_present(self, token_ids: List[int]) -> List[int]: """Do not add eos again if user already added it.""" if token_ids and token_ids[-1] == self.eos_token_id: warnings.warn( f"This sequence already has {self.eos_token}. In future versions this behavior may lead to duplicated" " eos tokens being added." ) return token_ids return token_ids + [self.eos_token_id] def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. """ eos = [self.eos_token_id] if token_ids_1 is None: return len(token_ids_0 + eos) * [0] return len(token_ids_0 + eos + token_ids_1 + eos) * [0] def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. """ token_ids_0 = self._add_eos_if_not_present(token_ids_0) if token_ids_1 is None: return token_ids_0 token_ids_1 = self._add_eos_if_not_present(token_ids_1) return token_ids_0 + token_ids_1 def __getstate__(self): state = self.__dict__.copy() state["sp_model"] = None return state def __setstate__(self, d): self.__dict__ = d # for backward compatibility if not hasattr(self, "sp_model_kwargs"): self.sp_model_kwargs = {} self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(self.vocab_file) def _tokenize(self, text: str, **kwargs) -> List[str]: """Take as input a string and return a list of strings (tokens) for words/sub-words""" return self.sp_model.encode(text, out_type=str) def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" if token.startswith("", token) num = int(match.group(1)) return self.vocab_size - num - 1 return self.sp_model.piece_to_id(token) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" if index < self.sp_model.get_piece_size(): token = self.sp_model.IdToPiece(index) else: token = f"" return token def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" current_sub_tokens = [] out_string = "" prev_is_special = False for token in tokens: # make sure that special tokens are not decoded using sentencepiece model if token in self.all_special_tokens: if not prev_is_special: out_string += " " out_string += self.sp_model.decode(current_sub_tokens) + token prev_is_special = True current_sub_tokens = [] else: current_sub_tokens.append(token) prev_is_special = False out_string += self.sp_model.decode(current_sub_tokens) return out_string.strip() def save_vocabulary(self, save_directory, filename_prefix=None): if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None out_vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file): copyfile(self.vocab_file, out_vocab_file) elif not os.path.isfile(self.vocab_file): with open(out_vocab_file, "wb") as fi: content_spiece_model = self.sp_model.serialized_model_proto() fi.write(content_spiece_model) return out_vocab_file @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class T5PegasusTokenizer(BertTokenizer): """Tokenizer for T5 Pegasus, which is suitable for Chinese dataset. Based on word granularity, if the word does not appear in the vocabulary file, then call BERT native Tokenizer. Args: vocab_file(str): Path to vocab file. pre_tokenizer(function): Default using jieba to cut words """ def __init__(self, vocab_file, *args, pre_tokenizer=lambda x: jieba.cut(x, HMM=False), **kwargs): super(T5PegasusTokenizer, self).__init__(vocab_file, *args, **kwargs) self.pre_tokenizer = pre_tokenizer def _tokenize(self, text, **kwargs): split_tokens = [] for word in self.pre_tokenizer(text): if word in self.vocab_dict: split_tokens.append(word) else: split_tokens.extend(super()._tokenize(word)) return split_tokens def convert_tokens_to_string(self, tokens: list): """Convert the tokens to the string""" return " ".join(tokens) def _decode(self, token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=None, **kwargs): ids = self.convert_ids_to_tokens( token_ids, skip_special_tokens=skip_special_tokens ) return self.convert_tokens_to_string(ids) pengcheng.mind-test/mindformers/models/utils.py000066400000000000000000000023241471400134000222340ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Check Model Input Config.""" import mindspore.common.dtype as mstype from ..version_control import get_cell_reuse def convert_mstype(ms_type: str = "float16"): """Convert the string type to MindSpore type.""" if isinstance(ms_type, mstype.Float): return ms_type if ms_type == "float16": return mstype.float16 if ms_type == "float32": return mstype.float32 raise KeyError(f"Supported data type keywords include: " f"[float16, float32], but get {ms_type}") def cell_reuse(): return get_cell_reuse() pengcheng.mind-test/mindformers/models/vit/000077500000000000000000000000001471400134000213235ustar00rootroot00000000000000pengcheng.mind-test/mindformers/models/vit/__init__.py000066400000000000000000000016641471400134000234430ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Vit Model API.""" from .vit import ViTForImageClassification, ViTModel from .vit_config import ViTConfig from .vit_processor import ViTProcessor, ViTImageProcessor __all__ = ['ViTForImageClassification', 'ViTModel', 'ViTConfig', 'ViTProcessor', 'ViTImageProcessor'] pengcheng.mind-test/mindformers/models/vit/convert_weight.py000066400000000000000000000064441471400134000247340ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/facebook""" import argparse import torch import mindspore as ms def convert_weight(torch_pth_path="mae_pretrain_vit_base.pth", ms_ckpt_path="mae_pretrain_vit_base.ckpt"): """ convert mae_vit_base_p16 weights from pytorch to mindspore pytorch and GPU required. """ param_dict = torch.load(torch_pth_path, map_location=torch.device("cpu")) new_dict = [] for k, v in param_dict["model"].items(): if k in ("cls_token", "mask_token"): k += "s" if "head" not in k: k = "vit." + k if "norm" in k: if "fc_norm" not in k: k = k.replace("norm", "layernorm") if "weight" in k: k = k.replace("weight", "gamma") elif "bias" in k: k = k.replace("bias", "beta") if "mlp" in k: k = k.replace("mlp", "output") if "fc1" in k: k = k.replace("fc1", "mapping") if "weight" in k: v = v.transpose(-1, 0) elif "fc2" in k: k = k.replace("fc2", "projection") if "weight" in k: v = v.transpose(-1, 0) if "attn" in k: k = k.replace("attn", "attention") if "proj" in k: k = k.replace("proj", "projection") if "weight" in k: v = v.transpose(-1, 0) if "qkv" not in k: new_dict.append({"name": k, "data": ms.Tensor(v.numpy())}) else: data = ms.Tensor(v.numpy()) length = data.shape[0] // 3 new_dict.append({"name": k.replace(".qkv", ".dense1"), "data": data[:length]}) new_dict.append({"name": k.replace(".qkv", ".dense2"), "data": data[length:length*2]}) new_dict.append({"name": k.replace(".qkv", ".dense3"), "data": data[length*2:]}) ms.save_checkpoint(new_dict, ms_ckpt_path) print("Weights conversion completes. ") if __name__ == "__main__": parser = argparse.ArgumentParser(description="mae vit weight convert script") parser.add_argument("--torch_pth_path", type=str, default="mae_pretrain_vit_base.pth", required=True, help="The torch checkpoint path.") parser.add_argument("--ms_ckpt_path", type=str, required=True, default="mae_pretrain_vit_base.ckpt", help="The output mindspore checkpoint path.") opt = parser.parse_args() convert_weight(opt.torch_pth_path, opt.ms_ckpt_path) pengcheng.mind-test/mindformers/models/vit/vit.py000066400000000000000000000316741471400134000225120ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/facebookresearch/mae # ============================================================================ """ViT Model.""" import math import numpy as np from mindspore import load_param_into_net, Parameter, nn from mindspore import ops as P from mindspore import dtype as mstype import mindspore.common.initializer as weight_init from mindformers.mindformer_book import MindFormerBook from mindformers.core.loss import build_loss from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.models.vit.vit_modules import Block, LayerNorm, Linear, Dropout, PixelShuffle from mindformers.models.vit.vit_modules import PatchEmbed from mindformers.models.vit.vit_config import ViTConfig @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTModel(BaseModel): """ Vision Transformer with support for patch or hybrid CNN input stage. The supported model name could be selected from ViTConfig.show_support_list(). Args: config (ViTConfig): the config of Vit model. Examples: >>> # input model name, load model and weights >>> model_a = ViTModel.from_pretrained('vit_base_p16') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('vit_base_p16') >>> model_b = ViTModel(config) """ _support_list = MindFormerBook.get_model_support_list()['vit'] def __init__(self, config=None): config = config if config else ViTConfig() super().__init__(config) self.use_moe = (config.moe_config.expert_num > 1) parallel_config = config.parallel_config dp = parallel_config.data_parallel self.global_pool = config.use_mean_pooling self.patch_embed = PatchEmbed(img_size=config.image_size, patch_size=config.patch_size, in_features=config.in_chans, out_features=config.embed_dim, parallel_config=parallel_config) self.cls_tokens = Parameter( weight_init.initializer(weight_init.TruncatedNormal(sigma=config.initializer_range), (1, 1, config.embed_dim)), requires_grad=True) num_patches = self.patch_embed.num_patches seq_length = num_patches + 1 self.seq_length = seq_length self.num_patches = num_patches self.num_masked = num_patches - seq_length + 1 self.pos_embed = Parameter( weight_init.initializer(weight_init.TruncatedNormal(sigma=config.initializer_range), (1, seq_length, config.embed_dim)), requires_grad=True) # stochastic depth decay rule hdr = [x.item() for x in np.linspace(0, config.drop_path_rate, config.depth)] parallel_config_args = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config self.blocks = nn.CellList([ Block(hidden_size=config.embed_dim, ffn_hidden_size=config.intermediate_size, seq_length=seq_length, drop_rate=config.drop_rate, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=hdr[i], layer_norm_eps=config.layer_norm_eps, qkv_bias=config.qkv_bias, init_values=config.init_values, weight_init='XavierUniform', layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, window_size=None, num_heads=config.num_heads, hidden_act=config.hidden_act, post_layernorm_residual=config.post_layernorm_residual, param_init_type=config.param_init_type, parallel_config=parallel_config_args) for i in range(config.depth)]) self.add = P.Add().shard(((dp, 1, 1), (1, 1, 1))) self.cast = P.Cast() self.tile = P.Tile().shard(((dp, 1, 1),)) self.cat = P.Concat(axis=1) self.fc_norm = LayerNorm((config.embed_dim,), eps=1e-6).shard(((dp, 1, 1),)) self.reduce_mean = P.ReduceMean().shard(((dp, 1, 1),)) self.dropout = Dropout(keep_prob=(1. - config.drop_rate)) self.dropout.shard(((dp, 1, 1),)) self.stride_slice = P.StridedSlice().shard(((dp, 1, 1),)) self.init_weights_vit() self.fix_init_weight() def fix_init_weight(self): """fix init weight""" def rescale(param, layer_id): values = param.data / (math.sqrt(2.0 * layer_id)) param.set_data(values) for layer_id, block in enumerate(self.blocks): if self.use_moe: rescale(block.attention.projection.weight, layer_id + 1) rescale(block.output.ffn.projection.weight, layer_id + 1) else: rescale(block.attention.projection.weight, layer_id + 1) rescale(block.output.projection.weight, layer_id + 1) def init_weights_vit(self): """init weights vit ViT weight initialization, original timm impl (for reproducibility) """ for name, cell in self.cells_and_names(): if isinstance(cell, Linear): cell.weight.set_data(weight_init.initializer( weight_init.TruncatedNormal(sigma=self.config.initializer_range), cell.weight.shape, cell.weight.dtype)) if isinstance(cell, Linear) and cell.bias is not None: cell.bias.set_data(weight_init.initializer(weight_init.Zero(), cell.bias.shape, cell.bias.dtype)) elif isinstance(cell, (LayerNorm, nn.LayerNorm)): cell.gamma.set_data(weight_init.initializer(weight_init.One(), cell.gamma.shape, cell.gamma.dtype)) cell.beta.set_data(weight_init.initializer(weight_init.Zero(), cell.beta.shape, cell.beta.dtype)) if name == "patch_embed.proj": cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) def no_weight_decay(self): return {'pos_embed', 'cls_tokens'} def load_pretrained(self, params_dict): return load_param_into_net(self, params_dict) def construct_without_pool(self, image, mask=None): """construct of vit without pool""" tokens = self.patch_embed(image, mask) batch_size = image.shape[0] cls_tokens = self.tile(self.cls_tokens, (batch_size, 1, 1)) tokens = self.cat((cls_tokens, tokens)) if self.pos_embed is not None: tokens = self.add(tokens, self.pos_embed) x = self.dropout(tokens) encoder_input_mask = P.Ones()((batch_size, self.seq_length, self.seq_length), mstype.float32) for block in self.blocks: x = block(x, encoder_input_mask) return x def construct(self, image): """construct of vit""" x = self.construct_without_pool(image) b, s, c = x.shape if self.global_pool: x = self.stride_slice( x, (0, 1, 0), (b, s, c), (1, 1, 1) ) x = self.reduce_mean(x, 1) out = self.fc_norm(x) else: out = self.stride_slice( x, (0, 0, 0), (b, 1, c), (1, 1, 1) ) return out @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTForImageClassification(BaseModel): """ Vision Transformer with support for patch or hybrid CNN input stage. The supported model name could be selected from ViTConfig.show_support_list(). Args: config (ViTConfig): the config of Vit model. Examples: >>> # input model name, load model and weights >>> model_a = ViTForImageClassification.from_pretrained('vit_base_p16') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('vit_base_p16') >>> model_b = ViTForImageClassification(config) """ _support_list = MindFormerBook.get_model_support_list()['vit'] def __init__(self, config=None): config = config if config else ViTConfig() super().__init__(config) self.vit = ViTModel(config) self.head = Linear( config.embed_dim, config.num_classes, weight_init=weight_init.TruncatedNormal(sigma=2e-5), compute_dtype=mstype.float32).to_float(mstype.float32) self.loss = build_loss(class_name=config.loss_type) self.load_checkpoint(config) def construct(self, image, target=None): """construct of vit""" out = self.vit(image) out = self.head(out) if not self.training: return out, target loss = self.loss(out, target) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTForMaskedImageModeling(BaseModel): """ Vision Transformer with support for patch or hybrid CNN input stage. The supported model name could be selected from ViTConfig.show_support_list(). Args: config (ViTConfig): the config of Vit model. Examples: >>> # input model name, load model and weights >>> model_a = ViTForMaskedImageModeling.from_pretrained('vit_base_p16') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('vit_base_p16') >>> model_b = ViTForMaskedImageModeling(config) """ _support_list = MindFormerBook.get_model_support_list()['vit'] def __init__(self, config=None): config = config if config else ViTConfig() super().__init__(config) self.vit = ViTModel(config) self.vit.patch_embed = PatchEmbed(img_size=config.image_size, patch_size=config.patch_size, in_features=config.in_chans, out_features=config.embed_dim, use_mask=True, parallel_config=config.parallel_config) self.decoder = nn.CellList( nn.Conv2d( in_channels=config.embed_dim, out_channels=config.encoder_stride ** 2 * config.in_chans, kernel_size=1, ), PixelShuffle(config.encoder_stride), ) self.transpose = P.Transpose() self.reshape = P.Reshape() self.expand_dims = P.ExpandDims() self.l1_loss = nn.L1Loss(reduction='none') # Initialize weights and apply final processing self.init_weights_vit() def init_weights_vit(self): for _, cell in self.cells_and_names(): if isinstance(cell, nn.Conv2d): cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) def construct(self, image, mask=None): """construct of vit for MIM""" x = self.vit.construct_without_pool(image) b, s, c = x.shape height = width = math.floor(s ** 0.5) x = self.reshape(self.transpose(x, (0, 2, 1)), (b, c, height, width)) reconstruct_images = self.decoder(x) if not self.training: return reconstruct_images size = self.config.image_size // self.config.patch_size mask = self.reshape(mask, (-1, size, size)) mask = P.repeat_elements(mask, self.config.patch_size, 1) mask = P.repeat_elements(mask, self.config.patch_size, 2) mask = self.expand_dims(mask, 1) reconstruction_loss = self.l1_loss(image, reconstruct_images) masked_im_loss = (reconstruction_loss * mask).sum() / (mask.sum() + 1e-5) / self.config.in_chans return masked_im_loss pengcheng.mind-test/mindformers/models/vit/vit_config.py000066400000000000000000000110041471400134000240200ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Mae Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.modules.transformer.moe import MoEConfig, default_moe_config from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_config import BaseConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class ViTConfig(BaseConfig): """ Config for ViT model Examples: >>> # init a config with a model name >>> config_a = ViTConfig.from_pretrained('vit_base_p16') >>> # init a config with a config path >>> import os >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), >>> 'configs', 'vit', 'run_vit_base_p16_224_100ep.yaml') >>> config_b = ViTConfig.from_pretrained(config_path) >>> # init a config with args >>> config_c = ViTConfig( >>> patch_size=16, >>> in_chans=3, >>> ... >>> ) """ _support_list = MindFormerBook.get_config_support_list()['vit'] def __init__(self, image_size: int = 224, patch_size: int = 16, num_channels: int = 3, initializer_range: float = 0.02, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, qkv_bias: bool = True, hidden_act: str = "gelu", post_layernorm_residual: bool = False, layer_norm_eps: float = 1e-12, attention_probs_dropout_prob: float = 0.0, hidden_dropout_prob: float = 0.0, drop_path_rate: float = 0.1, use_mean_pooling: bool = True, num_labels: int = 1000, loss_type: str = "SoftTargetCrossEntropy", encoder_stride: int = 16, checkpoint_name_or_path: str = '', layernorm_compute_type: mstype = mstype.float32, softmax_compute_type: mstype = mstype.float32, param_init_type: mstype = mstype.float32, parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs): super().__init__(**kwargs) self.image_size = image_size self.patch_size = patch_size self.in_chans = num_channels self.initializer_range = initializer_range self.embed_dim = hidden_size self.depth = num_hidden_layers self.num_heads = num_attention_heads self.intermediate_size = intermediate_size self.qkv_bias = qkv_bias self.hidden_act = hidden_act self.post_layernorm_residual = post_layernorm_residual self.layer_norm_eps = layer_norm_eps self.attention_dropout_rate = attention_probs_dropout_prob self.drop_rate = hidden_dropout_prob self.drop_path_rate = drop_path_rate self.num_classes = num_labels self.use_mean_pooling = use_mean_pooling self.loss_type = loss_type self.encoder_stride = encoder_stride self.checkpoint_name_or_path = checkpoint_name_or_path self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.param_init_type = param_init_type self.parallel_config = parallel_config self.moe_config = moe_config pengcheng.mind-test/mindformers/models/vit/vit_modules.py000066400000000000000000001111521471400134000242300ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Modules of ViTForImageClassification, including Linear, Block, MLP, Attention, PatchEmbed, etc. """ import math import numpy as np from mindspore import nn, Parameter, Tensor, context, ParallelMode import mindspore.common.dtype as mstype from mindspore.ops import operations as P, constexpr from mindspore.ops import functional as F import mindspore.common.initializer as init from mindspore.parallel._utils import _get_parallel_mode from mindformers.modules import layers, FeedForward from mindformers.modules.transformer.moe import default_moe_config, _check_moe_config from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, _check_config @constexpr def gen_shape(x_shape, ndim): return (x_shape,) + (1,) * (ndim + 1) def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False): """ grid_size: int of the grid height and width return: pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token) """ grid_h = np.arange(grid_size, dtype=np.float32) grid_w = np.arange(grid_size, dtype=np.float32) grid = np.meshgrid(grid_w, grid_h) # here w goes first grid = np.stack(grid, axis=0) grid = grid.reshape([2, 1, grid_size, grid_size]) pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid) if cls_token: pos_embed = np.concatenate([np.zeros([1, embed_dim]), pos_embed], axis=0) return pos_embed def get_2d_sincos_pos_embed_from_grid(embed_dim, grid): assert embed_dim % 2 == 0 # use half of dimensions to encode grid_h emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2) emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2) emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D) return emb def get_1d_sincos_pos_embed_from_grid(embed_dim, pos): """ embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D) """ assert embed_dim % 2 == 0 omega = np.arange(embed_dim // 2, dtype=np.float32) omega /= embed_dim / 2. omega = 1. / 10000 ** omega # (D/2,) pos = pos.reshape(-1) # (M,) out = np.einsum('m,d->md', pos, omega) # (M, D/2), outer product emb_sin = np.sin(out) # (M, D/2) emb_cos = np.cos(out) # (M, D/2) emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D) return emb class LayerNorm(layers.LayerNorm): # pylint: disable=W0212 r""" A self-defined layer norm operation using reduce sum and reduce mean. """ def __init__(self, normalized_shape, eps=1e-6, param_init_type=mstype.float32): super(LayerNorm, self).__init__( normalized_shape, eps=eps, param_init_type=param_init_type) class Identity(nn.Cell): def construct(self, x): return x class RelativePositionBias(nn.Cell): """relative position bias""" def __init__(self, window_size, num_heads): super(RelativePositionBias, self).__init__() self.window_size = window_size # cls to token & token to cls & cls to cls self.num_relative_distance = (2 * window_size[0] - 1) * (2 * window_size[1] - 1) + 3 self.relative_position_bias_table = Parameter( init.initializer( init.TruncatedNormal(sigma=.02), (self.num_relative_distance, num_heads)), name='relative_position_bias_table') # get pair-wise relative position index for each token inside the window coords_h = Tensor(np.arange(window_size[0]), mstype.int32) coords_w = Tensor(np.arange(window_size[1]), mstype.int32) coords = P.Stack(axis=0)(P.Meshgrid(indexing='ij')((coords_h, coords_w))) # 2, Wh, Ww coords_flatten = P.Flatten()(coords) # 2, Wh*Ww relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww relative_coords = P.Transpose()(relative_coords, (1, 2, 0)).asnumpy() # Wh*Ww, Wh*Ww, 2 relative_coords[:, :, 0] += window_size[0] - 1 relative_coords[:, :, 1] += window_size[1] - 1 relative_coords[:, :, 0] *= 2 * window_size[1] - 1 relative_position_index = \ np.zeros(((window_size[0] * window_size[1] + 1),) * 2, dtype=int) relative_position_index[1:, 1:] = np.sum(relative_coords, axis=-1) # Wh*Ww, Wh*Ww relative_position_index[0, 0:] = self.num_relative_distance - 3 relative_position_index[0:, 0] = self.num_relative_distance - 2 relative_position_index[0, 0] = self.num_relative_distance - 1 relative_position_index = Tensor(relative_position_index, mstype.int32) self.relative_position_index = Parameter( relative_position_index, requires_grad=False, name="relative_position_index") self.reshape = P.Reshape() self.transpose = P.Transpose() self.gather = P.Gather() def construct(self): relative_position_index = self.relative_position_index.view(-1) relative_position_bias = self.gather(self.relative_position_bias_table, relative_position_index, 0) relative_position_bias = self.reshape( relative_position_bias, (self.window_size[0] * self.window_size[1] + 1, self.window_size[0] * self.window_size[1] + 1, -1)) relative_position_bias = self.transpose(relative_position_bias, (2, 0, 1)) return relative_position_bias class Linear(layers.Linear): # pylint: disable=W0212 r""" Linear function for RingMo. """ def __init__(self, in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, transpose_b=True, expert_num=1, outer_batch=1, activation_compute_type=mstype.float16, param_init_type=mstype.float32, compute_dtype=mstype.float16): super(Linear, self).__init__( in_channels, out_channels, weight_init=weight_init, bias_init=bias_init, has_bias=has_bias, activation=activation, transpose_b=transpose_b, expert_num=expert_num, outer_batch=outer_batch, param_init_type=param_init_type, compute_dtype=compute_dtype) self.activation_compute_type = activation_compute_type def construct(self, x): """construct of layer""" out_shape = P.Shape()(x)[:-1] + (self.out_channels,) x = P.Reshape()(x, (-1, self.in_channels)) if self.expert_flag: x = P.Reshape()(x, (self.outer_batch, self.expert_num, -1, self.in_channels)) weight = self.cast(self.weight, self.dtype) x = self.matmul(x, weight) if self.has_bias: x = self.bias_add(x, self.cast(self.bias, self.dtype)) if self.activation_flag: x = self.activation(self.cast(x, self.activation_compute_type)) x = self.cast(x, self.dtype) output = P.Reshape()(x, out_shape) return output class Dropout(layers.Dropout): # pylint: disable=W0212 r""" A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for context training. """ def __init__(self, keep_prob=0.5, dtype=mstype.float32): super(Dropout, self).__init__(keep_prob=keep_prob, dtype=dtype) class DropPath(nn.Cell): """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" def __init__(self, drop_prob): super(DropPath, self).__init__() self.drop = Dropout(keep_prob=1 - drop_prob) self.mask = Tensor(np.ones(1,), dtype=mstype.float32) self.tile = P.Tile() self.mul = P.Mul() def construct(self, x): if not self.training: return x mask = self.tile(self.mask, (x.shape[0],) + (1,) * (x.ndim-1)) out = self.drop(mask) out = self.mul(out, x) return out def shard(self, strategy): self.mul.shard(strategy) class Block(nn.Cell): """Block of ringmo""" def __init__(self, hidden_size, ffn_hidden_size, num_heads, seq_length, drop_rate=0., attention_dropout_rate=0., hidden_dropout_rate=0., layer_norm_eps=1e-6, qkv_bias=True, window_size=None, post_layernorm_residual=False, init_values=None, weight_init='XavierUniform', layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', moe_config=default_moe_config, parallel_config=default_dpmp_config): super(Block, self).__init__() _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.seq_length = seq_length self.hidden_size = hidden_size self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None self.layernorm1 = LayerNorm((hidden_size,), eps=layer_norm_eps).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,), eps=layer_norm_eps).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) parallel_config_args = parallel_config.dpmp if self.use_moe else parallel_config self.attention = Attention(src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, window_size=window_size, num_heads=num_heads, weight_init=weight_init, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, qkv_bias=qkv_bias, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=parallel_config_args) self.output = MLP(hidden_size=hidden_size, dropout_rate=drop_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, weight_init=weight_init, hidden_act=hidden_act, use_dropout=False, parallel_config=parallel_config) if init_values is not None: self.gamma_1 = Parameter( Tensor(init_values * np.ones((hidden_size,)), mstype.float32), name="gamma1", requires_grad=True) self.gamma_2 = Parameter( Tensor(init_values * np.ones((hidden_size,)), mstype.float32), name="gamma2", requires_grad=True) else: self.gamma_1, self.gamma_2 = None, None self.mul_gamma = P.Mul().shard(((parallel_config.data_parallel, 1), (1,))) self.drop_path = DropPath(hidden_dropout_rate) self.drop_path.shard(((parallel_config.data_parallel, 1),)) self.drop_path3d = DropPath(hidden_dropout_rate) self.drop_path3d.shard(((parallel_config.data_parallel, 1, 1),)) self.mul = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.reshape = P.Reshape() def construct(self, x, input_mask, rel_pos_bias=None): """construct of Block""" x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) attention = self.attention( input_x, input_x, input_x, input_mask, rel_pos_bias) if self.gamma_1 is not None: attention = self.mul_gamma(attention, self.gamma_1) if len(x_shape) == 3: attention = P.Reshape()(attention, x_shape) attention = self.drop_path3d(attention) attention = P.Reshape()(attention, (-1, x_shape[-1])) else: attention = self.drop_path(attention) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) mlp_logit = self.output(output_x) if self.gamma_2 is not None: mlp_logit = self.mul_gamma(mlp_logit, self.gamma_2) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) mlp_logit = self.drop_path3d(mlp_logit) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: mlp_logit = self.drop_path(mlp_logit) if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) return output class MLP(FeedForward): r"""MLP for ring-mo.""" def __init__(self, hidden_size, ffn_hidden_size=None, out_features=None, dropout_rate=0., hidden_act='gelu', weight_init='XavierUniform', use_dropout=True, param_init_type=mstype.float32, parallel_config=default_dpmp_config): ffn_hidden_size = ffn_hidden_size or hidden_size super(MLP, self).__init__( hidden_size, ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, param_init_type=mstype.float32, parallel_config=parallel_config) mp = parallel_config.model_parallel dp = parallel_config.data_parallel out_features = out_features or hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=hidden_size, out_channels=ffn_hidden_size, activation=hidden_act, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=ffn_hidden_size, out_channels=out_features, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - dropout_rate) self.dropout.shard(((dp, 1),)) self.dropout_3d = Dropout(1 - dropout_rate) self.dropout_3d.shard(((dp, 1, 1),)) self.use_dropout = use_dropout def construct(self, x): """construct of mlp""" x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) if self.use_dropout: if len(F.shape(hidden)) == 3: hidden = self.dropout_3d(hidden) elif len(F.shape(hidden)) == 2: hidden = self.dropout(hidden) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) return output class Attention(nn.Cell): r""" This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following """ def __init__(self, src_seq_length, tgt_seq_length, hidden_size, num_heads, window_size=None, hidden_dropout_rate=0., attention_dropout_rate=0., qkv_bias=True, weight_init='XavierUniform', compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(Attention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] _check_config(parallel_config) self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - hidden_dropout_rate) self.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout = Dropout(1 - attention_dropout_rate) self.prob_dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) if window_size: self.relative_position_bias = RelativePositionBias(window_size, num_heads) else: self.window_size = None self.relative_position_bias_table = None self.relative_position_index = None self.relative_position_bias = None self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1, 1), (1, 1, 1))) self.expand_dims_rpb = P.ExpandDims().shard(((1, 1, 1),)) self.add_rpb = P.Add().shard(((parallel_config.data_parallel, 1, 1, 1), (1, 1, 1, 1))) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, rel_pos_bias=None): """construct of attention""" ori_shape = F.shape(query_tensor) batch_size = F.shape(attention_mask)[0] query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, attention_mask, rel_pos_bias) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) return output def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( F.reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = F.reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask, rel_pos_bias): r"""Get the weighted score along the seq_length.""" # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] factor = P.Cast()(self.scale_factor, P.DType()(query)) query = self.real_div(query, factor) key = self.real_div(key, factor) score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) score = P.Cast()(score, self.softmax_dtype) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(score)), P.Cast()(attention_mask, P.DType()(score))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, score) # add window cross module if self.relative_position_bias is not None: relative_position_bias = self.expand_dims_rpb(self.relative_position_bias(), 0) attention_scores = self.add_rpb(attention_scores, relative_position_bias) if rel_pos_bias is not None: attention_scores = self.add_3d(attention_scores, rel_pos_bias) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = P.Cast()(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge def _merge_heads(self, x): """ convert a 4d input to a 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor class PatchEmbed(nn.Cell): """Construct the embeddings from patch, position embeddings.""" def __init__(self, img_size=224, patch_size=16, in_features=3, out_features=768, norm_layer=False, use_mask=False, parallel_config=None): super(PatchEmbed, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 img_size = (img_size, img_size) patch_size = (patch_size, patch_size) self.grid_size = (img_size[0] // patch_size[0], img_size[1] // patch_size[1]) self.num_patches = self.grid_size[0] * self.grid_size[1] self.proj = nn.Conv2d( in_channels=in_features, out_channels=out_features, kernel_size=patch_size, stride=patch_size, weight_init=init.TruncatedNormal(sigma=0.02), has_bias=True, pad_mode='pad') self.proj.conv2d.shard(((dp, 1, 1, 1), (1, 1, 1, 1))) self.proj.bias_add.shard(((dp, 1, 1, 1), (1,))) self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1),)) # usually not use norm self.norm = LayerNorm((out_features,), eps=1e-6).shard(((dp, 1, 1),)) if norm_layer else Identity() self.mask_token = Parameter(init.initializer(init.Zero(), (1, 1, out_features)), requires_grad=True) if use_mask else None self.tile = P.Tile() self.mul = P.Mul() self.add = P.Add() def construct(self, x, mask=None): """construct""" # True x: bs False x: bs * dp x = self.proj(x) b, c, h, w = x.shape x = self.reshape(x, (b, c, h * w)) x = self.transpose(x, (0, 2, 1)) x = self.norm(x) if mask is not None: mask_tokens = self.tile(self.mask_token, (b, self.num_patches, 1)) x = self.mul(x, (1.0 - mask)) x = self.add(x, self.mul(mask_tokens, mask)) return x def get_kernel_size(patch_size): """ input: 2^i & i <= 5 | 14 output: a list of kernel size """ x = None y = None z = None ans = False for i in range(1, patch_size + 1): if patch_size % i == 0: x = i mul_y_z = patch_size // i for j in range(1, mul_y_z + 1): if mul_y_z % j == 0: y = j z = mul_y_z // j if x >= y >= z: ans = True break if ans: break if not ans: raise ValueError(patch_size) return [x, y, z] class Patchify(nn.Cell): """Patchify""" def __init__(self, patch_size, parallel_config=None): super(Patchify, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.patch_size = patch_size self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, img): p = self.patch_size bs, channels, h, w = img.shape x = self.reshape(img, (bs, channels, h // p, p, w // p, p)) x = self.transpose(x, (0, 2, 4, 1, 3, 5)) patches = self.reshape(x, (bs, (h // p) * (w // p), channels * p * p)) return patches class UnPatchify(nn.Cell): """UnPatchify""" def __init__(self, patch_size, seq_length, parallel_config=None): super(UnPatchify, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.p = patch_size self.h = self.w = int(seq_length ** .5) assert self.h * self.w == seq_length self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, x): bs = x.shape[0] x = self.reshape(x, (bs, self.h, self.w, self.p, self.p, 3)) x = self.transpose(x, (0, 5, 1, 3, 2, 4)) images = self.reshape(x, (bs, 3, self.h * self.p, self.w * self.p)) return images class PixelShuffle(nn.Cell): """PixelShuffle""" def __init__(self, upscale_factor): super().__init__() self.upscale_factor = upscale_factor self.reshape = P.Reshape() self.transpose = P.Transpose() def construct(self, x): b, _, h, w = x.shape x = self.reshape(x, (b, -1, self.upscale_factor, self.upscale_factor, h, w)) x = self.transpose(x, (0, 1, 4, 2, 5, 3)) x = self.reshape(x, (b, -1, h * self.upscale_factor, w * self.upscale_factor)) return x pengcheng.mind-test/mindformers/models/vit/vit_processor.py000066400000000000000000000111201471400134000245710ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ ViTProcessor """ import numpy as np from PIL import Image from mindspore import Tensor from mindspore.dataset.vision import CenterCrop, ToTensor, Normalize, Rescale from mindformers.mindformer_book import MindFormerBook from mindformers.dataset import Resize from mindformers.dataset.base_dataset import BaseDataset from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class ViTImageProcessor(BaseImageProcessor): """ ViTImageProcessor. Args: image_resolution (int): the target size. """ def __init__(self, size=224): super().__init__(image_resolution=size) self.resize = Resize(256, interpolation='cubic') self.center_crop = CenterCrop(size) self.to_tensor = ToTensor() self.rescale = Rescale(1.0 / 255.0, 0.0) self.normalize = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], is_hwc=False) def preprocess(self, images, **kwargs): """ Preprocess required by base processor. Args: images (tensor, PIL.Image, numpy.array, list): a batch of images. Return: A 4-rank tensor for a batch of images. """ images = self._format_inputs(images) res = [] for image in images: image = self.resize(image) image = self.center_crop(image) image = self.to_tensor(image) image = self.normalize(image) res.append(image) return Tensor(res) def _format_inputs(self, inputs): """ Transform image classification inputs into (bz, h, w, c) or (h, w, c) numpy array. Args: inputs (tensor, numpy.array, PIL.Image, list, BaseDataset): for numpy or tensor input, the channel could be (bz, c, h, w), (c, h, w) or (bz, h, w, c), (h, w, c); for list, the item could be PIL.Image, numpy.array, Tensor; for BaseDataset, return without any operations. Return: transformed images: for PIL.Image, numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for list, return a numpy array for each element; for BaseDataset, it is returned directly. """ if not isinstance(inputs, (list, Image.Image, Tensor, np.ndarray, BaseDataset)): raise TypeError("input type is not Tensor, numpy, Image, list of Image or MindFormer BaseDataset") if isinstance(inputs, list): return [self._format_inputs(item) for item in inputs] if isinstance(inputs, Image.Image): inputs = np.array(inputs) if isinstance(inputs, Tensor): inputs = inputs.asnumpy() if isinstance(inputs, np.ndarray): if len(inputs.shape) == 3: inputs = np.expand_dims(inputs, 0) inputs = self._chw2hwc(inputs) elif len(inputs.shape) == 4: inputs = self._chw2hwc(inputs) else: raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(inputs.shape)}") return inputs @staticmethod def _chw2hwc(inputs): if inputs.shape[-1] != 3: inputs = inputs.transpose(0, 2, 3, 1) return inputs @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class ViTProcessor(BaseProcessor): """ Vit processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['vit'] def __init__(self, image_processor=None, return_tensors='ms'): super().__init__( image_processor=image_processor, return_tensors=return_tensors ) pengcheng.mind-test/mindformers/modules/000077500000000000000000000000001471400134000207065ustar00rootroot00000000000000pengcheng.mind-test/mindformers/modules/__init__.py000066400000000000000000000016421471400134000230220ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Transformers API.""" from .transformer import * from .layers import * from .local_block_sparse_attention import * __all__ = [] __all__.extend(transformer.__all__) __all__.extend(layers.__all__) __all__.extend(local_block_sparse_attention.__all__) pengcheng.mind-test/mindformers/modules/__pycache__/000077500000000000000000000000001471400134000231165ustar00rootroot00000000000000pengcheng.mind-test/mindformers/modules/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005471471400134000273140ustar00rootroot00000000000000a ?cf@sHdZddlTddlTddlTgZeejeejeejdS)zMindFormers Transformers API.)*N)__doc__ transformerlayersZlocal_block_sparse_attention__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/modules/__init__.pys  pengcheng.mind-test/mindformers/modules/__pycache__/activation.cpython-39.pyc000066400000000000000000001512401471400134000277130ustar00rootroot00000000000000a ?cf@sdZddlmZddlZddlmZddlm Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZgdZGdddeZGdddeZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&Gd d!d!eZ'Gd"d#d#eZ(Gd$d%d%eZ)Gd&d'd'eZ*Gd(d)d)eZ+Gd*d+d+eZ,Gd,d-d-eZ-Gd.d/d/eZ.Gd0d1d1eZ/Gd2d3d3eZ0Gd4d5d5eZ1Gd6d7d7eZ2Gd8d9d9eZ3Gd:d;d;eZ4Gdd?d?eZ6Gd@dAdAeZ7GdBdCdCeZ8GdDdEdEeZ9GdFdGdGeZ:GdHdIdIeZ;e e"e!e#e%e&e(e*e+e,e-e.e/e$e0e1e2e'e3e4e5e7e8e9e:dJZ>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-2.0, -1.0, 1.0, 2.0]), mindspore.float32) >>> celu = nn.CELU() >>> output = celu(x) >>> print(output) [-0.86466473 -0.63212055 1. 2. ] ?cs tt|tj|d|_dS)zInitialize CELU.)alphaN)superr*__init__PCeLUceluselfr/ __class__I/home/ma-user/work/mindformers-baichuan/mindformers/modules/activation.pyr1nsz CELU.__init__cCs ||SN)r4r6xr9r9r: constructsszCELU.construct)r.__name__ __module__ __qualname____doc__r1r> __classcell__r9r9r7r:r*As,r*cs*eZdZdZdfdd ZddZZS)ra: Softmin activation function, which is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of multi-classification, and the purpose is to show the results of multi-classification in the form of probability. Calculate the value of the exponential function for the elements of the input Tensor on the `axis`, and then normalized to lie in range [0, 1] and sum up to 1. Softmin is defined as: .. math:: \text{softmin}(x_{i}) = \frac{\exp(-x_i)}{\sum_{j=0}^{n-1}\exp(-x_j)}, where :math:`x_{i}` is the :math:`i`-th slice in the given dimension of the input Tensor. Args: axis (Union[int, tuple[int]]): The axis to apply Softmin operation, if the dimension of input `x` is x.ndim, the range of axis is `[-x.ndim, x.ndim)`. -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - Tensor for computing Softmin functions with data type of float16 or float32. Outputs: Tensor, which has the same type and shape as `x` with values in the range [0,1]. Raises: TypeError: If `axis` is neither an int nor a tuple. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is a tuple whose length is less than 1. ValueError: If `axis` is a tuple whose elements are not all in the range [-x.ndim, x.ndim). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> # axis = -1(default), and the sum of return value is 1.0. >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> softmin = nn.Softmin() >>> output = softmin(x) >>> print(output) [0.2341 0.636 0.0862 0.01165 0.03168 ] cstt|||_dS)zInitialize Softmin.N)r0rr1axisr6rFr7r9r:r1szSoftmin.__init__cCstj||jSr;)rfunctionsoftminrFr<r9r9r:r>szSoftmin.construct)rEr?r9r9r7r:rws-rcs8eZdZdZfddZeeddZddZZ S)ra Softmax function applied to 2D features data. Applies `Softmax` to each location :math:`(c, h, w)` with an input Tensor of shape :math:`(C, H, W)` . Inputs: - **x** (Tensor) - Tensor of shape :math:`(N, C_{in}, H_{in}, W_{in})` or :math:`(C_{in}, H_{in}, W_{in})`. Outputs: Tensor, which has the same type and shape as `x` with values in the range[0,1]. Raises: TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `data_format` is neither 'NCHW' nor 'CHW'. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[[[0.1, 0.2]], [[0.3, 0.4]], [[0.6, 0.5]]]]), mindspore.float32) >>> softmax2d = nn.Softmax2d() >>> output = softmax2d(x) >>> print(output) [[[[0.25838965 0.28001308]] [[0.31559783 0.34200877]] [[0.42601252 0.37797815]]]] cs*tt|tjdd|_t|_dS)zInitialize Softmax2d.)rFN)r0rr1r2rsoftmaxShapeshaper6r7r9r:r1szSoftmax2d.__init__cCs*t|}|dvr&td|d|ddS)N)For 'z/', the in_shape must have 3 or 4 dims, but got .)len ValueError)rMcls_namedimr9r9r:_check_input_dimszSoftmax2d._check_input_dimcCs"||}|||j||Sr;)rMrWrUrK)r6r=x_shaper9r9r:r>s zSoftmax2d.construct) r@rArBrCr1 staticmethodrrWr>rDr9r9r7r:rs   rcs*eZdZdZdfdd ZddZZS)ra Softmax activation function, which is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of multi-classification, the purpose is to show the results of multi-classification in the form of probability. Calculate the value of the exponential function for the elements of the input Tensor on the `axis`, and then normalized to lie in range [0, 1] and sum up to 1. Softmax is defined as: .. math:: \text{softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_{j=0}^{n-1}\exp(x_j)}, where :math:`x_{i}` is the :math:`i`-th slice in the given dimension of the input Tensor. Args: axis (Union[int, tuple[int]]): The axis to apply Softmax operation, if the dimension of input `x` is x.ndim, the range of axis is `[-x.ndim, x.ndim)`, -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - The input of Softmax with data type of float16 or float32. Outputs: Tensor, which has the same type and shape as `x` with values in the range[0,1]. Raises: TypeError: If `axis` is neither an int nor a tuple. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is a tuple whose length is less than 1. ValueError: If `axis` is a tuple whose elements are not all in range [-len(x), len(x)). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> # axis = -1(default), and the sum of return value is 1.0. >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> softmax = nn.Softmax() >>> output = softmax(x) >>> print(output) [0.03168 0.01166 0.0861 0.636 0.2341 ] rEcstt|t||_dS)zInitialize Softmax.N)r0rr1r2rKrGr7r9r:r1szSoftmax.__init__cCs ||Sr;)rKr<r9r9r:r>szSoftmax.construct)rEr?r9r9r7r:rs-rcs*eZdZdZdfdd ZddZZS)ra: Applies the LogSoftmax function to n-dimensional input tensor. The input is transformed by the Softmax function and then by the log function to lie in range[-inf,0). Logsoftmax is defined as: .. math:: \text{logsoftmax}(x_i) = \log \left(\frac{\exp(x_i)}{\sum_{j=0}^{n-1} \exp(x_j)}\right), Args: axis (int): The axis to apply LogSoftmax operation, -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - The input of LogSoftmax, with float16 or float32 data type. Outputs: Tensor, which has the same type and shape as `x` with output values in the range[-inf,0). Raises: TypeError: If `axis` is not an int. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is not in range [-len(x), len(x)). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> log_softmax = nn.LogSoftmax() >>> output = log_softmax(x) >>> print(output) [[-5.00672150e+00 -6.72150636e-03 -1.20067215e+01] [-7.00091219e+00 -1.40009127e+01 -9.12250078e-04]] rEcstt|t||_dS)zInitialize LogSoftmax.N)r0rr1r2 log_softmaxrGr7r9r:r1AszLogSoftmax.__init__cCs ||Sr;)rZr<r9r9r:r>FszLogSoftmax.construct)rEr?r9r9r7r:rs(rcs*eZdZdZdfdd ZddZZS)r%ah Exponential Linear Unit activation function. Applies the exponential linear unit function element-wise. The activation function is defined as: .. math:: E_{i} = \begin{cases} x_i, &\text{if } x_i \geq 0; \cr \alpha * (\exp(x_i) - 1), &\text{otherwise.} \end{cases} where :math:`x_i` represents the element of the input and :math:`\alpha` represents the `alpha` parameter. ELU Activation Function Graph: .. image:: images/ELU.png :align: center Args: alpha (float): The alpha value of ELU, the data type is float. Default: ``1.0`` . Inputs: - **x** (Tensor) - The input of ELU is a Tensor of any dimension with data type of float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `alpha` is not equal to 1.0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float32) >>> elu = nn.ELU() >>> result = elu(x) >>> print(result) [-0.63212055 -0.86466473 0. 2. 1.] r.cstt|t||_dS)zInitialize ELU.N)r0r%r1r2Eluelur5r7r9r:r1{sz ELU.__init__cCs ||Sr;)r\r<r9r9r:r>sz ELU.construct)r.r?r9r9r7r:r%Js0r%cs(eZdZdZfddZddZZS)ra4 Rectified Linear Unit activation function. .. math:: \text{ReLU}(x) = (x)^+ = \max(0, x), It returns element-wise :math:`\max(0, x)`. Specially, the neurons with the negative output will be suppressed and the active neurons will stay the same. ReLU Activation Function Graph: .. image:: images/ReLU.png :align: center Inputs: - **x** (Tensor) - The input of ReLU is a Tensor of any dimension. The data type is `number `_ . Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is not a number. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16) >>> relu = nn.ReLU() >>> output = relu(x) >>> print(output) [0. 2. 0. 2. 0.] cstt|t|_dS)zInitialize ReLU.N)r0rr1r2relurNr7r9r:r1sz ReLU.__init__cCs ||Sr;)r]r<r9r9r:r>szReLU.constructr?r9r9r7r:rs' rcs(eZdZdZfddZddZZS)ra Compute ReLU6 activation function. ReLU6 is similar to ReLU with a upper limit of 6, which if the inputs are greater than 6, the outputs will be suppressed to 6. It computes element-wise as .. math:: Y = \min(\max(0, x), 6). The input is a Tensor of any valid shape. ReLU6 Activation Function Graph: .. image:: images/ReLU6.png :align: center Inputs: - **x** (Tensor) - The input of ReLU6 with data type of float16 or float32. Outputs: Tensor, which has the same type as `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> relu6 = nn.ReLU6() >>> output = relu6(x) >>> print(output) [0. 0. 0. 2. 1.] cstt|t|_dS)zInitialize ReLU6.N)r0rr1r2relu6rNr7r9r:r1szReLU6.__init__cCs ||Sr;)r^r<r9r9r:r>szReLU6.constructr?r9r9r7r:rs) rcs*eZdZdZdfdd ZddZZS)r"a3 Leaky ReLU activation function. The activation function is defined as: .. math:: \text{leaky_relu}(x) = \begin{cases}x, &\text{if } x \geq 0; \cr {\alpha} * x, &\text{otherwise.}\end{cases} where :math:`\alpha` represents the `alpha` parameter. For more details, see `Rectifier Nonlinearities Improve Neural Network Acoustic Models `_. LeakyReLU Activation Function Graph: .. image:: images/LeakyReLU.png :align: center Args: alpha (Union[int, float]): Slope of the activation function at x < 0. Default: ``0.2`` . Inputs: - **x** (Tensor) - The input of LeakyReLU is a Tensor of any dimension. Outputs: Tensor, has the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float or an int. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> leaky_relu = nn.LeakyReLU() >>> output = leaky_relu(x) >>> print(output) [[-0.2 4. -1.6] [ 2. -1. 9. ]] 皙?cs`tt|td|ttg|jt |_ t |_ ||_ t|_|j dkr\t|_dS)zInitialize LeakyReLU.r/N)r0r"r1 validatorcheck_value_typefloatintrUr2 GreaterEqual greater_equalMulmulr/Maximum select_opMinimumr5r7r9r:r1s    zLeakyReLU.__init__cCs2tt|jt|}||||}|Sr;)r2CastFscalar_to_tensorr/DTyperj)r6r=Z alpha_arrayoutr9r9r:r>"szLeakyReLU.construct)r_r?r9r9r7r:r"s. r"cs*eZdZdZdfdd ZddZZS) raR Randomized Leaky ReLU activation function. The activation function is defined as: .. math:: \text{RReLU}(x_{ji}) = \begin{cases}x_{ji}, &\text{if } x_{ji} \geq 0; \cr {\alpha_{ji}} * x_{ji}, &\text{otherwise.}\end{cases} where :math:`\alpha_{ji}` ~ :math:`U(l, u)`, :math:`l \le u`. Applies the RReLU function elementally, as described in the paper: `Empirical Evaluation of Rectified Activations in Convolution Network `_ . Args: lower (Union[int, float]): Slope of the activation function at x < 0. Default: ``1 / 8`` . upper (Union[int, float]): Slope of the activation function at x < 0. Default: ``1 / 3`` . Inputs: - **x** (Tensor) - The input of RReLU is a Tensor of any dimension. Outputs: Tensor, after RReLU, has the same type and shape as the `x`. Raises: TypeError: If `lower` is not a float or an int. TypeError: If `upper` is not a float or an int. TypeError: If `x` is not a Tensor. TypeError: If `x` is not a Tensor of mindspore.float16 or mindpore.float32. ValueError: If `lower` is greater than upper. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0], [2.0, 0]]), mindspore.float32) >>> r_relu = nn.RReLU() >>> output = r_relu(x) >>> print(output) [[-0.31465699 4. ] [ 2. 0. ]] ?UUUUUU?cstt|td|ttg|jtd|ttg|j||kr`td|jd|d|dt |t j d|_ t |t j d|_ t|_dS)NupperlowerzFor zP, the value of 'upper' must be greater than or equal to 'lower', but got upper: z , lower: z. r)r0rr1rarbrcrdrUrTr mstypefloat32rtrsr2Signsign)r6rtrsr7r9r:r1Xs  zRReLU.__init__c Cst|tstdt|d|j}|j}||}|dd}|dd}t ||j |j  |}||d}||} | |} | S)zRReLU construct.z1For 'rrelu', the input must be a Tensor, but got rRNrrE) isinstancer TypeErrortyperMrrxclipruniformrtrsastype) r6r=sizer sign_matrixnegative_filterpositive_filtermask negative_mask total_maskrpr9r9r:r>cs     zRReLU.construct)rqrrr?r9r9r7r:r(s/ rcs(eZdZdZfddZddZZS)ra Activation function SeLU (Scaled exponential Linear Unit). SeLU Activation Function Graph: .. image:: images/SeLU.png :align: center Refer to :func:`mindspore.ops.selu` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> selu = nn.SeLU() >>> output = selu(input_x) >>> print(output) [[-1.1113307 4.202804 -1.7575096] [ 2.101402 -1.7462534 9.456309 ]] cstt|t|_dS)zInitialize SeLUN)r0rr1r2selurNr7r9r:r1sz SeLU.__init__cCs ||Sr;)rr6input_xr9r9r:r>szSeLU.constructr?r9r9r7r:rss rc@seZdZdZddZdS)ra Sigmoid Linear Unit activation function. Applies the silu linear unit function element-wise. .. math:: \text{SiLU}(x) = x * \sigma(x), where :math:`x_i` is input, :math:`\sigma(x)` is Sigmoid function. .. math:: \text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)}, SiLU Activation Function Graph: .. image:: images/SiLU.png :align: center Inputs: - **x** (Tensor) - Input with the data type float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16) >>> silu = nn.SiLU() >>> output = silu(x) >>> print(output) [-0.269 1.762 -0.1423 1.762 -0.269] cCs tj|Sr;)rrHsilur<r9r9r:r>szSiLU.constructNr@rArBrCr>r9r9r9r:rs+rcs(eZdZdZfddZddZZS)rae Applies the Tanh function element-wise, returns a new tensor with the hyperbolic tangent of the elements of input, The input is a Tensor with any valid shape. Tanh function is defined as: .. math:: tanh(x_i) = \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = \frac{\exp(2x_i) - 1}{\exp(2x_i) + 1}, where :math:`x_i` is an element of the input Tensor. Tanh Activation Function Graph: .. image:: images/Tanh.png :align: center Inputs: - **x** (Tensor) - Tensor of any dimension, input with data type of float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([1, 2, 3, 2, 1]), mindspore.float16) >>> tanh = nn.Tanh() >>> output = tanh(x) >>> print(output) [0.7617 0.964 0.995 0.964 0.7617] cstt|t|_dS)zInitialize Tanh.N)r0rr1r2tanhrNr7r9r:r1sz Tanh.__init__cCs ||Sr;)rr<r9r9r:r>szTanh.constructr?r9r9r7r:rs' rc@seZdZdZddZdS)ra Tanhshrink activation function. The tanhshrink function is evaluated by element and returns a new tensor. Tanh function is defined as: .. math:: tanhshrink(x_i) =x_i- \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = x_i-\frac{\exp(2x_i) - 1}{\exp(2x_i) + 1}, where :math:`x_i` is an element of the input Tensor. Inputs: - **x** (Tensor) - Tensor of any dimension. Outputs: Tensor, with the same shape as the `x`. Raises: TypeError: If `x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([1, 2, 3, 2, 1]), ms.float16) >>> tanhshrink = nn.Tanhshrink() >>> output = tanhshrink(x) >>> print(output) [0.2383 1.036 2.004 1.036 0.2383] cCs t|Sr;)rm tanhshrinkr<r9r9r:r>szTanhshrink.constructNrr9r9r9r:rs$rcs*eZdZdZdfdd ZddZZS) ra Applies the Hardtanh function element-wise. The activation function is defined as: .. math:: \text{Hardtanh}(x) = \begin{cases} 1, & \text{ if } x > 1; \\ -1, & \text{ if } x < -1; \\ x, & \text{ otherwise. } \end{cases} Linear region range :math:`[-1, 1]` can be adjusted using `min_val` and `max_val`. Hardtanh Activation Function Graph: .. image:: images/Hardtanh.png :align: center Note: On Ascend, data type of float16 might lead to accidental accuracy problem. Args: min_val (Union[int, float]): Minimum value of the linear region range. Default: ``-1.0`` . max_val (Union[int, float]): Maximum value of the linear region range. Default: ``1.0`` . Inputs: - **x** (Tensor) - Input Tensor with data type of float16 or float32. On CPU and Ascend support dimension 0-7D. On GPU support dimension 0-4D. Outputs: Tensor, with the same dtype and shape as `x`. Raises: TypeError: If `x` is not a Tensor. TypeError: If dtype of `x` is neither float16 nor float32. TypeError: If dtype of `min_val` is neither float nor int. TypeError: If dtype of `max_val` is neither float nor int. ValueError: If `min_val` is not less than `max_val`. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hardtanh = nn.Hardtanh(min_val=-1.0, max_val=1.0) >>> output = hardtanh(x) >>> print(output) [-1. -1. 0. 1. 1.] r.csBtt|||_||_|j|jkr>td|jd|jdS)zInitialize Hardtanh.z:For Hardtanh, min_val should be less than max_val,but got z and N)r0rr1min_valmax_valrT)r6rrr7r9r:r1Us zHardtanh.__init__cCst||j|jSr;)rmhardtanhrrr<r9r9r:r>^szHardtanh.construct)rr.r?r9r9r7r:r s4 rcs*eZdZdZdfdd ZddZZS)ra Gaussian error linear unit activation function. Applies GELU function to each element of the input. The input is a Tensor with any valid shape. GELU is defined as: .. math:: GELU(x_i) = x_i*P(X < x_i), where :math:`P` is the cumulative distribution function of standard Gaussian distribution and :math:`x_i` is the element of the input. GELU Activation Function Graph: .. image:: images/GELU.png :align: center Args: approximate (bool): Whether to enable approximation. Default: ``True`` . If `approximate` is ``True``, The gaussian error linear activation is: :math:`0.5 * x * (1 + tanh(\sqrt(2 / \pi) * (x + 0.044715 * x^3)))` else, it is: :math:`x * P(X <= x) = 0.5 * x * (1 + erf(x / \sqrt(2)))`, where P(X) ~ N(0, 1). Inputs: - **x** (Tensor) - The input of GELU with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> gelu = nn.GELU() >>> output = gelu(x) >>> print(output) [[-1.5880802e-01 3.9999299e+00 -3.1077917e-21] [ 1.9545976e+00 -2.2918017e-07 9.0000000e+00]] >>> gelu = nn.GELU(approximate=False) >>> # CPU not support "approximate=False", using "approximate=True" instead >>> output = gelu(x) >>> print(output) [[-1.5865526e-01 3.9998732e+00 -0.0000000e+00] [ 1.9544997e+00 -1.4901161e-06 9.0000000e+00]] Tcsxtt|t|d|j||_|jr6t|_ n>t |_ t |_ tdtj|_tdtj|_tdtj|_dS)zInitialize GELU. approximate?r.g@N)r0rr1ra check_boolrUrr2GeLUgeluErferfSqrtsqrtr rurvconst0const1const2)r6rr7r9r:r1s   z GELU.__init__c CsT|jr||S|t|j|jt|j|j|||t|j |jSr;) rrrmcastrrrrrrr<r9r9r:r>s   zGELU.construct)Tr?r9r9r7r:rbs=rcs(eZdZdZfddZddZZS)ra< Fast Gaussian error linear unit activation function. Applies FastGelu function to each element of the input. The input is a Tensor with any valid shape. FastGelu is defined as: .. math:: FastGelu(x_i) = \frac {x_i} {1 + \exp(-1.702 * \left| x_i \right|)} * \exp(0.851 * (x_i - \left| x_i \right|)) where :math:`x_i` is the element of the input. FastGelu Activation Function Graph: .. image:: images/FastGelu.png :align: center Inputs: - **x** (Tensor) - The input of FastGelu with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> fast_gelu = nn.FastGelu() >>> output = fast_gelu(x) >>> print(output) [[-1.5418735e-01 3.9921875e+00 -9.7473649e-06] [ 1.9375000e+00 -1.0052517e-03 8.9824219e+00]] cstt|t|_dS)zInitialize FastGelu.N)r0rr1r2FastGeLU fast_gelurNr7r9r:r1szFastGelu.__init__cCs ||Sr;)rr<r9r9r:r>szFastGelu.constructr?r9r9r7r:rs+ rcs(eZdZdZfddZddZZS)ral Sigmoid activation function. Applies sigmoid-type activation element-wise. Sigmoid function is defined as: .. math:: \text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)}, where :math:`x_i` is the element of the input. Sigmoid Activation Function Graph: .. image:: images/Sigmoid.png :align: center Inputs: - **input_x** (Tensor) - Tensor of any dimension, the data type is float16, float32, float64, complex64 or complex128. Outputs: Tensor, with the same type and shape as the `input_x`. Raises: TypeError: If dtype of `input_x` is not float16, float32, float64, complex64 or complex128. TypeError: If `input_x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> sigmoid = nn.Sigmoid() >>> output = sigmoid(x) >>> print(output) [0.2688 0.11914 0.5 0.881 0.7305 ] cstt|t|_dS)zInitialize Sigmoid.N)r0rr1r2sigmoidrNr7r9r:r1szSigmoid.__init__cCs ||Sr;)rr<r9r9r:r>szSigmoid.constructr?r9r9r7r:rs+ rcs(eZdZdZfddZddZZS)raj Softsign activation function. Softsign Activation Function Graph: .. image:: images/Softsign.png :align: center Refer to :func:`mindspore.ops.softsign` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([0, -1, 2, 30, -30]), mindspore.float32) >>> softsign = nn.Softsign() >>> output = softsign(x) >>> print(output) [ 0. -0.5 0.6666667 0.9677419 -0.9677419] cstt|t|_dS)zInitialize Softsign.N)r0rr1r2softsignrNr7r9r:r18szSoftsign.__init__cCs ||Sr;)rr<r9r9r:r>=szSoftsign.constructr?r9r9r7r:rs rcs4eZdZdZeddd fdd Zdd ZZS) r a PReLU activation function. Applies the PReLU function element-wise. PReLU is defined as: .. math:: PReLU(x_i)= \max(0, x_i) + w * \min(0, x_i), where :math:`x_i` is an element of an channel of the input. Here :math:`w` is a learnable parameter with a default initial value 0.25. Parameter :math:`w` has dimensionality of the argument channel. If called without argument channel, a single parameter :math:`w` will be shared across all channels. PReLU Activation Function Graph: .. image:: images/PReLU.png :align: center Args: channel (int): The elements number of parameter :math:`w`. It could be an int, and the value is 1 or the channels number of input tensor `x`. Default: ``1`` . w (Union[float, list, Tensor]): The initial value of parameter. It could be a float, a float list or a tensor has the same dtype as the input tensor `x`. Default: ``0.25`` . Inputs: - **x** (Tensor) - The input of PReLU with data type of float16 or float32. The shape is :math:`(N, *)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same dtype and shape as the `x`. Raises: TypeError: If `channel` is not an int. TypeError: If `w` is not one of a float, a float list, a float Tensor. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If the `x` is a 0-D or 1-D Tensor on Ascend. ValueError: If `channel` is less than 1. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[[[0.1, 0.6], [0.9, 0.9]]]]), mindspore.float32) >>> prelu = nn.PReLU() >>> output = prelu(x) >>> print(output) [[[[0.1 0.6] [0.9 0.9]]]] )attrsr`?cstt|t|d|jt|ttj fr\tj |ftj d}| |t |t j d}nt|trt||krtd|jdt|d|d|D],}t|ttj fstd|jd|dqt |t j d}nt|t rR|jt jt j fvrtd|jd|jdt|jd ks0|jd |krptd|jd |jd |dntd|jd t|jdt|dd|_t|_dS)zInitialize PReLU.channelrrQzg', the length of 'w' must be equal to the 'channel' when the 'w' is a list, but got the length of 'w': z, the 'channel': rRzE', all elements in 'w' must be float when the 'w' is a list, but got zQ', the dtype of 'w' must be float16 or float32 when the 'w' is a tensor, but got r`rz', the dimension of 'w' must be 1, and the elements number should be equal to the 'channel' when the 'w' is a tensor, but got 'w' shape z, the 'channel' z:', the 'w' only supported float, list and tensor, but got a)nameN)r0r r1racheck_positive_intrUryrcnprvemptyfillr rulistrSrTrfloat16rMrzr{r@rwr2prelu)r6rrtmpir7r9r:r1|sF               zPReLU.__init__cCs||t|j|jSr;)rrmrrrr<r9r9r:r>szPReLU.construct)r`r)r@rArBrCrr1r>rDr9r9r7r:r As: r cs(eZdZdZfddZddZZS)r$a Applies hswish-type activation element-wise. The input is a Tensor with any valid shape. Hard swish is defined as: .. math:: \text{hswish}(x_{i}) = x_{i} * \frac{ReLU6(x_{i} + 3)}{6}, HSwish Activation Function Graph: .. image:: images/HSwish.png :align: center Inputs: - **x** (Tensor) - The input of HSwish, data type must be float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hswish = nn.HSwish() >>> result = hswish(x) >>> print(result) [-0.3333 -0.3333 0. 1.667 0.6665] cstt|t|_dS)zInitialize HSwish.N)r0r$r1r2hswishrNr7r9r:r1szHSwish.__init__cCs ||Sr;)rr<r9r9r:r>szHSwish.constructr?r9r9r7r:r$s% r$cs(eZdZdZfddZddZZS)r#a Hard sigmoid activation function. Calculates the output according to the input elements. Hard sigmoid is defined as: .. math:: \text{hsigmoid}(x_{i}) = \max(0, \min(1, \frac{x_{i} + 3}{6})), HSigmoid Activation Function Graph: .. image:: images/HSigmoid.png :align: center Inputs: - **input_x** (Tensor) - The input of HSigmoid. Tensor of any dimension. Outputs: Tensor, with the same type and shape as the `input_x`. Raises: TypeError: If `input_x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hsigmoid = nn.HSigmoid() >>> result = hsigmoid(x) >>> print(result) [0.3333 0.1666 0.5 0.8335 0.6665] cstt|t|_dS)zInitialize HSigmoid.N)r0r#r1r2hsigmoidrNr7r9r:r1szHSigmoid.__init__cCs ||Sr;)rrr9r9r:r>szHSigmoid.constructr?r9r9r7r:r#s$ r#cs(eZdZdZfddZddZZS)r&aN Applies logsigmoid activation element-wise. The input is a Tensor with any valid shape. Logsigmoid is defined as: .. math:: \text{logsigmoid}(x_{i}) = \log(\frac{1}{1 + \exp(-x_i)}), where :math:`x_{i}` is the element of the input. LogSigmoid Activation Function Graph: .. image:: images/LogSigmoid.png :align: center Inputs: - **x** (Tensor) - The input of LogSigmoid with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> net = nn.LogSigmoid() >>> x = Tensor(np.array([1.0, 2.0, 3.0]), mindspore.float32) >>> output = net(x) >>> print(output) [-0.31326166 -0.12692806 -0.04858734] csDtt|t|_t|_t|_ t |_ t |_ dS)zInitialize LogSigmoid.N)r0r&r1r2rgrhExpexpAddadd ReciprocalrecLoglogrNr7r9r:r1&s     zLogSigmoid.__init__cCs:||d}||}||d}||}||}|S)NrEr`)rhrrrr)r6r neg_input exp_neg_inputexp_neg_input_1rec_exp_neg_input_1retr9r9r:r>/s      zLogSigmoid.constructr?r9r9r7r:r&s' r&cs*eZdZdZd fdd Zdd ZZS) r'a; Local Response Normalization. Refer to :func:`mindspore.ops.lrn` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[[[0.1], [0.2]], ... [[0.3], [0.4]]]]), mindspore.float32) >>> output = nn.LRN()(input_x) >>> print(output) [[[[0.09534626] [0.1825742 ]] [[0.2860388 ] [0.3651484 ]]]] r.rACROSS_CHANNELScs&tt|t||||||_dS)zInitialize LRN.N)r0r'r1NN_OPSlrn_op)r6 depth_radiusbiasr/beta norm_regionr7r9r:r1Osz LRN.__init__cCs ||Sr;)rrr9r9r:r>Tsz LRN.construct)rr.r.rrr?r9r9r7r:r'8sr'cs*eZdZdZdfdd ZddZZS)r(a Applies the SoftShrink function element-wise. .. math:: \text{SoftShrink}(x) = \begin{cases} x - \lambda, & \text{ if } x > \lambda \\ x + \lambda, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} SoftShrink Activation Function Graph: .. image:: images/Softshrink.png :align: center Args: lambd (float): the :math:`\lambda` must be no less than zero for the SoftShrink formulation. Default: ``0.5`` . Inputs: - **input_x** (Tensor) - The input of SoftShrink with data type of float16 or float32. Any number of additional dimensions. Outputs: Tensor, has the same shape and data type as `input_x`. Raises: TypeError: If lambd is not a float. TypeError: If input_x is not a Tensor. TypeError: If dtype of input_x is neither float16 nor float32. ValueError: If lambd is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[ 0.5297, 0.7871, 1.1754], [ 0.7836, 0.6218, -1.1542]]), mindspore.float16) >>> softshrink = nn.SoftShrink() >>> output = softshrink(input_x) >>> print(output) [[ 0.02979 0.287 0.676 ] [ 0.2837 0.1216 -0.6543 ]] rcstt|t||_dSr;)r0r(r1r2 softshrinkr6lambdr7r9r:r1szSoftShrink.__init__cCs||}|Sr;)r)r6routputr9r9r:r>s zSoftShrink.construct)rr?r9r9r7r:r(Xs0r(cs*eZdZdZdfdd ZddZZS)r)a* Hard Shrink activation function. Calculates the output according to the input elements. The formula is defined as follows: .. math:: \text{HardShrink}(x) = \begin{cases} x, & \text{ if } x > \lambda \\ x, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} HShrink Activation Function Graph: .. image:: images/HShrink.png :align: center Args: lambd (float): The threshold :math:`\lambda` defined by the Hard Shrink formula. Default: ``0.5`` . Inputs: - **input_x** (Tensor) - The input of Hard Shrink with data type of float16 or float32. Outputs: Tensor, the same shape and data type as the input. Raises: TypeError: If `lambd` is not a float. TypeError: If dtype of `input_x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[ 0.5, 1, 2.0], [0.0533,0.0776,-2.1233]]), mindspore.float32) >>> hshrink = nn.HShrink() >>> output = hshrink(input_x) >>> print(output) [[ 0. 1. 2. ] [ 0. 0. -2.1233]] rcstt|t||_dSr;)r0r)r1r2hshrinkrr7r9r:r1szHShrink.__init__cCs ||Sr;)rrr9r9r:r>szHShrink.construct)rr?r9r9r7r:r)s.r)cs(eZdZdZfddZddZZS)r+a4Thresholds each element of the input Tensor. The formula is defined as follows: .. math:: y = \begin{cases} x, &\text{ if } x > \text{threshold} \\ \text{value}, &\text{ otherwise } \end{cases} Args: threshold (Union[int, float]): The value to threshold at. value (Union[int, float]): The value to replace with when element is less than threshold. Inputs: - **input_x** (Tensor) - The input of Threshold with data type of float16 or float32. Outputs: Tensor, the same shape and data type as the input. Raises: TypeError: If `threshold` is not a float or an int. TypeError: If `value` is not a float or an int. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> m = nn.Threshold(0.1, 20) >>> inputs = Tensor([0.1, 0.2, 0.3], mindspore.float32) >>> outputs = m(inputs) >>> print(outputs) [ 20.0 0.2 0.3] cstt|||_||_dS)zInitialize Threshold.N)r0r+r1 thresholdvalue)r6rrr7r9r:r1szThreshold.__init__cCst||j|jSr;)rmrrrr9r9r:r>szThreshold.constructr?r9r9r7r:r+s& r+cs(eZdZdZfddZddZZS)r,a Computes MISH(A Self Regularized Non-Monotonic Neural Activation Function) of input tensors element-wise. Refer to :func:`mindspore.ops.mish` for more details. Mish Activation Function Graph: .. image:: images/Mish.png :align: center Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> mish = nn.Mish() >>> output = mish(x) >>> print(output) [[-3.03401530e-01 3.99741292e+00 -2.68321624e-03] [ 1.94395900e+00 -3.35762873e-02 9.00000000e+00]] cstdt|_dS)zInitialize Mish.r,N)r0r1rr,mishrNr7r9r:r1s z Mish.__init__cCs ||Sr;)rrr9r9r:r>szMish.constructr?r9r9r7r:r,s r,cs*eZdZdZdfdd ZddZZS)r-a The gated linear unit function. .. math:: {GLU}(a, b)= a \otimes \sigma(b) where :math:`a` is the first half of the input matrices and :math:`b` is the second half. Here :math:`\sigma` is the sigmoid function, and :math:`\otimes` is the Hadamard product. Args: axis (int): the axis to split the input. Default: ``-1`` , the last axis in `x`. Inputs: - **x** (Tensor) - :math:`(\ast_1, N, \ast_2)` where `*` means, any number of additional dimensions. Outputs: Tensor, the same dtype as the `x`, with the shape :math:`(\ast_1, M, \ast_2)` where :math:`M=N/2`. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> m = ms.nn.GLU() >>> input = ms.Tensor([[0.1,0.2,0.3,0.4],[0.5,0.6,0.7,0.8]]) >>> output = m(input) >>> print(output) [[0.05744425 0.11973753] [0.33409387 0.41398472]] rEcs0td||_tj|dd|_t|_dS)zInitialize GLU.r-)rF output_numN)r0r1rVr2SplitspiltrrrGr7r9r:r1>s z GLU.__init__cCs ||\}}||}||Sr;)rr)r6r=x1x2r9r9r:r>Es z GLU.construct)rEr?r9r9r7r:r-s r-)rIrK softmax2d logsoftmaxr]r^rrelurrrrrrr\rrr leakyrelurr logsigmoidrrrrcCsT|rd|dnd}|dur dS|tvrJt|dttd|dt|S)a Gets the activation function. Args: name (str): The name of the activation function. prim_name (Union[str, None]): The name of primitive. Default: ``None`` . Returns: Function, the activation function. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore.nn as nn >>> sigmoid = nn.get_activation('sigmoid') >>> print(sigmoid) Sigmoid<> rQz', theTheNz 'name' must be in z , but got rR) _activationKeyErrorrkeys)r prim_name msg_prefixr9r9r:r!hs "r!)N)>rC __future__rnumpyr mindsporerramindspore._extendsrmindspore.commonrrumindspore.common.parameterrmindspore.common.tensorr mindspore.opsr rmr r2Zmindspore.ops.operationsr rmindspore.nn.cellr rmindspore.ops.primitiver__all__r*rrrrr%rrr"rrrrrrrrrrr r$r#r&r'r(r)r+r,r-rr!r9r9r9r:s            !67372:13@K#01)BS55"`/.: :71#/pengcheng.mind-test/mindformers/modules/__pycache__/layers.cpython-39.pyc000066400000000000000000001040071471400134000270500ustar00rootroot00000000000000a ?cfE@s0dZddlmZddlmZmZddlZddlZddlZ ddl m Z m Z ddl mZddlmZmZddlmmZddlmZdd lmZdd lmZdd lmZdd lmZ dd l!m"Z"zddl#m$Z$Wne%yddl#m&Z$Yn0ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3gdZ4ddZ5ddZ6ddZ7GdddZ8e"ddZ9e"dd Z:e"d!d"Z;e"d#d$ZGd)d*d*e jZ?Gd+d,d,eZ@Gd-d.d.eZAGd/d0d0e jZBGd1d2d2e jZCGd3d4d4e jZDd5d6ZEd7ejFfd8d9ZGdS):zw The basic layer of the Transformer Networks. This is an experimental interface that is subject to change or deletion. )absolute_import)wrapspartialN)nncontext) Parameter) initializerTensor)_get_graph_seed)cell_attr_register)Cell) functional) operations) constexpr) Validator)_get_parallel_mode_is_sharding_propagation) ParallelMode)logger)get_activation)default_dpmp_configOpParallelConfigMoEParallelConfig)FixedSparseAttentionDropout LayerNormLinearcsfdd}|S)z)Check whether input data type is correct.cs8tjijtfdd}|S)Ncslj|i|}|j}dvr&dd|vr6|d}|D]\}}|vr>|||q>|i|S)Nkwargs)bind argumentsitems)argsr bound_values argument_dictnamevalue bound_typesfuncsigE/home/ma-user/work/mindformers-baichuan/mindformers/modules/layers.pywrapper?sz?_args_type_validator_check..type_check..wrapper)inspect signature bind_partialrr)r(r, type_args type_kwargsr&r+ type_check;s  z._args_type_validator_check..type_checkr*)r1r2r3r*r0r+_args_type_validator_check8sr4csfdd}|S)Ncsttjd}||t|SN) valid_types prim_name)rrcheck_type_nametyper%r$Z partial_check class_nametypesr*r+validator_check_funcUsz0_valid_type_checks..validator_check_funcr*r=r<r>r*r;r+_valid_type_checksSsr@csfdd}|S)Ncsttjd}|||Sr5)rrr8r:r;r*r+r>`sz1_valid_value_checks..validator_check_funcr*r?r*r;r+_valid_value_checks^srAc@s@eZdZdZeddZeddZeddZedd Zd S) _LayerInputCheckzI A input check class for the inputs of the transformer model. cCsTt|ts|g}d}|D]}t||krd}q|sPt|d|d|d|dS)aX Check the input shape's length is equal to the expected shape :param input_shape(list): a list of the tensor shapes. :param param_name(str): the name of the checked parameter. :param func_name(str): the name of the function. :param target_len: the expected length of the shape. :return: FT z shape length must be one of z dimension, but got shape ) isinstancelistlen ValueError) input_shape param_name func_name target_lenmatcheditemr*r*r+check_shape_lengthns  z#_LayerInputCheck.check_shape_lengthcCst|dts|g}t|tr&t|}t|||dd|Dd}|D]}||krHd}q^qH|st|d|d|d|dS) aE Check the input shape's is equal to the expected shape :param input_shape(list): a list of the tensor shapes. :param param_name(str): the name of the checked parameter. :param func_name(str): the name of the function. :param target_shape: the expected shape. :return: rcSsg|] }t|qSr*)rF).0rMr*r*r+ z6_LayerInputCheck.check_shape_equal..FTrCz shape must be one of ,but got )rDrEtuplerBrNrG)rHrIrJ target_shaperLrMr*r*r+check_shape_equals"    z"_LayerInputCheck.check_shape_equalc Cs8|||kr4t|d|d|d|d|| dS)z< Check whether the input_shape[dim] is equal to target valuerCz at  shape must be rRT)rGrHdimrIcls_nameZ target_valuer*r*r+check_shape_value_on_axiss  z*_LayerInputCheck.check_shape_value_on_axisc Cs|}|\}}t|trt|}t|||t|t|dg|d|krvtd|d|d|d|dd|d |d d krtd|d |d |d d|d t|d kr|d |d krtd|d |d |d|d d|d dS)z Check the input shape's is equal to the expected shape, the value on 0-th is viewed as batch, and the batch size will not be checked. zFor z, the last dimension of rVz,but got the last dimension z in .rz, the first dimension of z2 shape greater than 0,but got the first dimension z shape should be divisible by z, but got the first dimension T)rDrSrErBrNrFrG)rHrIrJrTlengthhiddenr*r*r+check_shape_equal_without_batchs8       z0_LayerInputCheck.check_shape_equal_without_batchN) __name__ __module__ __qualname____doc__ staticmethodrNrUrZrar*r*r*r+rBis   rBcCs^|sB|r"t|d|d|d|sZt|d|d|dn|sZt|d|ddS)z6 If the past is True, check whether the inputs is NonerCz must be z), if use_pat is False, but found a tensorz, if use_pat is False.z# must be tensor, if use_pat is TrueT) TypeError)use_pastrIrJ default_value is_tensor is_defaultr*r*r+_check_past_none_input_nonesrlcCst||||dSN)rr8) input_dtyperI allow_dtypesrYr*r*r+_check_input_dtypesrpcCst||||dSrm)rBrN)rHrIrJrKr*r*r+_check_input_shapesrqcCst||||dSrm)rBrUrHrIrJrTr*r*r+_check_shape_equalsrscCst|||||dSrm)rBrZrWr*r*r+_check_input_shape_valuesrtcCst||||dSrm)rBrarrr*r*r+ _check_shape_equal_without_batchsrucs@eZdZdZdejffdd ZddZddZd d Z Z S) rzd A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for parallel training. g?cstt||dks|dkr,td|td|tj|j t d|t g|j ||_ t ddv|_|jrtdd\}}||_||_||_t|_tj|j|jd |_t|_t|_n t||_dS) Nrr[zBdropout probability should be a number in range (0, 1], but got {}dtype keep_prob device_target)Ascenddropout)Seed0Seed1)superr__init__rGformatrcheck_subclassmstype number_typerYcheck_value_typefloatrwr get_context is_ascendr seed0seed1rvPShape get_shapeDropoutGenMaskdropout_gen_mask DropoutDoMaskdropout_do_maskCastcastrz)selfrwrvrr __class__r*r+r~s*   zDropout.__init__cCsn|js |S|jdkr|S|js0||\}}|S||}t|}||j|}|||}| |||S)zA Input: a tensor Returns: a tensor r[) trainingrwrrzrrDTyperrr)rxout_shapervrwoutputr*r*r+ constructs    zDropout.constructcCs d|jS)Nz keep_prob={})rrw)rr*r*r+ extend_reprszDropout.extend_reprcCs0|jr |j||j|n |j|dSrm)rrshardrrzrstrategyr*r*r+rs z Dropout.shard) rbrcrdrerfloat32r~rrr __classcell__r*r*rr+rs rcs:eZdZdZdejdffdd ZddZdd ZZ S) ra A self-defined layer norm operation using reduce sum and reduce mean Args: normalized_shape (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. param_init_type: The param init type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. gh㈵>Fcstt||tjtjfvr0tdt|||_ |j sNt jdd|d|_ t t d||ddd|_t t d||d dd|_t jd d |_t |_t |_t |_t |_t |_||_t |_t |_t |_ dS) Nz\The type of parameter 'param_init_type' should in [float32, float16], but got the type : {}.r\)begin_norm_axisbegin_params_axisepsilononesgammaF)r$parallel_optimizerzerosbetaT) keep_dims)!r}rr~rrfloat16rgrr9is_self_definedr layer_normrrrr ReduceMeanmeanSquaresquareSqrtsqrtSubsub1sub2AddaddepsMulmuladd2RealDivreal_div)rnormalized_shaperparam_init_typerrr*r+r~3s6       zLayerNorm.__init__cCs|jrj||d}|||}|||d}||||j}|||}|| ||j |j }n| ||j |j \}}}|S)z8 x : batch x seq_length x hidden_size r\) rrrrrrrrrrrrr)rrrdiffvarianceZ variance_epsrrr*r*r+rMs   zLayerNorm.constructcCs|jr|j||j||j||j|d|df|j|d|df|j|ddf|j|ddf|j |ddf|j |d|dfn|j |dddf|S)a3 Set the shard for the layer norm. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy (tuple): The strategy for the dropout. Should be the same shape as the inputs. Examples: >>> import mindspore >>> net = mindformers.modules.transformer.LayerNorm(normalized_shape=(1024, 10)) >>> net.shard(((10, 2, 1),)) rr*r[) rrrrrrrrrrrrrr*r*r+r\s   zLayerNorm.shard) rbrcrdrerrr~rrrr*r*rr+r$src seZdZdZeeejejejejejeje e j e j gde e j e j gdddddddddde j e j f fdd Z d d Zdd d ZZS)raU The dense connected layer. Once the parallel mode is enabled, the input shape should be 3-D tensor. Applies dense connected layer for the input. This layer implements the operation as: .. math:: \text{outputs} = \text{activation}(\text{X} * \text{kernel} + \text{bias}), where :math:`X` is the input tensors, :math:`\text{activation}` is the activation function passed as the activation argument (if passed in), :math:`\text{kernel}` is a weight matrix with the same data type as the :math:`X` created by the layer, and :math:`\text{bias}` is a bias vector with the same data type as the :math:`X` created by the layer (only if has_bias is True). Args: in_channels (int): The number of channels in the input space. out_channels (int): The number of channels in the output space. weight_init (Union[Tensor, str, Initializer, numbers.Number]): The trainable weight_init parameter. The dtype is same as `x`. The values of str refer to the function `initializer`. Default: 'normal'. bias_init (Union[Tensor, str, Initializer, numbers.Number]): The trainable bias_init parameter. The dtype is same as `x`. The values of str refer to the function `initializer`. Default: 'zeros'. has_bias (bool): Specifies whether the layer uses a bias vector. Default: True. activation (str): activate function applied to the output of the fully connected layer, eg. 'ReLU'. Default: None. expert_num (int): The number of experts used in this Linear. Here, for the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. outer_batch (int): The replication number of experts. The replication is effective only when MoE is applied. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. compute_dtype (dtype.Number): The computation type. Default: mstype.float16 Inputs: - **x** (Tensor) - Tensor of shape :math:`(*, in\_channels)`. The `in_channels` in `Args` should be equal to :math:`in\_channels` in `Inputs`. Outputs: Tensor of shape :math:`(*, out\_channels)`. Raises: TypeError: If `in_channels` or `out_channels` is not an int. TypeError: If `has_bias` is not a bool. TypeError: If `activation` is not one of str, Cell, Primitive, None. ValueError: If length of shape of `weight_init` is not equal to 2 or shape[0] of `weight_init` is not equal to `out_channels` or shape[1] of `weight_init` is not equal to `in_channels`. ValueError: If length of shape of `bias_init` is not equal to 1 or shape[0] of `bias_init` is not equal to `out_channels`. Supported Platforms: ``Ascend`` ``GPU`` ) in_channels out_channelshas_bias transpose_b expert_num outer_batchr compute_dtypenormalrTNr[c sFtt|||_||_t|tsH|dusHt|tj sHt d|dt|t r|j dksx|j d|ksx|j d|krtd|r||gn||g} ||_| |_| |_||_|jdkrd|_tt||jg| | dd |_tj|d |_n*d |_tt|| | dd |_tj|d |_ttjfvo6t o6|jdu|_|jdurZ|jdurZtd d|_ ||_!|j!rt|t r|j dks|j d|krtd |jrtt|d|jd|g| dd |_ ntt||g| dd |_ d |j _"t#|_$||_%t&|r ||_'nt|tr t(|n||_'|j'du|_)| |_*t+|_,dS)NzIFor Linear cell, the activation should str type or nn.Cell type, but got r]r^rr[zSThe shape of parameter 'weight_init' is error, please check shape of 'weight_init'.Tweight)r$)rFzD'expert_group_size' should be configured as an integer in MoEConfig.zOThe shape of parameter 'bias_init' is error, please check shape of 'bias_init'.bias)-r}rr~rrrDstr issubclassrr rgr ndimrrGrrexpert_group_sizer expert_flagrrrr BatchMatMulmatmulMatMulrr AUTO_PARALLELruse_expert_group_sizerrrrbias_addact_namecallable activationractivation_flagrvrr)rrr weight_init bias_initrrrrrrrr weight_shaperr*r+r~sf"  (     zLinear.__init__cCst|dd|jf}t|d|jf}|jrz|jdur^t|d|j|j|jf}nt||j |jd|jf}t |}| |j |j }| ||j }|||}|jr||| |j|j }|jr||}t ||}t||}|S)z%Forward process, x should be a tensorNr\T)rrrReshaperrrrrrFrvrrrrrrrr)rr out_shape ori_dtyperrr*r*r+rs"     zLinear.constructcCs|jj||d|jr"|j||jrt|jtr|jdkr^|j j |d|dfn|jdkr|j j |ddf|j j ||j j |ddf|j j||j j|n,|jdkrtdnt|j |j|n|jrt|j trt|j dr|r|d}t|d krHt|d|d d }n2t|d krrt|d|d |d d }ntd|j |ntd|j|S)a Set the shard for the linear. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy_matmul (tuple): The strategy for the matmul. Should be the same shape as the inputs. strategy_bias (tuple): The strategy for the bias_add. Should be the same shape as the inputs. strategy_activation (tuple): The strategy for the strategy_activation. Should be the same shape as the inputs. out_strategy_matmul (tuple): The out strategy for the matmul. Should be the same shape as the inputs. ) in_strategy out_strategy leakyrelur logsigmoidr* logsoftmaxzWThe 'LogSoftmax' function is not supported in semi auto parallel or auto parallel mode.activation_shardr^r[) data_parallelmodel_parallel)rexpert_parallelra'The user-defined activation function currently only supports the case where the input policy is 2 or 4, so that relevant policies can be extracted from it.To avoid this error, you need to add the function of extracting 'ParallelConfig' or 'OpParallelConfig' for the incoming strategy_activation zThe user passed the custom defined activation function %s. If the user want to enable shard for the activation cell, the user should set the shard for each primitives in the cell.)rrrrrrDrrlowerr select_oprexprreclogrGgetattrr hasattrrFrrrrwarning)rstrategy_matmul strategy_biasstrategy_activationout_strategy_matmulZ shard_tupleparallel_configr*r*r+r sB  z Linear.shard)NNN)rbrcrdrer r4rcheck_positive_int check_boolrArrrr~rrrr*r*rr+r{s82   <rc sjeZdZdZeejejejejejejeegdddde ffdd Z ddZ d d Z d d Z ZS) ra Fixed Sparse Attention Layer. This function contains the sparse attention primitives used in Sparse Transformers (see paper) `Generating Long Sequences with Sparse Transformers `_. Specifically, it includes the following: 1. A faster implementation of normal attention (the upper triangle is not computed, and many operations are fused). 2. An implementation of "strided" and "fixed" attention, as in the Sparse Transformers paper. Args: batch_size(int): Number of input batch size. num_heads(int): Number of attention heads. size_per_head(int): An integer determining embedding size of each attention head, only supports 64, 128 for now. block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines the size of such blocks, Block X Block. Only supports 64 for now. seq_length(int): length of input sequence, only supports 1024 for now. Default 1024. num_different_global_patterns(int): An integer determining the number of different global attentions layouts. While global attention can be fixed by which block/s are representative of any local window, since there are multi-heads, each head can use a different global representative, only supports 4 for now. Default 4. parallel_config(OpParallelConfig): The config of parallel setting, see `OpParallelConfig`. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **q** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, seq_length, hidden_size]): Sequence of queries to query the context. - **k** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, seq_length, hidden_size]): Sequence of queries to query the context. - **v** (Tensor) - Tensor value (:class:`mstype.fp16` [batch size, sequence length, Embedding Size]): Sequence of queries to query the context. - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp32`, :class:`mstype.fp16` [batch_size, seq_length, seq_length]): Lower triangular matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, seq_length, hidden_size] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules import FixedSparseAttention >>> from mindspore import Tensor >>> model = FixedSparseAttention(batch_size=2, ... num_heads=8, ... size_per_head=64, ... block_size=64) >>> q = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> k = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> v = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1024, 1024)), mstype.float32) >>> output = model(q, k, v, attention_mask) >>> print(output.shape) (2, 1024, 512) ) batch_size num_heads size_per_head block_size seq_lengthnum_different_global_patternsrrc stt||j|j}} || dkr>td|d| d||dkrbtd|d|jd||_||_|||_||_ ||_ |||_ ||_ |d|_ t|_t|d| dff|_t|dddf|dddff|_t|dddfd f|_td gtjd |_||_d d g} |jdkr} t!|j D],}| d|dddkrd| | |f<qԐqd| } | |jdd|j ddf}t|d}tj"|tj#ddddddddf|jdd}||j|j d|jdddf}t|tj|_$tt%| tj|_&tt'(|j |_)t*|j|j | |ddf| |ddfd|ddf|dddff|_+t,|| dddddf|| dddddf|| ddff|_-t.d|dddff|_/t|dddfdf|_0t|dddff|_1t|dddff|_2t|| ddddff|_3t|| ddff|_4t5| |ddfdf|_6t7|ddff|_8dS)NrzThe number of heads z6 must be a multiple of parallel_config.model_parallel r]zThe batch_size z5 must be a multiple of parallel_config.data_parallel rr[r[r[r[grv@rzaFor 'FixedSparseAttention', the class variable 'seq_length' must be 1024, but got the value : {}.z_For 'FixedSparseAttention', the class variable 'block_size' must be 64, but got the value : {}.zpFor 'FixedSparseAttention', the class variable 'num_different_global_patterns' must be 4, but got the value : {}zhFor 'FixedSparseAttention', the class variable 'size_per_head' only supports {}, but got the value : {}.gir^rr[axisrr*)9r}rr~rrrGrr hidden_sizerr block_numr global_sizerrreshape Transposer transposer batch_matmulrmultiplyr rr multiply_datarrnprrrangerepeatnewaxis global_masktrillocal_mask_trianglemathr scale_factor MatmulDDS matmul_dds DSDMatmul matmul_dsdrrmul1 transpose1 transpose2 transpose3 transpose4rdiv StridedSliceslice1)rrrrrrrrdpmpZsize_per_head_listZ local_onesZglobal_mask_originalijZglobal_mask_fxrrr*r+r~s          $     2$   :zFixedSparseAttention.__init__c Cstt|d|j|j|j|jgtt|dt j g|jtt|d|j|j|j|jgtt|dt j g|jtt|d|j|j|j|jgtt|dt j g|jtt|d|j|j|j|jgtt|dt j t j g|j| |||\}}}| |\}}||t|jt|}||t|jt|}|||||\}}||||} || d} t| d|j|j|jf} || d} t| d|j|j|jf} | S)zForward processqkvattention_mask)rr[r rr^r\rr^r[r )rsrrrYrrrrprvrrr_transpose_inputs_generate_attention_maskr)rrr!r#r'rrrr() rr0r1r2r3 local_maskr local_prob global_prob attentionattention_merger*r*r+rsB  zFixedSparseAttention.constructcCsn||d|j|jf}||d|jddf|j|j|jfd}||d|jf}t|}|d|jd|jf}|d|j|jdf}|||}|||}|||}t |j d} | || }| t tdtjt |tj} || |j} || d} || |j|j|j|jf} || |j|jdd|j|jddf} || d}|j}||fS) zd generate global attention mask and local attention mask from origin attention mask r\rr[r)g?r5r r )rrr+rrrrrr ExpandDimsrrrrrtuple_to_arrayrrr$rr%r&r)rr3 input_maskrHZlocal_shape_rightZlocal_shape_leftZlocal_mask_leftZlocal_mask_rightZlocal_attention_maskZlower_triangleZlocal_multiplied_outZ local_adderZlocal_mask_originalZ local_mask_fxr8rr*r*r+r7s>       z-FixedSparseAttention._generate_attention_maskc Cs|||dd|j|jddfd}|||dd|j|jddfd}|||dd|j|jddfd}|||fS)z4 do reshape and transpose to inputs r\r r )rr^r r[)rrrr)rr0r1r2r*r*r+r6s*z&FixedSparseAttention._transpose_inputs)rbrcrdrer4rrr@rrr~rr7r6rr*r*rr+rBs >  I #rcs,eZdZdZeffdd ZddZZS) AlibiTensor Link to paper: https://arxiv.org/abs/2108.12409 Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation: with l being a tensor, and a fixed value `softmax(l+a) = softmax(l)`. Based on https://github.com/ofirpress/attention_with_linear_biases/blob/a35aaca144e0eb6b789dfcb46784c4b8e31b7983/fairseq/models/transformer.py#L742 Args: seq_length(int) - length of sequence num_heads(int) - number of heads Inputs: attention_mask(Tensor) - Token-wise attention mask, this should be of shape (batch_size, max_seq_len). dtype(mstype) - dtype of the output tensor Returns: alibi(Tensor), ailibi tensor shaped (batch_size, num_heads, 1, max_seq_len) c stt||j}||_||_tt| t j |_ t |dff|_t |ddff|_t |dff|_t |dfdf|_t |df|dff|_t d|ddff|_dtt|}tjddt|d  tj d}tjdd|tjd}t||}||krtjddtd|d  tj d} t|||} tjddd| dtjd} tj |t| | gdd}t|dddft j |_!dS) Nr[r)r[r[r^r rrr )"r}r@r~rrrr rrrr minus_onerr=r expand_2d expand_3dCumSumcumsumrrrr mul_sloperfloorlog2arrayarangeint32powermin concatenateslopes) rrrrr,closest_power_of_2basepowersrP extra_basenum_remaining_heads extra_powersrr*r+r~<s.&   zAlibiTensor.__init__cCsV||d}|||j}|||}||d}||j|}||d|}|S) Note: alibi will added to the attention bias that will be applied to the query, key product of attention therefore alibi will have to be of shape (batch_size, num_heads, query_length, key_length) r\r[r^) rFrrBrrCrGrPrDastype)rr3rv arange_tensoralibir*r*r+r[s   zAlibiTensor.construct)rbrcrdrerr~rrr*r*rr+r@)sr@cs6eZdZdZfddZejfddZddZZ S) AlibiTensorV2rAc stt|||_||_tt| tj |_ t |_ t |_t |_t |_t |_t |_t |_t |_t |_t |_dtt|}tjddt|d  tj d}tj dd|tj!d}t"||}||krhtjddtd|d  tj d}t#|||}tj ddd|dtj!d} tj$|t"|| gdd}t|dddddftj |_%dS)Nr^r rr[rr )&r}r[r~rrr rrrrrBrr=rCrDrErFradd_2dadd_3drrrGrrrrmul_maskrrHrIrJrKrLrMrNrOrP) rrrrQrRrSrPrTrUrVrr*r+r~|s4          &   zAlibiTensorV2.__init__c Cs|j\}}||d}|ddddf }|||}||d}||d }|||}||d}||j|}| || ||d|df} | | | ||dd|f} | |S)rWr\Nr[rr^r[) rrFr\rCrr]rDrGrPr^rrX) rr3rvbsZseqlenrYZmax_posdiagrZZ alibi_maskr*r*r+rs      zAlibiTensorV2.constructcCs|j}|j}|j|dff|j|ddff|j|dff|j|df|dff|j|ddf|ddff|j|df|dff|j d|dddff|j |ddff|j ||ddf|dddffdS)z+Parallel strategy configuratiuon interface.r[)r[r[r[r[N) rrrCrrDrFr\r]rrGrr^)rrr,r-r*r*r+rszAlibiTensorV2.shard) rbrcrdrer~rrrrrr*r*rr+r[is "r[cCs\dd}t|r||Sdtt|}||td|dddd||S)z calculate slopes of alibi tensorcs6ddt|d  fddt|DS)Nr^r csg|]}|qSr*r*)rOr.ratiostartr*r+rPrQzG_get_interleave.._get_interleave_power_of_2..)rrIr)nr*rbr+_get_interleave_power_of_2sz3_get_interleave.._get_interleave_power_of_2r^rN)rrI is_integerrH_get_interleave)rerfrQr*r*r+rhs rhmscCs|dvsJd|dt|}tt|dd}t||d}tt|dd}t|||df}t|d}tt|dd}t|d}||}||}t|d}|dkr|St||dS) zbuild alibi tensor)rriz%return tensors must be 'np' or 'ms', z not support.r[rr_rr)rhr expand_dimsrKtilerarr )seq_lenrZreturn_tensorsrvrPZposition_pointrarZr*r*r+build_alibi_tensor_v2s"    rm)Hre __future__r functoolsrrr-rnumpyr mindsporerrmindspore.common.parameterrmindspore.common.initializerrr mindspore.common.dtypecommonrvrmindspore.common.seedr Zmindspore._extendsr mindspore.nn.cellr mindspore.opsr rrrmindspore.ops.primitivermindspore._checkparamr ImportError _checkparammindspore.parallel._utilsrrmindspore.contextrmindformers.tools.loggerrZmindformers.modules.activationr2mindformers.modules.transformer.op_parallel_configrrr__all__r4r@rArBrlrprqrsrtrurrrrr@r[rhrrmr*r*r*r+sf              V      ;WHh@Wpengcheng.mind-test/mindformers/modules/__pycache__/local_block_sparse_attention.cpython-39.pyc000066400000000000000000000256601471400134000334660ustar00rootroot00000000000000a ?cfN8@sdZddlmZddlmZmZddlZddlZddlZ ddl Z ddl m Z ddlmZddl mZddl mZddlmZzddlmZWneyddlmZYn0dd lmZmZd gZd adada d d Z!ddZ"ddZ#GdddZ$ddZ%Gdd d e j&Z'dS)z! A Local Block Sparse Attention. )absolute_import)wrapspartialN)Tensordtype)cell_attr_register) Validator)default_dpmp_configOpParallelConfigLocalBlockSparseAttentionFcsfdd}|S)z)Check whether input data type is correct.cs8tjijtfdd}|S)Ncslj|i|}|j}dvr&dd|vr6|d}|D]\}}|vr>|||q>|i|S)Nkwargs)bind argumentsitems)argsr bound_values argument_dictnamevalue bound_typesfuncsig[/home/ma-user/work/mindformers-baichuan/mindformers/modules/local_block_sparse_attention.pywrapper4sz?_args_type_validator_check..type_check..wrapper)inspect signature bind_partialrr)rr type_args type_kwargsrr type_check0s  z._args_type_validator_check..type_checkr)r!r"r#rr r_args_type_validator_check-sr$csfdd}|S)zWtypes should be a list of types, this function check if the type is in the valid dtypescsttjd}||t|SN) valid_types prim_name)rr check_type_nametyperr partial_check class_nametypesrrvalidator_check_funcJs z0_valid_type_checks..validator_check_funcrr.r-r/rr,r_valid_type_checksHsr1csfdd}|S)z\the value should be a list of types, this function check if the value is in the valid dtypescsttjd}|||Sr%)rr r(r*r,rrr/Ws z1_valid_value_checks..validator_check_funcrr0rr,r_valid_value_checksUsr2c@s(eZdZdZddZddZddZdS) InitGatherIndexa% A self-defined function to init the index of gather operation Args: seq_len(int): Length of input sequence. local_size(int): An integer determining the local size. Current implementation of sparse self-attention is based on local sparse matrices, reference sliding window attention in Longformer block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices, reference block sparse attention in FlashAttention cCs>||_||_||_|j|j|_|j|j|_|jd|_dS)N)seq_len local_size block_size group_numlocal_block_numprev_block_num)selfr5r6r7rrr__init__ms zInitGatherIndex.__init__cCst|j|j}t|jD]`}td||j|j}td||j|j|j|j}t|||||j|d|j<q|S)Nrr4) nponesr8r6rangemaxr:r7aranger;indexistartendrrrgen_kv_gather_indexus z#InitGatherIndex.gen_kv_gather_indexcCstd|j|jf}t|jD]h}td||j|j}td||j|j |j}t |||dd||j|d|jddf<q|S)Nr4r) r=r>r5r6r?r8r@r:r7r9rArBrrrgen_mask_gatherd_indexs z&InitGatherIndex.gen_mask_gatherd_indexN)__name__ __module__ __qualname____doc__r<rGrHrrrrr3bs  r3cCs0t|||}t|tjat|tjadS)N)r3rrGmstypeint32kv_indexrH mask_index)r5r6r7 gather_indexrrrinit_gather_indexs rRc sjeZdZdZeeejejejejee j e j gde e gddde j effdd Zd ddZZS) r a Local Block Sparse Attention Layer. This function contains the sliding window attention primitives used in Longformer (see paper) `Longformer: The Long-Document Transformer ` and block sparse attention primitives used in Longformer FlashAttention (see paper) `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ` Specifically, it includes the following: 1. A faster implementation of normal attention (the entire area of the upper triangle and part of the area of the lower triangle are not computed, and many operations are fused). 2. An implementation of "local" and "block" sparse attention, as in the Longformer and FlashAttention. Args: seq_len(int): Length of input sequence. size_per_head(int): The hidden size of the input. local_size(int): An integer determining the local size. Current implementation of sparse self-attention is based on local sparse matrices, reference sliding window attention in Longformer. block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices, reference sliding window attention in FlashAttention. dropout_rate(float): The dropout rate of the attention score. Default 0.1. softmax_compute_type(mstype.Number): The type of softmax computation module. Default mstype.float16. Should be mstype.float32 or mstype.float16. parallel_config(OpParallelConfig): The config of parallel setting, see `OpParallelConfig`. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **q** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **k** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **v** (Tensor) - Tensor value (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp16` [batch_size, seq_length, seq_length]): Lower triangular matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, head_num, seq_length, size_per_head] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules import LocalBlockSparseAttention >>> from mindspore import Tensor >>> model = LocalBlockSparseAttention(seq_len=4096, ... size_per_head=128, ... local_size=1024, ... block_size=128) >>> q = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> k = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> v = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 4096, 4096)), mstype.float16) >>> output = model(q, k, v, attention_mask) >>> print(output.shape) (2, 16, 4096, 128) )r5 size_per_headr6r7softmax_compute_typeparallel_configg?c srtt|||dkr.td|d|||dkrNtd|d|||krjtd|d|||krtd|d ||dks|d krtd |tst|||d attt||_ ||_ ||_ ||_ ||_ |j}|j} t|| d d fd f|_tjjd d|| d d d f|| d d d ff|_t|d d fdf|_t|d d ff|_t|d d d d f|| d d d ff|_t|| d d d ff|_|j dkrt j!d |j d|_"|j"j#|| d d d fftj|| d d d f|| d d d ff|_$tdg|j d|_%t&|_'t(|| d d fdf|_)t*|d d f|d d ff|_+dS)Nrz) keep_probgr),superr r< ValueErrorgather_index_initedrRrmathsqrt scale_factorr6r7 dropout_raterT data_parallelmodel_parallelPRealDivshardreal_divmsops BatchMatMulqk_bmmMul attn_mask_mul ExpandDimsattn_mask_expand_dimsAdd attn_mask_addSoftmaxsoftmaxnnDropoutattention_dropoutdropoutpv_bmm multiply_dataReshapereshapeGathergather1GatherDgather2) r;r5rSr6r7r_rTrUdpmp __class__rrr<s       z"LocalBlockSparseAttention.__init__NcCs|j\}}}}t|jt|} ||| }||| }||||||j|j|f}||t d}||||d|j |f}||t d}||||d|j |f}| ||} t| } t| |j } |durFt t|ddf} ||d| }|t||j |j} || d} || |dd|j|j f} || | } || }t|| }|jdkrt||}|||}||||||f}|S)zForward processNr4rW)shaperbCastr^DTypereryr7r{rOr6rirTtilerPr}rkrwrmrorqr_rtrv)r;qkvattention_maskbszhead_numr5head_dimfactorscore ori_dtypeZ mask_index_adderprobsoutputrrr constructsB           z#LocalBlockSparseAttention.construct)N)rIrJrKrLrr$r check_positive_intr2rMfloat16float32r1r r r<r __classcell__rrrrr s&: A)(rL __future__r functoolsrrrr\numpyr= mindsporerf mindspore.nnrr mindspore.opsrgrbrrrMmindspore._extendsrZmindspore._checkparamr ImportError _checkparam2mindformers.modules.transformer.op_parallel_configr r __all__r[rOrPr$r1r2r3rRCellr rrrrs4         (pengcheng.mind-test/mindformers/modules/activation.py000066400000000000000000001422701471400134000234270ustar00rootroot00000000000000# Copyright 2020-2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """activation""" from __future__ import absolute_import import numpy as np from mindspore import _checkparam as validator from mindspore._extends import cell_attr_register from mindspore.common import dtype as mstype from mindspore.common.parameter import Parameter from mindspore.common.tensor import Tensor from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.ops.operations import nn_ops as NN_OPS from mindspore.nn.cell import Cell from mindspore import ops from mindspore.ops.primitive import _primexpr __all__ = ['Softmin', 'Softmax', 'Softmax2d', 'LogSoftmax', 'ReLU', 'ReLU6', 'RReLU', 'SeLU', 'SiLU', 'Tanh', 'Tanhshrink', 'Hardtanh', 'GELU', 'FastGelu', 'Sigmoid', 'Softsign', 'PReLU', 'get_activation', 'LeakyReLU', 'HSigmoid', 'HSwish', 'ELU', 'LogSigmoid', 'LRN', 'SoftShrink', 'HShrink', 'CELU', 'Threshold', 'Mish', 'GLU' ] class CELU(Cell): r""" Continuously differentiable exponential linear units activation function. Applies the continuously differentiable exponential linear units function element-wise. .. math:: \text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1)) CELU Activation Function Graph: .. image:: images/CELU.png :align: center Args: alpha (float): The :math:`\alpha` value for the Celu formulation. Default: ``1.0`` . Inputs: - **x** (Tensor) - The input of CELU. The required dtype is float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float. ValueError: If `alpha` has the value of 0. TypeError: If `x` is not a Tensor. TypeError: If the dtype of 'input_x' is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-2.0, -1.0, 1.0, 2.0]), mindspore.float32) >>> celu = nn.CELU() >>> output = celu(x) >>> print(output) [-0.86466473 -0.63212055 1. 2. ] """ def __init__(self, alpha=1.0): """Initialize CELU.""" super(CELU, self).__init__() self.celu = P.CeLU(alpha=alpha) def construct(self, x): return self.celu(x) class Softmin(Cell): r""" Softmin activation function, which is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of multi-classification, and the purpose is to show the results of multi-classification in the form of probability. Calculate the value of the exponential function for the elements of the input Tensor on the `axis`, and then normalized to lie in range [0, 1] and sum up to 1. Softmin is defined as: .. math:: \text{softmin}(x_{i}) = \frac{\exp(-x_i)}{\sum_{j=0}^{n-1}\exp(-x_j)}, where :math:`x_{i}` is the :math:`i`-th slice in the given dimension of the input Tensor. Args: axis (Union[int, tuple[int]]): The axis to apply Softmin operation, if the dimension of input `x` is x.ndim, the range of axis is `[-x.ndim, x.ndim)`. -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - Tensor for computing Softmin functions with data type of float16 or float32. Outputs: Tensor, which has the same type and shape as `x` with values in the range [0,1]. Raises: TypeError: If `axis` is neither an int nor a tuple. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is a tuple whose length is less than 1. ValueError: If `axis` is a tuple whose elements are not all in the range [-x.ndim, x.ndim). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> # axis = -1(default), and the sum of return value is 1.0. >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> softmin = nn.Softmin() >>> output = softmin(x) >>> print(output) [0.2341 0.636 0.0862 0.01165 0.03168 ] """ def __init__(self, axis=-1): """Initialize Softmin.""" super(Softmin, self).__init__() self.axis = axis def construct(self, x): return ops.function.softmin(x, self.axis) class Softmax2d(Cell): r""" Softmax function applied to 2D features data. Applies `Softmax` to each location :math:`(c, h, w)` with an input Tensor of shape :math:`(C, H, W)` . Inputs: - **x** (Tensor) - Tensor of shape :math:`(N, C_{in}, H_{in}, W_{in})` or :math:`(C_{in}, H_{in}, W_{in})`. Outputs: Tensor, which has the same type and shape as `x` with values in the range[0,1]. Raises: TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `data_format` is neither 'NCHW' nor 'CHW'. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[[[0.1, 0.2]], [[0.3, 0.4]], [[0.6, 0.5]]]]), mindspore.float32) >>> softmax2d = nn.Softmax2d() >>> output = softmax2d(x) >>> print(output) [[[[0.25838965 0.28001308]] [[0.31559783 0.34200877]] [[0.42601252 0.37797815]]]] """ def __init__(self): """Initialize Softmax2d.""" super(Softmax2d, self).__init__() self.softmax = P.Softmax(axis=-3) self.shape = P.Shape() @staticmethod @_primexpr def _check_input_dim(shape, cls_name): dim = len(shape) if dim not in (3, 4): raise ValueError(f"For '{cls_name}', the in_shape must have 3 or 4 dims, but got {dim}.") def construct(self, x): x_shape = self.shape(x) self._check_input_dim(x_shape, self.cls_name) return self.softmax(x) class Softmax(Cell): r""" Softmax activation function, which is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of multi-classification, the purpose is to show the results of multi-classification in the form of probability. Calculate the value of the exponential function for the elements of the input Tensor on the `axis`, and then normalized to lie in range [0, 1] and sum up to 1. Softmax is defined as: .. math:: \text{softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_{j=0}^{n-1}\exp(x_j)}, where :math:`x_{i}` is the :math:`i`-th slice in the given dimension of the input Tensor. Args: axis (Union[int, tuple[int]]): The axis to apply Softmax operation, if the dimension of input `x` is x.ndim, the range of axis is `[-x.ndim, x.ndim)`, -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - The input of Softmax with data type of float16 or float32. Outputs: Tensor, which has the same type and shape as `x` with values in the range[0,1]. Raises: TypeError: If `axis` is neither an int nor a tuple. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is a tuple whose length is less than 1. ValueError: If `axis` is a tuple whose elements are not all in range [-len(x), len(x)). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> # axis = -1(default), and the sum of return value is 1.0. >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> softmax = nn.Softmax() >>> output = softmax(x) >>> print(output) [0.03168 0.01166 0.0861 0.636 0.2341 ] """ def __init__(self, axis=-1): """Initialize Softmax.""" super(Softmax, self).__init__() self.softmax = P.Softmax(axis) def construct(self, x): return self.softmax(x) class LogSoftmax(Cell): r""" Applies the LogSoftmax function to n-dimensional input tensor. The input is transformed by the Softmax function and then by the log function to lie in range[-inf,0). Logsoftmax is defined as: .. math:: \text{logsoftmax}(x_i) = \log \left(\frac{\exp(x_i)}{\sum_{j=0}^{n-1} \exp(x_j)}\right), Args: axis (int): The axis to apply LogSoftmax operation, -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - The input of LogSoftmax, with float16 or float32 data type. Outputs: Tensor, which has the same type and shape as `x` with output values in the range[-inf,0). Raises: TypeError: If `axis` is not an int. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is not in range [-len(x), len(x)). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> log_softmax = nn.LogSoftmax() >>> output = log_softmax(x) >>> print(output) [[-5.00672150e+00 -6.72150636e-03 -1.20067215e+01] [-7.00091219e+00 -1.40009127e+01 -9.12250078e-04]] """ def __init__(self, axis=-1): """Initialize LogSoftmax.""" super(LogSoftmax, self).__init__() self.log_softmax = P.LogSoftmax(axis) def construct(self, x): return self.log_softmax(x) class ELU(Cell): r""" Exponential Linear Unit activation function. Applies the exponential linear unit function element-wise. The activation function is defined as: .. math:: E_{i} = \begin{cases} x_i, &\text{if } x_i \geq 0; \cr \alpha * (\exp(x_i) - 1), &\text{otherwise.} \end{cases} where :math:`x_i` represents the element of the input and :math:`\alpha` represents the `alpha` parameter. ELU Activation Function Graph: .. image:: images/ELU.png :align: center Args: alpha (float): The alpha value of ELU, the data type is float. Default: ``1.0`` . Inputs: - **x** (Tensor) - The input of ELU is a Tensor of any dimension with data type of float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `alpha` is not equal to 1.0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float32) >>> elu = nn.ELU() >>> result = elu(x) >>> print(result) [-0.63212055 -0.86466473 0. 2. 1.] """ def __init__(self, alpha=1.0): """Initialize ELU.""" super(ELU, self).__init__() self.elu = P.Elu(alpha) def construct(self, x): return self.elu(x) class ReLU(Cell): r""" Rectified Linear Unit activation function. .. math:: \text{ReLU}(x) = (x)^+ = \max(0, x), It returns element-wise :math:`\max(0, x)`. Specially, the neurons with the negative output will be suppressed and the active neurons will stay the same. ReLU Activation Function Graph: .. image:: images/ReLU.png :align: center Inputs: - **x** (Tensor) - The input of ReLU is a Tensor of any dimension. The data type is `number `_ . Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is not a number. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16) >>> relu = nn.ReLU() >>> output = relu(x) >>> print(output) [0. 2. 0. 2. 0.] """ def __init__(self): """Initialize ReLU.""" super(ReLU, self).__init__() self.relu = P.ReLU() def construct(self, x): return self.relu(x) class ReLU6(Cell): r""" Compute ReLU6 activation function. ReLU6 is similar to ReLU with a upper limit of 6, which if the inputs are greater than 6, the outputs will be suppressed to 6. It computes element-wise as .. math:: Y = \min(\max(0, x), 6). The input is a Tensor of any valid shape. ReLU6 Activation Function Graph: .. image:: images/ReLU6.png :align: center Inputs: - **x** (Tensor) - The input of ReLU6 with data type of float16 or float32. Outputs: Tensor, which has the same type as `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> relu6 = nn.ReLU6() >>> output = relu6(x) >>> print(output) [0. 0. 0. 2. 1.] """ def __init__(self): """Initialize ReLU6.""" super(ReLU6, self).__init__() self.relu6 = P.ReLU6() def construct(self, x): return self.relu6(x) class LeakyReLU(Cell): r""" Leaky ReLU activation function. The activation function is defined as: .. math:: \text{leaky_relu}(x) = \begin{cases}x, &\text{if } x \geq 0; \cr {\alpha} * x, &\text{otherwise.}\end{cases} where :math:`\alpha` represents the `alpha` parameter. For more details, see `Rectifier Nonlinearities Improve Neural Network Acoustic Models `_. LeakyReLU Activation Function Graph: .. image:: images/LeakyReLU.png :align: center Args: alpha (Union[int, float]): Slope of the activation function at x < 0. Default: ``0.2`` . Inputs: - **x** (Tensor) - The input of LeakyReLU is a Tensor of any dimension. Outputs: Tensor, has the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float or an int. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> leaky_relu = nn.LeakyReLU() >>> output = leaky_relu(x) >>> print(output) [[-0.2 4. -1.6] [ 2. -1. 9. ]] """ def __init__(self, alpha=0.2): """Initialize LeakyReLU.""" super(LeakyReLU, self).__init__() validator.check_value_type('alpha', alpha, [float, int], self.cls_name) self.greater_equal = P.GreaterEqual() self.mul = P.Mul() self.alpha = alpha self.select_op = P.Maximum() if self.alpha > 1: self.select_op = P.Minimum() def construct(self, x): alpha_array = P.Cast()(F.scalar_to_tensor(self.alpha), P.DType()(x)) out = self.select_op(alpha_array * x, x) return out class RReLU(Cell): r""" Randomized Leaky ReLU activation function. The activation function is defined as: .. math:: \text{RReLU}(x_{ji}) = \begin{cases}x_{ji}, &\text{if } x_{ji} \geq 0; \cr {\alpha_{ji}} * x_{ji}, &\text{otherwise.}\end{cases} where :math:`\alpha_{ji}` ~ :math:`U(l, u)`, :math:`l \le u`. Applies the RReLU function elementally, as described in the paper: `Empirical Evaluation of Rectified Activations in Convolution Network `_ . Args: lower (Union[int, float]): Slope of the activation function at x < 0. Default: ``1 / 8`` . upper (Union[int, float]): Slope of the activation function at x < 0. Default: ``1 / 3`` . Inputs: - **x** (Tensor) - The input of RReLU is a Tensor of any dimension. Outputs: Tensor, after RReLU, has the same type and shape as the `x`. Raises: TypeError: If `lower` is not a float or an int. TypeError: If `upper` is not a float or an int. TypeError: If `x` is not a Tensor. TypeError: If `x` is not a Tensor of mindspore.float16 or mindpore.float32. ValueError: If `lower` is greater than upper. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0], [2.0, 0]]), mindspore.float32) >>> r_relu = nn.RReLU() >>> output = r_relu(x) >>> print(output) [[-0.31465699 4. ] [ 2. 0. ]] """ def __init__(self, lower=1 / 8, upper=1 / 3): super(RReLU, self).__init__() validator.check_value_type('upper', upper, [float, int], self.cls_name) validator.check_value_type('lower', lower, [float, int], self.cls_name) if lower > upper: raise ValueError(f"For {self.cls_name}, the value of 'upper' must be greater than or equal to 'lower', " f"but got upper: {upper}, lower: {lower}. ") self.lower = Tensor(lower, dtype=mstype.float32) self.upper = Tensor(upper, dtype=mstype.float32) self.sign = P.Sign() def construct(self, x): """RReLU construct.""" if not isinstance(x, Tensor): raise TypeError(f"For 'rrelu', the input must be a Tensor, but got {type(x)}.") size = x.shape dtype = x.dtype sign_matrix = self.sign(x) negative_filter = sign_matrix.clip(None, 0) positive_filter = sign_matrix.clip(0, None) mask = ops.uniform(size, self.lower, self.upper).astype(dtype) negative_mask = negative_filter * mask * -1 total_mask = negative_mask + positive_filter out = total_mask * x return out class SeLU(Cell): r""" Activation function SeLU (Scaled exponential Linear Unit). SeLU Activation Function Graph: .. image:: images/SeLU.png :align: center Refer to :func:`mindspore.ops.selu` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> selu = nn.SeLU() >>> output = selu(input_x) >>> print(output) [[-1.1113307 4.202804 -1.7575096] [ 2.101402 -1.7462534 9.456309 ]] """ def __init__(self): """Initialize SeLU""" super(SeLU, self).__init__() self.selu = P.SeLU() def construct(self, input_x): return self.selu(input_x) class SiLU(Cell): r""" Sigmoid Linear Unit activation function. Applies the silu linear unit function element-wise. .. math:: \text{SiLU}(x) = x * \sigma(x), where :math:`x_i` is input, :math:`\sigma(x)` is Sigmoid function. .. math:: \text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)}, SiLU Activation Function Graph: .. image:: images/SiLU.png :align: center Inputs: - **x** (Tensor) - Input with the data type float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16) >>> silu = nn.SiLU() >>> output = silu(x) >>> print(output) [-0.269 1.762 -0.1423 1.762 -0.269] """ def construct(self, x): return ops.function.silu(x) class Tanh(Cell): r""" Applies the Tanh function element-wise, returns a new tensor with the hyperbolic tangent of the elements of input, The input is a Tensor with any valid shape. Tanh function is defined as: .. math:: tanh(x_i) = \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = \frac{\exp(2x_i) - 1}{\exp(2x_i) + 1}, where :math:`x_i` is an element of the input Tensor. Tanh Activation Function Graph: .. image:: images/Tanh.png :align: center Inputs: - **x** (Tensor) - Tensor of any dimension, input with data type of float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([1, 2, 3, 2, 1]), mindspore.float16) >>> tanh = nn.Tanh() >>> output = tanh(x) >>> print(output) [0.7617 0.964 0.995 0.964 0.7617] """ def __init__(self): """Initialize Tanh.""" super(Tanh, self).__init__() self.tanh = P.Tanh() def construct(self, x): return self.tanh(x) class Tanhshrink(Cell): r""" Tanhshrink activation function. The tanhshrink function is evaluated by element and returns a new tensor. Tanh function is defined as: .. math:: tanhshrink(x_i) =x_i- \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = x_i-\frac{\exp(2x_i) - 1}{\exp(2x_i) + 1}, where :math:`x_i` is an element of the input Tensor. Inputs: - **x** (Tensor) - Tensor of any dimension. Outputs: Tensor, with the same shape as the `x`. Raises: TypeError: If `x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([1, 2, 3, 2, 1]), ms.float16) >>> tanhshrink = nn.Tanhshrink() >>> output = tanhshrink(x) >>> print(output) [0.2383 1.036 2.004 1.036 0.2383] """ def construct(self, x): return F.tanhshrink(x) class Hardtanh(Cell): r""" Applies the Hardtanh function element-wise. The activation function is defined as: .. math:: \text{Hardtanh}(x) = \begin{cases} 1, & \text{ if } x > 1; \\ -1, & \text{ if } x < -1; \\ x, & \text{ otherwise. } \end{cases} Linear region range :math:`[-1, 1]` can be adjusted using `min_val` and `max_val`. Hardtanh Activation Function Graph: .. image:: images/Hardtanh.png :align: center Note: On Ascend, data type of float16 might lead to accidental accuracy problem. Args: min_val (Union[int, float]): Minimum value of the linear region range. Default: ``-1.0`` . max_val (Union[int, float]): Maximum value of the linear region range. Default: ``1.0`` . Inputs: - **x** (Tensor) - Input Tensor with data type of float16 or float32. On CPU and Ascend support dimension 0-7D. On GPU support dimension 0-4D. Outputs: Tensor, with the same dtype and shape as `x`. Raises: TypeError: If `x` is not a Tensor. TypeError: If dtype of `x` is neither float16 nor float32. TypeError: If dtype of `min_val` is neither float nor int. TypeError: If dtype of `max_val` is neither float nor int. ValueError: If `min_val` is not less than `max_val`. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hardtanh = nn.Hardtanh(min_val=-1.0, max_val=1.0) >>> output = hardtanh(x) >>> print(output) [-1. -1. 0. 1. 1.] """ def __init__(self, min_val=-1.0, max_val=1.0): """Initialize Hardtanh.""" super(Hardtanh, self).__init__() self.min_val = min_val self.max_val = max_val if self.min_val >= self.max_val: raise ValueError(f"For Hardtanh, min_val should be less than max_val," f"but got {self.min_val} and {self.max_val}") def construct(self, x): return F.hardtanh(x, self.min_val, self.max_val) class GELU(Cell): r""" Gaussian error linear unit activation function. Applies GELU function to each element of the input. The input is a Tensor with any valid shape. GELU is defined as: .. math:: GELU(x_i) = x_i*P(X < x_i), where :math:`P` is the cumulative distribution function of standard Gaussian distribution and :math:`x_i` is the element of the input. GELU Activation Function Graph: .. image:: images/GELU.png :align: center Args: approximate (bool): Whether to enable approximation. Default: ``True`` . If `approximate` is ``True``, The gaussian error linear activation is: :math:`0.5 * x * (1 + tanh(\sqrt(2 / \pi) * (x + 0.044715 * x^3)))` else, it is: :math:`x * P(X <= x) = 0.5 * x * (1 + erf(x / \sqrt(2)))`, where P(X) ~ N(0, 1). Inputs: - **x** (Tensor) - The input of GELU with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> gelu = nn.GELU() >>> output = gelu(x) >>> print(output) [[-1.5880802e-01 3.9999299e+00 -3.1077917e-21] [ 1.9545976e+00 -2.2918017e-07 9.0000000e+00]] >>> gelu = nn.GELU(approximate=False) >>> # CPU not support "approximate=False", using "approximate=True" instead >>> output = gelu(x) >>> print(output) [[-1.5865526e-01 3.9998732e+00 -0.0000000e+00] [ 1.9544997e+00 -1.4901161e-06 9.0000000e+00]] """ def __init__(self, approximate=True): """Initialize GELU.""" super(GELU, self).__init__() validator.check_bool(approximate, 'approximate', self.cls_name) self.approximate = approximate if self.approximate: self.gelu = P.GeLU() else: self.erf = P.Erf() self.sqrt = P.Sqrt() self.const0 = Tensor(0.5, mstype.float32) self.const1 = Tensor(1.0, mstype.float32) self.const2 = Tensor(2.0, mstype.float32) def construct(self, x): if self.approximate: return self.gelu(x) return x * F.cast(self.const0, x.dtype) * (F.cast(self.const1, x.dtype) + \ self.erf(x / self.sqrt(F.cast(self.const2, x.dtype)))) class FastGelu(Cell): r""" Fast Gaussian error linear unit activation function. Applies FastGelu function to each element of the input. The input is a Tensor with any valid shape. FastGelu is defined as: .. math:: FastGelu(x_i) = \frac {x_i} {1 + \exp(-1.702 * \left| x_i \right|)} * \exp(0.851 * (x_i - \left| x_i \right|)) where :math:`x_i` is the element of the input. FastGelu Activation Function Graph: .. image:: images/FastGelu.png :align: center Inputs: - **x** (Tensor) - The input of FastGelu with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> fast_gelu = nn.FastGelu() >>> output = fast_gelu(x) >>> print(output) [[-1.5418735e-01 3.9921875e+00 -9.7473649e-06] [ 1.9375000e+00 -1.0052517e-03 8.9824219e+00]] """ def __init__(self): """Initialize FastGelu.""" super(FastGelu, self).__init__() self.fast_gelu = P.FastGeLU() def construct(self, x): return self.fast_gelu(x) class Sigmoid(Cell): r""" Sigmoid activation function. Applies sigmoid-type activation element-wise. Sigmoid function is defined as: .. math:: \text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)}, where :math:`x_i` is the element of the input. Sigmoid Activation Function Graph: .. image:: images/Sigmoid.png :align: center Inputs: - **input_x** (Tensor) - Tensor of any dimension, the data type is float16, float32, float64, complex64 or complex128. Outputs: Tensor, with the same type and shape as the `input_x`. Raises: TypeError: If dtype of `input_x` is not float16, float32, float64, complex64 or complex128. TypeError: If `input_x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> sigmoid = nn.Sigmoid() >>> output = sigmoid(x) >>> print(output) [0.2688 0.11914 0.5 0.881 0.7305 ] """ def __init__(self): """Initialize Sigmoid.""" super(Sigmoid, self).__init__() self.sigmoid = P.Sigmoid() def construct(self, x): return self.sigmoid(x) class Softsign(Cell): r""" Softsign activation function. Softsign Activation Function Graph: .. image:: images/Softsign.png :align: center Refer to :func:`mindspore.ops.softsign` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([0, -1, 2, 30, -30]), mindspore.float32) >>> softsign = nn.Softsign() >>> output = softsign(x) >>> print(output) [ 0. -0.5 0.6666667 0.9677419 -0.9677419] """ def __init__(self): """Initialize Softsign.""" super(Softsign, self).__init__() self.softsign = P.Softsign() def construct(self, x): return self.softsign(x) class PReLU(Cell): r""" PReLU activation function. Applies the PReLU function element-wise. PReLU is defined as: .. math:: PReLU(x_i)= \max(0, x_i) + w * \min(0, x_i), where :math:`x_i` is an element of an channel of the input. Here :math:`w` is a learnable parameter with a default initial value 0.25. Parameter :math:`w` has dimensionality of the argument channel. If called without argument channel, a single parameter :math:`w` will be shared across all channels. PReLU Activation Function Graph: .. image:: images/PReLU.png :align: center Args: channel (int): The elements number of parameter :math:`w`. It could be an int, and the value is 1 or the channels number of input tensor `x`. Default: ``1`` . w (Union[float, list, Tensor]): The initial value of parameter. It could be a float, a float list or a tensor has the same dtype as the input tensor `x`. Default: ``0.25`` . Inputs: - **x** (Tensor) - The input of PReLU with data type of float16 or float32. The shape is :math:`(N, *)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same dtype and shape as the `x`. Raises: TypeError: If `channel` is not an int. TypeError: If `w` is not one of a float, a float list, a float Tensor. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If the `x` is a 0-D or 1-D Tensor on Ascend. ValueError: If `channel` is less than 1. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[[[0.1, 0.6], [0.9, 0.9]]]]), mindspore.float32) >>> prelu = nn.PReLU() >>> output = prelu(x) >>> print(output) [[[[0.1 0.6] [0.9 0.9]]]] """ @cell_attr_register(attrs="") def __init__(self, channel=1, w=0.25): """Initialize PReLU.""" super(PReLU, self).__init__() validator.check_positive_int(channel, 'channel', self.cls_name) if isinstance(w, (float, np.float32)): tmp = np.empty((channel,), dtype=np.float32) tmp.fill(w) w = Tensor(tmp, dtype=mstype.float32) elif isinstance(w, list): if len(w) != channel: raise ValueError(f"For '{self.cls_name}', the length of 'w' must be equal to the 'channel' when " f"the 'w' is a list, but got the length of 'w': {len(w)}, the 'channel': {channel}.") for i in w: if not isinstance(i, (float, np.float32)): raise ValueError(f"For '{self.cls_name}', all elements in 'w' must be " f"float when the 'w' is a list, but got {i}.") w = Tensor(w, dtype=mstype.float32) elif isinstance(w, Tensor): if w.dtype not in (mstype.float16, mstype.float32): raise ValueError(f"For '{self.cls_name}', the dtype of 'w' must be float16 or " f"float32 when the 'w' is a tensor, but got {w.dtype}.") if len(w.shape) != 1 or w.shape[0] != channel: raise ValueError(f"For '{self.cls_name}', the dimension of 'w' must be 1, and the elements number " f"should be equal to the 'channel' when the 'w' is a tensor, " f"but got 'w' shape {w.shape}, the 'channel' {channel}.") else: raise TypeError(f"For '{self.cls_name}', the 'w' only supported float, list and tensor, " f"but got {type(w).__name__}.") self.w = Parameter(w, name='a') self.prelu = P.PReLU() def construct(self, x): return self.prelu(x, F.cast(self.w, x.dtype)) class HSwish(Cell): r""" Applies hswish-type activation element-wise. The input is a Tensor with any valid shape. Hard swish is defined as: .. math:: \text{hswish}(x_{i}) = x_{i} * \frac{ReLU6(x_{i} + 3)}{6}, HSwish Activation Function Graph: .. image:: images/HSwish.png :align: center Inputs: - **x** (Tensor) - The input of HSwish, data type must be float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hswish = nn.HSwish() >>> result = hswish(x) >>> print(result) [-0.3333 -0.3333 0. 1.667 0.6665] """ def __init__(self): """Initialize HSwish.""" super(HSwish, self).__init__() self.hswish = P.HSwish() def construct(self, x): return self.hswish(x) class HSigmoid(Cell): r""" Hard sigmoid activation function. Calculates the output according to the input elements. Hard sigmoid is defined as: .. math:: \text{hsigmoid}(x_{i}) = \max(0, \min(1, \frac{x_{i} + 3}{6})), HSigmoid Activation Function Graph: .. image:: images/HSigmoid.png :align: center Inputs: - **input_x** (Tensor) - The input of HSigmoid. Tensor of any dimension. Outputs: Tensor, with the same type and shape as the `input_x`. Raises: TypeError: If `input_x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hsigmoid = nn.HSigmoid() >>> result = hsigmoid(x) >>> print(result) [0.3333 0.1666 0.5 0.8335 0.6665] """ def __init__(self): """Initialize HSigmoid.""" super(HSigmoid, self).__init__() self.hsigmoid = P.HSigmoid() def construct(self, input_x): return self.hsigmoid(input_x) class LogSigmoid(Cell): r""" Applies logsigmoid activation element-wise. The input is a Tensor with any valid shape. Logsigmoid is defined as: .. math:: \text{logsigmoid}(x_{i}) = \log(\frac{1}{1 + \exp(-x_i)}), where :math:`x_{i}` is the element of the input. LogSigmoid Activation Function Graph: .. image:: images/LogSigmoid.png :align: center Inputs: - **x** (Tensor) - The input of LogSigmoid with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> net = nn.LogSigmoid() >>> x = Tensor(np.array([1.0, 2.0, 3.0]), mindspore.float32) >>> output = net(x) >>> print(output) [-0.31326166 -0.12692806 -0.04858734] """ def __init__(self): """Initialize LogSigmoid.""" super(LogSigmoid, self).__init__() self.mul = P.Mul() self.exp = P.Exp() self.add = P.Add() self.rec = P.Reciprocal() self.log = P.Log() def construct(self, input_x): neg_input = self.mul(input_x, -1) exp_neg_input = self.exp(neg_input) exp_neg_input_1 = self.add(exp_neg_input, 1) rec_exp_neg_input_1 = self.rec(exp_neg_input_1) ret = self.log(rec_exp_neg_input_1) return ret class LRN(Cell): r""" Local Response Normalization. Refer to :func:`mindspore.ops.lrn` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[[[0.1], [0.2]], ... [[0.3], [0.4]]]]), mindspore.float32) >>> output = nn.LRN()(input_x) >>> print(output) [[[[0.09534626] [0.1825742 ]] [[0.2860388 ] [0.3651484 ]]]] """ def __init__(self, depth_radius=5, bias=1.0, alpha=1.0, beta=0.5, norm_region="ACROSS_CHANNELS"): """Initialize LRN.""" super(LRN, self).__init__() self.lrn_op = NN_OPS.LRN(depth_radius, bias, alpha, beta, norm_region) def construct(self, input_x): return self.lrn_op(input_x) class SoftShrink(Cell): r""" Applies the SoftShrink function element-wise. .. math:: \text{SoftShrink}(x) = \begin{cases} x - \lambda, & \text{ if } x > \lambda \\ x + \lambda, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} SoftShrink Activation Function Graph: .. image:: images/Softshrink.png :align: center Args: lambd (float): the :math:`\lambda` must be no less than zero for the SoftShrink formulation. Default: ``0.5`` . Inputs: - **input_x** (Tensor) - The input of SoftShrink with data type of float16 or float32. Any number of additional dimensions. Outputs: Tensor, has the same shape and data type as `input_x`. Raises: TypeError: If lambd is not a float. TypeError: If input_x is not a Tensor. TypeError: If dtype of input_x is neither float16 nor float32. ValueError: If lambd is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[ 0.5297, 0.7871, 1.1754], [ 0.7836, 0.6218, -1.1542]]), mindspore.float16) >>> softshrink = nn.SoftShrink() >>> output = softshrink(input_x) >>> print(output) [[ 0.02979 0.287 0.676 ] [ 0.2837 0.1216 -0.6543 ]] """ def __init__(self, lambd=0.5): super(SoftShrink, self).__init__() self.softshrink = P.SoftShrink(lambd) def construct(self, input_x): output = self.softshrink(input_x) return output class HShrink(Cell): r""" Hard Shrink activation function. Calculates the output according to the input elements. The formula is defined as follows: .. math:: \text{HardShrink}(x) = \begin{cases} x, & \text{ if } x > \lambda \\ x, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} HShrink Activation Function Graph: .. image:: images/HShrink.png :align: center Args: lambd (float): The threshold :math:`\lambda` defined by the Hard Shrink formula. Default: ``0.5`` . Inputs: - **input_x** (Tensor) - The input of Hard Shrink with data type of float16 or float32. Outputs: Tensor, the same shape and data type as the input. Raises: TypeError: If `lambd` is not a float. TypeError: If dtype of `input_x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[ 0.5, 1, 2.0], [0.0533,0.0776,-2.1233]]), mindspore.float32) >>> hshrink = nn.HShrink() >>> output = hshrink(input_x) >>> print(output) [[ 0. 1. 2. ] [ 0. 0. -2.1233]] """ def __init__(self, lambd=0.5): super(HShrink, self).__init__() self.hshrink = P.HShrink(lambd) def construct(self, input_x): return self.hshrink(input_x) class Threshold(Cell): r"""Thresholds each element of the input Tensor. The formula is defined as follows: .. math:: y = \begin{cases} x, &\text{ if } x > \text{threshold} \\ \text{value}, &\text{ otherwise } \end{cases} Args: threshold (Union[int, float]): The value to threshold at. value (Union[int, float]): The value to replace with when element is less than threshold. Inputs: - **input_x** (Tensor) - The input of Threshold with data type of float16 or float32. Outputs: Tensor, the same shape and data type as the input. Raises: TypeError: If `threshold` is not a float or an int. TypeError: If `value` is not a float or an int. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> m = nn.Threshold(0.1, 20) >>> inputs = Tensor([0.1, 0.2, 0.3], mindspore.float32) >>> outputs = m(inputs) >>> print(outputs) [ 20.0 0.2 0.3] """ def __init__(self, threshold, value): """Initialize Threshold.""" super(Threshold, self).__init__() self.threshold = threshold self.value = value def construct(self, input_x): return F.threshold(input_x, self.threshold, self.value) class Mish(Cell): r""" Computes MISH(A Self Regularized Non-Monotonic Neural Activation Function) of input tensors element-wise. Refer to :func:`mindspore.ops.mish` for more details. Mish Activation Function Graph: .. image:: images/Mish.png :align: center Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> mish = nn.Mish() >>> output = mish(x) >>> print(output) [[-3.03401530e-01 3.99741292e+00 -2.68321624e-03] [ 1.94395900e+00 -3.35762873e-02 9.00000000e+00]] """ def __init__(self): """Initialize Mish.""" super().__init__("Mish") self.mish = NN_OPS.Mish() def construct(self, input_x): return self.mish(input_x) class GLU(Cell): r""" The gated linear unit function. .. math:: {GLU}(a, b)= a \otimes \sigma(b) where :math:`a` is the first half of the input matrices and :math:`b` is the second half. Here :math:`\sigma` is the sigmoid function, and :math:`\otimes` is the Hadamard product. Args: axis (int): the axis to split the input. Default: ``-1`` , the last axis in `x`. Inputs: - **x** (Tensor) - :math:`(\ast_1, N, \ast_2)` where `*` means, any number of additional dimensions. Outputs: Tensor, the same dtype as the `x`, with the shape :math:`(\ast_1, M, \ast_2)` where :math:`M=N/2`. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> m = ms.nn.GLU() >>> input = ms.Tensor([[0.1,0.2,0.3,0.4],[0.5,0.6,0.7,0.8]]) >>> output = m(input) >>> print(output) [[0.05744425 0.11973753] [0.33409387 0.41398472]] """ def __init__(self, axis=-1): """Initialize GLU.""" super().__init__("GLU") self.dim = axis self.spilt = P.Split(axis=axis, output_num=2) self.sigmoid = P.Sigmoid() def construct(self, x): x1, x2 = self.spilt(x) x2 = self.sigmoid(x2) return x1 * x2 _activation = { 'softmin': Softmin, 'softmax': Softmax, 'softmax2d': Softmax2d, 'logsoftmax': LogSoftmax, 'relu': ReLU, 'relu6': ReLU6, 'rrelu': RReLU, 'silu': SiLU, 'tanh': Tanh, 'tanhshrink': Tanhshrink, 'hardtanh': Hardtanh, 'gelu': GELU, 'fast_gelu': FastGelu, 'elu': ELU, 'sigmoid': Sigmoid, 'softsign': Softsign, 'prelu': PReLU, 'leakyrelu': LeakyReLU, 'hswish': HSwish, 'hsigmoid': HSigmoid, 'logsigmoid': LogSigmoid, 'softshrink': SoftShrink, 'hshrink': HShrink, 'threshold': Threshold, 'mish': Mish, } def get_activation(name, prim_name=None): """ Gets the activation function. Args: name (str): The name of the activation function. prim_name (Union[str, None]): The name of primitive. Default: ``None`` . Returns: Function, the activation function. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore.nn as nn >>> sigmoid = nn.get_activation('sigmoid') >>> print(sigmoid) Sigmoid<> """ msg_prefix = f"For '{prim_name}', the" if prim_name else "The" if name is None: return None if name not in _activation: raise KeyError(f"{msg_prefix} 'name' must be in {list(_activation.keys())}, but got {name}.") return _activation[name]() pengcheng.mind-test/mindformers/modules/layers.py000066400000000000000000001371051471400134000225660ustar00rootroot00000000000000# Copyright 2021 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ The basic layer of the Transformer Networks. This is an experimental interface that is subject to change or deletion. """ from __future__ import absolute_import from functools import wraps, partial import inspect import math import numpy as np from mindspore import nn, context from mindspore.common.parameter import Parameter from mindspore.common.initializer import initializer, Tensor import mindspore.common.dtype as mstype from mindspore.common.seed import _get_graph_seed from mindspore._extends import cell_attr_register from mindspore.nn.cell import Cell from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.ops.primitive import constexpr # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindspore.context import ParallelMode from mindformers.tools.logger import logger from mindformers.modules.activation import get_activation from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, OpParallelConfig, MoEParallelConfig __all__ = [ "FixedSparseAttention", "Dropout", "LayerNorm", "Linear" ] def _args_type_validator_check(*type_args, **type_kwargs): """Check whether input data type is correct.""" def type_check(func): sig = inspect.signature(func) bound_types = sig.bind_partial(*type_args, **type_kwargs).arguments @wraps(func) def wrapper(*args, **kwargs): nonlocal bound_types bound_values = sig.bind(*args, **kwargs) argument_dict = bound_values.arguments if "kwargs" in bound_types: bound_types = bound_types["kwargs"] if "kwargs" in argument_dict: argument_dict = argument_dict["kwargs"] for name, value in argument_dict.items(): if name in bound_types: bound_types[name](value, name) return func(*args, **kwargs) return wrapper return type_check def _valid_type_checks(types, class_name): # types should be a list of types, this function check if the type is in the valid dtypes def validator_check_func(value, name): # The args of Validator.check_type_name is (arg_name, arg_type, valid_types, prim_name) # as the input of _args_type_validator_check is fixed, so we need to manually change the input order partial_check = partial(Validator.check_type_name, valid_types=types, prim_name=class_name) return partial_check(name, type(value)) return validator_check_func def _valid_value_checks(types, class_name): # the value should be a list of types, this function check if the value is in the valid dtypes def validator_check_func(value, name): # The args of Validator.check_type_name is (arg_name, arg_type, valid_types, prim_name) # as the input of _args_type_validator_check is fixed, so we need to manually change the input order partial_check = partial(Validator.check_type_name, valid_types=types, prim_name=class_name) return partial_check(name, value) return validator_check_func class _LayerInputCheck: """ A input check class for the inputs of the transformer model. """ @staticmethod def check_shape_length(input_shape, param_name, func_name, target_len): """ Check the input shape's length is equal to the expected shape :param input_shape(list): a list of the tensor shapes. :param param_name(str): the name of the checked parameter. :param func_name(str): the name of the function. :param target_len: the expected length of the shape. :return: """ if not isinstance(target_len, list): target_len = [target_len] matched = False for item in target_len: if len(input_shape) == item: matched = True if not matched: raise ValueError(f"{func_name} {param_name} shape length must be one of {target_len} dimension, " f"but got shape {input_shape}") return True @staticmethod def check_shape_equal(input_shape, param_name, func_name, target_shape): """ Check the input shape's is equal to the expected shape :param input_shape(list): a list of the tensor shapes. :param param_name(str): the name of the checked parameter. :param func_name(str): the name of the function. :param target_shape: the expected shape. :return: """ if not isinstance(target_shape[0], list): target_shape = [target_shape] if isinstance(input_shape, tuple): input_shape = list(input_shape) _LayerInputCheck.check_shape_length(input_shape, param_name, func_name, [len(item) for item in target_shape]) matched = False for item in target_shape: if item == input_shape: matched = True break if not matched: raise ValueError(f"{func_name} {param_name} shape must be one of {target_shape}," f"but got {input_shape}") return True @staticmethod def check_shape_value_on_axis(input_shape, dim, param_name, cls_name, target_value): """ Check whether the input_shape[dim] is equal to target value""" if input_shape[dim] != target_value: raise ValueError(f"{cls_name} {param_name} at {dim} shape must be {target_value}," f"but got {input_shape[dim]}") return True @staticmethod def check_shape_equal_without_batch(input_shape, param_name, func_name, target_shape): """ Check the input shape's is equal to the expected shape, the value on 0-th is viewed as batch, and the batch size will not be checked. """ target_shape = target_shape length, hidden = target_shape if isinstance(input_shape, tuple): input_shape = list(input_shape) _LayerInputCheck.check_shape_length(input_shape, param_name, func_name, [len(target_shape), len(target_shape) + 1]) if input_shape[-1] != hidden: raise ValueError(f"For {func_name}, the last dimension of {param_name} shape must be {hidden}," f"but got the last dimension {input_shape[-1]} in {input_shape}.") if input_shape[0] == 0: raise ValueError(f"For {func_name}, the first dimension of {param_name} shape greater than 0," f"but got the first dimension {input_shape[0]} in {input_shape}.") if len(input_shape) == 2 and input_shape[0] % length != 0: raise ValueError(f"For {func_name}, the first dimension of {param_name} shape should be divisible " f"by {length}, " f"but got the first dimension {input_shape[0]} in {input_shape}.") return True @constexpr def _check_past_none_input_none(use_past, param_name, func_name, default_value, is_tensor, is_default): """ If the past is True, check whether the inputs is None""" if not use_past: if is_tensor: raise TypeError(f"{func_name} {param_name} must be {default_value}, if use_pat is False, but found " f"a tensor") if not is_default: raise TypeError(f"{func_name} {param_name} must be {default_value}, if use_pat is False.") else: if not is_tensor: raise TypeError(f"{func_name} {param_name} must be tensor, if use_pat is True") return True @constexpr def _check_input_dtype(input_dtype, param_name, allow_dtypes, cls_name): Validator.check_type_name(param_name, input_dtype, allow_dtypes, cls_name) @constexpr def _check_input_shape(input_shape, param_name, func_name, target_len): # check the input length _LayerInputCheck.check_shape_length(input_shape, param_name, func_name, target_len) @constexpr def _check_shape_equal(input_shape, param_name, func_name, target_shape): # check the input length _LayerInputCheck.check_shape_equal(input_shape, param_name, func_name, target_shape) @constexpr def _check_input_shape_value(input_shape, dim, param_name, cls_name, target_value): _LayerInputCheck.check_shape_value_on_axis(input_shape, dim, param_name, cls_name, target_value) @constexpr def _check_shape_equal_without_batch(input_shape, param_name, func_name, target_shape): _LayerInputCheck.check_shape_equal_without_batch(input_shape, param_name, func_name, target_shape) class Dropout(nn.Cell): r""" A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for parallel training. """ def __init__(self, keep_prob=0.5, dtype=mstype.float32): super(Dropout, self).__init__() if keep_prob <= 0 or keep_prob > 1: raise ValueError( "dropout probability should be a number in range (0, 1], but got {}".format( keep_prob)) Validator.check_subclass("dtype", dtype, mstype.number_type, self.cls_name) Validator.check_value_type('keep_prob', keep_prob, [float], self.cls_name) self.keep_prob = keep_prob self.is_ascend = context.get_context('device_target') in ["Ascend"] if self.is_ascend: seed0, seed1 = _get_graph_seed(0, "dropout") self.seed0 = seed0 self.seed1 = seed1 self.dtype = dtype self.get_shape = P.Shape() self.dropout_gen_mask = P.DropoutGenMask(Seed0=self.seed0, Seed1=self.seed1) self.dropout_do_mask = P.DropoutDoMask() self.cast = P.Cast() else: self.dropout = P.Dropout(keep_prob) def construct(self, x): r""" Input: a tensor Returns: a tensor """ if not self.training: return x if self.keep_prob == 1: return x if not self.is_ascend: out, _ = self.dropout(x) return out shape = self.get_shape(x) dtype = P.DType()(x) keep_prob = self.cast(self.keep_prob, dtype) output = self.dropout_gen_mask(shape, keep_prob) return self.dropout_do_mask(x, output, keep_prob) def extend_repr(self): return 'keep_prob={}'.format(self.keep_prob) def shard(self, strategy): if self.is_ascend: self.dropout_gen_mask.shard(strategy) self.dropout_do_mask.shard(strategy) else: self.dropout.shard(strategy) class LayerNorm(Cell): r""" A self-defined layer norm operation using reduce sum and reduce mean Args: normalized_shape (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. param_init_type: The param init type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, normalized_shape, eps=1e-5, param_init_type=mstype.float32, is_self_defined=False): super(LayerNorm, self).__init__() if param_init_type not in [mstype.float32, mstype.float16]: raise TypeError("The type of parameter 'param_init_type' should in [float32, float16], " "but got the type : {}.".format(type(param_init_type))) # Since the mindspore 1.10 version, the layernorm has been changed to P.LayerNorm self.is_self_defined = is_self_defined if not self.is_self_defined: self.layer_norm = P.LayerNorm(begin_norm_axis=-1, begin_params_axis=-1, epsilon=eps) self.gamma = Parameter(initializer('ones', normalized_shape, param_init_type), name="gamma", parallel_optimizer=False) self.beta = Parameter(initializer('zeros', normalized_shape, param_init_type), name="beta", parallel_optimizer=False) self.mean = P.ReduceMean(keep_dims=True) self.square = P.Square() self.sqrt = P.Sqrt() self.sub1 = P.Sub() self.sub2 = P.Sub() self.add = P.Add() self.eps = eps self.mul = P.Mul() self.add2 = P.Add() self.real_div = P.RealDiv() def construct(self, x): r""" x : batch x seq_length x hidden_size """ if self.is_self_defined: mean = self.mean(x, -1) diff = self.sub1(x, mean) variance = self.mean(self.square(diff), -1) variance_eps = self.sqrt(self.add(variance, self.eps)) output = self.real_div(diff, variance_eps) output = self.add2(self.mul(output, self.gamma), self.beta) else: output, _, _ = self.layer_norm(x, self.gamma, self.beta) return output def shard(self, strategy): r""" Set the shard for the layer norm. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy (tuple): The strategy for the dropout. Should be the same shape as the inputs. Examples: >>> import mindspore >>> net = mindformers.modules.transformer.LayerNorm(normalized_shape=(1024, 10)) >>> net.shard(((10, 2, 1),)) """ if self.is_self_defined: self.mean.shard(strategy) self.square.shard(strategy) self.sqrt.shard(strategy) self.sub1.shard((strategy[0], strategy[0])) self.sub2.shard((strategy[0], strategy[0])) self.add.shard((strategy[0], ())) self.mul.shard((strategy[0], (1,))) self.add2.shard((strategy[0], (1,))) self.real_div.shard((strategy[0], strategy[0])) else: self.layer_norm.shard((strategy[0], (1,), (1,))) return self class Linear(Cell): r""" The dense connected layer. Once the parallel mode is enabled, the input shape should be 3-D tensor. Applies dense connected layer for the input. This layer implements the operation as: .. math:: \text{outputs} = \text{activation}(\text{X} * \text{kernel} + \text{bias}), where :math:`X` is the input tensors, :math:`\text{activation}` is the activation function passed as the activation argument (if passed in), :math:`\text{kernel}` is a weight matrix with the same data type as the :math:`X` created by the layer, and :math:`\text{bias}` is a bias vector with the same data type as the :math:`X` created by the layer (only if has_bias is True). Args: in_channels (int): The number of channels in the input space. out_channels (int): The number of channels in the output space. weight_init (Union[Tensor, str, Initializer, numbers.Number]): The trainable weight_init parameter. The dtype is same as `x`. The values of str refer to the function `initializer`. Default: 'normal'. bias_init (Union[Tensor, str, Initializer, numbers.Number]): The trainable bias_init parameter. The dtype is same as `x`. The values of str refer to the function `initializer`. Default: 'zeros'. has_bias (bool): Specifies whether the layer uses a bias vector. Default: True. activation (str): activate function applied to the output of the fully connected layer, eg. 'ReLU'. Default: None. expert_num (int): The number of experts used in this Linear. Here, for the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. outer_batch (int): The replication number of experts. The replication is effective only when MoE is applied. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. compute_dtype (dtype.Number): The computation type. Default: mstype.float16 Inputs: - **x** (Tensor) - Tensor of shape :math:`(*, in\_channels)`. The `in_channels` in `Args` should be equal to :math:`in\_channels` in `Inputs`. Outputs: Tensor of shape :math:`(*, out\_channels)`. Raises: TypeError: If `in_channels` or `out_channels` is not an int. TypeError: If `has_bias` is not a bool. TypeError: If `activation` is not one of str, Cell, Primitive, None. ValueError: If length of shape of `weight_init` is not equal to 2 or shape[0] of `weight_init` is not equal to `out_channels` or shape[1] of `weight_init` is not equal to `in_channels`. ValueError: If length of shape of `bias_init` is not equal to 1 or shape[0] of `bias_init` is not equal to `out_channels`. Supported Platforms: ``Ascend`` ``GPU`` """ @cell_attr_register @_args_type_validator_check(in_channels=Validator.check_positive_int, out_channels=Validator.check_positive_int, has_bias=Validator.check_bool, transpose_b=Validator.check_bool, expert_num=Validator.check_positive_int, outer_batch=Validator.check_positive_int, param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "Linear"), compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "Linear")) def __init__(self, in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, transpose_b=True, expert_num=1, outer_batch=1, expert_group_size=None, param_init_type=mstype.float32, compute_dtype=mstype.float16): super(Linear, self).__init__() self.in_channels = in_channels self.out_channels = out_channels if not (isinstance(activation, str) or activation is None or issubclass(activation, nn.Cell)): raise TypeError(f"For Linear cell, the activation should str type or nn.Cell type, but got {activation}.") if isinstance(weight_init, Tensor) and (weight_init.ndim != 2 or weight_init.shape[0] != out_channels or weight_init.shape[1] != in_channels): raise ValueError("The shape of parameter 'weight_init' is error, please check shape of 'weight_init'.") weight_shape = [out_channels, in_channels] if transpose_b else [in_channels, out_channels] self.expert_num = expert_num self.outer_batch = outer_batch self.expert_group_size = expert_group_size self.transpose_b = transpose_b if self.expert_num > 1: self.expert_flag = True self.weight = Parameter(initializer(weight_init, [self.expert_num] + weight_shape, param_init_type), name="weight") self.matmul = P.BatchMatMul(transpose_b=transpose_b) else: self.expert_flag = False self.weight = Parameter(initializer(weight_init, weight_shape, param_init_type), name="weight") self.matmul = P.MatMul(transpose_b=transpose_b) self.use_expert_group_size = _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) \ and not _is_sharding_propagation() and self.expert_flag is True if self.use_expert_group_size is True and self.expert_group_size is None: raise ValueError("'expert_group_size' should be configured as an integer in MoEConfig.") self.bias = None self.has_bias = has_bias if self.has_bias: if isinstance(bias_init, Tensor) and (bias_init.ndim != 1 or bias_init.shape[0] != out_channels): raise ValueError("The shape of parameter 'bias_init' is error, please check shape of 'bias_init'.") if self.expert_flag: self.bias = Parameter(initializer(bias_init, [1, self.expert_num, 1, out_channels], param_init_type), name="bias") else: self.bias = Parameter(initializer(bias_init, [out_channels], param_init_type), name="bias") self.bias.parallel_optimizer = False self.bias_add = P.Add() self.act_name = activation if callable(activation): self.activation = activation() else: self.activation = get_activation(activation) if isinstance(activation, str) else activation self.activation_flag = self.activation is not None self.dtype = compute_dtype self.cast = P.Cast() def construct(self, x): """Forward process, x should be a tensor""" out_shape = P.Shape()(x)[:-1] + (self.out_channels,) x = P.Reshape()(x, (-1, self.in_channels)) if self.expert_flag: if self.use_expert_group_size is True: x = P.Reshape()(x, (-1, self.expert_num, self.expert_group_size, self.in_channels)) else: x = P.Reshape()(x, (self.outer_batch, self.expert_num, -1, self.in_channels)) ori_dtype = F.dtype(x) weight = self.cast(self.weight, self.dtype) x = self.cast(x, self.dtype) x = self.matmul(x, weight) if self.has_bias: x = self.bias_add(x, self.cast(self.bias, self.dtype)) if self.activation_flag: x = self.activation(x) x = F.cast(x, ori_dtype) output = P.Reshape()(x, out_shape) return output def shard(self, strategy_matmul, strategy_bias=None, strategy_activation=None, out_strategy_matmul=None): r""" Set the shard for the linear. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy_matmul (tuple): The strategy for the matmul. Should be the same shape as the inputs. strategy_bias (tuple): The strategy for the bias_add. Should be the same shape as the inputs. strategy_activation (tuple): The strategy for the strategy_activation. Should be the same shape as the inputs. out_strategy_matmul (tuple): The out strategy for the matmul. Should be the same shape as the inputs. """ self.matmul.shard(in_strategy=strategy_matmul, out_strategy=out_strategy_matmul) if self.has_bias: self.bias_add.shard(strategy_bias) if self.activation_flag and isinstance(self.act_name, str): # some operations has many primitives, need to manually set the shard if self.act_name.lower() == "leakyrelu": self.activation.select_op.shard((strategy_activation[0], strategy_activation[0])) elif self.act_name.lower() == "logsigmoid": self.activation.mul.shard((strategy_activation[0], ())) self.activation.exp.shard(strategy_activation) self.activation.add.shard((strategy_activation[0], ())) self.activation.rec.shard(strategy_activation) self.activation.log.shard(strategy_activation) elif self.act_name.lower() == "logsoftmax": raise ValueError("The 'LogSoftmax' function is not supported in semi auto parallel " "or auto parallel mode.") else: getattr(self.activation, self.act_name).shard(strategy_activation) elif self.activation_flag and isinstance(self.activation, Cell): if hasattr(self.activation, 'activation_shard') and strategy_activation: shard_tuple = strategy_activation[0] if len(shard_tuple) == 2: parallel_config = OpParallelConfig(data_parallel=shard_tuple[0], model_parallel=shard_tuple[1]) elif len(shard_tuple) == 4: parallel_config = MoEParallelConfig(data_parallel=shard_tuple[0], expert_parallel=shard_tuple[1], model_parallel=shard_tuple[2]) else: raise ValueError("The user-defined activation function currently only supports the case where the " "input policy is 2 or 4, so that relevant policies can be extracted from it." "To avoid this error, you need to add the function of extracting " "'ParallelConfig' or 'OpParallelConfig' for the incoming strategy_activation ") self.activation.activation_shard(parallel_config) else: logger.warning("The user passed the custom defined activation function %s. " "If the user want to enable shard for the activation cell, " "the user should set the shard for each primitives in the cell.", self.activation_flag) return self class FixedSparseAttention(nn.Cell): """ Fixed Sparse Attention Layer. This function contains the sparse attention primitives used in Sparse Transformers (see paper) `Generating Long Sequences with Sparse Transformers `_. Specifically, it includes the following: 1. A faster implementation of normal attention (the upper triangle is not computed, and many operations are fused). 2. An implementation of "strided" and "fixed" attention, as in the Sparse Transformers paper. Args: batch_size(int): Number of input batch size. num_heads(int): Number of attention heads. size_per_head(int): An integer determining embedding size of each attention head, only supports 64, 128 for now. block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines the size of such blocks, Block X Block. Only supports 64 for now. seq_length(int): length of input sequence, only supports 1024 for now. Default 1024. num_different_global_patterns(int): An integer determining the number of different global attentions layouts. While global attention can be fixed by which block/s are representative of any local window, since there are multi-heads, each head can use a different global representative, only supports 4 for now. Default 4. parallel_config(OpParallelConfig): The config of parallel setting, see `OpParallelConfig`. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **q** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, seq_length, hidden_size]): Sequence of queries to query the context. - **k** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, seq_length, hidden_size]): Sequence of queries to query the context. - **v** (Tensor) - Tensor value (:class:`mstype.fp16` [batch size, sequence length, Embedding Size]): Sequence of queries to query the context. - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp32`, :class:`mstype.fp16` [batch_size, seq_length, seq_length]): Lower triangular matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, seq_length, hidden_size] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules import FixedSparseAttention >>> from mindspore import Tensor >>> model = FixedSparseAttention(batch_size=2, ... num_heads=8, ... size_per_head=64, ... block_size=64) >>> q = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> k = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> v = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1024, 1024)), mstype.float32) >>> output = model(q, k, v, attention_mask) >>> print(output.shape) (2, 1024, 512) """ @_args_type_validator_check(batch_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, size_per_head=Validator.check_positive_int, block_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, num_different_global_patterns=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "FixedSparseAttention")) def __init__(self, batch_size, num_heads, size_per_head, block_size, seq_length=1024, num_different_global_patterns=4, parallel_config=default_dpmp_config): super(FixedSparseAttention, self).__init__() dp, mp = parallel_config.data_parallel, parallel_config.model_parallel if num_heads % mp != 0: raise ValueError(f"The number of heads {num_heads} must be a " f"multiple of parallel_config.model_parallel {mp}.") if batch_size % dp != 0: raise ValueError(f"The batch_size {batch_size} must be a " f"multiple of parallel_config.data_parallel {parallel_config.data_parallel}.") self.seq_length = seq_length self.batch_size = batch_size self.hidden_size = size_per_head * num_heads self.num_heads = num_heads self.block_size = block_size self.block_num = seq_length // block_size self.size_per_head = size_per_head self.global_size = seq_length // 4 self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, mp, 1),)) self.batch_matmul = P.BatchMatMul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.multiply = P.Mul().shard(((dp, 1, 1, 1), (1, 1, 1))) self.multiply_data = Tensor([-10000.0], dtype=mstype.float32) self.parallel_config = parallel_config size_per_head_list = [64, 128] if self.seq_length != 1024: raise ValueError("For 'FixedSparseAttention', the class variable 'seq_length' must be 1024, " "but got the value : {}.".format(seq_length)) if self.block_size != 64: raise ValueError("For 'FixedSparseAttention', the class variable 'block_size' must be 64, " "but got the value : {}.".format(block_size)) if num_different_global_patterns != 4: raise ValueError("For 'FixedSparseAttention', the class variable 'num_different_global_patterns' " "must be 4, but got the value : {}".format(num_different_global_patterns)) if self.size_per_head not in size_per_head_list: raise ValueError("For 'FixedSparseAttention', the class variable 'size_per_head' only supports {}, " "but got the value : {}.".format(size_per_head_list, self.size_per_head)) local_ones = np.ones((self.block_size, self.block_size), dtype=np.float16) global_mask_original = np.ones((self.seq_length, self.global_size), dtype=np.float16) for i in range(self.seq_length): for j in range(self.global_size): if i // 16 >= (j // 16 + 1) * 4: global_mask_original[i, j] = 0.0 global_mask_original = -10000 * global_mask_original global_mask_fx = global_mask_original.reshape((self.seq_length // 16, 16, self.global_size // 16, 16)) global_mask = np.transpose(global_mask_fx, (2, 0, 1, 3)) global_mask = np.repeat(global_mask[np.newaxis, :, :, :, :], self.batch_size, axis=0) global_mask = global_mask.reshape((self.batch_size * self.global_size // 16, self.seq_length // 16, 16, 16)) self.global_mask = Tensor(global_mask, mstype.float32) self.local_mask_triangle = Tensor(np.tril(local_ones), mstype.float32) self.scale_factor = Tensor((math.sqrt(self.size_per_head))) self.matmul_dds = P.MatmulDDS(self.batch_size, self.num_heads).shard(((mp, dp, 1, 1), (mp, dp, 1, 1), (1, dp, 1, 1), (dp, 1, 1, 1))) self.matmul_dsd = P.DSDMatmul().shard(((dp, mp, 1, 1, 1, 1, 1), (dp, mp, 1, 1, 1, 1, 1), (dp, mp, 1, 1))) self.sub1 = P.Sub().shard(((1,), (dp, 1, 1, 1))) self.mul1 = P.Mul().shard(((dp, 1, 1, 1), (1,))) self.transpose1 = P.Transpose().shard(((dp, 1, 1, 1),)) self.transpose2 = P.Transpose().shard(((dp, 1, 1, 1),)) self.transpose3 = P.Transpose().shard(((dp, mp, 1, 1, 1, 1),)) self.transpose4 = P.Transpose().shard(((dp, mp, 1, 1),)) self.div = P.RealDiv().shard(((mp, dp, 1, 1), ())) self.slice1 = P.StridedSlice().shard(((dp, 1, 1),)) def construct(self, q, k, v, attention_mask): """Forward process""" _check_shape_equal(F.shape(q), "q", self.cls_name, [self.batch_size, self.seq_length, self.hidden_size]) _check_input_dtype(F.dtype(q), "q", [mstype.float16], self.cls_name) _check_shape_equal(F.shape(k), "k", self.cls_name, [self.batch_size, self.seq_length, self.hidden_size]) _check_input_dtype(F.dtype(k), "k", [mstype.float16], self.cls_name) _check_shape_equal(F.shape(v), "v", self.cls_name, [self.batch_size, self.seq_length, self.hidden_size]) _check_input_dtype(F.dtype(v), "v", [mstype.float16], self.cls_name) _check_shape_equal(F.shape(attention_mask), "attention_mask", self.cls_name, [self.batch_size, self.seq_length, self.seq_length]) _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) q, k, v = self._transpose_inputs(q, k, v) local_mask, global_mask = self._generate_attention_mask(attention_mask) q = self.div(q, F.cast(self.scale_factor, F.dtype(q))) k = self.div(k, F.cast(self.scale_factor, F.dtype(k))) local_prob, global_prob = self.matmul_dds(q, k, local_mask, global_mask) attention = self.matmul_dsd(local_prob, global_prob, v) attention_merge = self.transpose3(attention, (0, 1, 3, 4, 2, 5)) attention_merge = F.reshape( attention_merge, (-1, self.num_heads, self.seq_length, self.size_per_head)) attention_merge = self.transpose4(attention_merge, (0, 2, 1, 3)) attention_merge = F.reshape( attention_merge, (-1, self.seq_length, self.size_per_head * self.num_heads)) return attention_merge def _generate_attention_mask(self, attention_mask): """ generate global attention mask and local attention mask from origin attention mask """ attention_mask = self.reshape(attention_mask, (-1, self.seq_length, self.seq_length)) input_mask = self.slice1(attention_mask, (0, self.seq_length - 1, 0), (self.batch_size, self.seq_length, self.seq_length), (1, 1, 1)) input_mask = self.reshape(input_mask, (-1, self.seq_length)) input_shape = P.Shape()(input_mask) # bs, seq_length # bs, block_num, 1, block_size local_shape_right = (input_shape[0], self.block_num, 1, self.block_size) # bs, block_num, block_size, 1 local_shape_left = (input_shape[0], self.block_num, self.block_size, 1) local_mask_left = self.reshape(input_mask, local_shape_left) local_mask_right = self.reshape(input_mask, local_shape_right) # bs, block_num, block_size, block_size local_attention_mask = self.batch_matmul(local_mask_left, local_mask_right) lower_triangle = P.ExpandDims()(self.local_mask_triangle, 0) local_attention_mask = self.multiply(local_attention_mask, lower_triangle) local_multiplied_out = self.sub1(P.Cast()(F.tuple_to_array((1.0,)), mstype.float32), P.Cast()(local_attention_mask, mstype.float32)) local_adder = self.mul1(local_multiplied_out, self.multiply_data) local_mask_original = self.transpose1(local_adder, (0, 2, 1, 3)) local_mask_original = self.reshape( local_mask_original, (self.batch_size * self.block_size, self.block_num * self.block_size)) local_mask_fx = self.reshape( local_mask_original, (self.batch_size * self.block_size // 16, 16, self.block_num * self.block_size // 16, 16)) local_mask = self.transpose2(local_mask_fx, (2, 0, 1, 3)) global_mask = self.global_mask return local_mask, global_mask def _transpose_inputs(self, q, k, v): """ do reshape and transpose to inputs """ q = self.transpose( self.reshape( q, (-1, 16, self.num_heads * self.size_per_head // 16, 16)), (2, 0, 1, 3)) k = self.transpose( self.reshape( k, (-1, 16, self.num_heads * self.size_per_head // 16, 16)), (2, 0, 1, 3)) v = self.transpose( self.reshape( v, (-1, 16, self.num_heads * self.size_per_head // 16, 16)), (0, 2, 3, 1)) return q, k, v class AlibiTensor(nn.Cell): """ Link to paper: https://arxiv.org/abs/2108.12409 Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation: with l being a tensor, and a fixed value `softmax(l+a) = softmax(l)`. Based on https://github.com/ofirpress/attention_with_linear_biases/blob/a35aaca144e0eb6b789dfcb46784c4b8e31b7983/fairseq/models/transformer.py#L742 Args: seq_length(int) - length of sequence num_heads(int) - number of heads Inputs: attention_mask(Tensor) - Token-wise attention mask, this should be of shape (batch_size, max_seq_len). dtype(mstype) - dtype of the output tensor Returns: alibi(Tensor), ailibi tensor shaped (batch_size, num_heads, 1, max_seq_len) """ def __init__(self, seq_length, num_heads, parallel_config=default_dpmp_config): super(AlibiTensor, self).__init__() dp = parallel_config.data_parallel self.seq_length = seq_length self.num_heads = num_heads self.minus_one = Tensor(-np.ones(seq_length), mstype.float32) self.expand_2d = P.ExpandDims().shard(((dp, 1),)) self.expand_3d = P.ExpandDims().shard(((dp, 1, 1),)) self.cumsum = P.CumSum().shard(((dp, 1),)) self.add = P.Add().shard(((dp, 1), (1,))) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.mul_slope = P.Mul().shard(((1, 1), (dp, 1, 1))) # build slopes closest_power_of_2 = 2 ** math.floor(math.log2(num_heads)) base = np.array(2 ** (-(2 ** -(math.log2(closest_power_of_2) - 3))), dtype=np.float32) powers = np.arange(1, 1 + closest_power_of_2, dtype=np.int32) slopes = np.power(base, powers) if closest_power_of_2 != num_heads: extra_base = np.array( 2 ** (-(2 ** -(math.log2(2 * closest_power_of_2) - 3))), dtype=np.float32 ) num_remaining_heads = min(closest_power_of_2, num_heads - closest_power_of_2) extra_powers = np.arange(1, 1 + 2 * num_remaining_heads, 2, dtype=np.int32) slopes = np.concatenate([slopes, np.power(extra_base, extra_powers)], axis=0) self.slopes = Tensor(slopes[:, None], mstype.float32) # (num_heads, 1) def construct(self, attention_mask, dtype): """ Note: alibi will added to the attention bias that will be applied to the query, key product of attention therefore alibi will have to be of shape (batch_size, num_heads, query_length, key_length) """ arange_tensor = self.cumsum(attention_mask, -1) # (batch_size, seq_len) arange_tensor = self.add(arange_tensor, self.minus_one) # (batch_size, seq_len) arange_tensor = self.mul(arange_tensor, attention_mask) # (batch_size, seq_len) arange_tensor = self.expand_2d(arange_tensor, 1) # (batch_size, 1, seq_len) alibi = self.mul_slope(self.slopes, arange_tensor) # (batch_size, num_heads, seq_len) alibi = self.expand_3d(alibi, 2).astype(dtype) # (batch_size, num_heads, 1, seq_len) return alibi class AlibiTensorV2(nn.Cell): """ Link to paper: https://arxiv.org/abs/2108.12409 Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation: with l being a tensor, and a fixed value `softmax(l+a) = softmax(l)`. Based on https://github.com/ofirpress/attention_with_linear_biases/blob/a35aaca144e0eb6b789dfcb46784c4b8e31b7983/fairseq/models/transformer.py#L742 Args: seq_length(int) - length of sequence num_heads(int) - number of heads Inputs: attention_mask(Tensor) - Token-wise attention mask, this should be of shape (batch_size, max_seq_len). dtype(mstype) - dtype of the output tensor Returns: alibi(Tensor), ailibi tensor shaped (batch_size, num_heads, 1, max_seq_len) """ def __init__(self, seq_length, num_heads): super(AlibiTensorV2, self).__init__() self.seq_length = seq_length self.num_heads = num_heads self.minus_one = Tensor(-np.ones(seq_length), mstype.float32) self.expand_2d = P.ExpandDims() self.expand_3d = P.ExpandDims() self.cumsum = P.CumSum() self.add_2d = P.Add() self.add_3d = P.Add() self.mul = P.Mul() self.mul_slope = P.Mul() self.transpose = P.Transpose() self.reshape = P.Reshape() self.mul_mask = P.Mul() # build slopes closest_power_of_2 = 2 ** math.floor(math.log2(num_heads)) base = np.array(2 ** (-(2 ** -(math.log2(closest_power_of_2) - 3))), dtype=np.float32) powers = np.arange(1, 1 + closest_power_of_2, dtype=np.int32) slopes = np.power(base, powers) if closest_power_of_2 != num_heads: extra_base = np.array( 2 ** (-(2 ** -(math.log2(2 * closest_power_of_2) - 3))), dtype=np.float32 ) num_remaining_heads = min(closest_power_of_2, num_heads - closest_power_of_2) extra_powers = np.arange(1, 1 + 2 * num_remaining_heads, 2, dtype=np.int32) slopes = np.concatenate([slopes, np.power(extra_base, extra_powers)], axis=0) self.slopes = Tensor(slopes[None, :, None, None], mstype.float32) # (num_heads, 1) def construct(self, attention_mask, dtype=mstype.float32): """ Note: alibi will added to the attention bias that will be applied to the query, key product of attention therefore alibi will have to be of shape (batch_size, num_heads, query_length, key_length) """ bs, seqlen = attention_mask.shape arange_tensor = self.cumsum(attention_mask, -1) # (batch_size, seq_len) max_pos = -arange_tensor[:, -1:] # (batch_size, 1) arange_tensor = self.add_2d(arange_tensor, max_pos) # (batch_size, seq_len) arange_tensor = self.expand_2d(arange_tensor, 1) # (batch_size, 1, seq_len) diag = -self.transpose(arange_tensor, (0, 2, 1)) # (batch_size, seq_len, 1) arange_tensor = self.add_3d(arange_tensor, diag) # (batch_size, seq_len, seq_len) arange_tensor = self.expand_3d(arange_tensor, 1) # (batch_size, 1, seq_len, seq_len) alibi = self.mul_slope(self.slopes, arange_tensor) # (batch_size, num_heads, seq_len, seq_len) alibi_mask = self.mul_mask(alibi, self.reshape(attention_mask, (bs, 1, seqlen, 1))) # (batch_size, num_heads, seq_len, seq_len) alibi_mask = self.mul_mask(alibi_mask, self.reshape(attention_mask, (bs, 1, 1, seqlen))) # (batch_size, num_heads, seq_len, seq_len) return alibi_mask.astype(dtype) def shard(self, parallel_config): """Parallel strategy configuratiuon interface.""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.expand_2d.shard(((dp, 1),)) self.expand_3d.shard(((dp, 1, 1),)) self.cumsum.shard(((dp, 1),)) self.add_2d.shard(((dp, 1), (dp, 1))) self.add_3d.shard(((dp, 1, 1), (dp, 1, 1))) self.mul.shard(((dp, 1), (dp, 1))) self.mul_slope.shard(((1, 1, 1, 1), (dp, 1, 1, 1))) self.transpose.shard(((dp, 1, 1),)) self.mul_mask.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def _get_interleave(n): """calculate slopes of alibi tensor""" def _get_interleave_power_of_2(n): start = (2 ** (-2 ** -(math.log2(n) - 3))) ratio = start return [start * ratio ** i for i in range(n)] if math.log2(n).is_integer(): return _get_interleave_power_of_2(n) closest_power_of_2 = 2 ** math.floor(math.log2(n)) return _get_interleave_power_of_2(closest_power_of_2) + \ _get_interleave(2 * closest_power_of_2)[0::2][:n - closest_power_of_2] def build_alibi_tensor_v2(seq_len, num_heads, return_tensors='ms', dtype=mstype.float32): """build alibi tensor""" assert return_tensors in ['np', 'ms'],\ f"return tensors must be 'np' or 'ms', {return_tensors} not support." slopes = _get_interleave(num_heads) slopes = np.expand_dims(np.expand_dims(slopes, 1), 1) position_point = np.arange(seq_len) - seq_len + 1 position_point = np.expand_dims(np.expand_dims(position_point, 0), 0) position_point = np.tile(position_point, (num_heads, seq_len, 1)) diag = np.diag(position_point[0]) diag = np.expand_dims(np.expand_dims(diag, 0), 0) diag = np.transpose(diag, (0, 2, 1)) position_point = position_point - diag alibi = slopes * position_point alibi = np.expand_dims(alibi, 0) if return_tensors == 'np': return alibi return Tensor(alibi, dtype=dtype) pengcheng.mind-test/mindformers/modules/local_block_sparse_attention.py000066400000000000000000000341161471400134000271730ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ A Local Block Sparse Attention. """ from __future__ import absolute_import from functools import wraps, partial import inspect import math import numpy as np import mindspore as ms import mindspore.nn as nn import mindspore.ops as P from mindspore import Tensor from mindspore import dtype as mstype from mindspore._extends import cell_attr_register try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, OpParallelConfig __all__ = ["LocalBlockSparseAttention"] gather_index_inited = False kv_index = None mask_index = None def _args_type_validator_check(*type_args, **type_kwargs): """Check whether input data type is correct.""" def type_check(func): sig = inspect.signature(func) bound_types = sig.bind_partial(*type_args, **type_kwargs).arguments @wraps(func) def wrapper(*args, **kwargs): nonlocal bound_types bound_values = sig.bind(*args, **kwargs) argument_dict = bound_values.arguments if "kwargs" in bound_types: bound_types = bound_types["kwargs"] if "kwargs" in argument_dict: argument_dict = argument_dict["kwargs"] for name, value in argument_dict.items(): if name in bound_types: bound_types[name](value, name) return func(*args, **kwargs) return wrapper return type_check def _valid_type_checks(types, class_name): """types should be a list of types, this function check if the type is in the valid dtypes""" def validator_check_func(value, name): # The args of Validator.check_type_name is (arg_name, arg_type, valid_types, prim_name) # as the input of _args_type_validator_check is fixed, so we need to manually change the input order partial_check = partial(Validator.check_type_name, valid_types=types, prim_name=class_name) return partial_check(name, type(value)) return validator_check_func def _valid_value_checks(types, class_name): """the value should be a list of types, this function check if the value is in the valid dtypes""" def validator_check_func(value, name): # The args of Validator.check_type_name is (arg_name, arg_type, valid_types, prim_name) # as the input of _args_type_validator_check is fixed, so we need to manually change the input order partial_check = partial(Validator.check_type_name, valid_types=types, prim_name=class_name) return partial_check(name, value) return validator_check_func class InitGatherIndex: """ A self-defined function to init the index of gather operation Args: seq_len(int): Length of input sequence. local_size(int): An integer determining the local size. Current implementation of sparse self-attention is based on local sparse matrices, reference sliding window attention in Longformer block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices, reference block sparse attention in FlashAttention """ def __init__(self, seq_len, local_size, block_size): self.seq_len = seq_len self.local_size = local_size self.block_size = block_size self.group_num = self.seq_len // self.block_size self.local_block_num = self.local_size // self.block_size self.prev_block_num = self.local_block_num - 1 def gen_kv_gather_index(self): index = np.ones((self.group_num * self.local_size)) for i in range(self.group_num): start = max(0, i - self.prev_block_num) * self.block_size end = (max(0, i - self.prev_block_num) + self.local_size // self.block_size) * self.block_size index[i * self.local_size:(i + 1) * self.local_size] = np.arange( start, end) return index def gen_mask_gatherd_index(self): index = np.ones((1, self.seq_len, self.local_size)) for i in range(self.group_num): start = max(0, i - self.prev_block_num) * self.block_size end = (max(0, i - self.prev_block_num) + self.local_block_num) * self.block_size index[:, i * self.block_size:(i + 1) * self.block_size, :] = np.arange(start, end) return index def init_gather_index(seq_len, local_size, block_size): gather_index = InitGatherIndex(seq_len, local_size, block_size) global kv_index kv_index = Tensor(gather_index.gen_kv_gather_index(), mstype.int32) global mask_index mask_index = Tensor(gather_index.gen_mask_gatherd_index(), mstype.int32) class LocalBlockSparseAttention(nn.Cell): """ Local Block Sparse Attention Layer. This function contains the sliding window attention primitives used in Longformer (see paper) `Longformer: The Long-Document Transformer ` and block sparse attention primitives used in Longformer FlashAttention (see paper) `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ` Specifically, it includes the following: 1. A faster implementation of normal attention (the entire area of the upper triangle and part of the area of the lower triangle are not computed, and many operations are fused). 2. An implementation of "local" and "block" sparse attention, as in the Longformer and FlashAttention. Args: seq_len(int): Length of input sequence. size_per_head(int): The hidden size of the input. local_size(int): An integer determining the local size. Current implementation of sparse self-attention is based on local sparse matrices, reference sliding window attention in Longformer. block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices, reference sliding window attention in FlashAttention. dropout_rate(float): The dropout rate of the attention score. Default 0.1. softmax_compute_type(mstype.Number): The type of softmax computation module. Default mstype.float16. Should be mstype.float32 or mstype.float16. parallel_config(OpParallelConfig): The config of parallel setting, see `OpParallelConfig`. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **q** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **k** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **v** (Tensor) - Tensor value (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp16` [batch_size, seq_length, seq_length]): Lower triangular matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, head_num, seq_length, size_per_head] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules import LocalBlockSparseAttention >>> from mindspore import Tensor >>> model = LocalBlockSparseAttention(seq_len=4096, ... size_per_head=128, ... local_size=1024, ... block_size=128) >>> q = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> k = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> v = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 4096, 4096)), mstype.float16) >>> output = model(q, k, v, attention_mask) >>> print(output.shape) (2, 16, 4096, 128) """ @cell_attr_register @_args_type_validator_check( seq_len=Validator.check_positive_int, size_per_head=Validator.check_positive_int, local_size=Validator.check_positive_int, block_size=Validator.check_positive_int, softmax_compute_type=_valid_value_checks( [mstype.float16, mstype.float32], "LocalBlockSparseAttention"), parallel_config=_valid_type_checks([OpParallelConfig], "LocalBlockSparseAttention")) def __init__(self, seq_len, size_per_head, local_size, block_size, dropout_rate=0.1, softmax_compute_type=mstype.float16, parallel_config=default_dpmp_config): super(LocalBlockSparseAttention, self).__init__() if seq_len % block_size != 0: raise ValueError( f"block_size must be divisible by seq_len, " f"but got block_size {block_size} is not divisible by seq_len {seq_len}" ) if local_size % block_size != 0: raise ValueError( f"block_size must be divisible by local_size, " f"but got block_size {block_size} is not divisible by local_size {local_size}" ) if local_size >= seq_len: raise ValueError( f"local_size must be less than seq_len, " f"but got local_size {local_size} is not less than seq_len {seq_len}" ) if block_size > local_size: raise ValueError( f"block_size must be less than (or equal to) local_size, " f"but got block_size {local_size} is not less than (or equal to) local_size {local_size}" ) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError( f"dropout probability should be a number in range [0, 1), but got {dropout_rate}" ) global gather_index_inited if not gather_index_inited: init_gather_index(seq_len, local_size, block_size) gather_index_inited = True self.scale_factor = Tensor(math.sqrt(math.sqrt(size_per_head))) self.local_size = local_size self.block_size = block_size self.dropout_rate = dropout_rate self.softmax_compute_type = softmax_compute_type dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.real_div = P.RealDiv().shard(((dp, mp, 1, 1), ())) self.qk_bmm = ms.ops.BatchMatMul(transpose_b=True).shard( ((dp, mp, 1, 1, 1), (dp, mp, 1, 1, 1))) self.attn_mask_mul = P.Mul().shard(((dp, 1, 1), (1,))) self.attn_mask_expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.attn_mask_add = P.Add().shard( ((dp, 1, 1, 1, 1), (dp, mp, 1, 1, 1))) self.softmax = P.Softmax().shard(((dp, mp, 1, 1, 1),)) if self.dropout_rate > 1e-5: self.attention_dropout = nn.Dropout(keep_prob=1 - self.dropout_rate) self.attention_dropout.dropout.shard(((dp, mp, 1, 1, 1),)) self.pv_bmm = ms.ops.BatchMatMul().shard( ((dp, mp, 1, 1, 1), (dp, mp, 1, 1, 1))) self.multiply_data = Tensor([-10000.0], dtype=self.softmax_compute_type) self.reshape = P.Reshape() self.gather1 = P.Gather().shard(((dp, mp, 1, 1), (1,))) self.gather2 = P.GatherD().shard(((dp, 1, 1), (dp, 1, 1))) def construct(self, q, k, v, attention_mask=None): """Forward process""" bsz, head_num, seq_len, head_dim = q.shape factor = P.Cast()(self.scale_factor, P.DType()(q)) q = self.real_div(q, factor) k = self.real_div(k, factor) q = self.reshape(q, (bsz, head_num, seq_len // self.block_size, self.block_size, head_dim)) # kv: [bsz, head_num, seq_len, head_dim] -> [bsz, head_num, seq_len // block_size, local_size, head_dim] k = self.gather1(k, kv_index, 2) k = self.reshape(k, (bsz, head_num, -1, self.local_size, head_dim)) v = self.gather1(v, kv_index, 2) v = self.reshape(v, (bsz, head_num, -1, self.local_size, head_dim)) # q * k.T score = self.qk_bmm(q, k) ori_dtype = P.DType()(score) score = P.Cast()(score, self.softmax_compute_type) # softmax if attention_mask is not None: mask_index_ = P.tile(mask_index, (bsz, 1, 1)) attention_mask = self.gather2( attention_mask, 2, mask_index_) # [bsz, seq_len, local_size] adder = self.attn_mask_mul( P.Cast()(attention_mask, self.softmax_compute_type), self.multiply_data) adder = self.attn_mask_expand_dims(adder, 1) adder = self.reshape( adder, (bsz, 1, -1, self.block_size, self.local_size)) score = self.attn_mask_add(adder, score) probs = self.softmax(score) probs = P.Cast()(probs, ori_dtype) if self.dropout_rate > 1e-5: probs = self.attention_dropout(probs) # p * v output = self.pv_bmm(probs, v) output = self.reshape(output, (bsz, head_num, seq_len, head_dim)) return output pengcheng.mind-test/mindformers/modules/transformer/000077500000000000000000000000001471400134000232505ustar00rootroot00000000000000pengcheng.mind-test/mindformers/modules/transformer/.ipynb_checkpoints/000077500000000000000000000000001471400134000270415ustar00rootroot00000000000000pengcheng.mind-test/mindformers/modules/transformer/.ipynb_checkpoints/transformer-checkpoint.py000066400000000000000000005737651471400134000341310ustar00rootroot00000000000000# Copyright 2021-2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Transformer Networks. This is interface that is subject to change or deletion. """ from __future__ import absolute_import import math import numpy as np from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore.common.initializer import initializer from mindspore import nn from mindspore import context import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.nn.cell import Cell try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import log as logger from mindspore.parallel._utils import _get_parallel_mode from mindspore.context import ParallelMode from mindformers.modules.layers import LayerNorm, Linear, \ _args_type_validator_check, _valid_type_checks, _valid_value_checks, \ _check_past_none_input_none, _check_input_dtype from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, _PipeLineConfig, OpParallelConfig, \ _Config, _check_config, MoEParallelConfig from mindformers.modules.transformer.moe import default_moe_config, MoE, _check_moe_config from mindformers.version_control import get_dropout from mindformers.tools.logger import _LogActionOnce __all__ = [ "AttentionMask", "AttentionMaskHF", "VocabEmbedding", "MultiHeadAttention", "FeedForward", "TransformerEncoder", "TransformerDecoder", "TransformerEncoderLayer", "TransformerDecoderLayer", "Transformer", "TransformerOpParallelConfig", "EmbeddingOpParallelConfig", "TransformerRecomputeConfig"] class EmbeddingOpParallelConfig(_Config): r""" The parallel config of :class:`VocabEmbedding` for the setting data parallel or model parallel for the embedding table. Args: data_parallel(int): The data parallel way. The input data will be sliced into n parts for embedding layer according to this value. Default: 1. model_parallel(int): The model parallel way. The embedding table parameters will be sliced at 0-th axis according to the model parallel way. Default: 1. vocab_emb_dp(bool): Shard embedding in model parallel or data parallel. If True, the embedding lookup will be a data parallel style training and model_parallel value will be ignored. If false, the embedding table will be sharded into n parts at the 0-th dimension row slice of the embedding table, where the n is the model parallel way determined by this parameter. Default: True Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import EmbeddingOpParallelConfig >>> config=EmbeddingOpParallelConfig(data_parallel=1, model_parallel=1, vocab_emb_dp=True) """ def __init__(self, data_parallel=1, model_parallel=1, use_seq_parallel=False, select_recompute=False, vocab_emb_dp=True): self._dp_mp_config = OpParallelConfig(data_parallel=data_parallel, use_seq_parallel=use_seq_parallel, model_parallel=model_parallel, select_recompute=select_recompute) Validator.check_bool(vocab_emb_dp, "vocab_emb_dp") self.vocab_emb_dp = vocab_emb_dp self.use_seq_parallel = use_seq_parallel self.select_recompute = select_recompute @property def data_parallel(self): return self._dp_mp_config.data_parallel @data_parallel.setter def data_parallel(self, value): self._dp_mp_config.data_parallel = value @property def model_parallel(self): return self._dp_mp_config.model_parallel @model_parallel.setter def model_parallel(self, value): self._dp_mp_config.model_parallel = value @property def vocab_emb_dp(self): return self._vocab_emb_dp @vocab_emb_dp.setter def vocab_emb_dp(self, value): Validator.check_bool(value, "vocab_emb_dp") self._vocab_emb_dp = value @property def dp_mp_config(self): return self._dp_mp_config class TransformerRecomputeConfig(_Config): r""" TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) """ def __init__(self, recompute=False, select_recompute=False, parallel_optimizer_comm_recompute=False, mp_comm_recompute=True, recompute_slice_activation=False): Validator.check_bool(recompute, "recompute") Validator.check_bool(parallel_optimizer_comm_recompute, "parallel_optimizer_comm_recompute") Validator.check_bool(mp_comm_recompute, "mp_comm_recompute") Validator.check_bool(select_recompute, "select_recompute") Validator.check_bool(recompute_slice_activation, "recompute_slice_activation") self._recompute = recompute self._select_recompute = select_recompute self._parallel_optimizer_comm_recompute = parallel_optimizer_comm_recompute self._mp_comm_recompute = mp_comm_recompute self._recompute_slice_activation = recompute_slice_activation @property def recompute(self): return self._recompute @recompute.setter def recompute(self, value): Validator.check_bool(value, "recompute") self._recompute = value @property def select_recompute(self): return self._select_recompute @select_recompute.setter def select_recompute(self, value): Validator.check_bool(value, "select_recompute") self._select_recompute = value @property def parallel_optimizer_comm_recompute(self): return self._parallel_optimizer_comm_recompute @parallel_optimizer_comm_recompute.setter def parallel_optimizer_comm_recompute(self, value): Validator.check_bool(value, "parallel_optimizer_comm_recompute") self._parallel_optimizer_comm_recompute = value @property def mp_comm_recompute(self): return self._mp_comm_recompute @mp_comm_recompute.setter def mp_comm_recompute(self, value): Validator.check_bool(value, "mp_comm_recompute") self._mp_comm_recompute = value @property def recompute_slice_activation(self): return self._recompute_slice_activation @recompute_slice_activation.setter def recompute_slice_activation(self, value): Validator.check_bool(value, "recompute_slice_activation") self._recompute_slice_activation = value default_transformer_recompute_config = TransformerRecomputeConfig() class TransformerOpParallelConfig(_Config): r""" TransformerOpParallelConfig for setting parallel configuration, such as the data parallel and model parallel. Note: Except the recompute argument, other arguments will **not** be effective when the user doesn't set auto_parallel_context to `SEMI_AUTO_PARALLEL` or `AUTO_PARALLEL`. The micro_batch_num must be greater than or equal to pipeline_stage when training. The data_parallel\*model_parallel \*pipeline_stage must be equal or less equal to the device. When setting the pipeline stage and optimizer_shard, the config will overwrite the auto_parallel_context. When given the 8 devices and the data_parallel is 1 and model_parallel is 1, the calculation will be repeated on each device. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. recompute (Union[TransformerRecomputeConfig, bool]): The configuration of recomputation for the transformer block. Default: An instance of TransformerRecomputeConfig with default values. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> recompute_config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) >>> config=TransformerOpParallelConfig(data_parallel=1, model_parallel=1, recompute=recompute_config) """ def __init__(self, data_parallel=1, model_parallel=1, expert_parallel=1, pipeline_stage=1, micro_batch_num=1, recompute=default_transformer_recompute_config, use_seq_parallel=False, optimizer_shard=False, gradient_aggregation_group=4, vocab_emb_dp=True): self.recompute = recompute self.select_recompute = recompute.select_recompute self.use_seq_parallel = use_seq_parallel self.optimizer_shard = optimizer_shard self.gradient_aggregation_group = gradient_aggregation_group self._embed_dp_mp_config = EmbeddingOpParallelConfig( data_parallel=data_parallel, model_parallel=model_parallel, vocab_emb_dp=vocab_emb_dp, use_seq_parallel=use_seq_parallel, select_recompute=recompute.select_recompute) self._pp_config = _PipeLineConfig(pipeline_stage=pipeline_stage, micro_batch_num=micro_batch_num) self._moe_config = MoEParallelConfig( data_parallel=data_parallel, model_parallel=model_parallel, select_recompute=recompute.select_recompute, expert_parallel=expert_parallel, use_seq_parallel=use_seq_parallel) @property def recompute(self): return self._recompute @recompute.setter def recompute(self, value): if not isinstance(value, TransformerRecomputeConfig) and not isinstance(value, bool): raise TypeError(f"recompute must be a TransformerRecomputeConfig/bool, but got {type(value).__name__}.") if isinstance(value, bool): logger.warning(f"TransformerRecomputeConfig is recommended as the recompute configuration type.") self._recompute = value @property def vocab_emb_dp(self): return self._embed_dp_mp_config.vocab_emb_dp @vocab_emb_dp.setter def vocab_emb_dp(self, value): self._embed_dp_mp_config.vocab_emb_dp = value @property def gradient_aggregation_group(self): return self._gradient_aggregation_group @gradient_aggregation_group.setter def gradient_aggregation_group(self, value): Validator.check_positive_int(value, "gradient_aggregation_group") self._gradient_aggregation_group = value @property def micro_batch_num(self): return self._pp_config.micro_batch_num @micro_batch_num.setter def micro_batch_num(self, value): self._pp_config.micro_batch_num = value @property def model_parallel(self): return self._embed_dp_mp_config.model_parallel @model_parallel.setter def model_parallel(self, value): self._embed_dp_mp_config.model_parallel = value self._moe_config.model_parallel = value @property def data_parallel(self): return self._embed_dp_mp_config.data_parallel @data_parallel.setter def data_parallel(self, value): self._embed_dp_mp_config.data_parallel = value self._moe_config.data_parallel = value @property def expert_parallel(self): return self._moe_config.expert_parallel @expert_parallel.setter def expert_parallel(self, value): self._moe_config.expert_parallel = value @property def pipeline_stage(self): return self._pp_config.pipeline_stage @pipeline_stage.setter def pipeline_stage(self, value): self._pp_config.pipeline_stage = value @property def optimizer_shard(self): return self._optimizer_shard @optimizer_shard.setter def optimizer_shard(self, value): Validator.check_bool(value, "optimizer_shard") self._optimizer_shard = value context.set_auto_parallel_context(enable_parallel_optimizer=value) @property def embedding_dp_mp_config(self): return self._embed_dp_mp_config @property def dp_mp_config(self): return self._embed_dp_mp_config.dp_mp_config @property def moe_parallel_config(self): return self._moe_config default_transformer_config = TransformerOpParallelConfig() default_embedding_parallel_config = EmbeddingOpParallelConfig() class FeedForward(Cell): r""" The multilayer perceptron with two linear layers with dropout applied at final output. The first linear will project the input dimension from hidden_size to ffn_hidden_size. The second linear will project the dimension from ffn_hidden_size to hidden_size. The first linear is sharded on the relative dimension, and the second linear is sharded on the output dimension. The overview process can be: .. math:: Dropout((xW_1+b_1)W_2 + b_2) where the :math:`W_1, W_2, b_1` and :math:`b_2` are trainable parameters. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see examples. Default: gelu. expert_num (int): The number of experts used in Linear. For the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. param_init_type (dtype.Number): The parameter initialization type. Should be mstype.float32 or mstype.float16. Default: mstype.float32. parallel_config (OpParallelConfig, MoEParallelConfig): The config of parallel setting, see `OpParallelConfig` or `MoEParallelConfig`. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: TypeError: `hidden_act` is not a string or nn.Cell. TypeError: `parallel_config` is not a subclass of OpParallelConfig. ValueError: `ffn_hidden_size` is not a multiple of the model parallel way. ValueError: `hidden_size` is not a multiple of the model parallel way. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import FeedForward >>> from mindspore import dtype as mstype >>> from mindspore import Tensor, nn >>> import mindspore.ops as ops >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 2 using custom hidden activation >>> class MyActivationNoShard(nn.Cell): ... def __init__(self): ... super(MyActivationNoShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationNoShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 3 using custom hidden activation with activation_shard >>> # If user wantss to run on the SEMI/AUTO parallel mode, the custom activation must provide >>> # a class function named activation_shard. It accepts the argument parallel_config (OpParallelConfig, >>> # MoEParallelConfig) and set the shard for the primitives used in the construct. >>> class MyActivationWithShard(nn.Cell): ... def __init__(self): ... super(MyActivationWithShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) ... def activation_shard(self, parallel_config): ... self.add.shard(((parallel_config.data_parallel, parallel_config.model_parallel), ())) >>> >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationWithShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) """ @_LogActionOnce(m_logger=logger, key='FeedForward', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, dropout_rate=Validator.check_non_negative_float, param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "FeedForward")) def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act='gelu', expert_num=1, expert_group_size=None, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(FeedForward, self).__init__() if hidden_act is None or not (isinstance(hidden_act, str) or issubclass(hidden_act, nn.Cell)): raise TypeError(f"For FeedForward cell, the hidden_act should str type or nn.Cell type, " f"but got {hidden_act}.") if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = parallel_config.data_parallel // ep if ffn_hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'FeedForward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1))) self.projection.bias.parallel_optimizer = False self.dropout = get_dropout(dropout_rate) self.dropout_3d = get_dropout(dropout_rate) self.dropout_4d = get_dropout(dropout_rate) self.cast = P.Cast() else: _check_config(parallel_config) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = parallel_config.data_parallel // ep if ffn_hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'FeedForward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.mapping.shard(strategy_matmul=((dp, ep, 1, 1), (ep, 1, mp)), strategy_bias=((dp, ep, 1, mp), (1, ep, 1, mp)), strategy_activation=((dp, ep, 1, mp),)) else: self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1)), strategy_bias=((dp, ep, 1, 1), (1, ep, 1, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = get_dropout(dropout_rate) self.dropout_3d = get_dropout(dropout_rate) self.dropout_4d = get_dropout(dropout_rate) self.dropout.dropout.shard(((dp, 1),)) self.dropout_3d.dropout.shard(((dp, 1, 1),)) self.dropout_4d.dropout.shard(((dp, ep, 1, 1),)) self.cast = P.Cast() def construct(self, x): """Forward process of the FeedForward""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) else: output = self.dropout_4d(output) return output class AttentionMask(Cell): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length) with 1 and 0, where 1 indicates the current position is a valid token, otherwise not. Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMask >>> from mindspore import Tensor >>> mask = AttentionMask(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [0. 0. 0. 0]]] """ @_LogActionOnce(m_logger=logger, key='AttentionMask', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(seq_length=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "AttentionMask")) def __init__(self, seq_length, parallel_config=default_dpmp_config): super(AttentionMask, self).__init__() self.seq_length = seq_length self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.reshape = P.Reshape() self.mul = P.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.expand_dim = P.ExpandDims().shard(((1, 1),)) ones = np.ones(shape=(seq_length, seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (1, 1, 1))) def construct(self, input_mask): """Forward process of the AttentionMask""" _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) input_mask = P.Cast()(self.not_equal(input_mask, 0), mstype.float16) input_shape = P.Shape()(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) # Mask the padded inputs mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.mul(mask_left, mask_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask class AttentionMaskHF(Cell): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length). Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMaskHF >>> from mindspore import Tensor >>> mask = AttentionMaskHF(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [1. 1. 1. 1]]] """ @_LogActionOnce(m_logger=logger, key='AttentionMaskHF', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(seq_length=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "AttentionMaskHF")) def __init__(self, seq_length, parallel_config=default_dpmp_config): super(AttentionMaskHF, self).__init__() self.seq_length = seq_length self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.reshape = P.Reshape() self.mul = P.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.expand_dim = P.ExpandDims().shard(((1, 1),)) ones = np.ones(shape=(seq_length, seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (1, 1, 1))) def construct(self, input_mask): """Forward process of the AttentionMaskHF""" _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) input_mask = P.Cast()(P.OnesLike()(input_mask), mstype.float16) input_shape = P.Shape()(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) # Mask the padded inputs mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.mul(mask_left, mask_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask class VocabEmbedding(Cell): """ The embedding lookup table from the 0-th dim of the parameter table. When the parallel_config.vocab_emb_dp is True and in the `AUTO_PARALLEL` mode, the embedding lookup will be trained by the data parallel way, as the parameters will be repeated on each device. If false, the embedding table will be sharded into n parts at the 0-th dimension of the embedding table, where the n is the model parallel way determined by `parallel_config.model_parallel` (EmbeddingOpParallelConfig). Note: When `AUTO_PARALLEL` or `SEMI_AUTO_PARALLEL` mode is enabled, this layer support only 2-d dimension inputs, as the shard is designed for 2d inputs. Args: vocab_size (int): Size of the dictionary of embeddings. embedding_size (int): The size of each embedding vector. parallel_config (EmbeddingOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. param_init (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: Tuple, a tuple contains (`output`, `embedding_table`) - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). - **embedding_table** (Tensor) - The embedding table with shape (vocab_size, embedding_size). Raises: ValueError: If the parallel_config.vocab_emb_dp is True, the vocab size is not a multiple of parallel_config.model_parallel ValueError: `vocab_size` is not a positive value. ValueError: `embedding_size` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import VocabEmbedding >>> from mindspore import Tensor >>> from mindspore import dtype as mstype >>> model = VocabEmbedding(vocab_size=30, embedding_size=30) >>> tensor = Tensor(np.ones((20, 15)), mstype.int32) >>> output, table = model(tensor) >>> print(output.shape) (20, 15, 30) >>> print(table.shape) (30, 30) """ @_LogActionOnce(m_logger=logger, key='VocabEmbedding', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(vocab_size=Validator.check_positive_int, embedding_size=Validator.check_positive_int, parallel_config=_valid_type_checks([EmbeddingOpParallelConfig], "VocabEmbedding")) def __init__(self, vocab_size, embedding_size, parallel_config=default_embedding_parallel_config, param_init='normal'): super(VocabEmbedding, self).__init__() _check_config(parallel_config) self.vocab_size = vocab_size self.embedding_size = embedding_size self.embedding_table = Parameter(initializer(param_init, [self.vocab_size, self.embedding_size]), name='embedding_table', parallel_optimizer=False) if parallel_config.vocab_emb_dp: self.gather = P.Gather().shard(((1, 1), (parallel_config.data_parallel, 1))) logger.info(f"Using {parallel_config.data_parallel} data parallel for the embedding lookup.") else: if self.vocab_size % parallel_config.model_parallel != 0: raise ValueError(f"The vocab size of the embedding {self.vocab_size} must be a " f"multiple of parallel_config.model_parallel {parallel_config.model_parallel}.") self.gather = P.Gather().shard(((parallel_config.model_parallel, 1), (parallel_config.data_parallel, 1))) logger.info(f"Using {parallel_config.data_parallel} data parallel and {parallel_config.model_parallel} " f"model parallel for the embedding lookup.") def construct(self, input_ids): _check_input_dtype(F.dtype(input_ids), "input_ids", [mstype.int32], self.cls_name) output = self.gather(self.embedding_table, input_ids, 0) return output, self.embedding_table.value() class MultiHeadAttention(Cell): r""" This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following .. math:: MultiHeadAttention(query, key, vector) = Concat(head_1, \dots, head_h)W^O where :math:`head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)`. The default is with a bias. if query, key and value tensor is same, then it will be self attention. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The sequence length of the query vector. tgt_seq_length(int): The sequence length of the key and value vector. hidden_size(int): The hidden size of the input. num_heads(int): The number of the heads. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. compute_dtype(dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. softmax_compute_type(dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. param_init_type(dtype.Number): The parameter initialization type of the module. Default mstype.float32. Should be mstype.float32 or mstype.float16. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **query_tensor** (Tensor) - The query vector with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **key_tensor** (Tensor) - The key vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **value_tensor** (Tensor) - The value vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, size_per_head, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, size_per_head). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import MultiHeadAttention >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> model = MultiHeadAttention(batch_size=None, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3) >>> from_tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 20, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> # We need to prepare the memory parameters for saving key and value states firstly. >>> model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3, use_past=True) >>> key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), mstype.float16) >>> value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), mstype.float16) >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> # Set is_first_iteration=True to generate the full memory states >>> model.add_flags_recursive(is_first_iteration=True) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> from_tensor = Tensor(np.ones((2, 1, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 1, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1, 20)), mstype.float16) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than the >>> # full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 1, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) """ @_LogActionOnce(m_logger=logger, key='MultiHeadAttention', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), parallel_config=_valid_type_checks([OpParallelConfig], "MultiHeadAttention"), use_past=Validator.check_bool) def __init__(self, batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, hidden_dropout_rate=0.1, attention_dropout_rate=0.1, compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, use_past=False, parallel_config=default_dpmp_config): super(MultiHeadAttention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] self.dp = parallel_config.data_parallel self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) if batch_size: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul() self.real_div = P.RealDiv() self.sub = P.Sub() self.mul = P.Mul() self.add = P.Add() # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.use_past = use_past self.dropout = get_dropout(hidden_dropout_rate) self.prob_dropout = get_dropout(attention_dropout_rate) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_cast = P.Cast() self.softmax_reshape = P.Reshape() self.expand_dims = P.ExpandDims() # Query self.dense1 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) # Key self.dense2 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) # Value self.dense3 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) else: _check_config(parallel_config) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.use_past = use_past self.dropout = get_dropout(hidden_dropout_rate) self.prob_dropout = get_dropout(attention_dropout_rate) self.dropout.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout.dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) self.softmax_cast = P.Cast() self.softmax_reshape = P.Reshape() self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.dropout.dropout.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.select_recompute: # _attn中涉及的关键算子使用重计算逻辑，通常配合序列并行使用，会有较好的性能提升效果 self.batch_matmul.recompute() self.sub.recompute() self.add.recompute() self.merger_head_transpose.recompute() self.softmax_reshape.recompute() self.prob_dropout.recompute() self.softmax_cast.recompute() self.softmax.softmax.recompute() self.softmax_3d.recompute() def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" self._check_inputs(query_tensor, key_tensor, value_tensor, attention_mask, key_past, value_past, batch_valid_length) ori_shape = F.shape(query_tensor) batch_size = self._get_batch_size_from_query(query_tensor) query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) ori_dtype = F.dtype(query_tensor) query_tensor = F.cast(query_tensor, self.dtype) key_tensor = F.cast(key_tensor, self.dtype) value_tensor = F.cast(value_tensor, self.dtype) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, self._get_seq_length_under_incremental(self.src_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if attention_mask is not None and len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # key and value for current token(s) key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key, self.expand_dims(valid_length_vector, 2)) value_present = self.mul1(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = F.cast(self.equal(valid_length, self.range), self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) current_value = self.mul1(self.tile(value, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add(key_past, current_key) value = self.add(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value attention_mask = F.reshape(self.attention_mask, (self.seq_length, self.seq_length, 1, 1)) layer_present = (key_present, value_present) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, attention_mask) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) output = F.cast(output, ori_dtype) return output, layer_present def _get_batch_size_from_query(self, query): r"""Get the batch size from query tensor""" # For the incremental prediction, the seq length for the input is 1. if len(F.shape(query)) == 2 and ((self.use_past and self.is_first_iteration) or (not self.use_past)): return F.shape(query)[0] // self.src_seq_length return F.shape(query)[0] def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _check_inputs(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): r"""Check inputs""" _check_input_dtype(F.dtype(query_tensor), "query_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(key_tensor), "key_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_tensor), "value_tensor", [mstype.float32, mstype.float16], self.cls_name) if attention_mask is not None: _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) key_is_tensor = isinstance(key_past, Tensor) value_is_tensor = isinstance(value_past, Tensor) batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) key_is_default = key_past is None value_is_default = value_past is None batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "key_past", self.cls_name, None, key_is_tensor, key_is_default) _check_past_none_input_none(self.use_past, "value_past", self.cls_name, None, value_is_tensor, value_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(key_past), "key_past", [mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_past), "value_past", [mstype.float16], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( self.softmax_reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = self.softmax_reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] factor = P.Cast()(self.scale_factor, P.DType()(query)) query = self.real_div(query, factor) key = self.real_div(key, factor) score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) attention_scores = self.softmax_cast(score, self.softmax_dtype) # for input size of (bs, 1) namely the second graph, # the shape of attention_mask matrix should be (bs, 1, 1, seq_length) if attention_mask is not None: if self.use_past and not self.is_first_iteration: # Calculate the current total token current_index = self.reducesum(F.cast(self.not_equal(self.slice(key, (0, 0, 0, 0), (F.shape(query)[0], 1, 1, self.seq_length), (1, 1, 1, 1)), 0), mstype.float32), (1, 2, 3)) # Get the precise position index index = self.sub1(F.cast(current_index, mstype.int32), 1) index = F.reshape(index, (-1, 1, 1)) # Calculate the attention_mask matrix via the position index attention_mask = F.cast(self.tensor_le(self.range, index), mstype.int32) attention_mask = self.expand_dims(attention_mask, 2) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(attention_scores)), P.Cast()(attention_mask, P.DType()(attention_scores))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, attention_scores) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = self.softmax_cast(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge class TransformerEncoderLayer(Cell): r""" Transformer Encoder Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. seq_length(int): The input sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, seq_length), (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoderLayer >>> from mindspore import Tensor >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) """ @_LogActionOnce(m_logger=logger, key='TransformerEncoderLayer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "TransformerEncoderLayer"), use_past=Validator.check_bool) def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', use_past=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerEncoderLayer, self).__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) self.batch_size = batch_size if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, parallel_config=attention_parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, parallel_config=attention_parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.add.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1), (parallel_config.data_parallel * parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.recompute.select_recompute: # 此处会消耗较大内存，开启后会损失一部分计算性能 self.layernorm2.layer_norm.recompute() if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, x, input_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(x, input_mask, init_reset, batch_valid_length) x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) if self.post_layernorm_residual: input_x = x else: input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) output = F.reshape(output, (-1, x_shape[-1])) output = self.layernorm1(output) output = F.reshape(output, x_shape) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) output = self.layernorm1(output) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present def _check_input(self, x, input_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) if input_mask is not None: _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True class TransformerDecoderLayer(Cell): r""" Transformer Decoder Layer. This is an implementation of the single layer of the transformer decoder layer, including self-attention, cross attention and feedward layer. When the encoder_output is None, the cross attention will not be effective. Args: hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, tgt_seq_length, hidden_size] or [batch_size * tgt_seq_length, hidden_size]. - **decoder_mask** (Tensor) - The attention mask for decoder with shape [batch_size, src_seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]. - **layer_present** (Tuple) - A tuple, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoderLayer >>> from mindspore import Tensor >>> model = TransformerDecoderLayer(batch_size=2, hidden_size=64, ffn_hidden_size=64, num_heads=2, ... src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(past[0].shape) (2, 2, 32, 10) >>> print(past[1].shape) (2, 2, 10, 32) >>> print(past[2].shape) (2, 2, 32, 20) >>> print(past[3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='TransformerDecoderLayer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "TransformerDecoderLayer"), use_past=Validator.check_bool) def __init__(self, hidden_size, ffn_hidden_size, num_heads, batch_size, src_seq_length, tgt_seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, use_past=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerDecoderLayer, self).__init__() _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) config_to_attention = parallel_config.dpmp if self.use_moe else parallel_config if batch_size or use_past: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}." .format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=config_to_attention) # Cross attention with the output of encoder as memory tensor self.cross_attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, param_init_type=param_init_type, parallel_config=config_to_attention) self.cross_attention_layernorm = LayerNorm((hidden_size,)).to_float( layernorm_compute_type) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add() self.add_3d = P.Add() self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}." .format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) self.attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=config_to_attention) # Cross attention with the output of encoder as memory tensor self.cross_attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, param_init_type=param_init_type, parallel_config=config_to_attention) self.cross_attention_layernorm = LayerNorm((hidden_size,)).to_float( layernorm_compute_type) self.cross_attention_layernorm.shard(((parallel_config.data_parallel, 1),)) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.add.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1), (parallel_config.data_parallel * parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.recompute.select_recompute: # 此处会消耗较大内存，开启后会损失一部分计算性能 self.layernorm2.layer_norm.recompute() if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_stats, decoder_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(hidden_stats, decoder_mask, encoder_output, memory_mask, init_reset, batch_valid_length) # the returned shape is [bs, seq_length, embedding_size] or [bs * seq_length, embedding_size] hidden_shape = F.shape(hidden_stats) hidden_stats = F.reshape(hidden_stats, (-1, hidden_shape[-1])) input_x = self.layernorm1(hidden_stats) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, decoder_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(hidden_stats, attention) middle_output = None if encoder_output is not None: middle_output = self.cross_attention_layernorm(x) middle_output = F.cast(middle_output, self.dtype) encoder_output = F.cast(encoder_output, self.dtype) cross_attn_output, cross_layer_present = self.cross_attention(middle_output, encoder_output, encoder_output, memory_mask, self.key_past, self.value_past, batch_valid_length) layer_present += cross_layer_present if self.post_layernorm_residual: x = self.add(middle_output, cross_attn_output) else: x = self.add(x, cross_attn_output) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(hidden_shape) == 3: output_x = P.Reshape()(output_x, hidden_shape) mlp_logit = P.Reshape()(mlp_logit, hidden_shape) x = P.Reshape()(x, hidden_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, hidden_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present def _check_input(self, hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(hidden_states), "hidden_states", [mstype.float32, mstype.float16], self.cls_name) if attention_mask is not None: _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) if encoder_output is not None: _check_input_dtype(F.dtype(encoder_output), "encoder_output", [mstype.float32, mstype.float16], self.cls_name) if memory_mask is not None: _check_input_dtype(F.dtype(memory_mask), "memory_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(init_reset), "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _get_lambda_func(total_layer=None): r""" A wrapper function of specifying pipeline stage and gradient aggregation fusion. If the total layer is not None, for example, set in the transformer model, the pipeline stage setting function will be `(layer_id + 0) // (total_layers / parallel_config.pipeline_stage)` for the encoder and, `(layer_id + offset) // (total_layers / parallel_config.pipeline_stage)` for the decoder, where `offset` is the layers in the encoder. """ def _set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs an offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # override the layers if total_layer: layers = total_layer # Used for the pipeline's stages setting if layers < parallel_config.pipeline_stage: raise ValueError(f"layers {layers} must be larger than pipeline stage {parallel_config.pipeline_stage}") pp_dis = max(layers // parallel_config.pipeline_stage, 1) # the pipeline stage must be in [0, parallel_config.pipeline_stage - 1] pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(layers // parallel_config.gradient_aggregation_group, 1) network.set_comm_fusion((layer_id + offset) // dis + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not parallel_config.select_recompute: network.recompute() else: if parallel_config.recompute.recompute and not parallel_config.recompute.select_recompute: paralel_op_comm_compute = parallel_config.recompute.parallel_optimizer_comm_recompute network.recompute(parallel_optimizer_comm_recompute=paralel_op_comm_compute, mp_comm_recompute=parallel_config.recompute.mp_comm_recompute, recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) return _set_parallel_configure_for_layer class TransformerEncoder(Cell): r""" Transformer Encoder module with multi-layer stacked of `TransformerEncoderLayer`, including multihead self attention and feedforward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. num_layers(int): The layers of the `TransformerEncoderLayer` hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. seq_length(int): The seq_length of the input tensor. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default: 0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default: mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default: mstype.float32. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. offset(int): The initial layer index for the `encoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default: False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_states** (Tensor) - Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size]. - **attention_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple contains the Tensor the projected key and value vector with shape ((batch_size, num_heads, size_per_head, seq_length), and (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoder >>> from mindspore import Tensor >>> model = TransformerEncoder(batch_size=2, num_layers=2, hidden_size=8, ffn_hidden_size=64, ... seq_length=16, num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(len(past)) 2 >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoder(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, num_layers=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) """ @_LogActionOnce(m_logger=logger, key='TransformerEncoder', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, num_layers=Validator.check_positive_int, offset=Validator.check_non_negative_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "TransformerEncoder"), use_past=Validator.check_bool) def __init__(self, batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, offset=0, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerEncoder, self).__init__() _check_config(parallel_config) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) if batch_size or use_past: Validator.check_positive_int(batch_size) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class TransformerDecoder(Cell): r""" Transformer Decoder module with multi-layer stacked of `TransformerDecoderLayer`, including multihead self attention, cross attention and feedforward layer. Args: num_layers(int): The layers of the `TransformerDecoderLayer`. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. offset(int): The initial layer index for the `decoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size] - **attention_mask** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size] - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoder >>> from mindspore import Tensor >>> model = TransformerDecoder(batch_size=2, num_layers=1, hidden_size=64, ffn_hidden_size=64, ... num_heads=2, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(past)) 1 >>> print(past[0][0].shape) (2, 2, 32, 10) >>> print(past[0][1].shape) (2, 2, 10, 32) >>> print(past[0][2].shape) (2, 2, 32, 20) >>> print(past[0][3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='TransformerDecoder', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, num_layers=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, offset=Validator.check_non_negative_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "TransformerDecoder"), use_past=Validator.check_bool) def __init__(self, num_layers, batch_size, hidden_size, ffn_hidden_size, src_seq_length, tgt_seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', lambda_func=None, use_past=False, offset=0, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerDecoder, self).__init__() _check_moe_config(moe_config, parallel_config) _check_config(parallel_config) self.use_moe = (moe_config.expert_num > 1) if batch_size or use_past: Validator.check_positive_int(batch_size) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, use_past=use_past, param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, use_past=use_past, param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss # Loop through each self-attention layer for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class Transformer(Cell): r""" Transformer module including encoder and decoder. The difference with the original implements is the module use the residual addition before the layer normalization. And the default hidden act is `gelu`. The details can be found in `Attention is all you need `_. Note: This is an experimental interface that is subject to change or deletion. Args: hidden_size(int): The hidden size of the input. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The seq_length of the encoder's input tensor. tgt_seq_length(int): The seq_length of the decoder's input tensor. encoder_layers(int): The layers of the `TransformerEncoderLayer`. Default 3. decoder_layers(int): The layers of the `TransformerDecoderLayer`. Default 3. num_heads(int): The number of the heads. Default: 2. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. lambda_func: A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // ((encoder_layers + decoder_layers) / pipeline_stage)`. Default None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **encoder_inputs** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. - **encoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the encoder module. - **decoder_inputs** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], this should be none if the decoder layer is 0. - **decoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the decoder module. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The output of the encoder with shape [batch_size, seq_length, hidden_size], this should be none if the decoder layer is 0 or the user wants no mask. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `encoder_layer_present`, `decoder_layer_present`, `accum_loss`) - **output** (Tensor) - If there is only encoder, the output logit of the encoder layer. The shape is [batch, src_seq_length, hidden_size] or [batch * src_seq_length, hidden_size], if there are encoder and decoders, the output is from the decoder layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size]. - **encoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). - **decoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)), and the projected key and value vector in cross attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). If the decoder is not set, the returned value will be None. - **accum_loss** (Tensor) - A Tensor indicates an auxiliary loss to minimize the mean square of the data part routed to each expert, and only returned if the number of experts is greater than 1. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import Transformer >>> from mindspore import Tensor >>> model = Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, ... ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, en_past, de_past = model(encoder_input_value, encoder_input_mask, decoder_input_value, ... decoder_input_mask, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(en_past)) 1 >>> print(len(de_past)) 2 >>> print(en_past[0][0].shape) (2, 2, 32, 20) >>> print(en_past[0][1].shape) (2, 2, 20, 32) >>> print(de_past[0][0].shape) (2, 2, 32, 10) >>> print(de_past[0][1].shape) (2, 2, 10, 32) >>> print(de_past[0][2].shape) (2, 2, 32, 20) >>> print(de_past[0][3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='Transformer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, encoder_layers=Validator.check_positive_int, decoder_layers=Validator.check_non_negative_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "Transformer"), use_past=Validator.check_bool) def __init__(self, hidden_size, batch_size, ffn_hidden_size, src_seq_length, tgt_seq_length, encoder_layers=3, decoder_layers=3, num_heads=2, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(Transformer, self).__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.batch_size = batch_size self.hidden_size = hidden_size self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past if encoder_layers <= 0 < decoder_layers: raise ValueError(f"Transformer doest support encoder layer {encoder_layers} and decoder" f"layer {decoder_layers}, please use TransformerDecoder") if encoder_layers > 0 and decoder_layers > 0 and use_past: raise ValueError(f"The {self.cls_name} with encoder and decoder does not support use_past=True.") # The shard setting of Transformer is set within the TransformerEncoderLayer if not lambda_func: lambda_func = _get_lambda_func(total_layer=encoder_layers + decoder_layers) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) if encoder_layers > 0: self.encoder = TransformerEncoder(num_layers=encoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=src_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, lambda_func=lambda_func, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config) else: self.encoder = None # Offset is needed as the encoder has consumed some flags. # so the decoder need to increase the flags based on the encoder layer self.decoder = None if decoder_layers > 0: self.decoder = TransformerDecoder(num_layers=decoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, lambda_func=lambda_func, use_past=use_past, param_init_type=param_init_type, offset=encoder_layers, moe_config=moe_config, parallel_config=parallel_config) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.batch_size = batch_size self.hidden_size = hidden_size self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past if encoder_layers <= 0 < decoder_layers: raise ValueError(f"Transformer doest support encoder layer {encoder_layers} and decoder" f"layer {decoder_layers}, please use TransformerDecoder") if encoder_layers > 0 and decoder_layers > 0 and use_past: raise ValueError(f"The {self.cls_name} with encoder and decoder does not support use_past=True.") logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") # The shard setting of Transformer is set within the TransformerEncoderLayer if not lambda_func: lambda_func = _get_lambda_func(total_layer=encoder_layers + decoder_layers) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) if encoder_layers > 0: self.encoder = TransformerEncoder(num_layers=encoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=src_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, lambda_func=lambda_func, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config) else: self.encoder = None # Offset is needed as the encoder has consumed some flags. # so the decoder need to increase the flags based on the encoder layer self.decoder = None if decoder_layers > 0: self.decoder = TransformerDecoder(num_layers=decoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, lambda_func=lambda_func, use_past=use_past, param_init_type=param_init_type, offset=encoder_layers, moe_config=moe_config, parallel_config=parallel_config) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, encoder_inputs, encoder_masks, decoder_inputs=None, decoder_masks=None, memory_mask=None, init_reset=True, batch_valid_length=None): """process process""" encoder_output = None output = None encoder_layer_present = None decoder_layer_present = None accum_loss = self.aux_loss if self.encoder is not None: if self.use_moe: encoder_output, encoder_layer_present, encoder_aux_loss = self.encoder(encoder_inputs, encoder_masks, init_reset, batch_valid_length) accum_loss = self.add(accum_loss, encoder_aux_loss) else: encoder_output, encoder_layer_present = self.encoder(encoder_inputs, encoder_masks, init_reset, batch_valid_length) output = encoder_output if self.decoder is not None: # decoder mask should be created outside of the model if self.use_moe: decoder_output, decoder_layer_present, decoder_aux_loss = self.decoder(decoder_inputs, decoder_masks, encoder_output, memory_mask, init_reset, batch_valid_length) accum_loss = self.add(accum_loss, decoder_aux_loss) else: decoder_output, decoder_layer_present = self.decoder(decoder_inputs, decoder_masks, encoder_output, memory_mask, init_reset, batch_valid_length) output = decoder_output if self.use_moe: return output, encoder_layer_present, decoder_layer_present, accum_loss return output, encoder_layer_present, decoder_layer_present pengcheng.mind-test/mindformers/modules/transformer/__init__.py000066400000000000000000000024671471400134000253720ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ NOTE: Transformer Networks. This is an experimental interface that is subject to change or deletion. """ from __future__ import absolute_import from .transformer import AttentionMask, VocabEmbedding, MultiHeadAttention, \ FeedForward, TransformerEncoder, TransformerDecoder, TransformerEncoderLayer, TransformerDecoderLayer, \ Transformer, TransformerOpParallelConfig, EmbeddingOpParallelConfig, TransformerRecomputeConfig from .transformer import * from .moe import MoEConfig from .op_parallel_config import OpParallelConfig __all__ = [] __all__.extend(transformer.__all__) __all__.extend(op_parallel_config.__all__) __all__.extend(moe.__all__) pengcheng.mind-test/mindformers/modules/transformer/__pycache__/000077500000000000000000000000001471400134000254605ustar00rootroot00000000000000pengcheng.mind-test/mindformers/modules/transformer/__pycache__/__init__.cpython-39.pyc000066400000000000000000000016241471400134000316530ustar00rootroot00000000000000a ?cf7@sdZddlmZddlmZmZmZmZmZm Z m Z m Z m Z m Z mZmZddlTddlmZddlmZgZeejeejeejdS) zn NOTE: Transformer Networks. This is an experimental interface that is subject to change or deletion. )absolute_import) AttentionMaskVocabEmbeddingMultiHeadAttention FeedForwardTransformerEncoderTransformerDecoderTransformerEncoderLayerTransformerDecoderLayer TransformerTransformerOpParallelConfigEmbeddingOpParallelConfigTransformerRecomputeConfig)*) MoEConfig)OpParallelConfigN)__doc__ __future__r transformerrrrrrr r r r r rrZmoerZop_parallel_configr__all__extendrrS/home/ma-user/work/mindformers-baichuan/mindformers/modules/transformer/__init__.pys 8    pengcheng.mind-test/mindformers/modules/transformer/__pycache__/moe.cpython-39.pyc000066400000000000000000000500571471400134000307000ustar00rootroot00000000000000a ?cf@sTdZddlmZddlmZddlZddlZddlmZddl m m Z ddl mmZzddlmZWneyddlmZYn0ddlmZddlmZdd lmZdd lmZdd lmZdd l m!Z!dd l"m#Z#m$Z$ddl%m&Z&dgZ'GdddZ(e(Z)dddZ*eddZ+GdddeZ,GdddeZ-GdddeZ.dS)zs Note: Mixture of Expert (MoE) structure. This is an experimental interface that is subject to change or deletion. )absolute_import)divisionN)Tensor) Validator) operations) functional) constexpr)Cell)Dense) ParallelMode)_get_parallel_mode_is_sharding_propagation)default_moeparallel_config MoEConfigc@seZdZdZd dd ZdS) raz The configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import MoEConfig >>> moe_config = MoEConfig(expert_num=4, capacity_factor=5.0, aux_loss_factor=0.05, num_experts_chosen=1, ... expert_group_size=64, group_wise_a2a=True, comp_comm_parallel=False, ... comp_comm_parallel_degree=2) 皙?皙?NFc Cst|dt|dt|dt|dt|dt|dt|d|durht|d|d krtd |d |d krtd |d ||krtd |d ||_||_||_||_||_ ||_ ||_ ||_ dS)N expert_numcapacity_factoraux_loss_factornum_experts_chosengroup_wise_a2acomp_comm_parallelcomp_comm_parallel_degreeexpert_group_size?z@'capacity_factor' must be equal to or greater than 1.0, but got .z1'aux_loss_factor' must be less than 1.0, but got zC'num_experts_chosen' must not be larger than 'expert_num', but got ) rcheck_positive_intcheck_positive_float check_bool ValueErrorrrrrrrrr) selfrrrrrrrrr#N/home/ma-user/work/mindformers-baichuan/mindformers/modules/transformer/moe.py__init__Ns:           zMoEConfig.__init__)rrrrNFFr)__name__ __module__ __qualname____doc__r%r#r#r#r$r-s c Cst|ts tdt|jd|jdk}|dur6dS|j|jdkrxtdt|jdt|jd |jd |jd t }||jdkrtd |d |jd|j |jdkrtd|j d |jd|j |j |krtd|j d|j d|ddS)z4 check if MoE with right configuration. z7'moe_config' must be an instance of MoEConfig, but got rrFNrz$When using MoE, the 'expert_num' in z2 must be a multiple of 'expert_parallel' value in z , but got z for 'expert_num' and z for 'expert_parallel'.z device_num: z( must be a multiple of expert_parallel: zdata parallel: z when using MoE.z"The product of the data parallel: z and model parallel: z! should be less than device_num: ) isinstancer TypeErrortyper&rexpert_parallelr!Dget_group_size data_parallelmodel_parallel) moe_configparallel_configuse_moe device_numr#r#r$_check_moe_configos8        r6cCst||||S)N)mathceil)ktokens_per_groupr expert_dimr#r#r$calculate_expert_capacitysr<csDeZdZdZdejeeffdd ZddZ ddZ d d Z Z S) MoEa The mixture of experts (MoE) implementation. The implementation includes a router and a FeedForward layer. The router dispatches tokens to experts in FeedForward, then FeedForward does computation, and the final output is obtained by multiplying FeedForward's output and router's combine weight. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. Default: gelu. param_init_type (dtype.Number): The parameter initialization type. Can be dtype.float32 or dtype.float16. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(MoEParallelConfig): The parallel config for MoE, see `MoEParallelConfig`. Default `default_moeparallel_config`, an instance of `MoEParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size]`. geluc spttttjfvrtr|_|j_ |j _ |j _ |j _ |j _ |j_|j_|j_|j_|j_|j_|j_jdkrjjksd_ddlm}|||||j j ||d_t_t_t !jdff_"t !jddff_#t !djddff_$t !ddjdff_%t&!jddfjddff_'t&!jddfjddff_(t)_*t+||dd|d_,t-_.t/d!t0fd d t1jD_2t/d !djddfdjddff_3tj4d jd !djddff_5t6!jdddff_7t6!djddff_8t6!jdddff_9t6!djjdff_:t6!jdjdff_;n|_|j_ |j _ |j _ |j _ |j_|j_|j_|j_|j_|j_|j_jdkrDjjksJd_ddlm}|||||j ||d _t_t_t !jdff_"t !jddff_#t !djddff_$t !ddjdff_%t&!jddfjddff_'t&!jddfjddff_(t)!d_*t+||dd|d_,t-_.t/d!t0fdd t1jD_2t/d !djddfdjddff_3tj4d jd !djddff_5t6!jdddff_7t6!djddff_8t6!jdddff_9t6!djjdff_:t6!jdjdff_;dS)NrFr) FeedForward) hidden_sizeffn_hidden_size dropout_rate hidden_actrrparam_init_typer3T)d_modelr2routing_policytrainingr3c3s|]}jdddfVqdSrNdp.0_r"r#r$ zMoE.__init__..r)axis output_num)r@rArBrCrrDr3r#r#c3s|]}jdddfVqdSrIrJrLrOr#r$rPrQ)<superr=r%r r AUTO_PARALLELr r@rr;rrrrr0dp_grouprKr-epr1mprrrZmindformers.modules.transformerr?ffnPReshapereshapeShapeshape Transposeshardtranspose_2dimtranspose_3dimtranspose_4dimtranspose_4dim_dp BatchMatMulbatch_mm batch_mm2MulmulRouterrouterCastcastConcattuplerangeconcat concat_dpSplitsplit StridedSlice stride_slicestride_slice_dpstride_slice_epstride_slice_dp_mpstride_slice_ep_mp) r"r@rArBrCrDr2r3r? __class__rOr$r%s   $$  **$"   $$ **$z MoE.__init__cCsd}|jr||jdkr(|j||j}|dkrd||7}||d|j|j||jfd}|||f}||d|j|j||jfd}||d|j|j||jfd}| |d|j|j||jfd}| ||j|j||jf}| |}| ||j|j||jf}|jr||d|j|j||jfd}||d|j|j||jfd}||d|j|j||jfd}|dkr||8}||d|j|j||jfd}| |d}|S)z$ Computing the FFN. rrrrrrrrr)rrHrr) rrYrxr;rWr@rsrzr{ryr]rZrd)r" expert_inputcapacitypad_size pad_tensor expert_outputr#r#r$ ffn_infer sf   z MoE.ffn_inferc Csd}||jdkrV|j||j}||7}||d|j|j||jfd}|||f}||j}g}||D]}|||}||qn| |} |dkr||8}| | d|j|j|j|fd} | S)zB Split and overlap FFN compute and communication. rr~r) rrxr;rWr@rsrurappendrrrw) r"rrrrZ sub_capacity output_listZsub_expert_inputZsub_expert_outputrr#r#r$ffn_parallel_inferGs,    zMoE.ffn_parallel_inferc Cst|}||d|jf}||}|d|j}|||j||jf}t|j||j|j}| |\}}}| |d}|||j||j|f}| |t |}| ||} || |j|j|j|f} || |j|j|j|f} || d} || |j||j|jf} || d} |jr2|| |} n || |} || |j|j|j|f} |||j||j|f}| |d}| |t | }|| |} | | d} || |d|df} || |} ||j|}| |fS)forward processr)rrr)rr)rrrrHr)Fr_r]r@rWr<rrr;rlrcrndtypergrbrerrrrhrjr) r" input_tensor input_shapeZ bs_and_dmodelr:expert_capacitydispatch_tensorcombine_tensoraux_lossrrZcombined_outputr#r#r$ constructdsV             z MoE.construct) r&r'r(r)mstypefloat32default_moe_configrr%rrr __classcell__r#r#r|r$r=sc:r=cs*eZdZdZdfdd ZddZZS) rka A router backbone used to calculate logits of each token, which should be cascaded by router implementations mapping tokens to experts. when moe_config.num_experts_chosen = 1, use top1 routing; when moe_config.num_experts_chosen > 1, use topk routing Args: d_model (int): The hidden size of each token. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). routing_policy: The policy of mapping tokens to experts. Default: topkRouter training (bool): The value indicating whether is in training phase. parallel_config: The parallel-related configuration. Inputs: - **input_tensor** (Tensor) - Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, hidden\_size)`. Outputs: Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim)`. NTcstt||j}||_|j|_|j|_|j|_||_ ||_ d|_ d|_ t tjd|j d|j |f|_t|j|jdd|_|jj|dfdft|_t|_|j durt||||d|_n||_ttjfvrt s|j|ddf|ffdS)Ng{Gz?rF) in_channels out_channelshas_bias)rr)rEr2rGr3)!rUrkr%r0rErr;rrrGrF noisy_policyZ noisy_epsilonrnprandomuniformnoiser densematmulrar[rirjrmrn TopkRouterrlr r rVr )r"rEr2rFrGr3rKr|r#r$r%s,$    zRouter.__init__cCs@||tj}|jdkr,|jr,|||j}||}||S)NZjitter) rnrrrrGrjrrrl)r"r router_logitsr#r#r$rs  zRouter.construct)NTN)r&r'r(r)r%rrr#r#r|r$rks  rkcs:eZdZdZd fdd ZddZdd Zd d ZZS) ra A router implementation which maps each tokens to the topk expert. Args: d_model (int): The hidden size of each token. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). training (bool): The value indicating whether is in training phase. config: The parallel-related configuration. Inputs: - **input_tensor** (Tensor) - Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, hidden\_size)`. Outputs: Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim, expert\_capacity)`, Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim, expert\_capacity)`, Tensor of shape :math:`(1)`. TNcstt|ttjfvrRtrR|j}||_|j |_ |j |_ ||_ ||_ d|_t|_t|_t|_tjdd|_tjddd|_|j|_t|_t|_t|_tdt j!|_"tdt j!|_#tj$dd|_%tj$dd|_&tj$dd|_'t(|_)t(|_*t(|_+t(|_,t(|_-t(|_.t(|_/t(0|ddf|ddff|_1t(0|dddf|dddff|_2t3|_4t5|_6t5|_7t8|_9t8|_:t8|_;t8|_|_?tj@d d |_AtB|_CtjDdd|_EtjDd d|_FtjDd d|_GtH|_ItH|_Jt8|_Ktdt j!|_Ln2|j}||_|j |_ |j |_ ||_ ||_ d|_t|_t|_t|_tjdd0|ddff|_tjddd0|ddff|_|j|_t0|ddfd d f|_t0|ddfd d f|_t0|dddfd d f|_tdt j!|_"tdt j!|_#tj$dd0|ddff|_%tj$dd0|ddff|_&tj$dd0|dff|_'t(0|df|dff|_)t(0d |_*t(0d |_+t(0|ddf|ddff|_,t(0|ddf|ddff|_-t(0|df|dff|_.t(0|df|dff|_/t(0|ddf|ddff|_1t(0|dddf|dddff|_2t30|dddfd f|_4t50|ddf|ddff|_6t50|dddf|dddff|_7t80|ddf|ddff|_9t80|ddfd f|_:t80|dddf|dddff|_;t80|df|dff|_0d |ddff|_?tj@d d 0|ddff|_AtB0|ddfd f|_CtjDdd0|ddff|_EtjDd d0|ddff|_FtjDd d0|dddff|_GtH0|dff|_ItH0|ddff|_Jt80d |_Ktdt j!|_LdS) Nr)rRF)rR keep_dimsr)rrT) exclusiver#rT)MrUrr%r r rVr r0rErr;rrGrWrr[rmrnr\r]r^r_SoftmaxsoftmaxArgMaxWithValueargmaxrOneHotonehotonehot2onehot3rrron_value off_value ReduceMean reduce_mean reduce_mean2 reduce_mean3rirjmul2mul3mul4mul5mul6mul7ramul8mul9NotEqual not_equalRealDivdiv1div2Addaddadd1add2add3add4SubsubCumSumcumsumLessless ReduceSum reduce_sumreduce_sum_keepreduce_sum_keep2 ExpandDimsexpandexpand2 add_scala init_loss)r"rEr2rGr3rKr|r#r$r%s              $                   $ $ $zTopkRouter.__init__c Cs||}||d|df}||}|d|j}t|j||j|j}|||j||jf}d}d}|j}d} d} ||} t |jD]<} | | | |j |} | | \} }|| |j|j |j}|| ||| dd}|||||}||||| | }|d|d|d|df\}}}}|||}|||}|| ||d} |||}|||d|| |j|j |j}|||d|||t j!||j |j}|"| |} q|#|#| dd}|$| |%|d} |&| d}|| |fS) rrrg& .>rrrHr)'r_r]rWr<rrr;rrrqrrrrrrrrrr_auxiliary_loss_maskout_overflowed_tokensrrrrrrrrrrnrint32rrrrr)r"rZrouter_logits_shape logits_shaper:rZaccum_expert_maskZaccum_expert_gatelossZ mask_countZaccum_combine_tensor router_probexpert_chosen_indexZ expert_index expert_gate expert_maskZrouter_prob_normaloutputexpert_mask_flatposition_in_expertrZcombine_tensor_sumrr#r#r$r^sR     $    zTopkRouter.constructcCsJ||d}||d}|||}||}||||j|j}|S)z2 Computing the load balance loss. r)rrrjrrrr;)r"rrZ density_1Zdensity_1_proxyrr#r#r$rs     zTopkRouter._auxiliary_lossc Csl||d}|dkr |||}|||}|||}|||}||d} ||| }||| |f} | S)zJ Keeping only the tokens that fit within expert_capacity. rrr)rrrrrrr) r"rrrlast_numrrrZ less_resultrrr#r#r$rs        z%TopkRouter._maskout_overflowed_tokens)TN) r&r'r(r)r%rrrrr#r#r|r$rsm9 r)NN)/r) __future__rrr7numpyrmindspore.common.tensorrmindspore.common.dtypecommonrr"mindspore.communication.management communication managementr.mindspore._checkparamr ImportError _checkparam mindspore.opsrr[rrmindspore.ops.primitivermindspore.nn.cellr Zmindspore.nn.layerr mindspore.contextr mindspore.parallel._utilsr r 2mindformers.modules.transformer.op_parallel_configr__all__rrr6r<r=rkrr#r#r#r$s>           ?  @pengcheng.mind-test/mindformers/modules/transformer/__pycache__/op_parallel_config.cpython-39.pyc000066400000000000000000000162341471400134000337360ustar00rootroot00000000000000a ?cf!!@sdZddlmZzddlmZWney>ddlmZYn0ddlmZddl m m Z ddl mZddlmZddlmZd gZGd d d ZGd d d eZGdd d eZGdddeZeZeZddZdS)zx Parallel Config for the Parallel Training This is an experimental interface that is subject to change and/or deletion. )absolute_import) ValidatorN)context) ParallelMode)_get_parallel_mode)logOpParallelConfigc@seZdZdZddZdS)_Configz A basic class of the configurecCs0d}|jD]\}}d||}||7}q|S)Nz[ParallelConfig] z{}:{} )__dict__itemsformat)selfinfokvZvar_infor]/home/ma-user/work/mindformers-baichuan/mindformers/modules/transformer/op_parallel_config.py__str__'s   z_Config.__str__N)__name__ __module__ __qualname____doc__rrrrrr $sr c@steZdZdZdddZeddZejddZed d Zejd d Zed d Z e jdd Z eddZ dS)MoEParallelConfiga MoEParallelConfig for MoE structure, which includes setting data parallel, model parallel and expert parallel. Args: data_parallel (int): The data parallel way. Default: 1 model_parallel (int): The model parallel way. Default: 1 expert_parallel (int): The expert parallel way. Default: 1 Supported Platforms: ``Ascend`` FcCsdt|dt|dt|dt|dt|dt||||d|_||_||_||_dS)N data_parallelmodel_parallelexpert_paralleluse_seq_parallelselect_recompute)rrrr)rcheck_positive_int check_boolr_dpmprrr)r rrrrrrrr__init__;s     zMoEParallelConfig.__init__cCs|jjSN)r!rr rrrrJszMoEParallelConfig.data_parallelcCst|d||j_dSNr)rrr!rr valuerrrrNs cCs|jjSr#)r!rr$rrrrSsz MoEParallelConfig.model_parallelcCst|d||j_dSNr)rrr!rr&rrrrWs cCs|jSr#)_expert_parallelr$rrrr\sz!MoEParallelConfig.expert_parallelcCst|d||_dS)Nr)rrr)r&rrrr`s cCs|jS)z Get the configuration for dpmp )r!r$rrrdpmpeszMoEParallelConfig.dpmpN)rrrFF) rrrrr"propertyrsetterrrr*rrrrr/s$        rc@sNeZdZdZd ddZeddZejddZed d Zejd d Zd S)ra OpParallelConfig for the setting data parallel and model parallel. Args: data_parallel (int): The data parallel way. Default: 1 model_parallel (int): The model parallel way. Default: 1 Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules import OpParallelConfig >>> config=OpParallelConfig(data_parallel=1, model_parallel=1) rFcCsLt|dt|dt|dt|d||_||_||_||_dS)Nrrrr)rrr rrrr)r rrrrrrrr"zs    zOpParallelConfig.__init__cCs|jSr#)_data_parallelr$rrrrszOpParallelConfig.data_parallelcCst|d||_dSr%)rrr-r&rrrrs cCs|jSr#)_model_parallelr$rrrrszOpParallelConfig.model_parallelcCst|d||_dSr()rrr.r&rrrrs N)rrFF) rrrrr"r+rr,rrrrrrks    c@sNeZdZdZd ddZeddZejddZedd Zejd d Zd S) _PipeLineConfiga PPConfig for the setting data parallel, model parallel Args: pipeline_stage (int): The number of the pipeline stages. Default: 1 micro_batch_num (int): The model parallel way. Default: 1 Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> config=_PipeLineConfig(pipeline_stage=1, micro_batch_num=1) rcCs(t|dt|d||_||_dS)Npipeline_stagemicro_batch_num)rrr0r1)r r0r1rrrr"s  z_PipeLineConfig.__init__cCs|jSr#)_pipeline_stager$rrrr0sz_PipeLineConfig.pipeline_stagecCs"t|d||_tj|ddS)Nr0)pipeline_stages)rrr2rset_auto_parallel_contextr&rrrr0s cCs|jSr#)_micro_batch_numr$rrrr1sz_PipeLineConfig.micro_batch_numcCst|d||_dS)Nr1)rrr5r&rrrr1s N)rr) rrrrr"r+r0r,r1rrrrr/s    r/c Cstd}t|dr6||jkr6td|d|jdttjtjfv}|sPdSt }td}|j |j ||krtd|j d |j d |d |d t|d r|r||j krtd|d|j ddS)z7 Check if micro_batch_num >= pipeline_stage r3r0zThe pipeline stage z= in auto_parallel_context is not equal to the pipeline_stage z in the config.Nenable_parallel_optimizerz!The product of the data parallel z, model parallel z pipeline stages z should be less than device_num .optimizer_shardzThe optimizer shard z> in auto_parallel_context is not equal to the optimizer_shard zS in the OpParallelConfig. Please check the optimizer_shard to make them consistent.)rget_auto_parallel_contexthasattrr0 ValueErrorrrSEMI_AUTO_PARALLEL AUTO_PARALLELDget_group_sizerrr8loggerwarning)configr0is_auto_parallel device_numr8rrr _check_configs0     rE)r __future__rmindspore._checkparamr ImportError _checkparam mindsporer"mindspore.communication.management communication managementr>mindspore.contextrmindspore.parallel._utilsrrr@__all__r rrr/default_dpmp_configZdefault_moeparallel_configrErrrrs&       <,)pengcheng.mind-test/mindformers/modules/transformer/__pycache__/transformer.cpython-39.pyc000066400000000000000000003323621471400134000324640ustar00rootroot00000000000000a ?cf@sNdZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddl mZddlmmZdd lmZdd lmZdd lmZzdd lmZWneyddlmZYn0dd l mZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6ddl7m8Z8ddl9m:Z:gdZ;Gddde0ZGddde0Z?e?Z@e<ZAGdddeZBGdddeZCGd d!d!eZDGd"d#d#eZEGd$d%d%eZFGd&d'd'eZGGd(d)d)eZHd2d*d+ZIGd,d-d-eZJGd.d/d/eZKGd0d1d1eZLdS)3zZ Note: Transformer Networks. This is interface that is subject to change or deletion. )absolute_importN)Tensor) Parameter) initializer)nn)context) operations) functional)Cell) Validator)log)_get_parallel_mode) ParallelMode) LayerNormLinear_args_type_validator_check_valid_type_checks_valid_value_checks_check_past_none_input_none_check_input_dtype)default_dpmp_config_PipeLineConfigOpParallelConfig_Config _check_configMoEParallelConfig)default_moe_configMoE_check_moe_config) get_dropout)_LogActionOnce) AttentionMaskAttentionMaskHFVocabEmbeddingMultiHeadAttention FeedForwardTransformerEncoderTransformerDecoderTransformerEncoderLayerTransformerDecoderLayer TransformerTransformerOpParallelConfigEmbeddingOpParallelConfigTransformerRecomputeConfigc@steZdZdZdddZeddZejd dZed d Zejd d Zed dZ e jddZ eddZ dS)r,a The parallel config of :class:`VocabEmbedding` for the setting data parallel or model parallel for the embedding table. Args: data_parallel(int): The data parallel way. The input data will be sliced into n parts for embedding layer according to this value. Default: 1. model_parallel(int): The model parallel way. The embedding table parameters will be sliced at 0-th axis according to the model parallel way. Default: 1. vocab_emb_dp(bool): Shard embedding in model parallel or data parallel. If True, the embedding lookup will be a data parallel style training and model_parallel value will be ignored. If false, the embedding table will be sharded into n parts at the 0-th dimension row slice of the embedding table, where the n is the model parallel way determined by this parameter. Default: True Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import EmbeddingOpParallelConfig >>> config=EmbeddingOpParallelConfig(data_parallel=1, model_parallel=1, vocab_emb_dp=True) FTcCs4t||||d|_t|d||_||_||_dS)N) data_paralleluse_seq_parallelmodel_parallelselect_recompute vocab_emb_dp)r _dp_mp_configr check_boolr3r0r2)selfr/r1r0r2r3r7V/home/ma-user/work/mindformers-baichuan/mindformers/modules/transformer/transformer.py__init__Ys z"EmbeddingOpParallelConfig.__init__cCs|jjSNr4r/r6r7r7r8r/esz'EmbeddingOpParallelConfig.data_parallelcCs ||j_dSr:r;r6valuer7r7r8r/iscCs|jjSr:r4r1r<r7r7r8r1msz(EmbeddingOpParallelConfig.model_parallelcCs ||j_dSr:r?r=r7r7r8r1qscCs|jSr:) _vocab_emb_dpr<r7r7r8r3usz&EmbeddingOpParallelConfig.vocab_emb_dpcCst|d||_dS)Nr3)r r5r@r=r7r7r8r3ys cCs|jSr:)r4r<r7r7r8 dp_mp_config~sz&EmbeddingOpParallelConfig.dp_mp_configN)r.r.FFT) __name__ __module__ __qualname____doc__r9propertyr/setterr1r3rAr7r7r7r8r,Bs&       r,c@seZdZdZdddZeddZejddZed d Zejd d Zed d Z e jdd Z eddZ e jddZ eddZ e jddZ dS)r-a TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) FTcCs^t|dt|dt|dt|dt|d||_||_||_||_||_dS)N recompute!parallel_optimizer_comm_recomputemp_comm_recomputer2recompute_slice_activation)r r5 _recompute_select_recompute"_parallel_optimizer_comm_recompute_mp_comm_recompute_recompute_slice_activation)r6rHr2rIrJrKr7r7r8r9s     z#TransformerRecomputeConfig.__init__cCs|jSr:rLr<r7r7r8rHsz$TransformerRecomputeConfig.recomputecCst|d||_dS)NrH)r r5rLr=r7r7r8rHs cCs|jSr:)rMr<r7r7r8r2sz+TransformerRecomputeConfig.select_recomputecCst|d||_dS)Nr2)r r5rMr=r7r7r8r2s cCs|jSr:)rNr<r7r7r8rIsz>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> recompute_config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) >>> config=TransformerOpParallelConfig(data_parallel=1, model_parallel=1, recompute=recompute_config) r.FTc Cs^||_|j|_||_||_| |_t||| ||jd|_t||d|_t |||j||d|_ dS)N)r/r1r3r0r2)pipeline_stagemicro_batch_num)r/r1r2expert_parallelr0) rHr2r0optimizer_shardgradient_aggregation_groupr,_embed_dp_mp_configr _pp_configr _moe_config) r6r/r1rUrSrTrHr0rVrWr3r7r7r8r9s z$TransformerOpParallelConfig.__init__cCs|jSr:rQr<r7r7r8rHsz%TransformerOpParallelConfig.recomputecCsHt|ts*t|ts*tdt|jdt|tr>td||_dS)Nz=recompute must be a TransformerRecomputeConfig/bool, but got .zNTransformerRecomputeConfig is recommended as the recompute configuration type.) isinstancer-bool TypeErrortyperBloggerwarningrLr=r7r7r8rHs   cCs|jjSr:rXr3r<r7r7r8r3sz(TransformerOpParallelConfig.vocab_emb_dpcCs ||j_dSr:rbr=r7r7r8r3 scCs|jSr:)_gradient_aggregation_groupr<r7r7r8rW$sz6TransformerOpParallelConfig.gradient_aggregation_groupcCst|d||_dS)NrW)r check_positive_intrcr=r7r7r8rW(s cCs|jjSr:rYrTr<r7r7r8rT-sz+TransformerOpParallelConfig.micro_batch_numcCs ||j_dSr:rer=r7r7r8rT1scCs|jjSr:)rXr1r<r7r7r8r15sz*TransformerOpParallelConfig.model_parallelcCs||j_||j_dSr:)rXr1rZr=r7r7r8r19scCs|jjSr:)rXr/r<r7r7r8r/>sz)TransformerOpParallelConfig.data_parallelcCs||j_||j_dSr:)rXr/rZr=r7r7r8r/BscCs|jjSr:rZrUr<r7r7r8rUGsz+TransformerOpParallelConfig.expert_parallelcCs ||j_dSr:rfr=r7r7r8rUKscCs|jjSr:rYrSr<r7r7r8rSOsz*TransformerOpParallelConfig.pipeline_stagecCs ||j_dSr:rgr=r7r7r8rSSscCs|jSr:)_optimizer_shardr<r7r7r8rVWsz+TransformerOpParallelConfig.optimizer_shardcCs"t|d||_tj|ddS)NrV)enable_parallel_optimizer)r r5rhrset_auto_parallel_contextr=r7r7r8rV[s cCs|jSr:)rXr<r7r7r8embedding_dp_mp_configasz2TransformerOpParallelConfig.embedding_dp_mp_configcCs|jjSr:)rXrAr<r7r7r8rAesz(TransformerOpParallelConfig.dp_mp_configcCs|jSr:)rZr<r7r7r8moe_parallel_configisz/TransformerOpParallelConfig.moe_parallel_configN)rBrCrDrE$default_transformer_recompute_configr9rFrHrGr3rWrTr1r/rUrSrVrkrArlr7r7r7r8r+s^%                      r+c s~eZdZdZeedeejfvde e j e j e j e ejejgdeeegdddddejeffdd Zd d ZZS) r%a/ The multilayer perceptron with two linear layers with dropout applied at final output. The first linear will project the input dimension from hidden_size to ffn_hidden_size. The second linear will project the dimension from ffn_hidden_size to hidden_size. The first linear is sharded on the relative dimension, and the second linear is sharded on the output dimension. The overview process can be: .. math:: Dropout((xW_1+b_1)W_2 + b_2) where the :math:`W_1, W_2, b_1` and :math:`b_2` are trainable parameters. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see examples. Default: gelu. expert_num (int): The number of experts used in Linear. For the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. param_init_type (dtype.Number): The parameter initialization type. Should be mstype.float32 or mstype.float16. Default: mstype.float32. parallel_config (OpParallelConfig, MoEParallelConfig): The config of parallel setting, see `OpParallelConfig` or `MoEParallelConfig`. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: TypeError: `hidden_act` is not a string or nn.Cell. TypeError: `parallel_config` is not a subclass of OpParallelConfig. ValueError: `ffn_hidden_size` is not a multiple of the model parallel way. ValueError: `hidden_size` is not a multiple of the model parallel way. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import FeedForward >>> from mindspore import dtype as mstype >>> from mindspore import Tensor, nn >>> import mindspore.ops as ops >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 2 using custom hidden activation >>> class MyActivationNoShard(nn.Cell): ... def __init__(self): ... super(MyActivationNoShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationNoShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 3 using custom hidden activation with activation_shard >>> # If user wantss to run on the SEMI/AUTO parallel mode, the custom activation must provide >>> # a class function named activation_shard. It accepts the argument parallel_config (OpParallelConfig, >>> # MoEParallelConfig) and set the shard for the primitives used in the construct. >>> class MyActivationWithShard(nn.Cell): ... def __init__(self): ... super(MyActivationWithShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) ... def activation_shard(self, parallel_config): ... self.add.shard(((parallel_config.data_parallel, parallel_config.model_parallel), ())) >>> >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationWithShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) m_loggerkey no_warning) hidden_sizeffn_hidden_size dropout_rateparam_init_typeparallel_configgelur.Nc  stt||dus,t|ts>> import numpy as np >>> from mindformers.modules.transformer import AttentionMask >>> from mindspore import Tensor >>> mask = AttentionMask(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [0. 0. 0. 0]]] rn seq_lengthrvcstt|||_t|jdfdf|_t |_ t |jddf|jddff|_ t d|_tj||fd}tt|tj|_t|jddfdf|_dSNr.r7r.r.rr.r.r.)rr!r9rrNotEqualrr/ not_equalReshapereshape BatchMatMulmul ExpandDims expand_dimnponesrtrilrrlower_triangle_maskMulmultiplyr6rrvrrr7r8r9s zAttentionMask.__init__c Cstt|dtjtjg|jt| |dtj}t |}|dd|df}|d}| ||}| ||}| ||}| |jd}|||}|S)z$Forward process of the AttentionMask input_maskrr.r)rrrrrrrrrrShaperrrrr r6r input_shapeZ shape_right shape_left mask_leftZ mask_rightattention_maskZlower_traiangler7r7r8rs    zAttentionMask.constructrBrCrDrEr r`r rrrr rdrrrr9rrr7r7rr8r!hs'   r!csZeZdZdZeedeejfvde e j e e gddeffdd ZddZZS)r"a Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length). Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMaskHF >>> from mindspore import Tensor >>> mask = AttentionMaskHF(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [1. 1. 1. 1]]] rnrcstt|||_t|jdfdf|_t |_ t |jddf|jddff|_ t d|_tj||fd}tt|tj|_t|jddfdf|_dSr)rr"r9rrrrr/rrrrrrrrrrrrrrrrrrr7r8r9s zAttentionMaskHF.__init__c Cstt|dtjtjg|jtt |tj}t |}|dd|df}|d}| ||}| ||}| ||}| |jd}|||}|S)z&Forward process of the AttentionMaskHFrrr.r)rrrrrrrrrOnesLikerrrrrrrr7r7r8rs    zAttentionMaskHF.constructrr7r7rr8r"s&   r"cs`eZdZdZeedeejfvde e j e j e e gddedffdd ZddZZS) r#a The embedding lookup table from the 0-th dim of the parameter table. When the parallel_config.vocab_emb_dp is True and in the `AUTO_PARALLEL` mode, the embedding lookup will be trained by the data parallel way, as the parameters will be repeated on each device. If false, the embedding table will be sharded into n parts at the 0-th dimension of the embedding table, where the n is the model parallel way determined by `parallel_config.model_parallel` (EmbeddingOpParallelConfig). Note: When `AUTO_PARALLEL` or `SEMI_AUTO_PARALLEL` mode is enabled, this layer support only 2-d dimension inputs, as the shard is designed for 2d inputs. Args: vocab_size (int): Size of the dictionary of embeddings. embedding_size (int): The size of each embedding vector. parallel_config (EmbeddingOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. param_init (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: Tuple, a tuple contains (`output`, `embedding_table`) - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). - **embedding_table** (Tensor) - The embedding table with shape (vocab_size, embedding_size). Raises: ValueError: If the parallel_config.vocab_emb_dp is True, the vocab size is not a multiple of parallel_config.model_parallel ValueError: `vocab_size` is not a positive value. ValueError: `embedding_size` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import VocabEmbedding >>> from mindspore import Tensor >>> from mindspore import dtype as mstype >>> model = VocabEmbedding(vocab_size=30, embedding_size=30) >>> tensor = Tensor(np.ones((20, 15)), mstype.int32) >>> output, table = model(tensor) >>> print(output.shape) (20, 15, 30) >>> print(table.shape) (30, 30) rn) vocab_sizeembedding_sizervnormalcstt|t|||_||_tt||j|jgddd|_|j rvt d|j dff|_td|j dnf|j|jdkrtd |jd |jd t |jdf|j dff|_td|j d |jd dS)Nembedding_tableF)namerrr.zUsing z( data parallel for the embedding lookup.rz The vocab size of the embedding z6 must be a multiple of parallel_config.model_parallel r[z data parallel and z) model parallel for the embedding lookup.)rr#r9rrrrrrr3rGatherrr/gatherr`infor1r)r6rrrv param_initrr7r8r94s    zVocabEmbedding.__init__cCs8tt|dtjg|j||j|d}||jfS)N input_idsr) rrrrint32rrrr>)r6rrr7r7r8rMszVocabEmbedding.construct)rBrCrDrEr r`r rrrr rdrr,!default_embedding_parallel_configr9rrr7r7rr8r#s6  r#cseZdZdZeedeejfvde e j e j e j e j e j e j e ejejgde ejejgde ejejgdeegde jd ddejejejdeffdd Zdd d Zd d Zd dZdddZddZddZddZddZZS)r$a This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following .. math:: MultiHeadAttention(query, key, vector) = Concat(head_1, \dots, head_h)W^O where :math:`head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)`. The default is with a bias. if query, key and value tensor is same, then it will be self attention. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The sequence length of the query vector. tgt_seq_length(int): The sequence length of the key and value vector. hidden_size(int): The hidden size of the input. num_heads(int): The number of the heads. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. compute_dtype(dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. softmax_compute_type(dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. param_init_type(dtype.Number): The parameter initialization type of the module. Default mstype.float32. Should be mstype.float32 or mstype.float16. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **query_tensor** (Tensor) - The query vector with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **key_tensor** (Tensor) - The key vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **value_tensor** (Tensor) - The value vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, size_per_head, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, size_per_head). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import MultiHeadAttention >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> model = MultiHeadAttention(batch_size=None, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3) >>> from_tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 20, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> # We need to prepare the memory parameters for saving key and value states firstly. >>> model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3, use_past=True) >>> key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), mstype.float16) >>> value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), mstype.float16) >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> # Set is_first_iteration=True to generate the full memory states >>> model.add_flags_recursive(is_first_iteration=True) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> from_tensor = Tensor(np.ones((2, 1, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 1, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1, 20)), mstype.float16) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than the >>> # full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 1, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) rn) rr num_headssrc_seq_lengthtgt_seq_lengthattention_dropout_ratehidden_dropout_rate compute_dtypesoftmax_compute_typerurvuse_past皙?Fc sZ tt|tddv|_| j|_tt j t j fv|_ |rHt |tt j fvrt| ||_||_||_||_|dks|dkrtd||dks|dkrtd|||dkrtd|||| jdkrtd|| jd |_t||d || d |_|jj| jdfd f| j| jf| jdffd d |jj_t|_t|_ t!|_"||_#||j#|_$tj%dd|_&tj%dd|_'t(dg| d|_)t*|_+t,|_-t.|_/t0|_1t2|_3t(t45t45|j$|_6| |_7t8||_9t8||_:t;<=| |_>t;<=| |_?t@|_At!|_BtC|_Dt|||| d|_Et|||| d|_Ft|||| d|_G||_H| |_I|j7rtJK|"ddd} t(tJL| |ddftMjN|_O||_Pt(tJQtJjR|jP|jPfdtMjN|_StTd|_UtVd|_WtXd|_YtCd|_DtZd|_[t2d|_3t\d|_]t.d|_^t_d|_Lt`d|_at0d|_bn\t| ||_||_||_||_|dks|dkrtd||dks|dkrtd|||dkrtd|||| jdkr:td|| jd |_t||d || d |_|jj| jdfd f| j| jf| jdffd d |jj_t| jd| jdff|_t| j| jddff|_ t!|_"||_#||j#|_$tj%dd|_&tj%dd|_'t(dg| d|_)t*| j| jddf| j| jddff|_+t,| j| jddfdf|_-t.d | jdddff|_/t0| jdddfd f|_1t2| jdddf| j| jddff|_3t(t45t45|j$|_6| |_7t8||_9t8||_:|j9j9| jdff|j:j9| j| jddfft;<=| |_>|j>j>| j| jddfft;<=| |_?|j?j>| j| jdfft@|_At!|_BtC| jddff|_Dt|||| d|_E|jEj| jdf| jdff| j| jf| jffdt|||| d|_F|jFj| jdf| jdff| j| jf| jffdt|||| d|_G|jGj| jdf| jdff| j| jf| jffd||_H| |_I|j7rtJK|"ddd} t(tJL| |ddftMjN|_O||_Pt(tJQtJjR|jP|jPfdtMjN|_StTd|_UtVd|_WtXd|_YtCd|_DtZd|_[t2d|_3t\d|_]t.d|_^t_d|_Lt`d|_at0d|_b| jcr|j9j9| j| jdff|jj| j| jdfd f| j| jf| jdff| j| jdffd| jd rV|j+e|j/e|j3e|j e|jBe|j:e|jAe|j>j>e|j?edS)N device_target)Ascendrr.zuFor 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be in range [0, 1.0), but got the value : {}.zxFor 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be in range [0, 1.0), but got the value : {}.zFor 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple of 'num_heads', but got the hidden_size is {} and the num_heads is {}.zFor 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of 'parallel_config.model_parallel', but got the num_heads is {} and the parallel_config.model_parallel is {}.TF)rxryr{rrur)rrr)axisrg)r)rrurr.r.r.r.rr7)r)rrrr)rr7r7rrrZout_strategy_matmul)frr$r9r get_context _is_ascendr/rr rSEMI_AUTO_PARALLELrZis_parallel_moder rdrrrrr batch_sizerrr1is_first_iterationrrrrrr Transpose transposemerger_head_transposerrn_head size_per_headConcatZconcat_kZconcat_vr multiply_datar batch_matmulRealDivreal_divSubsubrrAddaddmathsqrt scale_factorrrr prob_dropoutrSoftmaxto_floatsoftmax softmax_3dr softmax_castsoftmax_reshaper expand_dimsdense1dense2dense3r softmax_dtyperarangetilerrrangerrrr StridedSliceslicerr ReduceSum reducesum LessEqual tensor_leEqualequalsub1TileLesslessmul1r0r2rH)r6rrrrrrrrrrrurrvZ seq_rangerr7r8r9s                  $            $         zMultiHeadAttention.__init__Nc Cs||||||||t|}||} ||||\}}}t|} t||j}t||j}t||j}||} ||} | |} | t | | | |j |j|jfd} | t | | | |j|j|jfd} | t | | | |j|j|jfd} |dur2tt|dkr2||d}| }| }|jrH|jrt||j|ddd|j}|| ||d}|| ||d}n|d}| |d}t|||j|j}||| ddd|jf||d}||| dd|jdf||d}|||} |||} | }| }t |j|j|jddf}||f}|| | | |}||}| |}t ||}t|| }||fS) z)Forward process of the MultiHeadAttentionrrr.r)rrrr.Nrr.rrrr.r.)! _check_inputsrr_get_batch_size_from_query_convert_to_2d_tensorrrrr r rr!_get_seq_length_under_incrementalrrrrrrrrrrviewrrr rrr_attnrr)r6 query_tensor key_tensor value_tensorrkey_past value_pastbatch_valid_length ori_shaper ori_dtypequeryrpr> key_present value_presentZvalid_length_vector valid_lengthZ current_key current_value layer_present attentionrr7r7r8rs          "         zMultiHeadAttention.constructcCsFtt|dkr8|jr|js$|js8t|d|jSt|dS)z$Get the batch size from query tensorrr)rrrrrr)r6r,r7r7r8r3s$z-MultiHeadAttention._get_batch_size_from_querycCs|jr|jsdS|S)zxReturn the length of the tensor. For the incremental prediction, the seq length for the input is 1. r.)rr)r6lengthr7r7r8r!:s z4MultiHeadAttention._get_seq_length_under_incrementalcCsRtt|dtjtjg|jtt|dtjtjg|jtt|dtjtjg|j|durtt|dtjtjg|jt|t}t|t} t|t} |du} |du} |du} t |j d|jd|| t |j d|jd| | t |j d|jd| | |j rNtt|dtjg|jtt|dtjg|jtt|dtj g|jd S) Check inputsr$r%r&Nrr'r(r)T rrrrrrrr\rrrr)r6r$r%r&rr'r(r)Z key_is_tensorZvalue_is_tensorbatch_valid_length_is_tensorZkey_is_defaultZvalue_is_defaultbatch_is_defaultr7r7r8rBs6   z MultiHeadAttention._check_inputscCsdt|}t|d|df}t|}t|d|df}t|}t|d|df}|||fS)z"convert a nd tensor to a 2d tensorr)rrr)r6r$r%r& query_shape key_shape value_shaper7r7r8r ^s   z(MultiHeadAttention._convert_to_2d_tensorcCs<||d}t|}d|d|df}|||}|S)z convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output rr)rrrr)r6rx_shape new_shapeZx_merger7r7r8 _merge_headsis   zMultiHeadAttention._merge_headsc Cs^|jr|jtjks|js$||}n6t|}||||dd|df}|||}|S)z For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. rr) rr rrrrrrr)r6attention_scoresattention_probsrr7r7r8_softmaxzs   zMultiHeadAttention._softmaxcCszt|jt|}|||}|||}|||}t|}|||j}|dur@|jr|j s| t | ||dt |ddd|jfddtjd} |t | tjd} t | d} t ||j| tj}||d}|tt d t|t|t|} || |j} || |}||} || |} || } || |}| |}|S) ay Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores N)rrrrrr.r)r.rrrr)g?)!rrrDTyperrrr rrrrrrrrrrrrrrrrrrtuple_to_arrayrrrrArr>)r6r,rpr>rfactorscorer+r? current_indexindexZ multiplu_outZadderr@Zweighted_valuesZattention_merger7r7r8r#sH              zMultiHeadAttention._attn)NNN)NNN)rBrCrDrEr r`r rrrr rdrrrrrrrr5rr9rrr!rr r>rAr#rr7r7rr8r$SsZ~    | U  r$cseZdZdZeedeejfvde e j e j e j e j e j e j e j eejejgdeejejgdeejejgdeeegde j d dddejejejddeef fdd Zdd d Zd dZZS)r(a Transformer Encoder Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. seq_length(int): The input sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, seq_length), (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoderLayer >>> from mindspore import Tensor >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) rn) rrrrsrrrpost_layernorm_residuallayernorm_compute_typerrurvrrFrwc stt||s| r t|||_ttjfvrt |||j dkr^t d ||j ||j dkr~t d ||j ||j dkrt d ||j t |||jdk|_| |_||_||_t|f| |_t|f| |_|jr|jn|}t|||||||| | | |d |_|jr<|js|jj?j |j!|j dfdf|j!|j f|j dff|j!|j dffd|jj@j@ |j!|j dffntAd|jBddS)NrzFor 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the 'parallel_config.model_parallel', but got the num_heads is {} and parallel_config.model_parallel is {}.zFor 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config. model_parallel is {}.zFor 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} and parallel_config. model_parallel is {}.r.) rrrrrrrrrrurrvrrrtrsrur moe_configrv)rrrtrsrurrvrrrr'rr(rrrrrrThe B only support sharding propagation or semi-auto parallel mode now.)Crr(r9r rdrr rrrr1rrrr|use_moerrrrrr layernorm1 layernorm2dpmpr$r2rrr%rHrrrr/radd_3drrrr'r(rrrrrrr9r:rrrzerosrr rrAssignassignr0rHr2 layer_normrr RuntimeErrorr)r6rrrrsrrrrrHrIrrurrrKrvZattention_parallel_configrrr7r8r96s\"      $       $      z TransformerEncoderLayer.__init__NTc Cs|||||t|}t|d|df}|jr:|}n ||}t||j}d}d}|jr| |j | |j t||j|j }| |j | |j t||j|j }t ||}t ||}||||||j |j |\} } |jr||| }n ||| }||} t| |j} d} |jr:|| \} } n || } d}d}|jr| \}}| |j ||j }| |j ||j }t ||}t ||}t | |} t | |} t|dkr:t| |} t| |} t||}|jr,|| | }t|d|df}||}t||}n ||| }n8|jrZ|| | }||}n ||| }t||}|jr|| | fS|| fSforward processrNr) _check_inputrrrrHrQrrrrWr'rr(dependr2rrRrPrrrrrT)r6rr init_resetr)r<input_x key_reset value_resetr2r1output_xaux_loss mlp_logit value_update key_updater-r.rr7r7r8r st  ""                  z!TransformerEncoderLayer.constructc Cstt|dtjtjg|j|durDtt|dtjtjg|jt|t}|du}t|t}|du}t |j d|jd||t |j d|jd|||j rtt|dtj g|jdS)r4rNrTr^r)r5) r6rrr^r)init_reset_is_tensorinit_reset_is_defaultr6r7r7r7r8r\^s   z$TransformerEncoderLayer._check_input)NTNrBrCrDrEr r`r rrrr rdrr5rrrrrrrrrr9rr\rr7r7rr8r(sNp    B Ur(cseZdZdZeedeejfvde e j e j e j e j e j e j e j e j eejejgdeejejgdeejejgdeeegde j d ddddejejejdeef fdd Zdd d Zd dZZS)r)a Transformer Decoder Layer. This is an implementation of the single layer of the transformer decoder layer, including self-attention, cross attention and feedward layer. When the encoder_output is None, the cross attention will not be effective. Args: hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, tgt_seq_length, hidden_size] or [batch_size * tgt_seq_length, hidden_size]. - **decoder_mask** (Tensor) - The attention mask for decoder with shape [batch_size, src_seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]. - **layer_present** (Tuple) - A tuple, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoderLayer >>> from mindspore import Tensor >>> model = TransformerDecoderLayer(batch_size=2, hidden_size=64, ffn_hidden_size=64, num_heads=2, ... src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(past[0].shape) (2, 2, 32, 10) >>> print(past[1].shape) (2, 2, 10, 32) >>> print(past[2].shape) (2, 2, 32, 20) >>> print(past[3].shape) (2, 2, 20, 32) rn) rrrrsrrrrrHrIrrurvrrFrwc sztt|t|||jdk|_|jr0|jn|}|s<| rFt|t t j fvrt |||j dkr~td||j ||j dkrtd||j ||j dkrtd||j | rtd|jd||_| |_| |_||_||_| |_||_t|f| |_t|f| |_t|||||||| | | |d |_t|||||||| | | |d |_t|f| |_|jrt|||| |||d |_ nt!||||| |d |_ | |_"t#$|_%t#$|_&t'j(|_)d|_*d|_+|jrvt#,-d |_.t#/-d |_0t#1-d |_2||}||||f|_3||||f|_4t5t6t7j8|j3d|j)dd|_*t5t6t7j8|j4d|j)dd|_+t#9-d|_:t#;-d|_nt t j fvrdt |||j dkrtd||j ||j dkrtd||j ||j dkr(td||j | r@td|jd||_| |_| |_||_||_| |_||_t|f| |_|j-|j?dfft|f| |_|j-|j?dfft|||||||| | | |d |_t|||||||| | | |d |_t|f| |_|j-|j?dff|jr>t|||| |||d |_ nt!||||| |d |_ | |_"t#$-|j?df|j?dff|_%t#$-|j?ddf|j?ddff|_&t'j(|_)d|_*d|_+|jr~t#,-d |_.t#/-d |_0t#1-d |_2||}||||f|_3||||f|_4t5t6t7j8|j3d|j)dd|_*t5t6t7j8|j4d|j)dd|_+t#9-d|_:t#;-d|_|j@rv|j%-|j?|j df|j?|j dff|j-|j?|j dff|j-|j?|j dff|jAjBr|jjCA|jsv|j jDj-|j?|j dfdf|j?|j f|j dff|j?|j dffd|j jEjE-|j?|j dffntFd|jddS)Nr.rzFor 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by 'parallel_config.model_parallel', but got the num_heads is {} and parallel_config.model_parallel is {}.zFor 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config.model_parallel is {}.zFor 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} and parallel_config.model_parallel is {}.rNz does not support use_past=True.) rrrrrrrrrrrurv) rrrrrrrrrrrurvrJ)rrrtrsrrurvrrrr'rLr(rrMrrrrO)Grr)r9rr|rPrSr rdr rrrr1rrrrrrrrrrrrrQrRr$r2cross_attentioncross_attention_layernormrrr%rHrrrrTrrrr'r(rrrrrrrr9r:rrrrUrr rrrVrWr/r0rHr2rXrrrY)r6rrrsrrrrrrrHrrIrrurrKrvZconfig_to_attentionrrr7r8r9s#              $      z TransformerDecoderLayer.__init__NTc Cs|||||||t|}t|d|df}||}t||j}d} d} |jr||j | |j t||j|j } ||j | |j t||j|j } t || }t || }| |||||j |j |\} } |jr||| } n ||| } d}|durz|| }t||j}t||j}||||||j |j |\}}| |7} |jrn|||} n || |} || }t||j}d}|jr||\}}n ||}d}d}|jr| \}}||j ||j }||j ||j }t || }t || }t ||}t ||}t|dkrt||}t||}t| |} |jrv|||}n || |}n.|jr|||}n || |}t||}|jr|| |fS|| fSrZ)r\rrrrQrrrrWr'rr(r]r2rHrrkrjrRrPrrrrrT)r6Z hidden_statsZ decoder_maskencoder_output memory_maskr^r)Z hidden_shaper_r`rar2r1rZ middle_outputZcross_attn_outputZcross_layer_presentrbrcrdrerfr-r.rr7r7r8rs  ""               z!TransformerDecoderLayer.constructc Cs tt|dtjtjg|j|durDtt|dtjtjg|j|durjtt|dtjtjg|j|durtt|dtjtjg|jt|t}|du}t|t} |du} t |j d|jd||t |j d|jd| | |j rtt|dtj g|jtt|dtj g|jdS) r4 hidden_statesNrrlrmTr^r)) rrrrrrrr\rrrbool_r) r6rnrrlrmr^r)rgrhr6r7r7r7r8r\+ s6    z$TransformerDecoderLayer._check_input)NNTNrir7r7rr8r)rsVZ    m ar)csfdd}|S)a A wrapper function of specifying pipeline stage and gradient aggregation fusion. If the total layer is not None, for example, set in the transformer model, the pipeline stage setting function will be `(layer_id + 0) // (total_layers / parallel_config.pipeline_stage)` for the encoder and, `(layer_id + offset) // (total_layers / parallel_config.pipeline_stage)` for the decoder, where `offset` is the layers in the encoder. c sr}||jkr(td|d|jt||jd}t||||jd}||_t||jd}||||dt|jtr|jr|j s|n0|jjr|jj s|jj }|j||jj |jj ddS)a Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs an offset, if there are other modules in the net. layers(int) - The total layers used for the model. zlayers z$ must be larger than pipeline stage r.)rIrJrKN) rSrmaxminrWset_comm_fusionr\rHr]r2rIrJrK) networklayer_idoffsetrvlayersZpp_disZpp_iddisZparalel_op_comm_compute total_layerr7r8!_set_parallel_configure_for_layerP s$     z;_get_lambda_func.._set_parallel_configure_for_layerr7)ryrzr7rxr8_get_lambda_funcG s $r{cseZdZdZeedeejfvde e j e j e j e j e j e j e j e j e jeejejgdeejejgdeejejgdeegde jdddddejejejdddeef fd d Zdd d ZZS)r&a! Transformer Encoder module with multi-layer stacked of `TransformerEncoderLayer`, including multihead self attention and feedforward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. num_layers(int): The layers of the `TransformerEncoderLayer` hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. seq_length(int): The seq_length of the input tensor. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default: 0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default: mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default: mstype.float32. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. offset(int): The initial layer index for the `encoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default: False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_states** (Tensor) - Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size]. - **attention_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple contains the Tensor the projected key and value vector with shape ((batch_size, num_heads, size_per_head, seq_length), and (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoder >>> from mindspore import Tensor >>> model = TransformerEncoder(batch_size=2, num_layers=2, hidden_size=8, ffn_hidden_size=64, ... seq_length=16, num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(len(past)) 2 >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoder(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, num_layers=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) rn)rrrrsr num_layersrurrrHrIrrurvrrrwFNrcstt|t|t|||jdk|_|s4|r>t||jrJ|j n|j }t t j fvrt|_tdtj|_||_t|_t|D]R}t||||||| | || | | |||d}|st}||||||d|j|qnt t j fvrtd|_tdtj|_td||_t|_t|D]V}t||||||| | || | | |||d}|sjt}||||||d|j|q4nt d|j!ddS) Nr.)rrrrsrrrrIrrrrHrurrKrvrtrvrurvr7r7For parallel mode, sharding propagation is recommended, you can use it by setting 'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, search_mode="sharding_propagation")' and 'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'rNrO)"rr&r9rrr|rPr rdrlrAr rrrrrrrrrcr|rCellListblocksrr(r{appendrr`rarYr)r6rr|rrrsrrrrrrHrIrru lambda_funcrurrKrvconfig_to_layeriblockrr7r8r9 s'         zTransformerEncoder.__init__Tc Csd}|jrZ|j}t|jD]4}|j|||||\}}} ||f}||| }q|||fSt|jD]&}|j|||||\}}||f}qd||fSr[r7rPrcrr|rr) r6rnrr^r) present_layer accum_lossrpresentrcr7r7r8ra s(      zTransformerEncoder.construct)TNrBrCrDrEr r`r rrrr rdcheck_non_negative_intrr5rrrrrr+rdefault_transformer_configr9rrr7r7rr8r&w sR{    Zr&cseZdZdZeedeejfvde e j e j e j e j e j e j e j e j e j e jeejejgdeejejgdeejejgdeegde jddddejejejddddeef fd d Zdd d ZZS)r'aS Transformer Decoder module with multi-layer stacked of `TransformerDecoderLayer`, including multihead self attention, cross attention and feedforward layer. Args: num_layers(int): The layers of the `TransformerDecoderLayer`. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. offset(int): The initial layer index for the `decoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size] - **attention_mask** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size] - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoder >>> from mindspore import Tensor >>> model = TransformerDecoder(batch_size=2, num_layers=1, hidden_size=64, ffn_hidden_size=64, ... num_heads=2, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(past)) 1 >>> print(past[0][0].shape) (2, 2, 32, 10) >>> print(past[0][1].shape) (2, 2, 10, 32) >>> print(past[0][2].shape) (2, 2, 32, 20) >>> print(past[0][3].shape) (2, 2, 20, 32) rn)rrrrsrr|rrurrrHrIrrurvrrFrwNrcstt|t||t||jdk|_|s4|r>t||jrJ|j n|j }t t j fvrt|_tdtj|_||_t|_t|D]T}t||||||| || | ||| | ||d}|st}||||||d|j|qnt t j fvrtd|_tdtj|_td||_t|_t|D]X}t||||||| || | ||| | ||d}|snt}||||||d|j|q6nt d|j!ddS) Nr.r})rrrrsrrrrrrIrrrrurHrKrvr~rrrNrO)"rr'r9rrr|rPr rdrlrAr rrrrrrrrrcr|rrrrr)r{rrr`rarYr)r6r|rrrrsrrrrrrHrIrrurrrrurKrvrrrrr7r8r9 s)         zTransformerDecoder.__init__Tc Csd}|jr^|j}t|jD]8} |j| ||||||\}} } || f}||| }q|||fSt|jD]*} |j| ||||||\}} || f}qh||fSrr) r6rnrrlrmr^r)rrrrrcr7r7r8rU s0      zTransformerDecoder.construct)NNTNrr7r7rr8r'y sXf    `r'cseZdZdZeedeejfvde e j e j e j e j e j e j e j e j e j e jeejejgdeejejgdeejejgdeegde jddddddddejejejd deeffd d Zdd dZZS)r*a# Transformer module including encoder and decoder. The difference with the original implements is the module use the residual addition before the layer normalization. And the default hidden act is `gelu`. The details can be found in `Attention is all you need `_. Note: This is an experimental interface that is subject to change or deletion. Args: hidden_size(int): The hidden size of the input. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The seq_length of the encoder's input tensor. tgt_seq_length(int): The seq_length of the decoder's input tensor. encoder_layers(int): The layers of the `TransformerEncoderLayer`. Default 3. decoder_layers(int): The layers of the `TransformerDecoderLayer`. Default 3. num_heads(int): The number of the heads. Default: 2. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. lambda_func: A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // ((encoder_layers + decoder_layers) / pipeline_stage)`. Default None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **encoder_inputs** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. - **encoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the encoder module. - **decoder_inputs** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], this should be none if the decoder layer is 0. - **decoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the decoder module. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The output of the encoder with shape [batch_size, seq_length, hidden_size], this should be none if the decoder layer is 0 or the user wants no mask. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `encoder_layer_present`, `decoder_layer_present`, `accum_loss`) - **output** (Tensor) - If there is only encoder, the output logit of the encoder layer. The shape is [batch, src_seq_length, hidden_size] or [batch * src_seq_length, hidden_size], if there are encoder and decoders, the output is from the decoder layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size]. - **encoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). - **decoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)), and the projected key and value vector in cross attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). If the decoder is not set, the returned value will be None. - **accum_loss** (Tensor) - A Tensor indicates an auxiliary loss to minimize the mean square of the data part routed to each expert, and only returned if the number of experts is greater than 1. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import Transformer >>> from mindspore import Tensor >>> model = Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, ... ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, en_past, de_past = model(encoder_input_value, encoder_input_mask, decoder_input_value, ... decoder_input_mask, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(en_past)) 1 >>> print(len(de_past)) 2 >>> print(en_past[0][0].shape) (2, 2, 32, 20) >>> print(en_past[0][1].shape) (2, 2, 20, 32) >>> print(de_past[0][0].shape) (2, 2, 32, 10) >>> print(de_past[0][1].shape) (2, 2, 10, 32) >>> print(de_past[0][2].shape) (2, 2, 32, 20) >>> print(de_past[0][3].shape) (2, 2, 20, 32) rn)rrrrsrencoder_layersdecoder_layersrrrrHrIrrurvrrrrrwFNcstt||s|r t|ttjfvrlt|||_ ||_ ||_ ||_ ||_ |dkrj|krnntd|d|d|dkr|dkr|rtd|jd|st||d}t|||jdk|_t|_td tj|_|dkr"t||||||| | | | || |||||d |_nd|_d|_|dkrt|||||||| | | | | |||||||d |_nxttjfvrt|||_ ||_ ||_ ||_ ||_ |dkr|krnntd|d|d|dkr|dkr|rtd|jdt d |st||d}t|||jdk|_t!d |_td tj|_|dkrt||||||| | | | || |||||d |_nd|_d|_|dkrt|||||||| | | | | |||||||d |_nt"d|jddS)Nrz(Transformer doest support encoder layer z and decoderlayer z, please use TransformerDecoderrNz9 with encoder and decoder does not support use_past=True.rxr.r})r|rrrrsrrrrrrIrrHrurrrKrv)r|rrrrsrrrrrrrHrIrrrrururKrvrrrO)#rr*r9r rdr rrrrrrrrrrrr{rr|rPrrrrrrrcr&encoderdecoderr'r`rarrY)r6rrrrsrrrrrrrrrHrIrrurrrKrvrr7r8r9 s'                   zTransformer.__init__TcCsd}d} d} d} |j} |jdurb|jrJ|||||\}} } || | } n|||||\}} |} |jdur|jr|||||||\}} }|| |} n|||||||\}} |} |jr| | | | fS| | | fS)zprocess processN)rcrrPrr)r6Zencoder_inputsZ encoder_masksZdecoder_inputsZ decoder_masksrmr^r)rlrZencoder_layer_presentZdecoder_layer_presentrZencoder_aux_lossZdecoder_outputZdecoder_aux_lossr7r7r8r s@      zTransformer.construct)NNNTNrr7r7rr8r*s s\|     r*)N)MrE __future__rrnumpyrmindspore.common.tensorrmindspore.common.parameterrmindspore.common.initializerr mindsporerrZmindspore.common.dtypecommonrr mindspore.opsrrr rmindspore.nn.cellr Zmindspore._checkparamr ImportError _checkparamr r`mindspore.parallel._utilsr mindspore.contextrZmindformers.modules.layersrrrrrrr2mindformers.modules.transformer.op_parallel_configrrrrrrZ#mindformers.modules.transformer.moerrrZmindformers.version_controlrmindformers.tools.loggerr __all__r,r-rmr+rrr%r!r"r#r$r(r)r{r&r'r*r7r7r7r8sr             $   ARwKJVv0X 0{pengcheng.mind-test/mindformers/modules/transformer/moe.py000066400000000000000000001132131471400134000244030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Mixture of Expert (MoE) structure. This is an experimental interface that is subject to change or deletion. """ from __future__ import absolute_import from __future__ import division import math import numpy as np from mindspore.common.tensor import Tensor import mindspore.common.dtype as mstype import mindspore.communication.management as D # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.ops.primitive import constexpr from mindspore.nn.cell import Cell from mindspore.nn.layer import Dense from mindspore.context import ParallelMode from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindformers.modules.transformer.op_parallel_config import default_moeparallel_config __all__ = [ "MoEConfig"] class MoEConfig: r""" The configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import MoEConfig >>> moe_config = MoEConfig(expert_num=4, capacity_factor=5.0, aux_loss_factor=0.05, num_experts_chosen=1, ... expert_group_size=64, group_wise_a2a=True, comp_comm_parallel=False, ... comp_comm_parallel_degree=2) """ def __init__(self, expert_num=1, capacity_factor=1.1, aux_loss_factor=0.05, num_experts_chosen=1, expert_group_size=None, group_wise_a2a=False, comp_comm_parallel=False, comp_comm_parallel_degree=2): Validator.check_positive_int(expert_num, "expert_num") Validator.check_positive_float(capacity_factor, "capacity_factor") Validator.check_positive_float(aux_loss_factor, "aux_loss_factor") Validator.check_positive_int(num_experts_chosen, "num_experts_chosen") Validator.check_bool(group_wise_a2a, "group_wise_a2a") Validator.check_bool(comp_comm_parallel, "comp_comm_parallel") Validator.check_positive_int(comp_comm_parallel_degree, "comp_comm_parallel_degree") if expert_group_size is not None: Validator.check_positive_int(expert_group_size, "expert_group_size") if capacity_factor < 1.0: raise ValueError(f"'capacity_factor' must be equal to or greater than 1.0, " f"but got {capacity_factor}.") if aux_loss_factor >= 1.0: raise ValueError(f"'aux_loss_factor' must be less than 1.0, " f"but got {aux_loss_factor}.") if num_experts_chosen > expert_num: raise ValueError(f"'num_experts_chosen' must not be larger than 'expert_num', " f"but got {num_experts_chosen}.") self.expert_num = expert_num self.capacity_factor = capacity_factor self.aux_loss_factor = aux_loss_factor self.num_experts_chosen = num_experts_chosen self.expert_group_size = expert_group_size self.group_wise_a2a = group_wise_a2a self.comp_comm_parallel = comp_comm_parallel self.comp_comm_parallel_degree = comp_comm_parallel_degree default_moe_config = MoEConfig() def _check_moe_config(moe_config=None, parallel_config=None): """ check if MoE with right configuration. """ if not isinstance(moe_config, MoEConfig): raise TypeError(f"'moe_config' must be an instance of MoEConfig, but got {type(moe_config).__name__}.") use_moe = (moe_config.expert_num > 1) if use_moe is False: return if moe_config.expert_num % parallel_config.expert_parallel != 0: raise ValueError(f"When using MoE, the 'expert_num' in {type(moe_config).__name__} must be a multiple " f"of 'expert_parallel' value in {type(parallel_config).__name__}, but got " f"{moe_config.expert_num} for 'expert_num' and {parallel_config.expert_parallel} for " f"'expert_parallel'.") device_num = D.get_group_size() if device_num % parallel_config.expert_parallel != 0: raise ValueError(f"device_num: {device_num} must be a multiple of expert_parallel: " f"{parallel_config.expert_parallel}.") if parallel_config.data_parallel % parallel_config.expert_parallel != 0: raise ValueError(f"data parallel: {parallel_config.data_parallel} must be a multiple of " f"expert_parallel: {parallel_config.expert_parallel} when using MoE.") if parallel_config.data_parallel * parallel_config.model_parallel > device_num: raise ValueError(f"The product of the data parallel: {parallel_config.data_parallel} and " f"model parallel: {parallel_config.model_parallel} " f"should be less than device_num: {device_num}.") @constexpr def calculate_expert_capacity(k, tokens_per_group, capacity_factor, expert_dim): return math.ceil(k * tokens_per_group * capacity_factor / expert_dim) class MoE(Cell): """ The mixture of experts (MoE) implementation. The implementation includes a router and a FeedForward layer. The router dispatches tokens to experts in FeedForward, then FeedForward does computation, and the final output is obtained by multiplying FeedForward's output and router's combine weight. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. Default: gelu. param_init_type (dtype.Number): The parameter initialization type. Can be dtype.float32 or dtype.float16. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(MoEParallelConfig): The parallel config for MoE, see `MoEParallelConfig`. Default `default_moeparallel_config`, an instance of `MoEParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size]`. """ def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act='gelu', param_init_type=mstype.float32, moe_config=default_moe_config, parallel_config=default_moeparallel_config): super(MoE, self).__init__() if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.hidden_size = hidden_size self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.aux_loss_factor = moe_config.aux_loss_factor self.num_experts_chosen = moe_config.num_experts_chosen self.expert_group_size = moe_config.expert_group_size self.dp_group = parallel_config.data_parallel self.dp = parallel_config.data_parallel self.ep = parallel_config.expert_parallel self.mp = parallel_config.model_parallel self.comp_comm_parallel = moe_config.comp_comm_parallel self.comp_comm_parallel_degree = moe_config.comp_comm_parallel_degree self.group_wise_a2a = moe_config.group_wise_a2a if not (self.mp > 1 and self.dp == self.ep): self.group_wise_a2a = False from mindformers.modules.transformer import FeedForward self.ffn = FeedForward(hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, expert_num=self.expert_dim, expert_group_size=self.expert_group_size, param_init_type=param_init_type, parallel_config=parallel_config) self.reshape = P.Reshape() self.shape = P.Shape() self.transpose_2dim = P.Transpose().shard(((self.dp, 1),)) self.transpose_3dim = P.Transpose().shard(((self.dp, 1, 1),)) self.transpose_4dim = P.Transpose().shard(((1, self.dp, 1, 1),)) self.transpose_4dim_dp = P.Transpose().shard(((1, 1, self.dp, 1),)) self.batch_mm = P.BatchMatMul().shard(((self.dp, 1, 1), (self.dp, 1, 1))) self.batch_mm2 = P.BatchMatMul().shard(((self.dp, 1, 1), (self.dp, 1, 1))) self.mul = P.Mul() self.router = Router(d_model=hidden_size, moe_config=moe_config, routing_policy=None, training=True, parallel_config=parallel_config) self.cast = P.Cast() self.concat = P.Concat(3).shard(tuple((self.dp, 1, 1, 1) for _ in range(self.comp_comm_parallel_degree))) self.concat_dp = P.Concat(2).shard(((1, self.dp, 1, 1), (1, self.dp, 1, 1))) self.split = P.Split(axis=2, output_num=self.comp_comm_parallel_degree).shard(((1, self.dp, 1, 1),)) self.stride_slice = P.StridedSlice().shard(((self.dp, 1, 1, 1),)) self.stride_slice_dp = P.StridedSlice().shard(((1, self.dp, 1, 1),)) self.stride_slice_ep = P.StridedSlice().shard(((self.ep, 1, 1, 1),)) self.stride_slice_dp_mp = P.StridedSlice().shard(((1, self.dp, self.mp, 1),)) self.stride_slice_ep_mp = P.StridedSlice().shard(((self.ep, 1, self.mp, 1),)) else: self.hidden_size = hidden_size self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.aux_loss_factor = moe_config.aux_loss_factor self.num_experts_chosen = moe_config.num_experts_chosen self.dp_group = parallel_config.data_parallel self.dp = parallel_config.data_parallel self.ep = parallel_config.expert_parallel self.mp = parallel_config.model_parallel self.comp_comm_parallel = moe_config.comp_comm_parallel self.comp_comm_parallel_degree = moe_config.comp_comm_parallel_degree self.group_wise_a2a = moe_config.group_wise_a2a if not (self.mp > 1 and self.dp == self.ep): self.group_wise_a2a = False from mindformers.modules.transformer import FeedForward self.ffn = FeedForward(hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, expert_num=self.expert_dim, param_init_type=param_init_type, parallel_config=parallel_config) self.reshape = P.Reshape() self.shape = P.Shape() self.transpose_2dim = P.Transpose().shard(((self.dp, 1),)) self.transpose_3dim = P.Transpose().shard(((self.dp, 1, 1),)) self.transpose_4dim = P.Transpose().shard(((1, self.dp, 1, 1),)) self.transpose_4dim_dp = P.Transpose().shard(((1, 1, self.dp, 1),)) self.batch_mm = P.BatchMatMul().shard(((self.dp, 1, 1), (self.dp, 1, 1))) self.batch_mm2 = P.BatchMatMul().shard(((self.dp, 1, 1), (self.dp, 1, 1))) self.mul = P.Mul().shard(((), ())) self.router = Router(d_model=hidden_size, moe_config=moe_config, routing_policy=None, training=True, parallel_config=parallel_config) self.cast = P.Cast() self.concat = P.Concat(3).shard(tuple((self.dp, 1, 1, 1) for _ in range(self.comp_comm_parallel_degree))) self.concat_dp = P.Concat(2).shard(((1, self.dp, 1, 1), (1, self.dp, 1, 1))) self.split = P.Split(axis=2, output_num=self.comp_comm_parallel_degree).shard(((1, self.dp, 1, 1),)) self.stride_slice = P.StridedSlice().shard(((self.dp, 1, 1, 1),)) self.stride_slice_dp = P.StridedSlice().shard(((1, self.dp, 1, 1),)) self.stride_slice_ep = P.StridedSlice().shard(((self.ep, 1, 1, 1),)) self.stride_slice_dp_mp = P.StridedSlice().shard(((1, self.dp, self.mp, 1),)) self.stride_slice_ep_mp = P.StridedSlice().shard(((self.ep, 1, self.mp, 1),)) def ffn_infer(self, expert_input, capacity): """ Computing the FFN. """ pad_size = 0 if self.group_wise_a2a: # If capacity can't div by mp, pad for mp shard. if capacity%self.mp != 0: pad_size = self.mp-(capacity%self.mp) if pad_size != 0: capacity += pad_size pad_tensor = self.stride_slice_dp(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, pad_size, self.hidden_size), (1, 1, 1, 1)) expert_input = self.concat_dp((expert_input, pad_tensor)) # capacity shard by mp expert_input = self.stride_slice_dp_mp(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # group-wise alltoall expert_input = self.stride_slice_ep_mp(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # allgather expert_input = self.stride_slice_ep(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) expert_input = self.reshape(expert_input, (self.expert_dim * self.dp_group * capacity, self.hidden_size)) # expert_output's shape: (self.expert_dim, self.dp_group*expert_capacity, self.hidden_size) expert_output = self.ffn(expert_input) expert_output = self.reshape(expert_output, (self.expert_dim, self.dp_group, capacity, self.hidden_size)) if self.group_wise_a2a: # capacity shard by mp expert_output = self.stride_slice_ep_mp(expert_output, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # group-wise alltoall expert_output = self.stride_slice_dp_mp(expert_output, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # allgather expert_output = self.stride_slice_dp(expert_output, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # Slice capacity back to org shape. if pad_size != 0: capacity -= pad_size expert_output = self.stride_slice_dp(expert_output, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # expert_output's shape: (self.dp_group, self.hidden_size, self.expert_dim, expert_capacity) expert_output = self.transpose_4dim(expert_output, (1, 3, 0, 2)) return expert_output def ffn_parallel_infer(self, expert_input, capacity): """ Split and overlap FFN compute and communication. """ # Pad capacity for comp_comm_parallel_degree split. pad_size = 0 if capacity%self.comp_comm_parallel_degree != 0: pad_size = self.comp_comm_parallel_degree-(capacity%self.comp_comm_parallel_degree) capacity += pad_size pad_tensor = self.stride_slice_dp(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, pad_size, self.hidden_size), (1, 1, 1, 1)) expert_input = self.concat_dp((expert_input, pad_tensor)) sub_capacity = capacity // self.comp_comm_parallel_degree output_list = [] for sub_expert_input in self.split(expert_input): sub_expert_output = self.ffn_infer(sub_expert_input, sub_capacity) output_list.append(sub_expert_output) expert_output = self.concat(output_list) # Slice capacity back to org shape. if pad_size != 0: capacity -= pad_size expert_output = self.stride_slice(expert_output, (0, 0, 0, 0), (self.dp_group, self.hidden_size, self.expert_dim, capacity), (1, 1, 1, 1)) return expert_output def construct(self, input_tensor): """forward process""" input_shape = F.shape(input_tensor) input_tensor = self.reshape(input_tensor, (-1, self.hidden_size)) bs_and_dmodel = self.shape(input_tensor) tokens_per_group = bs_and_dmodel[0] // self.dp_group input_tensor = self.reshape(input_tensor, (self.dp_group, tokens_per_group, self.hidden_size)) expert_capacity = calculate_expert_capacity(self.num_experts_chosen, tokens_per_group, self.capacity_factor, self.expert_dim) # dispatch_tensor's shape: (self.dp_group, tokens_per_group, self.expert_dim, expert_capacity) # combine_tensor's shape: (self.dp_group, tokens_per_group, self.expert_dim, expert_capacity) dispatch_tensor, combine_tensor, aux_loss = self.router(input_tensor) # after transpose, input_tensor's shape: (self.dp_group, self.hidden_size, tokens_per_group) input_tensor = self.transpose_3dim(input_tensor, (0, 2, 1)) dispatch_tensor = self.reshape(dispatch_tensor, (self.dp_group, tokens_per_group, self.expert_dim * expert_capacity)) dispatch_tensor = self.cast(dispatch_tensor, F.dtype(input_tensor)) # expert_input's shape: (self.dp_group, self.hidden_size, self.expert_dim * expert_capacity) expert_input = self.batch_mm(input_tensor, dispatch_tensor) expert_input = self.reshape(expert_input, (self.dp_group, self.hidden_size, self.expert_dim, expert_capacity)) # The following four ops are to implement transpose(expert_input, (2, 0, 3, 1)), for that a single transpose # has bad performance expert_input = self.reshape(expert_input, (self.dp_group * self.hidden_size, self.expert_dim * expert_capacity)) expert_input = self.transpose_2dim(expert_input, (1, 0)) expert_input = self.reshape(expert_input, (self.expert_dim, expert_capacity, self.dp_group, self.hidden_size)) # expert_input's shape: (self.expert_dim, self.dp_group, expert_capacity, self.hidden_size) expert_input = self.transpose_4dim_dp(expert_input, (0, 2, 1, 3)) # expert_output's shape: (self.dp_group, self.hidden_size, self.expert_dim, expert_capacity) if self.comp_comm_parallel: expert_output = self.ffn_parallel_infer(expert_input, expert_capacity) else: expert_output = self.ffn_infer(expert_input, expert_capacity) expert_output = self.reshape(expert_output, (self.dp_group, self.hidden_size, self.expert_dim * expert_capacity)) combine_tensor = self.reshape(combine_tensor, (self.dp_group, tokens_per_group, self.expert_dim * expert_capacity)) # combine_tensor's shape: (self.dp_group, self.expert_dim*expert_capacity, tokens_per_group) combine_tensor = self.transpose_3dim(combine_tensor, (0, 2, 1)) combine_tensor = self.cast(combine_tensor, F.dtype(expert_output)) # combined_output's shape: (self.dp_group, self.hidden_size, tokens_per_group) combined_output = self.batch_mm2(expert_output, combine_tensor) # combined_output's shape: (self.dp_group, tokens_per_group, self.hidden_size) combined_output = self.transpose_3dim(combined_output, (0, 2, 1)) combined_output = self.reshape(combined_output, (bs_and_dmodel[0], bs_and_dmodel[1])) combined_output = self.reshape(combined_output, input_shape) aux_loss = self.mul(self.aux_loss_factor, aux_loss) return combined_output, aux_loss class Router(Cell): r""" A router backbone used to calculate logits of each token, which should be cascaded by router implementations mapping tokens to experts. when moe_config.num_experts_chosen = 1, use top1 routing; when moe_config.num_experts_chosen > 1, use topk routing Args: d_model (int): The hidden size of each token. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). routing_policy: The policy of mapping tokens to experts. Default: topkRouter training (bool): The value indicating whether is in training phase. parallel_config: The parallel-related configuration. Inputs: - **input_tensor** (Tensor) - Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, hidden\_size)`. Outputs: Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim)`. """ def __init__(self, d_model, moe_config, routing_policy=None, training=True, parallel_config=None): super(Router, self).__init__() dp = parallel_config.data_parallel self.d_model = d_model self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.num_experts_chosen = moe_config.num_experts_chosen self.training = training self.routing_policy = routing_policy self.noisy_policy = None # candidate: ["jitter", "rsample", "None"] self.noisy_epsilon = 1e-2 self.noise = Tensor(np.random.uniform(1 - self.noisy_epsilon, 1 + self.noisy_epsilon, (d_model,))) self.dense = Dense(in_channels=self.d_model, out_channels=self.expert_dim, has_bias=False) self.dense.matmul.shard(((dp, 1), (1, 1))) self.mul = P.Mul() self.cast = P.Cast() if self.routing_policy is None: self.router = TopkRouter(d_model=d_model, moe_config=moe_config, training=training, parallel_config=parallel_config) else: self.router = routing_policy if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.mul.shard(((dp, 1, 1), (dp,))) def construct(self, input_tensor): input_tensor = self.cast(input_tensor, mstype.float32) if self.noisy_policy == "jitter" and self.training: # Here, we temporarily implement the multiplicative jitter this way, # for the lack of UniforReal parallel operator. input_tensor = self.mul(input_tensor, self.noise) router_logits = self.dense(input_tensor) return self.router(router_logits) class TopkRouter(Cell): r""" A router implementation which maps each tokens to the topk expert. Args: d_model (int): The hidden size of each token. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). training (bool): The value indicating whether is in training phase. config: The parallel-related configuration. Inputs: - **input_tensor** (Tensor) - Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, hidden\_size)`. Outputs: Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim, expert\_capacity)`, Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim, expert\_capacity)`, Tensor of shape :math:`(1)`. """ def __init__(self, d_model, moe_config, training=True, parallel_config=None): super(TopkRouter, self).__init__() if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): dp = parallel_config.data_parallel self.d_model = d_model self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.training = training self.dp_group = dp self.noisy_policy = None self.cast = P.Cast() self.reshape = P.Reshape() self.shape = P.Shape() self.softmax = P.Softmax(axis=-1) self.argmax = P.ArgMaxWithValue(axis=-1, keep_dims=False) self.num_experts_chosen = moe_config.num_experts_chosen self.onehot = P.OneHot() self.onehot2 = P.OneHot() self.onehot3 = P.OneHot() self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_mean = P.ReduceMean(keep_dims=False) self.reduce_mean2 = P.ReduceMean(keep_dims=False) self.reduce_mean3 = P.ReduceMean(keep_dims=False) self.mul = P.Mul() self.mul2 = P.Mul() self.mul3 = P.Mul() self.mul4 = P.Mul() self.mul5 = P.Mul() self.mul6 = P.Mul() self.mul7 = P.Mul() self.mul8 = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mul9 = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.not_equal = P.NotEqual() self.div1 = P.RealDiv() self.div2 = P.RealDiv() self.add = P.Add() self.add1 = P.Add() self.add2 = P.Add() self.add3 = P.Add() self.add4 = P.Add() self.sub = P.Sub() self.cumsum = P.CumSum(exclusive=True) self.less = P.Less() self.reduce_sum = P.ReduceSum(keep_dims=False) self.reduce_sum_keep = P.ReduceSum(keep_dims=True) self.reduce_sum_keep2 = P.ReduceSum(keep_dims=True) self.expand = P.ExpandDims() self.expand2 = P.ExpandDims() self.add_scala = P.Add() self.init_loss = Tensor(0.0, mstype.float32) else: dp = parallel_config.data_parallel self.d_model = d_model self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.training = training self.dp_group = dp self.noisy_policy = None self.cast = P.Cast() self.reshape = P.Reshape() self.shape = P.Shape() self.softmax = P.Softmax(axis=-1).shard(((dp, 1, 1,),)) self.argmax = P.ArgMaxWithValue(axis=-1, keep_dims=False).shard(((dp, 1, 1),)) self.num_experts_chosen = moe_config.num_experts_chosen self.onehot = P.OneHot().shard(((dp, 1, 1), (), ())) self.onehot2 = P.OneHot().shard(((dp, 1, 1), (), ())) self.onehot3 = P.OneHot().shard(((dp, 1, 1, 1), (), ())) self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_mean = P.ReduceMean(keep_dims=False).shard(((dp, 1, 1),)) self.reduce_mean2 = P.ReduceMean(keep_dims=False).shard(((dp, 1, 1),)) self.reduce_mean3 = P.ReduceMean(keep_dims=False).shard(((dp, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.mul2 = P.Mul().shard(((), ())) self.mul3 = P.Mul().shard(((), ())) self.mul4 = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mul5 = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mul6 = P.Mul().shard(((dp, 1), (dp, 1))) self.mul7 = P.Mul().shard(((dp, 1), (dp, 1))) self.mul8 = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mul9 = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.not_equal = P.NotEqual().shard(((dp, 1, 1, 1), ())) self.div1 = P.RealDiv().shard(((dp, 1, 1), (dp, 1, 1))) self.div2 = P.RealDiv().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.add = P.Add().shard(((dp, 1, 1), (dp, 1, 1))) self.add1 = P.Add().shard(((dp, 1, 1), ())) self.add2 = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.add3 = P.Add().shard(((dp, 1), (dp, 1))) self.add4 = P.Add().shard(((dp, 1, 1, 1), ())) self.sub = P.Sub().shard(((), (dp, 1, 1))) self.cumsum = P.CumSum(exclusive=True).shard(((dp, 1, 1),)) self.less = P.Less().shard(((dp, 1, 1), ())) self.reduce_sum = P.ReduceSum(keep_dims=False).shard(((dp, 1, 1),)) self.reduce_sum_keep = P.ReduceSum(keep_dims=True).shard(((dp, 1, 1),)) self.reduce_sum_keep2 = P.ReduceSum(keep_dims=True).shard(((dp, 1, 1, 1),)) self.expand = P.ExpandDims().shard(((dp, 1),)) self.expand2 = P.ExpandDims().shard(((dp, 1, 1),)) self.add_scala = P.Add().shard(((), ())) self.init_loss = Tensor(0.0, mstype.float32) def construct(self, router_logits): """forward process""" router_logits_shape = self.shape(router_logits) router_logits = self.reshape(router_logits, (-1, router_logits_shape[-1])) logits_shape = self.shape(router_logits) tokens_per_group = logits_shape[0] // self.dp_group expert_capacity = calculate_expert_capacity(self.num_experts_chosen, tokens_per_group, self.capacity_factor, self.expert_dim) router_logits = self.reshape(router_logits, (self.dp_group, tokens_per_group, self.expert_dim)) accum_expert_mask = 0 accum_expert_gate = 0 loss = self.init_loss mask_count = 0 accum_combine_tensor = 0 # Probabilities for each token of what expert is should be sent to router_prob = self.softmax(router_logits) for expert_chosen_index in range(self.num_experts_chosen): # for each token, set the router_prob of the selected experts to zero router_prob = self.mul4(router_prob, self.sub(self.on_value, accum_expert_mask)) # shape is : (dp_group, tokens_per_group) expert_index, expert_gate = self.argmax(router_prob) # expert_mask's shape: (dp_group, tokens_per_group, self.expert_dim) expert_mask = self.onehot(expert_index, self.expert_dim, self.on_value, self.off_value) # renormalize the rest prob to be of sum 1 router_prob_normal = self.div1(router_prob, self.add1(self.reduce_sum_keep(router_prob, -1), 1e-9)) # the balance loss is computed at each routing step loss = self.add_scala(loss, self._auxiliary_loss(expert_mask, router_prob_normal)) output = self._maskout_overflowed_tokens(expert_mask, expert_capacity, expert_gate, mask_count, expert_chosen_index) expert_mask, expert_gate, expert_mask_flat, position_in_expert = output[0], output[1], output[2], output[3] accum_expert_mask = self.add(accum_expert_mask, expert_mask) accum_expert_gate = self.add3(accum_expert_gate, expert_gate) mask_count = self.add(mask_count, self.reduce_sum_keep(expert_mask, 1)) # combine_tensor's shape: (dp_group, tokens_per_group) combine_tensor = self.mul7(expert_gate, expert_mask_flat) # combine_tensor's shape: (dp_group, tokens_per_group, self.expert_dim) combine_tensor = self.mul8(self.expand(combine_tensor, -1), self.onehot2(expert_index, self.expert_dim, self.on_value, self.off_value)) # combine_tensor's shape: (dp_group, tokens_per_group, self.expert_dim, self.expert_capacity) combine_tensor = self.mul9(self.expand2(combine_tensor, -1), self.onehot3(self.cast(position_in_expert, mstype.int32), expert_capacity, self.on_value, self.off_value)) accum_combine_tensor = self.add2(accum_combine_tensor, combine_tensor) # expert weights normalization combine_tensor_sum = self.reduce_sum_keep2(self.reduce_sum_keep2(accum_combine_tensor, -1), -2) accum_combine_tensor = self.div2(accum_combine_tensor, self.add4(combine_tensor_sum, 1e-9)) # dispatch_tensor is of boolean type. Here, using NotEqual instead of Cast, for that 'Cast to bool' has # bad performance dispatch_tensor = self.not_equal(accum_combine_tensor, 0.0) return dispatch_tensor, accum_combine_tensor, loss def _auxiliary_loss(self, expert_mask, router_prob): """ Computing the load balance loss. """ # density_1's shape: (dp_group, self.expert_dim) density_1 = self.reduce_mean(expert_mask, 1) # density_1_proxy's shape: (dp_group, self.expert_dim) density_1_proxy = self.reduce_mean2(router_prob, 1) loss = self.mul(density_1, density_1_proxy) loss = self.reduce_mean3(loss) loss = self.mul3(self.mul2(loss, self.expert_dim), self.expert_dim) return loss def _maskout_overflowed_tokens(self, expert_mask, expert_capacity, expert_gate, last_num, expert_chosen_index): """ Keeping only the tokens that fit within expert_capacity. """ cumsum = self.cumsum(expert_mask, 1) if expert_chosen_index > 0: cumsum = self.add(cumsum, last_num) # position_in_expert's shape: (dp_group, tokens_per_group, self.expert_dim) position_in_expert = self.mul4(cumsum, expert_mask) less_result = self.less(position_in_expert, expert_capacity) # expert_mask's shape: (dp_group, tokens_per_group, self.expert_dim) expert_mask = self.mul5(less_result, expert_mask) # expert_mask_flat's shape: (dp_group, tokens_per_group) expert_mask_flat = self.reduce_sum(expert_mask, -1) # Mask out the experts that have overflowed the expert_capacity. # expert_gate's shape: (dp_group, tokens_per_group) expert_gate = self.mul6(expert_gate, expert_mask_flat) output = (expert_mask, expert_gate, expert_mask_flat, position_in_expert) return output pengcheng.mind-test/mindformers/modules/transformer/op_parallel_config.py000066400000000000000000000204411471400134000274420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Parallel Config for the Parallel Training This is an experimental interface that is subject to change and/or deletion. """ from __future__ import absolute_import # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import context import mindspore.communication.management as D from mindspore.context import ParallelMode from mindspore.parallel._utils import _get_parallel_mode from mindspore import log as logger __all__ = [ "OpParallelConfig" ] class _Config: r""" A basic class of the configure""" def __str__(self): info = "[ParallelConfig]" + '\n' for k, v in self.__dict__.items(): var_info = "{}:{}\n".format(k, v) info += var_info return info class MoEParallelConfig(_Config): r""" MoEParallelConfig for MoE structure, which includes setting data parallel, model parallel and expert parallel. Args: data_parallel (int): The data parallel way. Default: 1 model_parallel (int): The model parallel way. Default: 1 expert_parallel (int): The expert parallel way. Default: 1 Supported Platforms: ``Ascend`` """ def __init__(self, data_parallel=1, model_parallel=1, expert_parallel=1, use_seq_parallel=False, select_recompute=False): Validator.check_positive_int(data_parallel, "data_parallel") Validator.check_positive_int(model_parallel, "model_parallel") Validator.check_positive_int(expert_parallel, "expert_parallel") Validator.check_bool(use_seq_parallel, "use_seq_parallel") Validator.check_bool(select_recompute, "select_recompute") self._dpmp = OpParallelConfig(data_parallel=data_parallel, model_parallel=model_parallel, use_seq_parallel=use_seq_parallel, select_recompute=select_recompute) self.expert_parallel = expert_parallel self.use_seq_parallel = use_seq_parallel self.select_recompute = select_recompute @property def data_parallel(self): return self._dpmp.data_parallel @data_parallel.setter def data_parallel(self, value): Validator.check_positive_int(value, "data_parallel") self._dpmp.data_parallel = value @property def model_parallel(self): return self._dpmp.model_parallel @model_parallel.setter def model_parallel(self, value): Validator.check_positive_int(value, "model_parallel") self._dpmp.model_parallel = value @property def expert_parallel(self): return self._expert_parallel @expert_parallel.setter def expert_parallel(self, value): Validator.check_positive_int(value, "expert_parallel") self._expert_parallel = value @property def dpmp(self): """ Get the configuration for dpmp """ return self._dpmp class OpParallelConfig(_Config): r""" OpParallelConfig for the setting data parallel and model parallel. Args: data_parallel (int): The data parallel way. Default: 1 model_parallel (int): The model parallel way. Default: 1 Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules import OpParallelConfig >>> config=OpParallelConfig(data_parallel=1, model_parallel=1) """ def __init__(self, data_parallel=1, model_parallel=1, use_seq_parallel=False, select_recompute=False): Validator.check_positive_int(data_parallel, "data_parallel") Validator.check_positive_int(model_parallel, "model_parallel") Validator.check_bool(use_seq_parallel, "use_seq_parallel") Validator.check_bool(select_recompute, "select_recompute") self.data_parallel = data_parallel self.model_parallel = model_parallel self.use_seq_parallel = use_seq_parallel self.select_recompute = select_recompute @property def data_parallel(self): return self._data_parallel @data_parallel.setter def data_parallel(self, value): Validator.check_positive_int(value, "data_parallel") self._data_parallel = value @property def model_parallel(self): return self._model_parallel @model_parallel.setter def model_parallel(self, value): Validator.check_positive_int(value, "model_parallel") self._model_parallel = value class _PipeLineConfig(_Config): r""" PPConfig for the setting data parallel, model parallel Args: pipeline_stage (int): The number of the pipeline stages. Default: 1 micro_batch_num (int): The model parallel way. Default: 1 Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> config=_PipeLineConfig(pipeline_stage=1, micro_batch_num=1) """ def __init__(self, pipeline_stage=1, micro_batch_num=1): Validator.check_positive_int(pipeline_stage, "pipeline_stage") Validator.check_positive_int(micro_batch_num, "micro_batch_num") self.pipeline_stage = pipeline_stage self.micro_batch_num = micro_batch_num @property def pipeline_stage(self): return self._pipeline_stage @pipeline_stage.setter def pipeline_stage(self, value): Validator.check_positive_int(value, "pipeline_stage") self._pipeline_stage = value context.set_auto_parallel_context(pipeline_stages=value) @property def micro_batch_num(self): return self._micro_batch_num @micro_batch_num.setter def micro_batch_num(self, value): Validator.check_positive_int(value, "micro_batch_num") self._micro_batch_num = value # In case the user doesn't pass a config as args. default_dpmp_config = OpParallelConfig() default_moeparallel_config = MoEParallelConfig() def _check_config(config): """ Check if micro_batch_num >= pipeline_stage """ # the config pipeline_stage is same with context.pipeline_stage pipeline_stage = context.get_auto_parallel_context("pipeline_stages") if hasattr(config, 'pipeline_stage') and pipeline_stage != config.pipeline_stage: raise ValueError( f"The pipeline stage {pipeline_stage} in auto_parallel_context is not equal to the pipeline_stage " f"{config.pipeline_stage}" f" in the config.") # make sure the following is in auto parallel mode is_auto_parallel = _get_parallel_mode() in (ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) if not is_auto_parallel: return device_num = D.get_group_size() optimizer_shard = context.get_auto_parallel_context("enable_parallel_optimizer") if config.data_parallel * config.model_parallel * pipeline_stage > device_num: raise ValueError(f"The product of the data parallel {config.data_parallel}, " f"model parallel {config.model_parallel} " f"pipeline stages {pipeline_stage} " f"should be less than device_num {device_num}.") # the config optimizer_shard is same with context.optimizer_shard if hasattr(config, "optimizer_shard") and optimizer_shard and optimizer_shard != config.optimizer_shard: logger.warning(f"The optimizer shard {optimizer_shard} in auto_parallel_context is not equal to the" f" optimizer_shard {config.optimizer_shard} in the OpParallelConfig. Please check the " f"optimizer_shard to make them consistent.") pengcheng.mind-test/mindformers/modules/transformer/transformer.py000066400000000000000000005737651471400134000262130ustar00rootroot00000000000000# Copyright 2021-2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Transformer Networks. This is interface that is subject to change or deletion. """ from __future__ import absolute_import import math import numpy as np from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore.common.initializer import initializer from mindspore import nn from mindspore import context import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.nn.cell import Cell try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import log as logger from mindspore.parallel._utils import _get_parallel_mode from mindspore.context import ParallelMode from mindformers.modules.layers import LayerNorm, Linear, \ _args_type_validator_check, _valid_type_checks, _valid_value_checks, \ _check_past_none_input_none, _check_input_dtype from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, _PipeLineConfig, OpParallelConfig, \ _Config, _check_config, MoEParallelConfig from mindformers.modules.transformer.moe import default_moe_config, MoE, _check_moe_config from mindformers.version_control import get_dropout from mindformers.tools.logger import _LogActionOnce __all__ = [ "AttentionMask", "AttentionMaskHF", "VocabEmbedding", "MultiHeadAttention", "FeedForward", "TransformerEncoder", "TransformerDecoder", "TransformerEncoderLayer", "TransformerDecoderLayer", "Transformer", "TransformerOpParallelConfig", "EmbeddingOpParallelConfig", "TransformerRecomputeConfig"] class EmbeddingOpParallelConfig(_Config): r""" The parallel config of :class:`VocabEmbedding` for the setting data parallel or model parallel for the embedding table. Args: data_parallel(int): The data parallel way. The input data will be sliced into n parts for embedding layer according to this value. Default: 1. model_parallel(int): The model parallel way. The embedding table parameters will be sliced at 0-th axis according to the model parallel way. Default: 1. vocab_emb_dp(bool): Shard embedding in model parallel or data parallel. If True, the embedding lookup will be a data parallel style training and model_parallel value will be ignored. If false, the embedding table will be sharded into n parts at the 0-th dimension row slice of the embedding table, where the n is the model parallel way determined by this parameter. Default: True Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import EmbeddingOpParallelConfig >>> config=EmbeddingOpParallelConfig(data_parallel=1, model_parallel=1, vocab_emb_dp=True) """ def __init__(self, data_parallel=1, model_parallel=1, use_seq_parallel=False, select_recompute=False, vocab_emb_dp=True): self._dp_mp_config = OpParallelConfig(data_parallel=data_parallel, use_seq_parallel=use_seq_parallel, model_parallel=model_parallel, select_recompute=select_recompute) Validator.check_bool(vocab_emb_dp, "vocab_emb_dp") self.vocab_emb_dp = vocab_emb_dp self.use_seq_parallel = use_seq_parallel self.select_recompute = select_recompute @property def data_parallel(self): return self._dp_mp_config.data_parallel @data_parallel.setter def data_parallel(self, value): self._dp_mp_config.data_parallel = value @property def model_parallel(self): return self._dp_mp_config.model_parallel @model_parallel.setter def model_parallel(self, value): self._dp_mp_config.model_parallel = value @property def vocab_emb_dp(self): return self._vocab_emb_dp @vocab_emb_dp.setter def vocab_emb_dp(self, value): Validator.check_bool(value, "vocab_emb_dp") self._vocab_emb_dp = value @property def dp_mp_config(self): return self._dp_mp_config class TransformerRecomputeConfig(_Config): r""" TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) """ def __init__(self, recompute=False, select_recompute=False, parallel_optimizer_comm_recompute=False, mp_comm_recompute=True, recompute_slice_activation=False): Validator.check_bool(recompute, "recompute") Validator.check_bool(parallel_optimizer_comm_recompute, "parallel_optimizer_comm_recompute") Validator.check_bool(mp_comm_recompute, "mp_comm_recompute") Validator.check_bool(select_recompute, "select_recompute") Validator.check_bool(recompute_slice_activation, "recompute_slice_activation") self._recompute = recompute self._select_recompute = select_recompute self._parallel_optimizer_comm_recompute = parallel_optimizer_comm_recompute self._mp_comm_recompute = mp_comm_recompute self._recompute_slice_activation = recompute_slice_activation @property def recompute(self): return self._recompute @recompute.setter def recompute(self, value): Validator.check_bool(value, "recompute") self._recompute = value @property def select_recompute(self): return self._select_recompute @select_recompute.setter def select_recompute(self, value): Validator.check_bool(value, "select_recompute") self._select_recompute = value @property def parallel_optimizer_comm_recompute(self): return self._parallel_optimizer_comm_recompute @parallel_optimizer_comm_recompute.setter def parallel_optimizer_comm_recompute(self, value): Validator.check_bool(value, "parallel_optimizer_comm_recompute") self._parallel_optimizer_comm_recompute = value @property def mp_comm_recompute(self): return self._mp_comm_recompute @mp_comm_recompute.setter def mp_comm_recompute(self, value): Validator.check_bool(value, "mp_comm_recompute") self._mp_comm_recompute = value @property def recompute_slice_activation(self): return self._recompute_slice_activation @recompute_slice_activation.setter def recompute_slice_activation(self, value): Validator.check_bool(value, "recompute_slice_activation") self._recompute_slice_activation = value default_transformer_recompute_config = TransformerRecomputeConfig() class TransformerOpParallelConfig(_Config): r""" TransformerOpParallelConfig for setting parallel configuration, such as the data parallel and model parallel. Note: Except the recompute argument, other arguments will **not** be effective when the user doesn't set auto_parallel_context to `SEMI_AUTO_PARALLEL` or `AUTO_PARALLEL`. The micro_batch_num must be greater than or equal to pipeline_stage when training. The data_parallel\*model_parallel \*pipeline_stage must be equal or less equal to the device. When setting the pipeline stage and optimizer_shard, the config will overwrite the auto_parallel_context. When given the 8 devices and the data_parallel is 1 and model_parallel is 1, the calculation will be repeated on each device. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. recompute (Union[TransformerRecomputeConfig, bool]): The configuration of recomputation for the transformer block. Default: An instance of TransformerRecomputeConfig with default values. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> recompute_config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) >>> config=TransformerOpParallelConfig(data_parallel=1, model_parallel=1, recompute=recompute_config) """ def __init__(self, data_parallel=1, model_parallel=1, expert_parallel=1, pipeline_stage=1, micro_batch_num=1, recompute=default_transformer_recompute_config, use_seq_parallel=False, optimizer_shard=False, gradient_aggregation_group=4, vocab_emb_dp=True): self.recompute = recompute self.select_recompute = recompute.select_recompute self.use_seq_parallel = use_seq_parallel self.optimizer_shard = optimizer_shard self.gradient_aggregation_group = gradient_aggregation_group self._embed_dp_mp_config = EmbeddingOpParallelConfig( data_parallel=data_parallel, model_parallel=model_parallel, vocab_emb_dp=vocab_emb_dp, use_seq_parallel=use_seq_parallel, select_recompute=recompute.select_recompute) self._pp_config = _PipeLineConfig(pipeline_stage=pipeline_stage, micro_batch_num=micro_batch_num) self._moe_config = MoEParallelConfig( data_parallel=data_parallel, model_parallel=model_parallel, select_recompute=recompute.select_recompute, expert_parallel=expert_parallel, use_seq_parallel=use_seq_parallel) @property def recompute(self): return self._recompute @recompute.setter def recompute(self, value): if not isinstance(value, TransformerRecomputeConfig) and not isinstance(value, bool): raise TypeError(f"recompute must be a TransformerRecomputeConfig/bool, but got {type(value).__name__}.") if isinstance(value, bool): logger.warning(f"TransformerRecomputeConfig is recommended as the recompute configuration type.") self._recompute = value @property def vocab_emb_dp(self): return self._embed_dp_mp_config.vocab_emb_dp @vocab_emb_dp.setter def vocab_emb_dp(self, value): self._embed_dp_mp_config.vocab_emb_dp = value @property def gradient_aggregation_group(self): return self._gradient_aggregation_group @gradient_aggregation_group.setter def gradient_aggregation_group(self, value): Validator.check_positive_int(value, "gradient_aggregation_group") self._gradient_aggregation_group = value @property def micro_batch_num(self): return self._pp_config.micro_batch_num @micro_batch_num.setter def micro_batch_num(self, value): self._pp_config.micro_batch_num = value @property def model_parallel(self): return self._embed_dp_mp_config.model_parallel @model_parallel.setter def model_parallel(self, value): self._embed_dp_mp_config.model_parallel = value self._moe_config.model_parallel = value @property def data_parallel(self): return self._embed_dp_mp_config.data_parallel @data_parallel.setter def data_parallel(self, value): self._embed_dp_mp_config.data_parallel = value self._moe_config.data_parallel = value @property def expert_parallel(self): return self._moe_config.expert_parallel @expert_parallel.setter def expert_parallel(self, value): self._moe_config.expert_parallel = value @property def pipeline_stage(self): return self._pp_config.pipeline_stage @pipeline_stage.setter def pipeline_stage(self, value): self._pp_config.pipeline_stage = value @property def optimizer_shard(self): return self._optimizer_shard @optimizer_shard.setter def optimizer_shard(self, value): Validator.check_bool(value, "optimizer_shard") self._optimizer_shard = value context.set_auto_parallel_context(enable_parallel_optimizer=value) @property def embedding_dp_mp_config(self): return self._embed_dp_mp_config @property def dp_mp_config(self): return self._embed_dp_mp_config.dp_mp_config @property def moe_parallel_config(self): return self._moe_config default_transformer_config = TransformerOpParallelConfig() default_embedding_parallel_config = EmbeddingOpParallelConfig() class FeedForward(Cell): r""" The multilayer perceptron with two linear layers with dropout applied at final output. The first linear will project the input dimension from hidden_size to ffn_hidden_size. The second linear will project the dimension from ffn_hidden_size to hidden_size. The first linear is sharded on the relative dimension, and the second linear is sharded on the output dimension. The overview process can be: .. math:: Dropout((xW_1+b_1)W_2 + b_2) where the :math:`W_1, W_2, b_1` and :math:`b_2` are trainable parameters. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see examples. Default: gelu. expert_num (int): The number of experts used in Linear. For the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. param_init_type (dtype.Number): The parameter initialization type. Should be mstype.float32 or mstype.float16. Default: mstype.float32. parallel_config (OpParallelConfig, MoEParallelConfig): The config of parallel setting, see `OpParallelConfig` or `MoEParallelConfig`. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: TypeError: `hidden_act` is not a string or nn.Cell. TypeError: `parallel_config` is not a subclass of OpParallelConfig. ValueError: `ffn_hidden_size` is not a multiple of the model parallel way. ValueError: `hidden_size` is not a multiple of the model parallel way. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import FeedForward >>> from mindspore import dtype as mstype >>> from mindspore import Tensor, nn >>> import mindspore.ops as ops >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 2 using custom hidden activation >>> class MyActivationNoShard(nn.Cell): ... def __init__(self): ... super(MyActivationNoShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationNoShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 3 using custom hidden activation with activation_shard >>> # If user wantss to run on the SEMI/AUTO parallel mode, the custom activation must provide >>> # a class function named activation_shard. It accepts the argument parallel_config (OpParallelConfig, >>> # MoEParallelConfig) and set the shard for the primitives used in the construct. >>> class MyActivationWithShard(nn.Cell): ... def __init__(self): ... super(MyActivationWithShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) ... def activation_shard(self, parallel_config): ... self.add.shard(((parallel_config.data_parallel, parallel_config.model_parallel), ())) >>> >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationWithShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) """ @_LogActionOnce(m_logger=logger, key='FeedForward', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, dropout_rate=Validator.check_non_negative_float, param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "FeedForward")) def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act='gelu', expert_num=1, expert_group_size=None, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(FeedForward, self).__init__() if hidden_act is None or not (isinstance(hidden_act, str) or issubclass(hidden_act, nn.Cell)): raise TypeError(f"For FeedForward cell, the hidden_act should str type or nn.Cell type, " f"but got {hidden_act}.") if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = parallel_config.data_parallel // ep if ffn_hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'FeedForward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1))) self.projection.bias.parallel_optimizer = False self.dropout = get_dropout(dropout_rate) self.dropout_3d = get_dropout(dropout_rate) self.dropout_4d = get_dropout(dropout_rate) self.cast = P.Cast() else: _check_config(parallel_config) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = parallel_config.data_parallel // ep if ffn_hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'FeedForward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.mapping.shard(strategy_matmul=((dp, ep, 1, 1), (ep, 1, mp)), strategy_bias=((dp, ep, 1, mp), (1, ep, 1, mp)), strategy_activation=((dp, ep, 1, mp),)) else: self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1)), strategy_bias=((dp, ep, 1, 1), (1, ep, 1, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = get_dropout(dropout_rate) self.dropout_3d = get_dropout(dropout_rate) self.dropout_4d = get_dropout(dropout_rate) self.dropout.dropout.shard(((dp, 1),)) self.dropout_3d.dropout.shard(((dp, 1, 1),)) self.dropout_4d.dropout.shard(((dp, ep, 1, 1),)) self.cast = P.Cast() def construct(self, x): """Forward process of the FeedForward""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) else: output = self.dropout_4d(output) return output class AttentionMask(Cell): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length) with 1 and 0, where 1 indicates the current position is a valid token, otherwise not. Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMask >>> from mindspore import Tensor >>> mask = AttentionMask(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [0. 0. 0. 0]]] """ @_LogActionOnce(m_logger=logger, key='AttentionMask', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(seq_length=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "AttentionMask")) def __init__(self, seq_length, parallel_config=default_dpmp_config): super(AttentionMask, self).__init__() self.seq_length = seq_length self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.reshape = P.Reshape() self.mul = P.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.expand_dim = P.ExpandDims().shard(((1, 1),)) ones = np.ones(shape=(seq_length, seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (1, 1, 1))) def construct(self, input_mask): """Forward process of the AttentionMask""" _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) input_mask = P.Cast()(self.not_equal(input_mask, 0), mstype.float16) input_shape = P.Shape()(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) # Mask the padded inputs mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.mul(mask_left, mask_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask class AttentionMaskHF(Cell): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length). Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMaskHF >>> from mindspore import Tensor >>> mask = AttentionMaskHF(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [1. 1. 1. 1]]] """ @_LogActionOnce(m_logger=logger, key='AttentionMaskHF', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(seq_length=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "AttentionMaskHF")) def __init__(self, seq_length, parallel_config=default_dpmp_config): super(AttentionMaskHF, self).__init__() self.seq_length = seq_length self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.reshape = P.Reshape() self.mul = P.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.expand_dim = P.ExpandDims().shard(((1, 1),)) ones = np.ones(shape=(seq_length, seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (1, 1, 1))) def construct(self, input_mask): """Forward process of the AttentionMaskHF""" _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) input_mask = P.Cast()(P.OnesLike()(input_mask), mstype.float16) input_shape = P.Shape()(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) # Mask the padded inputs mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.mul(mask_left, mask_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask class VocabEmbedding(Cell): """ The embedding lookup table from the 0-th dim of the parameter table. When the parallel_config.vocab_emb_dp is True and in the `AUTO_PARALLEL` mode, the embedding lookup will be trained by the data parallel way, as the parameters will be repeated on each device. If false, the embedding table will be sharded into n parts at the 0-th dimension of the embedding table, where the n is the model parallel way determined by `parallel_config.model_parallel` (EmbeddingOpParallelConfig). Note: When `AUTO_PARALLEL` or `SEMI_AUTO_PARALLEL` mode is enabled, this layer support only 2-d dimension inputs, as the shard is designed for 2d inputs. Args: vocab_size (int): Size of the dictionary of embeddings. embedding_size (int): The size of each embedding vector. parallel_config (EmbeddingOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. param_init (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: Tuple, a tuple contains (`output`, `embedding_table`) - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). - **embedding_table** (Tensor) - The embedding table with shape (vocab_size, embedding_size). Raises: ValueError: If the parallel_config.vocab_emb_dp is True, the vocab size is not a multiple of parallel_config.model_parallel ValueError: `vocab_size` is not a positive value. ValueError: `embedding_size` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import VocabEmbedding >>> from mindspore import Tensor >>> from mindspore import dtype as mstype >>> model = VocabEmbedding(vocab_size=30, embedding_size=30) >>> tensor = Tensor(np.ones((20, 15)), mstype.int32) >>> output, table = model(tensor) >>> print(output.shape) (20, 15, 30) >>> print(table.shape) (30, 30) """ @_LogActionOnce(m_logger=logger, key='VocabEmbedding', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(vocab_size=Validator.check_positive_int, embedding_size=Validator.check_positive_int, parallel_config=_valid_type_checks([EmbeddingOpParallelConfig], "VocabEmbedding")) def __init__(self, vocab_size, embedding_size, parallel_config=default_embedding_parallel_config, param_init='normal'): super(VocabEmbedding, self).__init__() _check_config(parallel_config) self.vocab_size = vocab_size self.embedding_size = embedding_size self.embedding_table = Parameter(initializer(param_init, [self.vocab_size, self.embedding_size]), name='embedding_table', parallel_optimizer=False) if parallel_config.vocab_emb_dp: self.gather = P.Gather().shard(((1, 1), (parallel_config.data_parallel, 1))) logger.info(f"Using {parallel_config.data_parallel} data parallel for the embedding lookup.") else: if self.vocab_size % parallel_config.model_parallel != 0: raise ValueError(f"The vocab size of the embedding {self.vocab_size} must be a " f"multiple of parallel_config.model_parallel {parallel_config.model_parallel}.") self.gather = P.Gather().shard(((parallel_config.model_parallel, 1), (parallel_config.data_parallel, 1))) logger.info(f"Using {parallel_config.data_parallel} data parallel and {parallel_config.model_parallel} " f"model parallel for the embedding lookup.") def construct(self, input_ids): _check_input_dtype(F.dtype(input_ids), "input_ids", [mstype.int32], self.cls_name) output = self.gather(self.embedding_table, input_ids, 0) return output, self.embedding_table.value() class MultiHeadAttention(Cell): r""" This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following .. math:: MultiHeadAttention(query, key, vector) = Concat(head_1, \dots, head_h)W^O where :math:`head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)`. The default is with a bias. if query, key and value tensor is same, then it will be self attention. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The sequence length of the query vector. tgt_seq_length(int): The sequence length of the key and value vector. hidden_size(int): The hidden size of the input. num_heads(int): The number of the heads. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. compute_dtype(dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. softmax_compute_type(dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. param_init_type(dtype.Number): The parameter initialization type of the module. Default mstype.float32. Should be mstype.float32 or mstype.float16. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **query_tensor** (Tensor) - The query vector with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **key_tensor** (Tensor) - The key vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **value_tensor** (Tensor) - The value vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, size_per_head, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, size_per_head). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import MultiHeadAttention >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> model = MultiHeadAttention(batch_size=None, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3) >>> from_tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 20, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> # We need to prepare the memory parameters for saving key and value states firstly. >>> model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3, use_past=True) >>> key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), mstype.float16) >>> value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), mstype.float16) >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> # Set is_first_iteration=True to generate the full memory states >>> model.add_flags_recursive(is_first_iteration=True) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> from_tensor = Tensor(np.ones((2, 1, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 1, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1, 20)), mstype.float16) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than the >>> # full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 1, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) """ @_LogActionOnce(m_logger=logger, key='MultiHeadAttention', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), parallel_config=_valid_type_checks([OpParallelConfig], "MultiHeadAttention"), use_past=Validator.check_bool) def __init__(self, batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, hidden_dropout_rate=0.1, attention_dropout_rate=0.1, compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, use_past=False, parallel_config=default_dpmp_config): super(MultiHeadAttention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] self.dp = parallel_config.data_parallel self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) if batch_size: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul() self.real_div = P.RealDiv() self.sub = P.Sub() self.mul = P.Mul() self.add = P.Add() # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.use_past = use_past self.dropout = get_dropout(hidden_dropout_rate) self.prob_dropout = get_dropout(attention_dropout_rate) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_cast = P.Cast() self.softmax_reshape = P.Reshape() self.expand_dims = P.ExpandDims() # Query self.dense1 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) # Key self.dense2 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) # Value self.dense3 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) else: _check_config(parallel_config) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.use_past = use_past self.dropout = get_dropout(hidden_dropout_rate) self.prob_dropout = get_dropout(attention_dropout_rate) self.dropout.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout.dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) self.softmax_cast = P.Cast() self.softmax_reshape = P.Reshape() self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.dropout.dropout.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.select_recompute: # _attn中涉及的关键算子使用重计算逻辑，通常配合序列并行使用，会有较好的性能提升效果 self.batch_matmul.recompute() self.sub.recompute() self.add.recompute() self.merger_head_transpose.recompute() self.softmax_reshape.recompute() self.prob_dropout.recompute() self.softmax_cast.recompute() self.softmax.softmax.recompute() self.softmax_3d.recompute() def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" self._check_inputs(query_tensor, key_tensor, value_tensor, attention_mask, key_past, value_past, batch_valid_length) ori_shape = F.shape(query_tensor) batch_size = self._get_batch_size_from_query(query_tensor) query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) ori_dtype = F.dtype(query_tensor) query_tensor = F.cast(query_tensor, self.dtype) key_tensor = F.cast(key_tensor, self.dtype) value_tensor = F.cast(value_tensor, self.dtype) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, self._get_seq_length_under_incremental(self.src_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if attention_mask is not None and len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # key and value for current token(s) key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key, self.expand_dims(valid_length_vector, 2)) value_present = self.mul1(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = F.cast(self.equal(valid_length, self.range), self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) current_value = self.mul1(self.tile(value, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add(key_past, current_key) value = self.add(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value attention_mask = F.reshape(self.attention_mask, (self.seq_length, self.seq_length, 1, 1)) layer_present = (key_present, value_present) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, attention_mask) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) output = F.cast(output, ori_dtype) return output, layer_present def _get_batch_size_from_query(self, query): r"""Get the batch size from query tensor""" # For the incremental prediction, the seq length for the input is 1. if len(F.shape(query)) == 2 and ((self.use_past and self.is_first_iteration) or (not self.use_past)): return F.shape(query)[0] // self.src_seq_length return F.shape(query)[0] def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _check_inputs(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): r"""Check inputs""" _check_input_dtype(F.dtype(query_tensor), "query_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(key_tensor), "key_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_tensor), "value_tensor", [mstype.float32, mstype.float16], self.cls_name) if attention_mask is not None: _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) key_is_tensor = isinstance(key_past, Tensor) value_is_tensor = isinstance(value_past, Tensor) batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) key_is_default = key_past is None value_is_default = value_past is None batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "key_past", self.cls_name, None, key_is_tensor, key_is_default) _check_past_none_input_none(self.use_past, "value_past", self.cls_name, None, value_is_tensor, value_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(key_past), "key_past", [mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_past), "value_past", [mstype.float16], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( self.softmax_reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = self.softmax_reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] factor = P.Cast()(self.scale_factor, P.DType()(query)) query = self.real_div(query, factor) key = self.real_div(key, factor) score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) attention_scores = self.softmax_cast(score, self.softmax_dtype) # for input size of (bs, 1) namely the second graph, # the shape of attention_mask matrix should be (bs, 1, 1, seq_length) if attention_mask is not None: if self.use_past and not self.is_first_iteration: # Calculate the current total token current_index = self.reducesum(F.cast(self.not_equal(self.slice(key, (0, 0, 0, 0), (F.shape(query)[0], 1, 1, self.seq_length), (1, 1, 1, 1)), 0), mstype.float32), (1, 2, 3)) # Get the precise position index index = self.sub1(F.cast(current_index, mstype.int32), 1) index = F.reshape(index, (-1, 1, 1)) # Calculate the attention_mask matrix via the position index attention_mask = F.cast(self.tensor_le(self.range, index), mstype.int32) attention_mask = self.expand_dims(attention_mask, 2) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(attention_scores)), P.Cast()(attention_mask, P.DType()(attention_scores))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, attention_scores) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = self.softmax_cast(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge class TransformerEncoderLayer(Cell): r""" Transformer Encoder Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. seq_length(int): The input sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, seq_length), (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoderLayer >>> from mindspore import Tensor >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) """ @_LogActionOnce(m_logger=logger, key='TransformerEncoderLayer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "TransformerEncoderLayer"), use_past=Validator.check_bool) def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', use_past=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerEncoderLayer, self).__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) self.batch_size = batch_size if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, parallel_config=attention_parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, parallel_config=attention_parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.add.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1), (parallel_config.data_parallel * parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.recompute.select_recompute: # 此处会消耗较大内存，开启后会损失一部分计算性能 self.layernorm2.layer_norm.recompute() if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, x, input_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(x, input_mask, init_reset, batch_valid_length) x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) if self.post_layernorm_residual: input_x = x else: input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) output = F.reshape(output, (-1, x_shape[-1])) output = self.layernorm1(output) output = F.reshape(output, x_shape) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) output = self.layernorm1(output) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present def _check_input(self, x, input_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) if input_mask is not None: _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True class TransformerDecoderLayer(Cell): r""" Transformer Decoder Layer. This is an implementation of the single layer of the transformer decoder layer, including self-attention, cross attention and feedward layer. When the encoder_output is None, the cross attention will not be effective. Args: hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, tgt_seq_length, hidden_size] or [batch_size * tgt_seq_length, hidden_size]. - **decoder_mask** (Tensor) - The attention mask for decoder with shape [batch_size, src_seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]. - **layer_present** (Tuple) - A tuple, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoderLayer >>> from mindspore import Tensor >>> model = TransformerDecoderLayer(batch_size=2, hidden_size=64, ffn_hidden_size=64, num_heads=2, ... src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(past[0].shape) (2, 2, 32, 10) >>> print(past[1].shape) (2, 2, 10, 32) >>> print(past[2].shape) (2, 2, 32, 20) >>> print(past[3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='TransformerDecoderLayer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "TransformerDecoderLayer"), use_past=Validator.check_bool) def __init__(self, hidden_size, ffn_hidden_size, num_heads, batch_size, src_seq_length, tgt_seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, use_past=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerDecoderLayer, self).__init__() _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) config_to_attention = parallel_config.dpmp if self.use_moe else parallel_config if batch_size or use_past: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}." .format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=config_to_attention) # Cross attention with the output of encoder as memory tensor self.cross_attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, param_init_type=param_init_type, parallel_config=config_to_attention) self.cross_attention_layernorm = LayerNorm((hidden_size,)).to_float( layernorm_compute_type) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add() self.add_3d = P.Add() self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}." .format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) self.attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=config_to_attention) # Cross attention with the output of encoder as memory tensor self.cross_attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, param_init_type=param_init_type, parallel_config=config_to_attention) self.cross_attention_layernorm = LayerNorm((hidden_size,)).to_float( layernorm_compute_type) self.cross_attention_layernorm.shard(((parallel_config.data_parallel, 1),)) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.add.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1), (parallel_config.data_parallel * parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.recompute.select_recompute: # 此处会消耗较大内存，开启后会损失一部分计算性能 self.layernorm2.layer_norm.recompute() if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_stats, decoder_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(hidden_stats, decoder_mask, encoder_output, memory_mask, init_reset, batch_valid_length) # the returned shape is [bs, seq_length, embedding_size] or [bs * seq_length, embedding_size] hidden_shape = F.shape(hidden_stats) hidden_stats = F.reshape(hidden_stats, (-1, hidden_shape[-1])) input_x = self.layernorm1(hidden_stats) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, decoder_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(hidden_stats, attention) middle_output = None if encoder_output is not None: middle_output = self.cross_attention_layernorm(x) middle_output = F.cast(middle_output, self.dtype) encoder_output = F.cast(encoder_output, self.dtype) cross_attn_output, cross_layer_present = self.cross_attention(middle_output, encoder_output, encoder_output, memory_mask, self.key_past, self.value_past, batch_valid_length) layer_present += cross_layer_present if self.post_layernorm_residual: x = self.add(middle_output, cross_attn_output) else: x = self.add(x, cross_attn_output) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(hidden_shape) == 3: output_x = P.Reshape()(output_x, hidden_shape) mlp_logit = P.Reshape()(mlp_logit, hidden_shape) x = P.Reshape()(x, hidden_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, hidden_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present def _check_input(self, hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(hidden_states), "hidden_states", [mstype.float32, mstype.float16], self.cls_name) if attention_mask is not None: _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) if encoder_output is not None: _check_input_dtype(F.dtype(encoder_output), "encoder_output", [mstype.float32, mstype.float16], self.cls_name) if memory_mask is not None: _check_input_dtype(F.dtype(memory_mask), "memory_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(init_reset), "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _get_lambda_func(total_layer=None): r""" A wrapper function of specifying pipeline stage and gradient aggregation fusion. If the total layer is not None, for example, set in the transformer model, the pipeline stage setting function will be `(layer_id + 0) // (total_layers / parallel_config.pipeline_stage)` for the encoder and, `(layer_id + offset) // (total_layers / parallel_config.pipeline_stage)` for the decoder, where `offset` is the layers in the encoder. """ def _set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs an offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # override the layers if total_layer: layers = total_layer # Used for the pipeline's stages setting if layers < parallel_config.pipeline_stage: raise ValueError(f"layers {layers} must be larger than pipeline stage {parallel_config.pipeline_stage}") pp_dis = max(layers // parallel_config.pipeline_stage, 1) # the pipeline stage must be in [0, parallel_config.pipeline_stage - 1] pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(layers // parallel_config.gradient_aggregation_group, 1) network.set_comm_fusion((layer_id + offset) // dis + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not parallel_config.select_recompute: network.recompute() else: if parallel_config.recompute.recompute and not parallel_config.recompute.select_recompute: paralel_op_comm_compute = parallel_config.recompute.parallel_optimizer_comm_recompute network.recompute(parallel_optimizer_comm_recompute=paralel_op_comm_compute, mp_comm_recompute=parallel_config.recompute.mp_comm_recompute, recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) return _set_parallel_configure_for_layer class TransformerEncoder(Cell): r""" Transformer Encoder module with multi-layer stacked of `TransformerEncoderLayer`, including multihead self attention and feedforward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. num_layers(int): The layers of the `TransformerEncoderLayer` hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. seq_length(int): The seq_length of the input tensor. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default: 0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default: mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default: mstype.float32. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. offset(int): The initial layer index for the `encoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default: False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_states** (Tensor) - Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size]. - **attention_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple contains the Tensor the projected key and value vector with shape ((batch_size, num_heads, size_per_head, seq_length), and (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoder >>> from mindspore import Tensor >>> model = TransformerEncoder(batch_size=2, num_layers=2, hidden_size=8, ffn_hidden_size=64, ... seq_length=16, num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(len(past)) 2 >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoder(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, num_layers=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) """ @_LogActionOnce(m_logger=logger, key='TransformerEncoder', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, num_layers=Validator.check_positive_int, offset=Validator.check_non_negative_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "TransformerEncoder"), use_past=Validator.check_bool) def __init__(self, batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, offset=0, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerEncoder, self).__init__() _check_config(parallel_config) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) if batch_size or use_past: Validator.check_positive_int(batch_size) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class TransformerDecoder(Cell): r""" Transformer Decoder module with multi-layer stacked of `TransformerDecoderLayer`, including multihead self attention, cross attention and feedforward layer. Args: num_layers(int): The layers of the `TransformerDecoderLayer`. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. offset(int): The initial layer index for the `decoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size] - **attention_mask** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size] - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoder >>> from mindspore import Tensor >>> model = TransformerDecoder(batch_size=2, num_layers=1, hidden_size=64, ffn_hidden_size=64, ... num_heads=2, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(past)) 1 >>> print(past[0][0].shape) (2, 2, 32, 10) >>> print(past[0][1].shape) (2, 2, 10, 32) >>> print(past[0][2].shape) (2, 2, 32, 20) >>> print(past[0][3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='TransformerDecoder', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, num_layers=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, offset=Validator.check_non_negative_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "TransformerDecoder"), use_past=Validator.check_bool) def __init__(self, num_layers, batch_size, hidden_size, ffn_hidden_size, src_seq_length, tgt_seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', lambda_func=None, use_past=False, offset=0, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerDecoder, self).__init__() _check_moe_config(moe_config, parallel_config) _check_config(parallel_config) self.use_moe = (moe_config.expert_num > 1) if batch_size or use_past: Validator.check_positive_int(batch_size) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, use_past=use_past, param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, use_past=use_past, param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss # Loop through each self-attention layer for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class Transformer(Cell): r""" Transformer module including encoder and decoder. The difference with the original implements is the module use the residual addition before the layer normalization. And the default hidden act is `gelu`. The details can be found in `Attention is all you need `_. Note: This is an experimental interface that is subject to change or deletion. Args: hidden_size(int): The hidden size of the input. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The seq_length of the encoder's input tensor. tgt_seq_length(int): The seq_length of the decoder's input tensor. encoder_layers(int): The layers of the `TransformerEncoderLayer`. Default 3. decoder_layers(int): The layers of the `TransformerDecoderLayer`. Default 3. num_heads(int): The number of the heads. Default: 2. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. lambda_func: A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // ((encoder_layers + decoder_layers) / pipeline_stage)`. Default None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **encoder_inputs** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. - **encoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the encoder module. - **decoder_inputs** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], this should be none if the decoder layer is 0. - **decoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the decoder module. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The output of the encoder with shape [batch_size, seq_length, hidden_size], this should be none if the decoder layer is 0 or the user wants no mask. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `encoder_layer_present`, `decoder_layer_present`, `accum_loss`) - **output** (Tensor) - If there is only encoder, the output logit of the encoder layer. The shape is [batch, src_seq_length, hidden_size] or [batch * src_seq_length, hidden_size], if there are encoder and decoders, the output is from the decoder layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size]. - **encoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). - **decoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)), and the projected key and value vector in cross attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). If the decoder is not set, the returned value will be None. - **accum_loss** (Tensor) - A Tensor indicates an auxiliary loss to minimize the mean square of the data part routed to each expert, and only returned if the number of experts is greater than 1. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import Transformer >>> from mindspore import Tensor >>> model = Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, ... ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, en_past, de_past = model(encoder_input_value, encoder_input_mask, decoder_input_value, ... decoder_input_mask, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(en_past)) 1 >>> print(len(de_past)) 2 >>> print(en_past[0][0].shape) (2, 2, 32, 20) >>> print(en_past[0][1].shape) (2, 2, 20, 32) >>> print(de_past[0][0].shape) (2, 2, 32, 10) >>> print(de_past[0][1].shape) (2, 2, 10, 32) >>> print(de_past[0][2].shape) (2, 2, 32, 20) >>> print(de_past[0][3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='Transformer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, encoder_layers=Validator.check_positive_int, decoder_layers=Validator.check_non_negative_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "Transformer"), use_past=Validator.check_bool) def __init__(self, hidden_size, batch_size, ffn_hidden_size, src_seq_length, tgt_seq_length, encoder_layers=3, decoder_layers=3, num_heads=2, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(Transformer, self).__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.batch_size = batch_size self.hidden_size = hidden_size self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past if encoder_layers <= 0 < decoder_layers: raise ValueError(f"Transformer doest support encoder layer {encoder_layers} and decoder" f"layer {decoder_layers}, please use TransformerDecoder") if encoder_layers > 0 and decoder_layers > 0 and use_past: raise ValueError(f"The {self.cls_name} with encoder and decoder does not support use_past=True.") # The shard setting of Transformer is set within the TransformerEncoderLayer if not lambda_func: lambda_func = _get_lambda_func(total_layer=encoder_layers + decoder_layers) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) if encoder_layers > 0: self.encoder = TransformerEncoder(num_layers=encoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=src_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, lambda_func=lambda_func, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config) else: self.encoder = None # Offset is needed as the encoder has consumed some flags. # so the decoder need to increase the flags based on the encoder layer self.decoder = None if decoder_layers > 0: self.decoder = TransformerDecoder(num_layers=decoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, lambda_func=lambda_func, use_past=use_past, param_init_type=param_init_type, offset=encoder_layers, moe_config=moe_config, parallel_config=parallel_config) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.batch_size = batch_size self.hidden_size = hidden_size self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past if encoder_layers <= 0 < decoder_layers: raise ValueError(f"Transformer doest support encoder layer {encoder_layers} and decoder" f"layer {decoder_layers}, please use TransformerDecoder") if encoder_layers > 0 and decoder_layers > 0 and use_past: raise ValueError(f"The {self.cls_name} with encoder and decoder does not support use_past=True.") logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") # The shard setting of Transformer is set within the TransformerEncoderLayer if not lambda_func: lambda_func = _get_lambda_func(total_layer=encoder_layers + decoder_layers) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) if encoder_layers > 0: self.encoder = TransformerEncoder(num_layers=encoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=src_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, lambda_func=lambda_func, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config) else: self.encoder = None # Offset is needed as the encoder has consumed some flags. # so the decoder need to increase the flags based on the encoder layer self.decoder = None if decoder_layers > 0: self.decoder = TransformerDecoder(num_layers=decoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, lambda_func=lambda_func, use_past=use_past, param_init_type=param_init_type, offset=encoder_layers, moe_config=moe_config, parallel_config=parallel_config) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, encoder_inputs, encoder_masks, decoder_inputs=None, decoder_masks=None, memory_mask=None, init_reset=True, batch_valid_length=None): """process process""" encoder_output = None output = None encoder_layer_present = None decoder_layer_present = None accum_loss = self.aux_loss if self.encoder is not None: if self.use_moe: encoder_output, encoder_layer_present, encoder_aux_loss = self.encoder(encoder_inputs, encoder_masks, init_reset, batch_valid_length) accum_loss = self.add(accum_loss, encoder_aux_loss) else: encoder_output, encoder_layer_present = self.encoder(encoder_inputs, encoder_masks, init_reset, batch_valid_length) output = encoder_output if self.decoder is not None: # decoder mask should be created outside of the model if self.use_moe: decoder_output, decoder_layer_present, decoder_aux_loss = self.decoder(decoder_inputs, decoder_masks, encoder_output, memory_mask, init_reset, batch_valid_length) accum_loss = self.add(accum_loss, decoder_aux_loss) else: decoder_output, decoder_layer_present = self.decoder(decoder_inputs, decoder_masks, encoder_output, memory_mask, init_reset, batch_valid_length) output = decoder_output if self.use_moe: return output, encoder_layer_present, decoder_layer_present, accum_loss return output, encoder_layer_present, decoder_layer_present pengcheng.mind-test/mindformers/pet/000077500000000000000000000000001471400134000200265ustar00rootroot00000000000000pengcheng.mind-test/mindformers/pet/__init__.py000066400000000000000000000014361471400134000221430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """models init""" from .pet_config import PetConfig from .tuners import * __all__ = ['PetConfig'] __all__.extend(tuners.__all__) pengcheng.mind-test/mindformers/pet/constants.py000066400000000000000000000021731471400134000224170ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Constants used by pet modules. """ import enum class PetType(str, enum.Enum): r""" The enum of pet methods. """ LORA = "lora" ADAPTER = "adapter" ADALORA = "low_rank_adapter" PREFIX_TUNING = "prefixtuning" class BaseModelInitType(str, enum.Enum): r""" The enum of base model initial. """ MODEL_NAME = "MODEL_NAME" MODEL_DIR = "MODEL_DIR" MODEL_CONFIG = "MODEL_CONFIG" MODEL_INSTANCE = "MODEL_INSTANCE" pengcheng.mind-test/mindformers/pet/pet_config.py000066400000000000000000000040551471400134000225210ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: The config module of Parameter Efficient Tuning module. """ from mindformers.models.base_config import BaseConfig from mindformers.models.utils import convert_mstype from mindformers.pet.constants import PetType __all__ = ['LoraConfig'] class PetConfig(BaseConfig): """ Pet model config class which defines the tuning algorithm and pretrained for tuning. """ def __init__(self, pet_type=PetType.LORA, base_mode=None, **kwargs): super().__init__(**kwargs) self.pet_type = pet_type self.base_mode = base_mode class LoraConfig(PetConfig): """ Lora tuning algorithm config. """ def __init__(self, lora_rank: int = 8, lora_alpha: int = 16, lora_dropout: float = 0.01, lora_a_init: str = 'normal', lora_b_init: str = 'zero', param_init_type: str = 'float16', compute_dtype: str = 'float16', **kwargs): super().__init__(pet_type=PetType.LORA, **kwargs) self.lora_rank = lora_rank self.lora_alpha = lora_alpha self.lora_dropout = lora_dropout self.lora_a_init = lora_a_init self.lora_b_init = lora_b_init self.param_init_type = convert_mstype(param_init_type) self.compute_dtype = convert_mstype(compute_dtype) pengcheng.mind-test/mindformers/pet/tuners/000077500000000000000000000000001471400134000213465ustar00rootroot00000000000000pengcheng.mind-test/mindformers/pet/tuners/.ipynb_checkpoints/000077500000000000000000000000001471400134000251375ustar00rootroot00000000000000pengcheng.mind-test/mindformers/pet/tuners/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000016731471400134000314040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for bert_pet.""" from .adalora_adapter import * from .ada_adapter import * from .lora_adapter import * from .prefix_tuning_adapter import * from .pet_adapter import * __all__ = ['AdaLoraAdapter', 'AdaAdapter', 'LoraAdapter', 'PetAdapter', 'PrefixTuningAdapter'] pengcheng.mind-test/mindformers/pet/tuners/.ipynb_checkpoints/ada_adapter-checkpoint.py000066400000000000000000000022301471400134000320600ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/pdf/1902.00751.pdf """ from mindspore import nn from .pet_adapter import PetAdapter from ..pet_config import PetConfig class AdaAdapter(PetAdapter): r""" AdaAdapter is the adapter to modify the pretrained model, which uses adapter tuning algorithm. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-test/mindformers/pet/tuners/.ipynb_checkpoints/adalora_adapter-checkpoint.py000066400000000000000000000022521471400134000327420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/abs/2106.04647 """ from mindspore import nn from .pet_adapter import PetAdapter from ..pet_config import PetConfig class AdaLoraAdapter(PetAdapter): r""" AdaLoraAdapter is the adapter to modify the pretrained model, which uses adapter lora tuning algorithm. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-test/mindformers/pet/tuners/.ipynb_checkpoints/lora_adapter-checkpoint.py000066400000000000000000000132121471400134000322720ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/abs/2106.09685 """ import re from mindspore import nn from tk.delta.lora import LoRADense from mindformers.modules.layers import Linear from mindformers.tools.logger import logger from .pet_adapter import PetAdapter from ..pet_config import PetConfig from ..utils import re_match_list def recursive_replace_dense_cell(net, config): """default replace all dense.""" # pylint: disable=W0212 for name, cell in net._cells.items(): if cell: # add white list spaces. if re_match_list(name, config.exclude_layers): continue if re.match(config.reg_rules, name): if not isinstance(cell, nn.Dense) and not isinstance(cell, Linear): continue in_channels = cell.in_channels out_channels = cell.out_channels dest_cell = LoRADense(in_channels=in_channels, out_channels=out_channels, lora_rank=config.lora_rank, lora_alpha=config.lora_alpha, lora_dropout=config.lora_dropout, has_bias=cell.has_bias, activation=cell.activation) # load weight of oriangal layers. dest_cell.matmul = cell.matmul dest_cell.weight = cell.weight if cell.has_bias: dest_cell.bias = cell.bias dest_cell.bias_add = cell.bias_add # Shard strategies now only support loaded by linear layers. if isinstance(cell, Linear): strategy_matmul = cell.matmul.in_strategy dest_cell.lora_dropout.dropout.shard((strategy_matmul[0],)) if cell.transpose_b: dest_cell.lora_a_matmul.shard((strategy_matmul[0], (1, strategy_matmul[1][1]))) dest_cell.lora_b_matmul.shard(((strategy_matmul[0][0], 1), (strategy_matmul[1][0], 1))) dest_cell.mul.shard(((strategy_matmul[0][0], strategy_matmul[1][0]), ())) dest_cell.add.shard(((strategy_matmul[0][0], strategy_matmul[1][0]), (strategy_matmul[0][0], strategy_matmul[1][0]))) else: dest_cell.lora_a_matmul.shard((strategy_matmul[0], (strategy_matmul[1][0], 1))) dest_cell.lora_b_matmul.shard(((strategy_matmul[0][0], 1), (1, strategy_matmul[1][1]))) dest_cell.mul.shard(((strategy_matmul[0][0], strategy_matmul[1][1]), ())) dest_cell.add.shard(((strategy_matmul[0][0], strategy_matmul[1][1]), (strategy_matmul[0][0], strategy_matmul[1][1]))) # pylint: disable=W0212 net._cells[name] = dest_cell else: recursive_replace_dense_cell(cell, config) return net class LoraAdapter(PetAdapter): r""" LoraAdapter is the adapter to modify the pretrained model, which uses lora tuning algorithm. Args: model (BaseModel): The base pretrained model of mindformers. pet_config (PetConfig): The configurition of the Pet model. Return: model (BaseModel): The model replace the linear layer with lora dense layer. Examples: 1.modify certain task of llama >>> from mindformers.pet.tuners.lora_adapter import LoraAdapter >>> class LlamaForCausalLMWithLora(LlamaForCausalLM): >>> def __init__(self, config: LlamaConfig = None, pet=None): >>> super().__init__(config) >>> # get Pet tuning model. >>> self.pet = pet >>> self.pet.pet_config.reg_rules = r'.*wq|.*wv' >>> self.model = LoraAdapter.get_pet_model(self.model, self.pet.pet_config) >>> # freeze pretrained model >>> PetAdapter.freeze_pretrained_model(self, self.pet.pet_type) 2.modify certain model of llama >>> from mindformers.pet.tuners.lora_adapter import LoraAdapter >>> from mindformers.model.llama import LlamaModel >>> from mindformers.pet.pet_config import LoraConfig >>> llama_model = LlamaModel() >>> pet_config = LoraConfig() >>> llama_pet_model = LoraAdapter.get_pet_model(llama_model, pet_config) """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): model = model if model else PetAdapter.get_pretrained_model(config) if config.reg_rules is None: logger.warning("Lora Adapter use default replace rules: \'.*dense*|*linear*\'") config.reg_rules = r'.*dense*|.*linear*' model = recursive_replace_dense_cell(model, config) return model pengcheng.mind-test/mindformers/pet/tuners/__init__.py000066400000000000000000000016731471400134000234660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for bert_pet.""" from .adalora_adapter import * from .ada_adapter import * from .lora_adapter import * from .prefix_tuning_adapter import * from .pet_adapter import * __all__ = ['AdaLoraAdapter', 'AdaAdapter', 'LoraAdapter', 'PetAdapter', 'PrefixTuningAdapter'] pengcheng.mind-test/mindformers/pet/tuners/ada_adapter.py000066400000000000000000000022301471400134000241420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/pdf/1902.00751.pdf """ from mindspore import nn from .pet_adapter import PetAdapter from ..pet_config import PetConfig class AdaAdapter(PetAdapter): r""" AdaAdapter is the adapter to modify the pretrained model, which uses adapter tuning algorithm. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-test/mindformers/pet/tuners/adalora_adapter.py000066400000000000000000000022521471400134000250240ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/abs/2106.04647 """ from mindspore import nn from .pet_adapter import PetAdapter from ..pet_config import PetConfig class AdaLoraAdapter(PetAdapter): r""" AdaLoraAdapter is the adapter to modify the pretrained model, which uses adapter lora tuning algorithm. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-test/mindformers/pet/tuners/lora_adapter.py000066400000000000000000000132121471400134000243540ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/abs/2106.09685 """ import re from mindspore import nn from tk.delta.lora import LoRADense from mindformers.modules.layers import Linear from mindformers.tools.logger import logger from .pet_adapter import PetAdapter from ..pet_config import PetConfig from ..utils import re_match_list def recursive_replace_dense_cell(net, config): """default replace all dense.""" # pylint: disable=W0212 for name, cell in net._cells.items(): if cell: # add white list spaces. if re_match_list(name, config.exclude_layers): continue if re.match(config.reg_rules, name): if not isinstance(cell, nn.Dense) and not isinstance(cell, Linear): continue in_channels = cell.in_channels out_channels = cell.out_channels dest_cell = LoRADense(in_channels=in_channels, out_channels=out_channels, lora_rank=config.lora_rank, lora_alpha=config.lora_alpha, lora_dropout=config.lora_dropout, has_bias=cell.has_bias, activation=cell.activation) # load weight of oriangal layers. dest_cell.matmul = cell.matmul dest_cell.weight = cell.weight if cell.has_bias: dest_cell.bias = cell.bias dest_cell.bias_add = cell.bias_add # Shard strategies now only support loaded by linear layers. if isinstance(cell, Linear): strategy_matmul = cell.matmul.in_strategy dest_cell.lora_dropout.dropout.shard((strategy_matmul[0],)) if cell.transpose_b: dest_cell.lora_a_matmul.shard((strategy_matmul[0], (1, strategy_matmul[1][1]))) dest_cell.lora_b_matmul.shard(((strategy_matmul[0][0], 1), (strategy_matmul[1][0], 1))) dest_cell.mul.shard(((strategy_matmul[0][0], strategy_matmul[1][0]), ())) dest_cell.add.shard(((strategy_matmul[0][0], strategy_matmul[1][0]), (strategy_matmul[0][0], strategy_matmul[1][0]))) else: dest_cell.lora_a_matmul.shard((strategy_matmul[0], (strategy_matmul[1][0], 1))) dest_cell.lora_b_matmul.shard(((strategy_matmul[0][0], 1), (1, strategy_matmul[1][1]))) dest_cell.mul.shard(((strategy_matmul[0][0], strategy_matmul[1][1]), ())) dest_cell.add.shard(((strategy_matmul[0][0], strategy_matmul[1][1]), (strategy_matmul[0][0], strategy_matmul[1][1]))) # pylint: disable=W0212 net._cells[name] = dest_cell else: recursive_replace_dense_cell(cell, config) return net class LoraAdapter(PetAdapter): r""" LoraAdapter is the adapter to modify the pretrained model, which uses lora tuning algorithm. Args: model (BaseModel): The base pretrained model of mindformers. pet_config (PetConfig): The configurition of the Pet model. Return: model (BaseModel): The model replace the linear layer with lora dense layer. Examples: 1.modify certain task of llama >>> from mindformers.pet.tuners.lora_adapter import LoraAdapter >>> class LlamaForCausalLMWithLora(LlamaForCausalLM): >>> def __init__(self, config: LlamaConfig = None, pet=None): >>> super().__init__(config) >>> # get Pet tuning model. >>> self.pet = pet >>> self.pet.pet_config.reg_rules = r'.*wq|.*wv' >>> self.model = LoraAdapter.get_pet_model(self.model, self.pet.pet_config) >>> # freeze pretrained model >>> PetAdapter.freeze_pretrained_model(self, self.pet.pet_type) 2.modify certain model of llama >>> from mindformers.pet.tuners.lora_adapter import LoraAdapter >>> from mindformers.model.llama import LlamaModel >>> from mindformers.pet.pet_config import LoraConfig >>> llama_model = LlamaModel() >>> pet_config = LoraConfig() >>> llama_pet_model = LoraAdapter.get_pet_model(llama_model, pet_config) """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): model = model if model else PetAdapter.get_pretrained_model(config) if config.reg_rules is None: logger.warning("Lora Adapter use default replace rules: \'.*dense*|*linear*\'") config.reg_rules = r'.*dense*|.*linear*' model = recursive_replace_dense_cell(model, config) return model pengcheng.mind-test/mindformers/pet/tuners/pet_adapter.py000066400000000000000000000054041471400134000242130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: PET Adapter is the base adapter class for Parameter Efficient Tuning of MindFormers. """ import os from mindspore import nn from tk.graph.freeze_utils import freeze_delta from mindformers.auto_class import AutoModel from mindformers.mindformer_book import MindFormerBook from ..pet_config import PetConfig from ..constants import BaseModelInitType, PetType class PetAdapter: r""" PetAdapter is the base class of adapter to modify the pretrained model. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): """Add efficient tuning parameters to ptm.""" raise NotImplementedError("should implemented by the certain tuning algorithm.") @classmethod def get_pretrained_model(cls, config): """ Get pretrained model from config. """ init_type = config.base_model.init_type if init_type == BaseModelInitType.MODEL_CONFIG: config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'clip', 'model_config', config.base_model.model_config) base_model = AutoModel.from_config(config_path) if config.base_model.model_ckpt is None: raise ValueError("init base from model config, the model ckpt cannot be None") elif init_type == BaseModelInitType.MODEL_NAME: base_model = AutoModel.from_pretrained(config.base_model_name) elif init_type == BaseModelInitType.MODEL_DIR: checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), config.base_model_dir_name) base_model = AutoModel.from_pretrained(checkpoint_dir) else: raise ValueError("type of init base model must in BaseModelInitType") return base_model @classmethod def freeze_pretrained_model(cls, model, pet_type: PetType): """ Freeze the parameters of ptm which no update in the tuning process. Notes: Refer to mindpet api. """ freeze_delta(model, pet_type) pengcheng.mind-test/mindformers/pet/tuners/prefix_tuning_adapter.py000066400000000000000000000022151471400134000263010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://aclanthology.org/2021.acl-long.353.pdf """ from mindspore import nn from mindformers.pet.tuners.pet_adapter import PetAdapter from mindformers.pet.pet_config import PetConfig class PrefixTuningAdapter(PetAdapter): r""" Prefix tuning implement. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-test/mindformers/pet/utils.py000066400000000000000000000021101471400134000215320ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: functions. """ import re from typing import Optional, List def re_match_list(mstr, patterns: Optional[List[str]] = None): """string regular match""" if patterns is None: return False for pattern in patterns: if not isinstance(pattern, str): raise TypeError(f"List item '{pattern}' is not a string.") if re.match(pattern, mstr): return True return False pengcheng.mind-test/mindformers/pipeline/000077500000000000000000000000001471400134000210435ustar00rootroot00000000000000pengcheng.mind-test/mindformers/pipeline/__init__.py000066400000000000000000000041111471400134000231510ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Pipeline API.""" from .pipeline import pipeline from .build_pipeline import build_pipeline from .base_pipeline import BasePipeline from .image_classification_pipeline import ImageClassificationPipeline from .zero_shot_image_classification_pipeline import ZeroShotImageClassificationPipeline from .image_to_text_generation_pipeline import ImageToTextGenerationPipeline from .translation_pipeline import TranslationPipeline from .fill_mask_pipeline import FillMaskPipeline from .text_classification_pipeline import TextClassificationPipeline from .token_classification_pipeline import TokenClassificationPipeline from .question_answering_pipeline import QuestionAnsweringPipeline from .text_generation_pipeline import TextGenerationPipeline from .masked_image_modeling_pipeline import MaskedImageModelingPipeline from .segment_anything_pipeline import SegmentAnythingPipeline __all__ = ['ZeroShotImageClassificationPipeline', 'ImageClassificationPipeline', 'ImageToTextGenerationPipeline', 'pipeline', 'BasePipeline'] __all__.extend(translation_pipeline.__all__) __all__.extend(fill_mask_pipeline.__all__) __all__.extend(text_classification_pipeline.__all__) __all__.extend(token_classification_pipeline.__all__) __all__.extend(question_answering_pipeline.__all__) __all__.extend(text_generation_pipeline.__all__) __all__.extend(masked_image_modeling_pipeline.__all__) __all__.extend(image_to_text_generation_pipeline.__all__) pengcheng.mind-test/mindformers/pipeline/__pycache__/000077500000000000000000000000001471400134000232535ustar00rootroot00000000000000pengcheng.mind-test/mindformers/pipeline/__pycache__/__init__.cpython-39.pyc000066400000000000000000000024761471400134000274540ustar00rootroot00000000000000a ?cfI@sdZddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZdd lmZdd lmZddlmZddlmZgdZee jee jeejeejeejeejeejee jdS)zMindFormers Pipeline API.)pipeline)build_pipeline) BasePipeline)ImageClassificationPipeline)#ZeroShotImageClassificationPipeline)ImageToTextGenerationPipeline)TranslationPipeline)FillMaskPipeline)TextClassificationPipeline)TokenClassificationPipeline)QuestionAnsweringPipeline)TextGenerationPipeline)MaskedImageModelingPipeline)SegmentAnythingPipeline)rrrrrN)__doc__rr base_pipelinerZimage_classification_pipelinerZ'zero_shot_image_classification_pipelinerZ!image_to_text_generation_pipelinerZtranslation_pipelinerZfill_mask_pipeliner Ztext_classification_pipeliner Ztoken_classification_pipeliner Zquestion_answering_pipeliner Ztext_generation_pipeliner Zmasked_image_modeling_pipelinerZsegment_anything_pipeliner__all__extendrrH/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/__init__.pys.                     pengcheng.mind-test/mindformers/pipeline/__pycache__/base_pipeline.cpython-39.pyc000066400000000000000000000166671471400134000305230ustar00rootroot00000000000000a ?cf3$@sdZddlmZmZddlmZmZddlZddl m Z ddl m Z m Z ddl mZmZmZmZddlmZmZdd lmZdd lmZd d lmZd d lmZmZmZGdddeZdS) BasePipeline)ABCabstractmethod)OptionalUnionN)tqdm)TensorModel)GeneratorDatasetVisionBaseDataset SourceDatasetMappableDataset) BatchDataset RepeatDataset)logger) print_dict) AutoModel) BaseModel BaseTokenizerBaseImageProcessorcseZdZdZiZdeeeefe e e e dfdd Z dee eeeejfe edddZed d Zeeeejefeeed d d Zeeefeeed ddZeeeeejefdddZeeeeejefdddZeeeeejefdddZeddZZS)ra Base Pipeline For All Task Pipelines Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): The tokenizer of model, it could be None if the model do not need tokenizer. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. N)model tokenizerimage_processorc stt|||_t|tr6||jvr6t||_ n:t|t rH||_ n(t|t r\|j |_ nt dt|d||_||_|jfi|\|_|_|_d|_|dd|_dS)NzGmodel should be str or inherited from BaseModel or Model, but got type .r batch_size)superr__init__r isinstancestr _support_listrfrom_pretrainednetworkrr predict_network TypeErrortyperr_sanitize_parameters_preprocess_params_forward_params_postprocess_params call_countpop _batch_size)selfrrrkwargs __class__M/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/base_pipeline.pyr4s     zBasePipeline.__init__)inputsrc Ks<|jfi|\}}}i|j|}i|j|}i|j|}t|ttttf}t|t }|j d7_ |j dkr|st d|dur|j durd}n|j }|dkr|sd}t d|rt dt|ttfs||}g} t|D]} | || |||qn(|r(|||||} n|||||} | S)aZCall Method Args: inputs (Union[GeneratorDataset, list, str, etc]): The inputs of pipeline, the type of inputs depends on task. batch_size (Optional[int]): The batch size for a GeneratorDataset input, for other types of inputs, the batch size would be set to 1 by default. zYou seem to be using the pipeline sequentially for numerous samples. In order to maximize efficiency please set input a mindspore.dataset.GeneratorDataset.Nz.batch_size is set to 1 for non-dataset inputs.zdataset is processing.)r&r'r(r)rr r r r listr*rinfor,rrbatchrcreate_dict_iteratorextend run_single run_multi) r-r3rr.preprocess_paramsforward_paramspostprocess_paramsZ is_datasetis_listoutputsitemsr1r1r2__call__Js@          zBasePipeline.__call__cKs tddS)zSanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. Raises: NotImplementedError: If the method is not implemented. z$_sanitize_parameters not implementedNNotImplementedError)r-Zpipeline_parametersr1r1r2r&s z!BasePipeline._sanitize_parameters)r3r=r>r?cCs:|j|fi|}|j|fi|}|j|fi|}|S)aRun Single method This function is used to run a single forward process for task. Args: inputs (Union[dict, str, etc]): The inputs of pipeline, the type of inputs depends on task. preprocess_params (dict): The parameter dict for preprocess. forward_params (dict): The parameter dict for model forward process. postprocess_params (dict): The parameter dict for postprocess. ) preprocessforward postprocess)r-r3r=r>r? model_inputs model_outputsrAr1r1r2r;szBasePipeline.run_singlec Cs(g}|D]}||||||q|S)aRun Multiple Method This function is used to run a list input for task. Args: inputs (Union[list, tuple, iterator]): The iterable input for pipeline. preprocess_params (dict): The parameter dict for preprocess. forward_params (dict): The parameter dict for model forward process. postprocess_params (dict): The parameter dict for postprocess. )r:r;)r-r3r=r>r?rAitemr1r1r2r<s  zBasePipeline.run_multi)r3cKs tddS)aDThe Preprocess For Task Args: inputs (Union[dict, str, etc]): The inputs of pipeline, the type of inputs depends on task. preprocess_params (dict): The parameter dict for preprocess. Raises: NotImplementedError: If the method is not implemented. zpreprocess not implemented.NrD)r-r3r=r1r1r2rFs zBasePipeline.preprocess)rIcKs tddS)aWThe Forward Process of Model Args: model_inputs (Union[dict, str, etc]): The output of preprocess, the type of model_inputs depends on task. forward_params (dict): The parameter dict for model forward. Raises: NotImplementedError: If the method is not implemented. zforward not implemented.NrD)r-rIr>r1r1r2rGs zBasePipeline.forward)rJcKs tddS)aZThe Postprocess of Task Args: model_outputs (Union[dict, str, etc]): The output of model forward, the type of model_outputs depends on task. postprocess_params (dict): The parameter dict for post process. Raises: NotImplementedError: If the method is not implemented. zpostprocess not implemented.NrD)r-rJr?r1r1r2rHs zBasePipeline.postprocesscCstd|jt|jdS)show_support_listzsupport list of %s is:N)rr7__name__rr )clsr1r1r2rLszBasePipeline.show_support_list)NN)N) rM __module__ __qualname____doc__r rrrr rrrrr r6rnparrayintrCrr&dictr;tupler<rFrGrH classmethodrL __classcell__r1r1r/r2r%s@  5    ) rQabcrrtypingrrnumpyrRr mindsporerr mindspore.datasetr r r r Z!mindspore.dataset.engine.datasetsrrmindformers.toolsrmindformers.mindformer_bookr auto_classrmodelsrrrrr1r1r1r2s    pengcheng.mind-test/mindformers/pipeline/__pycache__/build_pipeline.cpython-39.pyc000066400000000000000000000032741471400134000306760ustar00rootroot00000000000000a ?cf) @s2dZddlmZmZmZdeeeedddZdS) zBuild Pipeline API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfigNpipeline)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)aBuild Pipeline API. Instantiate the task pipeline from MindFormerRegister's registry. Args: config (dict): The task pipeline's config. default_args (dict): The default args of pipeline. module_type (str): The module type of MindFormerModuleType. Default: 'pipline'. class_name (str): The class name of task pipeline. Returns: The task pipeline instance by config. Examples: >>> from mindformers import build_pipeline >>> pipeline_config = {'type': 'zero_shot_image_classification', 'model': 'clip_vit_b_32', 'candidate_labels': ["sunflower", "tree", "dog", "cat", "toy"], 'hypothesis_template': "This is a photo of {}."} >>> classifier = build_pipeline(pipeline_config) >>> type(classifier) N)r) isinstancedictrrget_instance_from_cfgrPIPELINE get_instance)rrrr kwargsrN/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/build_pipeline.pybuild_pipelinesr)NNrN)__doc__mindformers.tools.registerrrrr strrrrrrspengcheng.mind-test/mindformers/pipeline/__pycache__/fill_mask_pipeline.cpython-39.pyc000066400000000000000000000114771471400134000315440ustar00rootroot00000000000000a ?cf.@sdZddlZddlZddlZddlmZmZmZddl m Z m Z ddl m Z ddlmZdd lmZmZdd lmZmZd gZejejd d Gdd d eZdS)TranslationPipelineN)opsTensorModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel TokenizerFillMaskPipeline fill_mask)aliascsReZdZdZedZdZdfdd ZddZ d d Z d d Z d dZ Z S)rz Pipeline for mask fill Args: model: a pretrained model (str or BaseModel) in _supproted_list. tokenizer : a tokenizer (None or Tokenizer) for text processing bertZfillmaskNc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdd|_tj||fi|dS)Nz@tokenizer should be inherited from PretrainedTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.) isinstancestr _support_listospathisdirrfrom_pretrained tokenizerrr TypeErrortype ValueError __class____name__rr input_textsuper__init__)selfmodelrkwargsr!R/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/fill_mask_pipeline.pyr%+s$      zFillMaskPipeline.__init__c std|vrtd|jjdi}gd}i}|D]}||vr0||||<q0gdfdd|D}|||fS)z=sanitize parameters for preprocess, forward, and postprocess. batch_sizezThe z? does not support batch inference, please remove the batch_size)top_ktop_p do_sample eos_token_idrepetition_penalty max_length)r-r.r/r0r1r2paddingcsi|]\}}|vr||qSr*r*).0kvZpreprocess_key_namer*r+ Rz9FillMaskPipeline._sanitize_parameters..)r r!r"getitems)r&pipeline_parameterspostprocess_paramsforward_key_nameforward_kwargsitempreprocess_paramsr*r7r+_sanitize_parametersBsz%FillMaskPipeline._sanitize_parameterscKst|tr*|d}t|tjr*|}t|tr8|nd|_|dd}|dd}|j |f||dd|}t }||dd ||d d ||d d |t|j j d d S) ax Preprocess of mask fill Args: inputs (url, PIL.Image, tensor, numpy): the image to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed image. textrr2r3ms)r2r3return_tensors input_idsrattention_masktoken_type_ids)rG input_mask token_type_idmasked_lm_positions) rdict mindsporerasnumpytolistrr#poprr ExpandDims mask_index)r&inputsrAr2r3 expand_dimsr*r*r+ preprocessUs$         zFillMaskPipeline.preprocesscKs$|dd|jfi|}d|iS)z Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. NoneN output_ids)rQnetwork)r& model_inputsforward_paramsrXr*r*r+forwardqs zFillMaskPipeline.forwardc Ks|dd}g}tj|dd}t|D]V\}}|jjt|gdd}|dd}||||f||d |j d |d|_ q*|d |j i|gS) z Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results rXr )axisT)skip_special_tokens r)scoretokenZ token_strz[MASK]sequence) rOnpargmax enumeraterdecodeintreplaceappendr#) r& model_outputsr=outputsZ tokens_dictZ max_tokensindZtokenidrbr*r*r+ postprocesss  zFillMaskPipeline.postprocess)N)r" __module__ __qualname____doc__r get_model_support_listr return_namer%rBrVr\rn __classcell__r*r*r)r+rs )rqos.pathrnumpyrdrNrrr auto_classrrmindformer_bookr base_pipeliner Ztools.registerr r modelsrr__all__registerPIPELINErr*r*r*r+s  pengcheng.mind-test/mindformers/pipeline/__pycache__/image_classification_pipeline.cpython-39.pyc000066400000000000000000000144331471400134000337330ustar00rootroot00000000000000a ?cf@sdZddlmZmZddlZddlmZddlm Z ddl m Z m Z ddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lmZd dlmZejej ddGdddeZ!dS)z"Image Classification Pipeline API.)OptionalUnionN)Image) operations)TensorModel) AutoProcessor AutoModel)MindFormerBook) BaseModelBaseImageProcessor) load_image)MindFormerRegisterMindFormerModuleType)labels) BasePipelineimage_classification)aliascseZdZdZedZdee e e fe e dfdd ZddZee ejeejfd d d Zed d dZddZZS)ImageClassificationPipelineafPipeline for image classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import ImageClassificationPipeline >>> from mindformers import ViTImageProcessor >>> processor = ViTImageProcessor(size=224) >>> classifier = ImageClassificationPipeline( ... model='vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> classifier(np.uint8(np.random.random((5, 3, 255, 255)))) [[{'score': 0.0016654134, 'label': 'matchstick'}, {'score': 0.0015071577, 'label': 'theater curtain'}, {'score': 0.0014839625, 'label': 'ocarina'}, {'score': 0.0014319294, 'label': 'abaya'}, {'score': 0.0014109017, 'label': 'bottlecap'}], ..., {'score': 0.0014109018, 'label': 'bottlecap'}]] rN)modelimage_processorc st|trh||jvrR|dur(t|j}t|tsFtdt|dt |}nt |d|jdt|t t fstdt|d|durt dt j|jddfd|i|dS) NzEimage_processor should be inherited from BaseImageProcessor, but got .zI is not supported by ImageClassificationForPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z=ImageClassificationFoPipeline requires for a image_processor.F)moder) isinstancestr _support_listrfrom_pretrainedrr TypeErrortyper ValueErrorr rsuper__init__ set_train)selfrrkwargs __class__]/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/image_classification_pipeline.pyr"Ds"       z$ImageClassificationPipeline.__init__cKs:i}i}ddg}|D]}||vr||||<q|i|fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. top_kcandidate_labels)get)r$pipeline_parameterspreprocess_paramspostprocess_paramsZ post_listitemr(r(r)_sanitize_parameters\sz0ImageClassificationPipeline._sanitize_parameters)inputscKs6t|tr|d}t|tr$t|}||}d|iS)zThe Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. imageimage_processed)rdictrr r)r$r2r.r4r(r(r) preprocessls   z&ImageClassificationPipeline.preprocess) model_inputscKs:|dd|d}||d}t|}d|iS)zThe Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. NoneNr4rprobs)popnetworkPSoftmaxasnumpy)r$r7forward_paramsr4Zlogits_per_imager9r(r(r)forwards  z#ImageClassificationPipeline.forwardc Ks|dd}|dd}|d}g}t|tr:t|}n"t|trJ|}ntdt||durltd|D]L}tt ||d d d } |dur| dt |t |} | d d | Dqp|S)zPostprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result. Return: classification results. r*r+imagenetr9zRThe candidate_labels should be dataset name (str) or custom labels (list) but got NzZThe custom candidate_labels is None or the input dataset labels name is not supported yet.cSs |d S)Nrr()xr(r(r)z9ImageClassificationPipeline.postprocess..)keycSsg|]\}}||dqS))scorelabelr().0Z score_itemrHr(r(r) sz;ImageClassificationPipeline.postprocess..) r:rrrr,listr rsortedzipminlenappend) r$ model_outputsr/r*r+scoresoutputsZ inputs_labelsrGZ sorted_resr(r(r) postprocesss*       z'ImageClassificationPipeline.postprocess)N)__name__ __module__ __qualname____doc__r get_pipeline_support_task_listkeysrrrr rrr r"r1rrnpndarrayr6r5r@rT __classcell__r(r(r&r)r!sr)"rXtypingrrnumpyr[PILr mindspore.opsrr< mindsporerrmindformers.auto_classrr mindformers.mindformer_bookr mindformers.modelsr r mindformers.tools.image_toolsr mindformers.tools.registerrrZmindformers.dataset.labelsr base_pipelinerregisterPIPELINErr(r(r(r)s      image_to_text_generation_pipeline.cpython-39.pyc000066400000000000000000000123651471400134000345640ustar00rootroot00000000000000pengcheng.mind-test/mindformers/pipeline/__pycache__a ?cfY@sdZddlmZmZddlZddlmZddlm Z m Z ddl m Z ddlmZmZddlmZdd lmZmZdd lmZdd lmZmZd d lmZdgZejejddGdddeZ dS)z"Image Classification Pipeline API.)OptionalUnionN)Image)TensorModel) operations) AutoProcessor AutoModel)MindFormerBook) BaseModelBaseImageProcessor load_image)MindFormerRegisterMindFormerModuleType) BasePipelineImageToTextGenerationPipelineimage_to_text_generation)aliascseZdZdZedZdee e e fe e dfdd ZddZee ejeejfd d d Zed d dZddZZS)ra'Pipeline for image to text generation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. rN)modelimage_processorc st|trh||jvrR|dur(t|j}t|tsFtdt|dt |}nt |d|jdt|t t fstdt|d|durt d| dd|_tj|jdd f||d |dS) NzEimage_processor should be inherited from BaseImageProcessor, but got .zH is not supported by ImageToTextGenerationPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z=ImageToTextGenerationPipeline requires for a image_processor.hypothesis_templatez{}F)mode)r tokenizer) isinstancestr _support_listrfrom_pretrainedrr TypeErrortyper ValueErrorr rpoprsuper__init__ set_train)selfrrrkwargs __class__a/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/image_to_text_generation_pipeline.pyr%3s$       z&ImageToTextGenerationPipeline.__init__c Ksi}i}i}dg}gd}gd}|D]}||vr&||||<q&|D]}||vrF||||<qF|D]}||vrf||||<qf|||fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. top_k)r max_lengthpadding)r-top_p do_sample eos_token_idrepetition_penaltyr.seed)get) r'pipeline_parameterspreprocess_paramspostprocess_paramsforward_params post_listpre_listZ forward_listitemr+r+r,_sanitize_parametersLs z2ImageToTextGenerationPipeline._sanitize_parameters)inputsc Kst|tr|d}t|tr$t|}t|trxz.r. r/rNrmsF)r.r/return_tensorsadd_special_tokens input_idsr)image_processedprompt_input_ids) rdictrrlistrr#rshaperlenP ExpandDims) r'r>r7rHr.r/rZ prompt_listZprompt_processedrIr+r+r, preprocessgs2       z(ImageToTextGenerationPipeline.preprocess) model_inputscKs&|d}|d}|j||}d|iS)zThe Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. rHrI output_ids)networkZgenerate_text_for_image)r'rQr9rHrIZoutput_ids_per_imager+r+r,forwardsz%ImageToTextGenerationPipeline.forwardcKs|d}|jj|dd}|S)NrRT)skip_special_tokens)rdecode)r' model_outputsr8rRoutputsr+r+r, postprocesssz)ImageToTextGenerationPipeline.postprocess)NN)__name__ __module__ __qualname____doc__r get_pipeline_support_task_listkeysrrrr rrr r%r=rrnpndarrayrPrJrTrY __classcell__r+r+r)r,r"s ))!r]typingrrnumpyr`PILr mindsporerr mindspore.opsrrNmindformers.auto_classrr mindformers.mindformer_bookr mindformers.modelsr r mindformers.tools.image_toolsrmindformers.tools.registerrr base_pipeliner__all__registerPIPELINErr+r+r+r,s     pengcheng.mind-test/mindformers/pipeline/__pycache__/masked_image_modeling_pipeline.cpython-39.pyc000066400000000000000000000125261471400134000340630ustar00rootroot00000000000000a ?cfu@sdZddlZddlmZmZddlZddlmZddl m Z m Z ddl m Z mZddlmZddlmZmZdd lmZdd lmZmZd d lmZd gZddlmZejejddGdd d eZ dS)z#Masked Image Modeling Pipeline API.N)OptionalUnion)Image)TensorModel) AutoProcessor AutoModel)MindFormerBook) BaseModelBaseImageProcessor) load_image)MindFormerRegisterMindFormerModuleType) BasePipelineMaskedImageModelingPipeline)LOCAL_DEFAULT_PATHmasked_image_modeling)aliascseZdZdZedZdee e e fe e dfdd ZddZee ejeejfd d d Zed d dZddZZS)ra Pipeline for masked image modeling Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import MaskedImageModelingPipeline >>> from mindformers import ViTMAEImageProcessor >>> processor = ViTMAEImageProcessor(size=224) >>> reconstructor = MaskedImageModelingPipeline( ... model='mae_vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> reconstructor(np.uint8(np.random.random((5, 3, 255, 255)))) rN)modelimage_processorc st|trh||jvrR|dur(t|j}t|tsFtdt|dt |}nt |d|jdt|t t fstdt|d|durt dt j|jddfd|i|dS) NzEimage_processor should be inherited from BaseImageProcessor, but got .zI is not supported by ImageClassificationForPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z;MaskedImageModelingPipeline requires for a image_processor.F)moder) isinstancestr _support_listrfrom_pretrainedrr TypeErrortyper ValueErrorr rsuper__init__ set_train)selfrrkwargs __class__^/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/masked_image_modeling_pipeline.pyr"As"       z$MaskedImageModelingPipeline.__init__cKs6i}i}g}|D]}||vr||||<q|i|fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. )get)r$pipeline_parameterspreprocess_paramspostprocess_params post_listitemr(r(r)_sanitize_parametersYsz0MaskedImageModelingPipeline._sanitize_parameters)inputscKs6t|tr|d}t|tr$t|}||}d|iS)aThe Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be reconstructed. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. imageimage_processed)rdictrr r)r$r1r,r3r(r(r) preprocessis   z&MaskedImageModelingPipeline.preprocess) model_inputscKs.|dd|d}|j|}|}d|iS)zThe Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. NoneNr3 images_array)popnetworkasnumpy)r$r6forward_paramsr3r8r(r(r)forward|s   z#MaskedImageModelingPipeline.forwardc Ks|d}g}t|D]\}}|d}t|jjj}t|jjj}|||}|d}|tj }t |} t j td|d} | | || | dq|S)zPostprocess Args: model_outputs (dict): Outputs of forward process. Return: classification results. r8)rrrZ output_imagez.jpg)infodata) enumerate transposenparrayr normalizemeanstdastypeuint8r fromarrayospathjoinrsaveappend) r$ model_outputsr-r8outputsiZ image_arrayrFrGZreconstruct_image image_pathr(r(r) postprocesss      z'MaskedImageModelingPipeline.postprocess)N)__name__ __module__ __qualname____doc__r get_pipeline_support_task_listkeysrrrr rrr r"r0rrrCndarrayr5r4r=rT __classcell__r(r(r&r)r$s)!rXrKtypingrrnumpyrCPILr mindsporerrmindformers.auto_classrrmindformers.mindformer_bookr mindformers.modelsr r mindformers.tools.image_toolsr mindformers.tools.registerr r base_pipeliner__all__Z tools.utilsrregisterPIPELINErr(r(r(r)s     pengcheng.mind-test/mindformers/pipeline/__pycache__/pipeline.cpython-39.pyc000066400000000000000000000075211471400134000275160ustar00rootroot00000000000000a ?cf @sdZddlmZddlmZmZmZddlmZddl m Z m Z m Z m Z mZmZmZddlmZddlmZdd lmZeZeZGd d d eZdeeeee eeeeffeeeeeeedddZddZd S)z pipeline )Enum)OptionalUnionTuple)Model) build_modelbuild_tokenizerbuild_processor BaseModel BaseTokenizerBaseImageProcessorBaseAudioProcessor)MindFormerBook)MindFormerConfig)build_pipelinec@seZdZdZdZdS)BackendmsZmsliteN)__name__ __module__ __qualname__MSMS_LITErrH/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/pipeline.pyr"srNr)taskmodel tokenizerimage_processoraudio_processorbackendc Ksl|tjjkr2ddlm}||||||fi|}n6|tjjkrXt|||||fi|}ntd|d|S)aPipeline for downstream tasks Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (Optional[Union[str, BaseModel]]): The model used for task. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. backend(str): The inference backend. Default "ms", now support ["ms", "mslite"]. Return: A task pipeline. Raises: KeyError: If the task or model is not supported. Examples: >>> from mindformers import pipeline >>> from mindformers.tools.image_tools import load_image >>> classifier = pipeline("zero_shot_image_classification", candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) >>> img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> classifier(img) [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, {'score': 2.396818e-06, 'label': 'cat'}]] r)get_mslite_pipelinezThe inference backend "z@" is not supported,please select a backend from ["ms", "mslite"])rrvalueZmindformers.inferencer!rget_ms_pipeline ValueError) rrrrrr kwargsr! task_pipelinerrrpipeline's'   r'c Ks|tvr$t|dtdt|trV|tvrLtdtd|d|}d}nd}tt||}|jj d|i|durt |j}|durt |j drt|j j}|durt |j d rt|j j}|durt|j j}tf|||||d |}|S) zget mindspore infer pipeline.z9 is not supported by pipeline. please select a task from .zmodel must be in z& when model's type is string, but get Ncommoncheckpoint_name_or_pathrr) class_namerrrr)SUPPORT_PIPELINESkeysKeyError isinstancestrSUPPORT_MODEL_NAMESrgetr model_configupdaterhasattr processorr rrrrr) rrrrrr% model_nameZpipeline_configr&rrrr#Ys>       r#)NNNNNr)__doc__enumrtypingrrr mindsporermindformers.modelsrrr r r r r mindformers.mindformer_bookrmindformers.tools.registerrrget_pipeline_support_task_listr,get_model_name_support_listr1rr0r'r#rrrrs2  $      2pengcheng.mind-test/mindformers/pipeline/__pycache__/question_answering_pipeline.cpython-39.pyc000066400000000000000000000277541471400134000335340ustar00rootroot00000000000000a ?cf.H@sdZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddlmZmZddlmZmZdd lmZd d lmZdd lmZmZdd lmZmZdgZejej ddGdddeZ!dS)QuestionAnsweringPipelineN)TensorModel)dtype)BasicTokenizer)convert_examples_to_features SquadExample) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel Tokenizerquestion_answering)aliascseZdZdZedZdfd d Zd d Z d dZ ddZ ddZ ddZ ddZddZddZddZddZZS) ragPipeline for token classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. id2label : a dict which maps label id to label str. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import QuestionAnsweringPipeline >>> from mindformers import AutoTokenizer, BertForQuestionAnswering, AutoConfig >>> input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] >>> tokenizer = AutoTokenizer.from_pretrained('qa_bert_base_uncased_squad') >>> qa_squad_config = AutoConfig.from_pretrained('qa_bert_base_uncased_squad') >>> model = BertForQuestionAnswering(qa_squad_config) >>> qa_pipeline = QuestionAnsweringPipeline(task='question_answering', ... model=model, ... tokenizer=tokenizer) >>> results = qa_pipeline(input_data) >>> print(results) [{'text': 'Berlin', 'score': 0.9941, 'start': 34, 'end': 40}] r@r c  slt|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdd|_d|_tdd|_||krt d |d |d |d kr t d |d |d kr"t d||| d<|| d<|| d<|| d<|| d<|| d<tj||fi| dS)Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.T) do_lower_casez`doc_stride` (z ) is larger than `max_seq_len` ()r z$top_k parameter should be >= 1 (got z-max_answer_len parameter should be >= 1 (got doc_stridemax_question_len max_seq_lentop_k n_best_sizemax_answer_len) isinstancestr _support_listospathisdirr from_pretrained tokenizerr r TypeErrortype ValueError __class____name__rrfeaturesexamplesrbasic_tokenizersuper__init__) selfmodelr*rrrr r!r"kwargsr.[/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/question_answering_pipeline.pyr4Cs@         z"QuestionAnsweringPipeline.__init__c sbd|vrtd|jjdgdgdfdd|D}fdd|D}|i|fS) z=sanitize parameters for preprocess, forward, and postprocess. batch_sizezThe z? does not support batch inference, please remove the batch_size)rrr)r r!r"csi|]\}}|vr||qSr9r9.0kv)preprocess_key_namer9r: uzBQuestionAnsweringPipeline._sanitize_parameters..csi|]\}}|vr||qSr9r9r<)postprocess_key_namer9r:rAvrB)r-r.r/items)r5pipeline_parameterspreprocess_paramspostprocess_paramsr9)rCr@r:_sanitize_parameterslsz.QuestionAnsweringPipeline._sanitize_parameterscKst|tstdd|vr"td|d\}}|}|}td||ddd}t|g|j|d|d|ddd }||_|g|_ |S) a# Preprocess of token classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output return_tensors ("ms"): the type of returned tensors Return: processed text. z!The type of inputs should be str.-zHThe inputs should contain context and question separeated by '-' syntax.NrrrF)r1r*rrr is_training) r#r$r-splitstripr rr*r0r1)r5inputsrF context_text question_textZ squad_exampler0r9r9r: preprocesszs&  z$QuestionAnsweringPipeline.preprocessc Ks|jdg}|D]}t|jtjdt|jtjdt|jtjdt|j tjdt|j tjdt|j tjdd}|j fi|\}}}t dgd}t|d} dd|dD} dd|dD} ||| | | dq|S) z Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. Fr) input_ids input_mask token_type_idstart_position end_position unique_id RawResult)rV start_logits end_logitscSsg|] }t|qSr9floatr=xr9r9r: rBz5QuestionAnsweringPipeline.forward..cSsg|] }t|qSr9rZr\r9r9r:r^rB)r6 set_trainrrQmstypeint32 expand_dimsrRrSrTrUrVnetwork collections namedtupleintappend) r5 model_inputsforward_params model_outputsfeatureidsstartendrWrVrXrYr9r9r:forwards&    z!QuestionAnsweringPipeline.forwardcKs.|d}|d}|d}tt}|jD]}||j|q(i}|D]} | || j<qFt|jD]\} } || } | | |||} | | | | |}g}d}|D]$}||j |j |s|j r|}q||}g}t|D]F\}}i}|j |d<t||d|d<|j|d<|j|d <||qq`|d|S) z Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results r r!r"Ntextscorermrn)rd defaultdictlistr0 example_indexrgrV enumerater1_get_prelim_predictions _get_nbest start_logit end_logitrp_compute_softmaxroundrmrn)r5rjrGr r!r"example_index_to_featuresrkunique_id_to_resultresultruexampler0prelim_predictionsnbest total_scoresbest_non_null_entryentryprobsresultsioutputr9r9r: postprocesss@        z%QuestionAnsweringPipeline.postprocessc Cstdgd}g}t|D]\}}|j|vr0q||j} || j|} || j|} | D]} | D]} | t|jkrvqb| t|jkrqb| |j vrqb| |j vrqb|j | dsqb| | krqb| | d}||krqb| ||| | | j| | j| dqbqZqt |dddd}|S) zget prelim predictionsPrelimPrediction) feature_index start_index end_indexryrzFr cSs |j|jS)N)ryrzr]r9r9r:rBzCQuestionAnsweringPipeline._get_prelim_predictions..Tkeyreverse)rdrervrV_get_best_indexesrXrYlentokenstoken_to_orig_maptoken_is_max_contextgetrgsorted)r5r0r~r"r!_PrelimPredictionrrrkr start_indexes end_indexesrrlengthr9r9r:rwsV      z1QuestionAnsweringPipeline._get_prelim_predictionsc Cs`tdgd}i}g}|D]}t||kr4q0||j} |jdkr| j|j|jd} | j|j} | j|j} |j| | d} d | }| dd}| dd}| }d | }d | }| ||}||j| | \}}||vrqd ||<nd}d ||<d \}}||||j|j||dq|sJ||d d d d t|dks\J|S)zget nbest predictionsNbestPrediction)rpryrzrmrnrr  z ##z##T)rempty)rpryrz)rdrerrrrrr doc_tokensjoinreplacerLrK_get_final_text_get_answer_indexrNrgryrz)r5rr0rr!_NbestPredictionseen_predictionsrpredrk tok_tokensorig_doc_start orig_doc_end orig_tokenstok_text orig_text final_textchar_start_indexchar_end_indexr9r9r:rx$sV             z$QuestionAnsweringPipeline._get_nbestc CsTd}t|dD]8\}}||kr&|}||kr:|t|}|t|d7}q||fS)Nrrr )rvrKr) r5rNrr char_indexrwordrrr9r9r:r[s z+QuestionAnsweringPipeline._get_answer_indexcCs||sgSd}|D]}|dus$||kr|}qg}d}|D]$}t||}||||7}q6g}|D]}|||qd|S)z,Compute softmax probability over raw logits.Nr)mathexprg)r5scores max_scorerr exp_scores total_sumr]rr9r9r:r{fs   z*QuestionAnsweringPipeline._compute_softmaxcCsdd}d|j|}||}|dkr0|S|t|d}||\}}||\} } t|t| krl|Si} t| D]\} } | | | <qzd}|| vr| |}||vr||}|dur|Sd}|| vr| |}||vr||}|dur|S|||d}|S)z;Project the tokenized prediction back to the original text.cSsPg}t}t|D](\}}|dkr&q||t|<||qd|}||fS)Nrr)rd OrderedDictrvrrgr)rpns_chars ns_to_s_maprcns_textr9r9r: _strip_spaces~s   z@QuestionAnsweringPipeline._get_final_text.._strip_spacesrrr N)rr2tokenizefindrsix iteritems)r5 pred_textrrrrTrU orig_ns_textorig_ns_to_s_map tok_ns_texttok_ns_to_s_maptok_s_to_ns_mapr tok_indexorig_start_positionns_start_positionorig_end_positionns_end_position output_textr9r9r:r|s:     z)QuestionAnsweringPipeline._get_final_textcCsJtt|dddd}g}t|D]"\}}||kr6qF||dq"|S)z"Get the n-best logits from a list.cSs|dS)Nr r9rr9r9r:rrBz=QuestionAnsweringPipeline._get_best_indexes..Trr)rrvrg)r5logitsr!index_and_score best_indexesrrrr9r9r:rsz+QuestionAnsweringPipeline._get_best_indexes)rrrr rr)r/ __module__ __qualname____doc__r get_pipeline_support_task_listkeysr%r4rHrProrrwrxrr{rr __classcell__r9r9r8r:r$s)' 127 3)"ros.pathr&rdrr mindsporerrmindspore.commonrr`mindformers.modelsrZ#dataset.dataloader.squad_dataloaderrr auto_classr r mindformer_bookr base_pipelinerZtools.registerrrmodelsrr__all__registerPIPELINErr9r9r9r:s     pengcheng.mind-test/mindformers/pipeline/__pycache__/segment_anything_pipeline.cpython-39.pyc000066400000000000000000000363011471400134000331370ustar00rootroot00000000000000a ?cfi@s dZddlmZmZddlZddlZddlZddlm Z ddlm Z m Z ddl m Z mZddlmZddlmZmZdd lmZmZdd lmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)d d l*m+Z+d gZ,ej-ej.ddGdd d e+Z/dS)z"Image Classification Pipeline API.)OptionalUnionN)ops)TensorModel) AutoProcessor AutoModel)MindFormerBook) BaseModelBaseImageProcessor)MindFormerRegisterMindFormerModuleType)MaskData area_from_rlebatch_iteratorbatched_mask_to_boxbox_xyxy_to_xywhbuild_all_layer_point_gridscalculate_stability_scorecoco_encode_rlegenerate_crop_boxesis_box_near_crop_edge mask_to_rleremove_small_regions rle_to_maskuncrop_boxes_xyxy uncrop_masks uncrop_pointsbox_areanms) BasePipelineSegmentAnythingPipelinesegment_anything)aliasc seZdZdZedZd-ee e e fe e dfdd ZddZd d Zd d Zd dZd.e ddddZd/ddZd0e dddZddddZddZd1ddd(d)Zeeeed*d+d,ZZS)2r"a[Pipeline for image segment Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import SegmentAnythingPipeline >>> from mindformers import ViTImageProcessor >>> processor = ViTImageProcessor(size=224) >>> classifier = ImageClassificationPipeline( ... model='vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> classifier(np.uint8(np.random.random((5, 3, 255, 255)))) [[{'score': 0.0016654134, 'label': 'matchstick'}, {'score': 0.0015071577, 'label': 'theater curtain'}, {'score': 0.0014839625, 'label': 'ocarina'}, {'score': 0.0014319294, 'label': 'abaya'}, {'score': 0.0014109017, 'label': 'bottlecap'}], ..., {'score': 0.0014109018, 'label': 'bottlecap'}]] r#N)modelimage_processorc st|trh||jvrR|dur(t|j}t|tsFtdt|dt |}nt |d|jdt|t t fstdt|d|durt dt j|jddfd|i||jfi|tjd d |_|dS) NzEimage_processor should be inherited from BaseImageProcessor, but got .zI is not supported by ImageClassificationForPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z=ImageClassificationFoPipeline requires for a image_processor.Fmoder&axis) isinstancestr _support_listrfrom_pretrainedr&r TypeErrortyper ValueErrorr rsuper__init__ set_train reset_configrStackstack reset_image)selfr%r&kwargs __class__Y/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/segment_anything_pipeline.pyr5Ws(        z SegmentAnythingPipeline.__init__c Ksddi}ddi}ddi}dg}|D]}||vr"||||<q"d|d<gd}|D]"}||vrR||||<d|d<qRgd}|D]}||vr||||<qdg} | D]}||vr||||<q|||fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. seg_imageFr7) points_per_sidepoints_per_batchpred_iou_threshstability_score_threshstability_score_offsetbox_nms_thresh crop_n_layerscrop_nms_threshcrop_overlap_ratiocrop_n_points_downscale_factor point_gridsmin_mask_region_area output_modeT)rAmultimask_output return_logits)get) r;pipeline_parameterspreprocess_paramsforward_paramspostprocess_paramsZpreprocess_list_0itemZpreprocess_list_1 forward_list post_listr?r?r@_sanitize_parametersts, z,SegmentAnythingPipeline._sanitize_parameterscKsi}|d}|s|dd}|durX||\}}}||d<d|d<||d<||d<n:|jsfJd|j}d|d<|j|d<|j|d<|j|d<|dd} |d d} |d d} |d d} |j|| | | | d \} } } } | |d <| |d<| |d <| |d<n|dd}|dus Jdt|trDt |}t |t j }|j dd} |dd}|rt|jfi|t| |j|j\}}| |d<g|d<g|d<g|d<g|d<g|d<g|d<g|d<g|d<t||D]\}}|\}}}}|||||ddf}|j dd}||t|ddddf}|j||}|j||dd\} } }}|d|j|d|j|d|j|d||d||d| |d| q|S)zThe Preprocess For Task Args: inputs (dict): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. rAimageNfeatures original_size input_sizez*image must be set when image not in inputspointslabelsboxesmasks)r\ point_coords point_labelsr`rarbrc mask_inputsz5An image must be supported, when 'seg_image' is True.r7FZ image_list features_listoriginal_size_listinput_size_listcrop_boxes_listpoints_for_image_listpoint_coords_listpoint_labels_listr*T)r\rb multi_seg)rQpreprocess_image is_image_setr\r[r]preprocess_promptsr-r.cv2imreadcvtColor COLOR_BGR2RGBshaper7rrHrJzip set_imagenparrayrLappend)r;inputsrS model_inputsrArZ input_imager\r]rbrcr`ra orig_sizer7 crop_boxesZ layer_idxscrop_boxlayer_idx0y0x1y1Z crop_imageZcrop_image_sizeZ points_scalepoints_for_image_r?r?r@ preprocesss                  z"SegmentAnythingPipeline.preprocessc! sB|dd}|dd}|d}|st|j|d|d|d|d |d |d |d |d ||d \}}}|||dS|d } | \} } |d} |d} |d}|d}t}t|dD]D\}}| |}||}| |}|||d|}|d|}t}t|j|||D]j\}}}|j||||||dd\}}}|}|}|}t|d|jd|jdf|d|j |jdddd}~|j dkr|d |j k}| |t |d!|jj |j|d"<|jdkr|d"|jk}| ||d!|jj k|d!<t|d!|d <t|d dd| | g}|sB| |t|d!| | |d!<t|d!|d#<|d!=||q|d }|d }t|||jd$} | | t|d |d <t|d%|d%<tfd&d'tt|d#D|d(<||qt|dkr>|d }d)t|d(}t|||jd$} | | |S)*zThe Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. rOTrPFrArZr[r]r\rbrcr`rd) rZr[r]r\rbrcr`rdrOrP)ra low_res_masksiou_predictionsrjrkrlrirfrhrg)r[r]r\rbrcrOrPr*)r*r rr+)ra iou_predsr^rrastability_scorerles iou_thresholdr^csg|]}qSr?r?).0rrr?r@ ez3SegmentAnythingPipeline.forward..r?)rQr%r enumeraterrCasnumpyreshaperurepeatrDfilterrZmask_thresholdrFrErrallrrcatrrIrrrxryrangelenr)!r;r|rTrOrPrArarrr~Zorig_hZorig_wrjrkrlridatarr[rbrcrr]r\Z data_cropZpoints_for_image_batchZpoint_coords_batchZpoint_labels_batch data_batch keep_mask input_boxes input_scores keep_by_nmsr?rr@forwards             (  zSegmentAnythingPipeline.forwardc Ksr|d}|sh|djd}|dkrF|D]\}}|d||<q*n|D]\}}|||<qN|S|jdkr|||jt|j|j}|jdkrdd|dD|d <n.|jd krd d|dD|d <n |d|d <g}t t |d D]|}|d |t |d|t |d | |d ||d| g|d|t |d| d} || q|S)zPostprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result. Return: classification results. rArarr coco_rlecSsg|] }t|qSr?)rrrler?r?r@rrz7SegmentAnythingPipeline.postprocess..rZ segmentations binary_maskcSsg|] }t|qSr?)rrr?r?r@rrr`rr^rr) segmentationareabboxZ predicted_iourbrr)ruitemsrrMpostprocess_small_regionsmaxrGrIrNrrrrtolistrVrz) r; model_outputsrUrAbskvZ curr_annsidxannr?r?r@ postprocessss@        z#SegmentAnythingPipeline.postprocessRGB) image_formatreturncCs~t|tr"t|}t|tj}|dvs:Jd|d||jjkrX|ddddf}|jdd}| |\}}|||fS)a Calculates the image embeddings for the provided image, allowing masks to be predicted with the 'predict' method. Arguments: image (np.ndarray): The image for calculating masks. Expects an image in HWC uint8 format, with pixel values in [0, 255]. image_format (str): The color format of the image, in ['RGB', 'BGR']. )rZBGRz+image_format must be in ['RGB', 'BGR'], is r'.Nr*re) r-r.rqrrrsrtnetworkrrur&)r;rZrr\r}r]r?r?r@rns     z(SegmentAnythingPipeline.preprocess_imageFcCs|dur|j}|dkr&|dddf}|dkr|durLtj|jdtjd}|jj||}tj|dd}tj|dd}|r| ddd}| }nJ|dkr|durtj|jd|jdftjd}|jj ||}nt dt |t jd}t |t jd}|durf|j}|dkr4|jj||}tj|dd}n$|dkrP|jj||}nt d t |t jd}|durtj|dd}t |t jd}||||fS) zpreprocess promptsNr rer)dtyper+zpoints's ndim < 3.zboxes's ndim must be 1 or 2.)ndimrxonesruint32r& transformZ apply_coords expand_dims transposeZapply_coords_batchr3rmsfloat32Z apply_boxesZapply_boxes_batch)r;r\rbrcr`rarmrr?r?r@rpsB      z*SegmentAnythingPipeline.preprocess_prompts)rcCs>||||\}}}|j||_||_||_d|_dS)NT)r:rnrZ image_encoderr[r\r]ro)r;rZrr}r\r]r?r?r@rws z!SegmentAnythingPipeline.set_image)rcCsd|_d|_d|_d|_dS)zResets the currently set image.FN)ror[r\r]r;r?r?r@r:sz#SegmentAnythingPipeline.reset_imagecCs"|jstd|jdusJ|jS)z Returns the image embeddings for the currently set image, with shape 1xCxHxW, where C is the embedding dimension and (H,W) are the embedding spatial dimension of SAM (typically C=256, H=W=64). zCAn image must be set with .set_image(...) to generate an embedding.N)ro RuntimeErrorr[rr?r?r@get_image_embeddings z+SegmentAnythingPipeline.get_image_embedding @)\(?ffffff?rffffff?r?r rcKsJ|p|dd|_|p |dd|_|p2|dd|_|pD|dd|_|pV|dd |_|ph|d d |_|pz|d d |_|p|dd |_| p|dd|_ | p|dd|_ | p|dd|_ | p|dd |_ | p|dd|_ |jdu|j duksJd|jdur(t|j|j|j |_ |j dvsFJd|j ddS)z reset configrBrrCrrrErrFrrGrrHrrIrJrrKr rLNrMrNrz>Exactly one of points_per_side or point_grid must be provided.)rZuncompressed_rlerzUnknown output_mode r')rQrBrCrDrErFrGrHrIrJrKrLrMrNr)r;rBrCrDrErFrGrHrIrJrKrLrMrNr<r?r?r@r7s:   z$SegmentAnythingPipeline.reset_config) mask_datamin_area nms_threshrcCs|ds |Sg}g}|dD]b}t|}t||dd\}}| } t||dd\}}| oZ| } |tj|dd|t| qtj|dd} t| } | tj } t |} t | | |d}|D]F}||dkrtj| |dd}t |d|d|<| ||d |<q| ||S) z Removes small disconnected regions and holes in masks, then reruns box NMS to remove any new duplicates. Edits mask_data in place. Requires open-cv as a dependency. rZholesr(Zislandsrr+rrr`)rrrzrxrfloat concatenaterastyperryrrr)r;rrrZ new_masksscoresrmaskchanged unchangedrar`rrrZi_mask mask_tensorr?r?r@rCs0       z1SegmentAnythingPipeline.postprocess_small_regions)N)r)NNNNF)r) rrrrrrrrrr Nrr)__name__ __module__ __qualname____doc__r get_pipeline_support_task_listkeysr/rr.r rrr r5rYrrrrnrprwr:rr7rintrr __classcell__r?r?r=r@r"4s\&ax9  1  0)0rtypingrrrqnumpyrx mindsporerrrrmindformers.auto_classrrmindformers.mindformer_bookr mindformers.modelsr r mindformers.tools.registerr r Zmindformers.models.samrrrrrrrrrrrrrrrrrr base_pipeliner!__all__registerPIPELINEr"r?r?r?r@s  P pengcheng.mind-test/mindformers/pipeline/__pycache__/text_classification_pipeline.cpython-39.pyc000066400000000000000000000212721471400134000336340ustar00rootroot00000000000000a ?cf(@sdZddlZddlZddlmZmZmZddl m Z m Z ddl m Z ddlmZdd lmZmZdd lmZmZdd lmZdgZejejd d GdddeZdS)TextClassificationPipelineN)opsTensorModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel Tokenizer)labelstext_classification)aliascsbeZdZdZedZdfdd ZddZ dd Z d d Z d d Z ddZ ddZZS)raPipeline for text classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import TextClassificationPipeline >>> from mindformers import AutoTokenizer, BertForMultipleChoice, AutoConfig >>> input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", ... "i don't know um do you do a lot of camping-I know exactly."] >>> tokenizer = AutoTokenizer.from_pretrained('txtcls_bert_base_uncased_mnli') >>> txtcls_mnli_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased_mnli') >>> model = BertForMultipleChoice(txtcls_mnli_config) >>> txtcls_pipeline = TextClassificationPipeline(task='text_classification', ... model=model, ... tokenizer=tokenizer, ... max_length=model.config.seq_length, ... padding="max_length") >>> results = txtcls_pipeline(input_data, top_k=1) >>> print(results) [[{'label': 'neutral', 'score': 0.9714198708534241}], [{'label': 'contradiction', 'score': 0.9967639446258545}]] rNc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdtj||fi|dS)Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.) isinstancestr _support_listospathisdirrfrom_pretrained tokenizerrr TypeErrortype ValueError __class____name__rrsuper__init__)selfmodelrkwargsr\/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/text_classification_pipeline.pyr"@s"      z#TextClassificationPipeline.__init__c sd|vrtd|jjdi}gd}i}|D]}||vr0||||<q0gdfdd|D}d|vr|d|d<d |vr|d |d <t|d std |d d t|||fS) z=sanitize parameters for preprocess, forward, and postprocess. batch_sizezThe z? does not support batch inference, please remove the batch_size)top_ktop_p do_sample eos_token_idrepetition_penalty max_length)r*r+r,r-r.r/paddingcsi|]\}}|vr||qSr'r').0kvpreprocess_key_namer'r( fzCTextClassificationPipeline._sanitize_parameters..r*datasetzThe dataset does not support z, but only support )rrr getitemsrkeys)r#pipeline_parameterspostprocess_paramsforward_key_nameforward_kwargsitempreprocess_paramsr'r4r(_sanitize_parametersVs&z/TextClassificationPipeline._sanitize_parametersc Cst|d}t|d}t|d}dd|D}dd|D}dgt|dgt|d}|dg|t|}||dd}t|}|dg|t|}dg|dg||} || |fS)a  process of two sentences relationship classification Args: inputs_zero (str): the first sentence inputs_one (str): the second sentence Return: processed inputs, mask, token_type about two sentences input_idscSsg|]}|dkr|qSrr'r1xr'r'r( r7z=TextClassificationPipeline.inputs_process..cSsg|]}|dkr|qSrDr'rEr'r'r(rGr7rr N)lenlistasnumpy) r# inputs_zero inputs_oneZ len_inputsZinputs_zero_inputZinputs_one_input token_typeinputsZlen_inputs_maskmaskr'r'r(inputs_processqs  z)TextClassificationPipeline.inputs_processcKsht|tstdt}|jjdkrb|j|fddi|}|d}|d}||d||ddSd |vrrtd |d }| d d }| d d }|j|df||dd|} |j|df||dd|} | | | \}} } i} t t j|t jd| d<t t j| t jd| d<t t j| t jd| d<t}|| dd|| dd|| ddddS)ae Preprocess of text classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed text. zInputs type must be str gpt2_txtclsreturn_tensorsmsrCattention_maskr)rCrT-z+two texts of text pair should be split by -r/r0)r/r0rRr )dtypetoken_type_idsN)rC input_mask token_type_id label_ids)rrrr ExpandDimsr$ model_namersplitpoprPr from_numpynparrayint32)r#rNrA expand_dimstokensrCrTr/r0rKrLrOrMZ inputs_finalr'r'r( preprocesssH        z%TextClassificationPipeline.preprocesscKs |dd|jfi|}|S)z Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. NoneN)r_network)r# model_inputsforward_params output_idsr'r'r(forwards z"TextClassificationPipeline.forwardcCs0tj|ddd}t||}||jdddS)NT)axiskeepdims)ramaxexpsum)r#outputsmaxesZ shifted_expr'r'r(softmaxsz"TextClassificationPipeline.softmaxc s|dd}|dd}|r6ddtt|DnddttdD|d}|}||}fd d t|D}|jd d d d|dur|d|}|gS)z Postprocess Args: model_outputs (dict): outputs of forward process. Return: Classification results r*Nr8cSsi|]\}}||qSr'r'r1idlabelr'r'r(r6r7z:TextClassificationPipeline.postprocess..cSsi|]\}}||qSr'r'rvr'r'r(r6r7mnlircs"g|]\}}||dqS))rxscore)r@)r1irzid2labelr'r(rGsz:TextClassificationPipeline.postprocess..cSs|dS)Nrzr')rFr'r'r(r7z8TextClassificationPipeline.postprocess..T)keyreverse)r_ enumeraterr9rJrusort)r# model_outputsr=r*r8rsscoresZ dict_scoresr'r|r( postprocesss     z&TextClassificationPipeline.postprocess)N)r __module__ __qualname____doc__r get_pipeline_support_task_listr;rr"rBrPrfrlrur __classcell__r'r'r&r(rs-)ros.pathrnumpyra mindsporerrr auto_classrrmindformer_bookr base_pipeliner Ztools.registerr r modelsrrdataset.labelsr__all__registerPIPELINErr'r'r'r(s   pengcheng.mind-test/mindformers/pipeline/__pycache__/text_generation_pipeline.cpython-39.pyc000066400000000000000000000116261471400134000327760ustar00rootroot00000000000000a ?cf@sdZddlZddlmZmZddlZddlmZmZddl m Z m Z ddl m Z dd lmZdd lmZmZdd lmZmZdgZd d ZejejddGdddeZdS)TextGenerationPipelineN)UnionOptional)TensorModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel BaseTokenizercCs&g}|D]}|t|q|S)N)appendr get_model_support_listget)Zsupport_model_list support_listZ support_modelrX/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/text_generation_pipeline.py_setup_support_list srtext_generation)aliascseZdZdZeddgZdZdeee e fe e dfdd Z d d Zeeeefd d d ZedddZedddZZS)raPipeline for Text Generation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and tokenizer's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.pipeline import TextGenerationPipeline >>> text_generate = TextGenerationPipeline("gpt2") >>> output = text_generate("I love Beijing, because ") gpt2glmrN)model tokenizerc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdtj||fi|dS)Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.) isinstancestr _support_listospathisdirrfrom_pretrainedrr r TypeErrortype ValueError __class____name__rrsuper__init__)selfrrkwargsr)rrr,>s"      zTextGenerationPipeline.__init__cKsfddg}i}|D]}||vr||||<qi}gd}i}|D]}||vr@||||<q@|||fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. keysadd_special_tokens)top_ktop_p do_sample eos_token_idrepetition_penalty max_lengthseed)r)r-pipeline_parameterspreprocess_keyspreprocess_paramsitempostprocess_paramsforward_key_nameforward_kwargsrrr_sanitize_parametersVsz+TextGenerationPipeline._sanitize_parameters)inputscKsv|dd}t|trZ|dd}d}|r6|d|n|}||}t|tjrZ|}|j|d|dd}d|iS) zThe Preprocess For Translation Args: inputs (Union[str, dict, Tensor]): The text to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed text. r1Tr0Ntext src_language)return_tensorsr1 input_ids)rrdict mindsporerasnumpytolistr)r-rAr;r1r0default_src_language_name feature_namerErrr preprocessks     z!TextGenerationPipeline.preprocess) model_inputscKs0|dd|d}|jj|fi|}d|iS)zThe Forward Process of Model Args: inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. NoneNrE output_ids)popnetworkgenerate)r-rMforward_paramsrErOrrrforwards zTextGenerationPipeline.forward) model_outputscKs$|jj|ddd}|jd|igS)zPostprocess Args: model_outputs (dict): Outputs of forward process. Return: translation results. rOT)skip_special_tokens_text)rdecode return_name)r-rUr=outputsrrr postprocesss z"TextGenerationPipeline.postprocess)N)r* __module__ __qualname____doc__rr!rYrr rrrrr,r@rFrrLrTr[ __classcell__rrr/rr's  )r^os.pathr"typingrrrGrr auto_classrr mindformer_bookr base_pipeliner Ztools.registerr rmodelsrr__all__rregisterPIPELINErrrrrs  pengcheng.mind-test/mindformers/pipeline/__pycache__/token_classification_pipeline.cpython-39.pyc000066400000000000000000000205431471400134000337700ustar00rootroot00000000000000a ?cf'@sdZddlZddlZddlmZmZddlm Z m Z ddl m Z ddl mZdd lmZmZdd lmZmZdgZejejd d Gd ddeZdS)TokenClassificationPipelineN)opsModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel Tokenizertoken_classification)aliascsZeZdZdZedZdfdd ZddZ dd Z d d Z d d Z ddZ ZS)ruPipeline for token classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. id2label : a dict which maps label id to label str. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import TokenClassificationPipeline >>> from mindformers import AutoTokenizer, BertForTokenClassification, AutoConfig >>> from mindformers.dataset.labels import cluener_labels >>> id2label = {label_id: label for label_id, label in enumerate(cluener_labels)} >>> input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。"] >>> tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') >>> ner_dense_cluener_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese_cluener') >>> model = BertForTokenClassification(ner_dense_cluener_config) >>> tokcls_pipeline = TokenClassificationPipeline(task='token_classification', ... model=model, ... id2label=id2label, ... tokenizer=tokenizer, ... max_length=model.config.seq_length, ... padding="max_length") >>> results = tokcls_pipeline(input_data) >>> print(results) [[{'entity_group': 'address', 'start': 6, 'end': 8, 'score': 0.52329, 'word': '日内瓦'}, {'entity_group': 'name', 'start': 12, 'end': 25, 'score': 0.83922, 'word': 'freresoltramar'}]] rNc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjd|durt |jjd||_d|_tj||fi|dS) Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.z6 requires for a dict which maps label id to label str.) isinstancestr _support_listospathisdirrfrom_pretrained tokenizerrr TypeErrortype ValueError __class____name__r rid2label input_textsuper__init__)selfmodelr rkwargsr]/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/token_classification_pipeline.pyr#Ds*      z$TokenClassificationPipeline.__init__c s`d|vrtd|jjddhddgfdd|D}fd d|D|ifS) z=sanitize parameters for preprocess, forward, and postprocess. batch_sizezThe z? does not support batch inference, please remove the batch_sizer max_lengthpaddingcsi|]\}}|vr||qSr(r(.0kv)preprocess_key_namer(r) jzDTokenClassificationPipeline._sanitize_parameters..csi|]\}}|vr||qSr(r(r-)postprocess_paramsr(r)r2kr3)rrritems)r$pipeline_parameterspreprocess_paramsr()r4r1r)_sanitize_parametersasz0TokenClassificationPipeline._sanitize_parameterscKs|t|tstd||_|dd}|dd}|j|f||dd|}t}||dd||d d||d dd S) af Preprocess of token classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed text. zInputs type must be strr+r,ms)r+r,return_tensors input_idsrattention_masktoken_type_ids)r< input_maskr>)rrrr!poprr ExpandDims)r$inputsr7r+r, expand_dimsr(r(r) preprocessos       z&TokenClassificationPipeline.preprocesscKs$|jd|jfi|}d|iS)z Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. Flogits)r% set_trainnetwork)r$ model_inputsforward_paramsrEr(r(r)forwards z#TokenClassificationPipeline.forwardc sT|d}tj|ddd}t||}||jddd}tj|dd}tj|dd}fdd|D} d d|D} g} t| | D]\} } g}| }|D]}i}|d |d <|d |d <|d|d<t| |d |dd |dd |d |d<t |dd|d<j |d |dd |d<| |q| |q| S)z Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results rET)axiskeepdimsr)rLcs&g|]}fdd|ddDqS)csg|]}j|qSr()r )r.id_r$r(r) r3zFTokenClassificationPipeline.postprocess...r rKr()r.pred_idsrOr(r)rPr3z;TokenClassificationPipeline.postprocess..cSsg|]}|ddqS)r rKr()r. best_scoresr(r(r)rPr3rZ entity_groupr startendscoreword) asnumpynpmaxexpsumargmaxtolistzipget_entities_biosroundr!append)r$ model_outputsr4rEmaxes shifted_expprobsbatch_pred_idsZbatch_best_scores pred_pathsrRZ total_resultZ pred_path best_scoreZ single_result pred_entities pred_entityZ entity_resultr(rOr) postprocesss2       z'TokenClassificationPipeline.postprocesscCs>g}gd}t|D]"\}}|drz|ddkr>||gd}||d<||d<|dd|d<||gd}|dr|ddkr||gd}||d<|dd|d<q|d r|ddkr|dd}||dkr||d<|t|dkr8||q|ddkr0||gd}q|S) a_Gets entities from sequence. note: BIOS Args: seq (list): sequence of labels. Returns: list: list of (chunk_type, chunk_start, chunk_end). Example: # >>> seq = ['B-PER', 'I-PER', 'O', 'S-LOC'] # >>> get_entity_bios(seq) [['PER', 0, 1], ['LOC', 3, 3]] )rKrKrKzS-rrKr -rzB-zI-) enumerate startswithrbsplitlen)r$seqchunkschunkindxtag entity_typer(r(r)r`s8            z-TokenClassificationPipeline.get_entities_bios)N)r __module__ __qualname____doc__rget_pipeline_support_task_listkeysrr#r8rDrJrlr` __classcell__r(r(r'r)rs!()rzos.pathrnumpyrY mindsporerr auto_classrrmindformer_bookr base_pipeliner Ztools.registerr r modelsr r__all__registerPIPELINErr(r(r(r)s  pengcheng.mind-test/mindformers/pipeline/__pycache__/translation_pipeline.cpython-39.pyc000066400000000000000000000111011471400134000321210ustar00rootroot00000000000000a ?cf|@sdZddlZddlmZmZddlZddlmZmZddl m Z m Z ddl m Z dd lmZdd lmZmZdd lmZmZdgZejejd d GdddeZdS)TranslationPipelineN)UnionOptional)TensorModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel BaseTokenizer translation)aliascseZdZdZedZdZdee e e fe e dfdd Zdd Zee eefd d d Zed ddZedddZZS)raPipeline for Translation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and tokenizer's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.pipeline import TranslationPipeline >>> translator = TranslationPipeline("t5_small") >>> output = translator("abc") t5rN)model tokenizerc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdtj||fi|dS)Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.) isinstancestr _support_listospathisdirrfrom_pretrainedrr r TypeErrortype ValueError __class____name__rrsuper__init__)selfrrkwargsr!T/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/translation_pipeline.pyr$7s"      zTranslationPipeline.__init__cKsddg}i}|D]}||vr||||<qi}gd}i}|D]}||vr>||||<q>|||fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. keys)top_ktop_p do_sample eos_token_idrepetition_penalty max_length)get)r%pipeline_parametersZpreprocess_keyspreprocess_paramsitempostprocess_paramsZforward_key_nameZforward_kwargsr(r(r)_sanitize_parametersOsz(TranslationPipeline._sanitize_parameters)inputscKsht|trN|dd}d}|r*|d|n|}||}t|tjrN|}|j|ddd}d|iS)zThe Preprocess For Translation Args: inputs (Union[str, dict, Tensor]): The text to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed text. r*NtextZ src_language)return_tensors input_ids)rdictr1 mindsporerasnumpytolistr)r%r7r3r*Zdefault_src_language_name feature_namer:r(r(r) preprocessds    zTranslationPipeline.preprocess) model_inputscKs0|dd|d}|jj|fi|}d|iS)zThe Forward Process of Model Args: inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. NoneNr: output_ids)popnetworkgenerate)r%rAforward_paramsr:rCr(r(r)forwardzs zTranslationPipeline.forward) model_outputscKs$|jj|ddd}|jd|igS)zPostprocess Args: model_outputs (dict): Outputs of forward process. Return: translation results. rCT)skip_special_tokens_text)rdecode return_name)r%rIr5outputsr(r(r) postprocesss zTranslationPipeline.postprocess)N)r" __module__ __qualname____doc__r get_model_support_listrrMrrrrrrr$r6r;rr@rHrO __classcell__r(r(r'r)r s  )rRos.pathrtypingrrr<rr auto_classrr mindformer_bookr base_pipeliner Ztools.registerr rmodelsrr__all__registerPIPELINErr(r(r(r)s  zero_shot_image_classification_pipeline.cpython-39.pyc000066400000000000000000000167221471400134000357530ustar00rootroot00000000000000pengcheng.mind-test/mindformers/pipeline/__pycache__a ?cf#@sdZddlmZmZddlmZddlmZddl m Z ddl m Z ddl mZmZmZddlmZmZdd lmZmZd d lmZd d lmZejejddGdddeZdS)#ZeroShotImageClassificationPipeline)UnionOptional)Model) operations)MindFormerBook) load_image) BaseModelBaseImageProcessor Tokenizer)MindFormerRegisterMindFormerModuleType) AutoProcessor AutoModel) BasePipeline) BaseTokenizerzero_shot_image_classification)aliascseZdZdZedZdee e e fe e e edfdd ZddZed d d Zed d dZedddZZS)raPipeline For Zero Shot Image Classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer for text processing. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model, tokenizer, and image_processor's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.tools.image_tools import load_image >>> from mindformers.pipeline import ZeroShotImageClassificationPipeline >>> classifier = ZeroShotImageClassificationPipeline( ... model='clip_vit_b_32', ... candidate_labels=["sunflower", "tree", "dog", "cat", "toy"], ... hypothesis_template="This is a photo of {}." ... ) >>> img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." ... "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> classifier(img) [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, {'score': 2.396818e-06, 'label': 'cat'}]] rN)model tokenizerimage_processorc st|tr||jvr|dur(t|j}t|tsFtdt|d|durZt|j }t|t sxtdt|dt |}nt |d|jdt|t tfstdt|d|durt d|durt dtj|||fi|dS)NzEimage_processor should be inherited from BaseImageProcessor, but got .z@tokenizer should be inherited from PretrainedTokenizer, but got zN is not supported by ZeroShotImageClassificationPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z=ZeroShotImageClassificationPipeline requires for a tokenizer.zCZeroShotImageClassificationPipeline requires for a image_processor.) isinstancestr _support_listrfrom_pretrainedrr TypeErrortyperr r ValueErrorr rsuper__init__)selfrrrkwargs __class__g/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/zero_shot_image_classification_pipeline.pyr"Bs2          z,ZeroShotImageClassificationPipeline.__init__cKsPi}i}gd}|D]}||vr||||<qd|vrF|d|d<|i|fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. )candidate_labelshypothesis_template max_lengthpaddingreturn_tensorstop_k)get)r#pipeline_parameterspreprocess_paramspostprocess_paramsZpre_listitemr'r'r(_sanitize_parameterscsz8ZeroShotImageClassificationPipeline._sanitize_parameters)inputsc  s|dd}|dd|dd}|dd}|dd }|durLtd dur\td t|trn|d }t|trt|}||}fd d|D}|j||||dd} || |dS)aPreprocess of ZeroShotImageClassificationPipeline Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. candidate_labels (List[str]): The candidate labels for classification. hypothesis_template (Optional[str]): Prompt for text input. max_length (Optional[int]): Max length of tokenizer's output padding (Optional[Union[False, "max_length"]]): Padding for max_length return_tensors (Optional["ms"]): The type of returned tensors Return: Processed data. Raises: ValueError: If candidate_labels or hypothesis_template is None. r)Nr*za picture of {}.r+Mr,r-mszTcandidate_labels are supposed for ZeroShotImageClassificationPipeline, but got None.zVhypothesis_template is supposed for ZeroShotImageClassificationPipeline, but got None.imagecsg|]}|qSr')format).0Zcandidate_labelr*r'r( szBZeroShotImageClassificationPipeline.preprocess..)r+r,r- input_ids)image_processedr=r))popr rdictrrrr) r#r5r1r)r+r,r-r> sentencesr=r'r;r( preprocessvs6         z.ZeroShotImageClassificationPipeline.preprocess) model_inputscKsJ|dd|d}|d}|||\}}t|}||ddS)zForward process Args: model_inputs (dict): Outputs of preprocess. Return: Probs dict. NoneNr>r=r))probsr))r?networkPSoftmaxasnumpy)r#rCforward_paramsr>r=logits_per_image_rEr'r'r(forwards z+ZeroShotImageClassificationPipeline.forward) model_outputsc Ksv|dd}|d}|d}g}|D]L}tt||ddd}|dur\|dt|t|}|dd |Dq$|S) zPostprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result Return: Classification results. r.Nr)rEcSs |d S)Nrr')xr'r'r(zAZeroShotImageClassificationPipeline.postprocess..)keycSsg|]\}}||dqS))scorelabelr')r: score_itemrTr'r'r(r<szCZeroShotImageClassificationPipeline.postprocess..)r?sortedzipminlenappend) r#rNr2r.labelsscoresoutputsrS sorted_resr'r'r( postprocesss   z/ZeroShotImageClassificationPipeline.postprocess)NN)__name__ __module__ __qualname____doc__rget_pipeline_support_task_listkeysrrrr rrrr r"r4r@rBrMr_ __classcell__r'r'r%r(rs!+N)rctypingrr mindsporer mindspore.opsrrGmindformers.mindformer_bookrmindformers.tools.image_toolsrmindformers.modelsr r r mindformers.tools.registerr r mindformers.auto_classrr base_pipelinermodelsrregisterPIPELINErr'r'r'r(s      pengcheng.mind-test/mindformers/pipeline/base_pipeline.py000066400000000000000000000220631471400134000242170ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/base.py # ============================================================================ """BasePipeline""" from abc import ABC, abstractmethod from typing import Optional, Union import numpy as np from tqdm import tqdm from mindspore import Tensor, Model from mindspore.dataset import ( GeneratorDataset, VisionBaseDataset, SourceDataset, MappableDataset ) from mindspore.dataset.engine.datasets import BatchDataset, RepeatDataset from mindformers.tools import logger from mindformers.mindformer_book import print_dict from ..auto_class import AutoModel from ..models import BaseModel, BaseTokenizer, BaseImageProcessor class BasePipeline(ABC): r""" Base Pipeline For All Task Pipelines Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): The tokenizer of model, it could be None if the model do not need tokenizer. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. """ _support_list = {} def __init__(self, model: Union[str, BaseModel, Model], tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): super(BasePipeline, self).__init__() self.model = model if isinstance(model, str) and model in self._support_list: self.network = AutoModel.from_pretrained(model) elif isinstance(model, BaseModel): self.network = model elif isinstance(model, Model): self.network = model.predict_network else: raise TypeError(f"model should be str or inherited from BaseModel or Model, but got type {type(model)}.") self.tokenizer = tokenizer self.image_processor = image_processor self._preprocess_params, self._forward_params, \ self._postprocess_params = self._sanitize_parameters(**kwargs) self.call_count = 0 self._batch_size = kwargs.pop("batch_size", None) def __call__(self, inputs: Union[GeneratorDataset, list, str, Tensor, np.array], batch_size: Optional[int] = None, **kwargs): r"""Call Method Args: inputs (Union[GeneratorDataset, list, str, etc]): The inputs of pipeline, the type of inputs depends on task. batch_size (Optional[int]): The batch size for a GeneratorDataset input, for other types of inputs, the batch size would be set to 1 by default. """ preprocess_params, forward_params, postprocess_params = self._sanitize_parameters(**kwargs) preprocess_params = {**self._preprocess_params, **preprocess_params} forward_params = {**self._forward_params, **forward_params} postprocess_params = {**self._postprocess_params, **postprocess_params} is_dataset = isinstance(inputs, ( GeneratorDataset, VisionBaseDataset, MappableDataset, SourceDataset)) is_list = isinstance(inputs, list) self.call_count += 1 if self.call_count == 20 and not is_dataset: logger.info("You seem to be using the pipeline sequentially for" " numerous samples. In order to maximize efficiency" " please set input a mindspore.dataset.GeneratorDataset.") if batch_size is None: if self._batch_size is None: batch_size = 1 else: batch_size = self._batch_size if batch_size > 1 and not is_dataset: batch_size = 1 logger.info("batch_size is set to 1 for non-dataset inputs.") if is_dataset: logger.info("dataset is processing.") if not isinstance(inputs, (BatchDataset, RepeatDataset)): inputs = inputs.batch(batch_size) outputs = [] for items in tqdm(inputs.create_dict_iterator()): outputs.extend(self.run_single(items, preprocess_params, forward_params, postprocess_params)) elif is_list: outputs = self.run_multi(inputs, preprocess_params, forward_params, postprocess_params) else: outputs = self.run_single(inputs, preprocess_params, forward_params, postprocess_params) return outputs @abstractmethod def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. Raises: NotImplementedError: If the method is not implemented. """ raise NotImplementedError("_sanitize_parameters not implemented") def run_single(self, inputs: Union[dict, str, np.array, Tensor], preprocess_params: dict, forward_params: dict, postprocess_params: dict): r"""Run Single method This function is used to run a single forward process for task. Args: inputs (Union[dict, str, etc]): The inputs of pipeline, the type of inputs depends on task. preprocess_params (dict): The parameter dict for preprocess. forward_params (dict): The parameter dict for model forward process. postprocess_params (dict): The parameter dict for postprocess. """ model_inputs = self.preprocess(inputs, **preprocess_params) model_outputs = self.forward(model_inputs, **forward_params) outputs = self.postprocess(model_outputs, **postprocess_params) return outputs def run_multi(self, inputs: Union[list, tuple], preprocess_params: dict, forward_params: dict, postprocess_params: dict): r"""Run Multiple Method This function is used to run a list input for task. Args: inputs (Union[list, tuple, iterator]): The iterable input for pipeline. preprocess_params (dict): The parameter dict for preprocess. forward_params (dict): The parameter dict for model forward process. postprocess_params (dict): The parameter dict for postprocess. """ outputs = [] for item in inputs: outputs.extend(self.run_single(item, preprocess_params, forward_params, postprocess_params)) return outputs @abstractmethod def preprocess(self, inputs: Union[dict, str, np.array, Tensor], **preprocess_params): r"""The Preprocess For Task Args: inputs (Union[dict, str, etc]): The inputs of pipeline, the type of inputs depends on task. preprocess_params (dict): The parameter dict for preprocess. Raises: NotImplementedError: If the method is not implemented. """ raise NotImplementedError("preprocess not implemented.") @abstractmethod def forward(self, model_inputs: Union[dict, str, np.array, Tensor], **forward_params): r"""The Forward Process of Model Args: model_inputs (Union[dict, str, etc]): The output of preprocess, the type of model_inputs depends on task. forward_params (dict): The parameter dict for model forward. Raises: NotImplementedError: If the method is not implemented. """ raise NotImplementedError("forward not implemented.") @abstractmethod def postprocess(self, model_outputs: Union[dict, str, np.array, Tensor], **postprocess_params): r"""The Postprocess of Task Args: model_outputs (Union[dict, str, etc]): The output of model forward, the type of model_outputs depends on task. postprocess_params (dict): The parameter dict for post process. Raises: NotImplementedError: If the method is not implemented. """ raise NotImplementedError("postprocess not implemented.") @classmethod def show_support_list(cls): """show_support_list""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) pengcheng.mind-test/mindformers/pipeline/build_pipeline.py000066400000000000000000000044511471400134000244050ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Pipeline API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_pipeline( config: dict = None, default_args: dict = None, module_type: str = 'pipeline', class_name: str = None, **kwargs): r"""Build Pipeline API. Instantiate the task pipeline from MindFormerRegister's registry. Args: config (dict): The task pipeline's config. default_args (dict): The default args of pipeline. module_type (str): The module type of MindFormerModuleType. Default: 'pipline'. class_name (str): The class name of task pipeline. Returns: The task pipeline instance by config. Examples: >>> from mindformers import build_pipeline >>> pipeline_config = {'type': 'zero_shot_image_classification', 'model': 'clip_vit_b_32', 'candidate_labels': ["sunflower", "tree", "dog", "cat", "toy"], 'hypothesis_template': "This is a photo of {}."} >>> classifier = build_pipeline(pipeline_config) >>> type(classifier) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.PIPELINE, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-test/mindformers/pipeline/fill_mask_pipeline.py000066400000000000000000000140561471400134000252510ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """TranslationPipeline""" import os.path # from tkinter import _Padding import numpy as np import mindspore from mindspore import ops, Tensor, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, Tokenizer __all__ = ['FillMaskPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="fill_mask") class FillMaskPipeline(BasePipeline): """ Pipeline for mask fill Args: model: a pretrained model (str or BaseModel) in _supproted_list. tokenizer : a tokenizer (None or Tokenizer) for text processing """ _support_list = MindFormerBook.get_model_support_list()['bert'] return_name = 'fillmask' def __init__(self, model, tokenizer=None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" PretrainedTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") self.input_text = "" super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): """sanitize parameters for preprocess, forward, and postprocess.""" if 'batch_size' in pipeline_parameters: raise ValueError(f"The {self.__class__.__name__} does not support batch inference, please remove the " f"batch_size") postprocess_params = {} forward_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length'] forward_kwargs = {} for item in forward_key_name: if item in pipeline_parameters: forward_kwargs[item] = pipeline_parameters.get(item) preprocess_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length', 'padding'] preprocess_params = {k: v for k, v in pipeline_parameters.items() if k in preprocess_key_name} return preprocess_params, forward_kwargs, postprocess_params def preprocess(self, inputs, **preprocess_params): """ Preprocess of mask fill Args: inputs (url, PIL.Image, tensor, numpy): the image to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed image. """ if isinstance(inputs, dict): inputs = inputs['text'] if isinstance(inputs, mindspore.Tensor): inputs = inputs.asnumpy().tolist() self.input_text = inputs if isinstance(inputs, str) else "" max_length = preprocess_params.pop("max_length", 128) padding = preprocess_params.pop("padding", "max_length") inputs = self.tokenizer(inputs, max_length=max_length, padding=padding, return_tensors="ms", **preprocess_params) expand_dims = ops.ExpandDims() return {"input_ids": expand_dims(inputs["input_ids"], 0), "input_mask": expand_dims(inputs["attention_mask"], 0), "token_type_id": expand_dims(inputs["token_type_ids"], 0), "masked_lm_positions": expand_dims(Tensor(self.tokenizer.mask_index), 0)} def forward(self, model_inputs, **forward_params): """ Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. """ forward_params.pop("None", None) output_ids = self.network(**model_inputs) return {"output_ids": output_ids} def postprocess(self, model_outputs, **postprocess_params): """ Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results """ outputs = model_outputs["output_ids"][-2].asnumpy() tokens_dict = [] max_tokens = np.argmax(outputs, axis=1) for ind, tokenid in enumerate(max_tokens): token = self.tokenizer.decode([int(tokenid),], skip_special_tokens=True) token = token.replace(' ', '') tokens_dict.append({'score': outputs[ind, tokenid], 'token': tokenid, 'token_str': token}) self.input_text = self.input_text.replace('[MASK]', token, 1) tokens_dict.append({'sequence': self.input_text}) return [tokens_dict,] pengcheng.mind-test/mindformers/pipeline/image_classification_pipeline.py000066400000000000000000000162301471400134000274410ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Pipeline API.""" from typing import Optional, Union import numpy as np from PIL import Image from mindspore.ops import operations as P from mindspore import Tensor, Model from mindformers.auto_class import AutoProcessor, AutoModel from mindformers.mindformer_book import MindFormerBook from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.image_tools import load_image from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.dataset.labels import labels from .base_pipeline import BasePipeline @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="image_classification") class ImageClassificationPipeline(BasePipeline): r"""Pipeline for image classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import ImageClassificationPipeline >>> from mindformers import ViTImageProcessor >>> processor = ViTImageProcessor(size=224) >>> classifier = ImageClassificationPipeline( ... model='vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> classifier(np.uint8(np.random.random((5, 3, 255, 255)))) [[{'score': 0.0016654134, 'label': 'matchstick'}, {'score': 0.0015071577, 'label': 'theater curtain'}, {'score': 0.0014839625, 'label': 'ocarina'}, {'score': 0.0014319294, 'label': 'abaya'}, {'score': 0.0014109017, 'label': 'bottlecap'}], ..., {'score': 0.0014109018, 'label': 'bottlecap'}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['image_classification'].keys() def __init__(self, model: Union[str, BaseModel, Model], image_processor: Optional[BaseImageProcessor] = None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ImageClassificationForPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if image_processor is None: raise ValueError("ImageClassificationFoPipeline" " requires for a image_processor.") super().__init__(model.set_train(mode=False), image_processor=image_processor, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {} postprocess_params = {} post_list = ["top_k", "candidate_labels"] for item in post_list: if item in pipeline_parameters: postprocess_params[item] = pipeline_parameters.get(item) return preprocess_params, {}, postprocess_params def preprocess(self, inputs: (Union[str, Image.Image, Tensor, np.ndarray]), **preprocess_params): r"""The Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. """ if isinstance(inputs, dict): inputs = inputs['image'] if isinstance(inputs, str): inputs = load_image(inputs) image_processed = self.image_processor(inputs) return {"image_processed": image_processed} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ forward_params.pop("None", None) image_processed = model_inputs["image_processed"] logits_per_image = self.network(image_processed)[0] probs = P.Softmax()(logits_per_image).asnumpy() return {"probs": probs} def postprocess(self, model_outputs, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result. Return: classification results. """ top_k = postprocess_params.pop("top_k", 3) candidate_labels = postprocess_params.pop("candidate_labels", 'imagenet') scores = model_outputs['probs'] outputs = [] if isinstance(candidate_labels, str): inputs_labels = labels.get(candidate_labels) elif isinstance(candidate_labels, list): inputs_labels = candidate_labels else: raise ValueError(f"The candidate_labels should be dataset name (str) or custom labels (list)" f" but got {type(candidate_labels)}") if inputs_labels is None: raise ValueError(f"The custom candidate_labels is None or " f"the input dataset labels name is not supported yet.") for score in scores: sorted_res = sorted(zip(score, inputs_labels), key=lambda x: -x[0]) if top_k is not None: sorted_res = sorted_res[:min(top_k, len(inputs_labels))] outputs.append([{"score": score_item, "label": label} for score_item, label in sorted_res]) return outputs pengcheng.mind-test/mindformers/pipeline/image_to_text_generation_pipeline.py000066400000000000000000000155311471400134000303520ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Pipeline API.""" from typing import Optional, Union import numpy as np from PIL import Image from mindspore import Tensor, Model from mindspore.ops import operations as P from mindformers.auto_class import AutoProcessor, AutoModel from mindformers.mindformer_book import MindFormerBook from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.image_tools import load_image from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from .base_pipeline import BasePipeline __all__ = ['ImageToTextGenerationPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="image_to_text_generation") class ImageToTextGenerationPipeline(BasePipeline): r"""Pipeline for image to text generation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. """ _support_list = MindFormerBook.get_pipeline_support_task_list()['image_to_text_generation'].keys() def __init__(self, model: Union[str, BaseModel, Model], image_processor: Optional[BaseImageProcessor] = None, tokenizer=None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ImageToTextGenerationPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if image_processor is None: raise ValueError("ImageToTextGenerationPipeline" " requires for a image_processor.") self.hypothesis_template = kwargs.pop("hypothesis_template", "{}") super().__init__(model.set_train(mode=False), image_processor=image_processor, tokenizer=tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {} postprocess_params = {} forward_params = {} post_list = ["top_k"] pre_list = ["hypothesis_template", "max_length", "padding"] forward_list = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length', 'seed'] for item in post_list: if item in pipeline_parameters: postprocess_params[item] = pipeline_parameters.get(item) for item in pre_list: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) for item in forward_list: if item in pipeline_parameters: forward_params[item] = pipeline_parameters.get(item) return preprocess_params, forward_params, postprocess_params def preprocess(self, inputs: (Union[str, Image.Image, Tensor, np.ndarray]), **preprocess_params): r"""The Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. """ if isinstance(inputs, dict): inputs = inputs['image'] if isinstance(inputs, str): inputs = load_image(inputs) if isinstance(inputs, list): inputs = [load_image(item) for item in inputs] image_processed = self.image_processor(inputs) max_length = preprocess_params.pop("max_length", 32) padding = preprocess_params.pop("padding", "max_length") hypothesis_template = preprocess_params.pop("hypothesis_template", None) if hypothesis_template is not None: self.hypothesis_template = hypothesis_template prompt_list = [self.hypothesis_template] * image_processed.shape[0] prompt_processed = self.tokenizer(prompt_list, max_length=max_length, padding=padding, return_tensors="ms", add_special_tokens=False, **preprocess_params) prompt_input_ids = prompt_processed["input_ids"] if len(prompt_input_ids.shape) == 1: prompt_input_ids = P.ExpandDims()(prompt_input_ids, 0) return {"image_processed": image_processed, "prompt_input_ids": prompt_input_ids} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ image_processed = model_inputs["image_processed"] prompt_input_ids = model_inputs["prompt_input_ids"] output_ids_per_image = self.network.generate_text_for_image(image_processed, prompt_input_ids) return {"output_ids": output_ids_per_image} def postprocess(self, model_outputs, **postprocess_params): output_ids = model_outputs["output_ids"] outputs = self.tokenizer.decode(output_ids, skip_special_tokens=True) return outputs pengcheng.mind-test/mindformers/pipeline/masked_image_modeling_pipeline.py000066400000000000000000000145651471400134000276010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Pipeline API.""" import os from typing import Optional, Union import numpy as np from PIL import Image from mindspore import Tensor, Model from mindformers.auto_class import AutoProcessor, AutoModel from mindformers.mindformer_book import MindFormerBook from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.image_tools import load_image from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from .base_pipeline import BasePipeline __all__ = ['MaskedImageModelingPipeline'] from ..tools.utils import LOCAL_DEFAULT_PATH @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="masked_image_modeling") class MaskedImageModelingPipeline(BasePipeline): r"""Pipeline for masked image modeling Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import MaskedImageModelingPipeline >>> from mindformers import ViTMAEImageProcessor >>> processor = ViTMAEImageProcessor(size=224) >>> reconstructor = MaskedImageModelingPipeline( ... model='mae_vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> reconstructor(np.uint8(np.random.random((5, 3, 255, 255)))) """ _support_list = MindFormerBook.get_pipeline_support_task_list()['masked_image_modeling'].keys() def __init__(self, model: Union[str, BaseModel, Model], image_processor: Optional[BaseImageProcessor] = None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ImageClassificationForPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if image_processor is None: raise ValueError("MaskedImageModelingPipeline" " requires for a image_processor.") super().__init__(model.set_train(mode=False), image_processor=image_processor, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {} postprocess_params = {} post_list = [] for item in post_list: if item in pipeline_parameters: postprocess_params[item] = pipeline_parameters.get(item) return preprocess_params, {}, postprocess_params def preprocess(self, inputs: (Union[str, Image.Image, Tensor, np.ndarray]), **preprocess_params): r"""The Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be reconstructed. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. """ if isinstance(inputs, dict): inputs = inputs['image'] if isinstance(inputs, str): inputs = load_image(inputs) image_processed = self.image_processor(inputs) return {"image_processed": image_processed} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ forward_params.pop("None", None) image_processed = model_inputs["image_processed"] images_array = self.network(*image_processed) images_array = images_array.asnumpy() return {"images_array": images_array} def postprocess(self, model_outputs, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. Return: classification results. """ images_array = model_outputs['images_array'] outputs = [] for i, image_array in enumerate(images_array): image_array = image_array.transpose((1, 2, 0)) mean = np.array(self.image_processor.normalize.mean) std = np.array(self.image_processor.normalize.std) image_array = image_array * std + mean image_array = image_array * 255 image_array = image_array.astype(np.uint8) reconstruct_image = Image.fromarray(image_array) image_path = os.path.join(LOCAL_DEFAULT_PATH, f'output_image{i}.jpg') reconstruct_image.save(image_path) outputs.append({"info": image_path, "data": reconstruct_image}) return outputs pengcheng.mind-test/mindformers/pipeline/pipeline.py000066400000000000000000000122351471400134000232250ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ pipeline """ from enum import Enum from typing import Optional, Union, Tuple from mindspore import Model from mindformers.models import build_model, build_tokenizer, build_processor, \ BaseModel, BaseTokenizer, BaseImageProcessor, BaseAudioProcessor from mindformers.mindformer_book import MindFormerBook from mindformers.tools.register import MindFormerConfig from .build_pipeline import build_pipeline SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() class Backend(Enum): MS = "ms" MS_LITE = "mslite" def pipeline( task: str = None, model: Optional[Union[str, BaseModel, Model, Tuple[str, str]]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, backend: str = "ms", **kwargs): r"""Pipeline for downstream tasks Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (Optional[Union[str, BaseModel]]): The model used for task. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. backend(str): The inference backend. Default "ms", now support ["ms", "mslite"]. Return: A task pipeline. Raises: KeyError: If the task or model is not supported. Examples: >>> from mindformers import pipeline >>> from mindformers.tools.image_tools import load_image >>> classifier = pipeline("zero_shot_image_classification", candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) >>> img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> classifier(img) [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, {'score': 2.396818e-06, 'label': 'cat'}]] """ if backend == Backend.MS_LITE.value: from mindformers.inference import get_mslite_pipeline task_pipeline = get_mslite_pipeline(task, model, tokenizer, image_processor, audio_processor, **kwargs) elif backend == Backend.MS.value: task_pipeline = get_ms_pipeline(task, model, tokenizer, image_processor, audio_processor, **kwargs) else: raise ValueError(f"The inference backend \"{backend}\" is not supported," f"please select a backend from [\"ms\", \"mslite\"]") return task_pipeline def get_ms_pipeline(task, model, tokenizer, image_processor, audio_processor, **kwargs): """get mindspore infer pipeline.""" if task not in SUPPORT_PIPELINES.keys(): raise KeyError(f"{task} is not supported by pipeline. please select" f" a task from {SUPPORT_PIPELINES.keys()}.") if isinstance(model, str): if model not in SUPPORT_MODEL_NAMES: raise KeyError( f"model must be in {SUPPORT_MODEL_NAMES} when model's type is string, but get {model}.") model_name = model model = None else: model_name = "common" pipeline_config = MindFormerConfig(SUPPORT_PIPELINES.get(task).get(model_name)) pipeline_config.model.model_config.update({"checkpoint_name_or_path": model_name}) if model is None: model = build_model(pipeline_config.model) if image_processor is None and hasattr(pipeline_config.processor, 'image_processor'): image_processor = build_processor(pipeline_config.processor.image_processor) if audio_processor is None and hasattr(pipeline_config.processor, 'audio_processor'): audio_processor = build_processor(pipeline_config.processor.audio_processor) if tokenizer is None: tokenizer = build_tokenizer(pipeline_config.processor.tokenizer) task_pipeline = build_pipeline(class_name=task, model=model, image_processor=image_processor, audio_processor=audio_processor, tokenizer=tokenizer, **kwargs) return task_pipeline pengcheng.mind-test/mindformers/pipeline/question_answering_pipeline.py000066400000000000000000000440561471400134000272370ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/lonePatient/daguan_2019_rank9/blob/master/pydatagrand/train/ner_utils.py # ============================================================================ """QuestionAnsweringPipeline""" import os.path import collections import math import six from mindspore import Tensor, Model from mindspore.common import dtype as mstype from mindformers.models import BasicTokenizer from ..dataset.dataloader.squad_dataloader import convert_examples_to_features, SquadExample from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, Tokenizer __all__ = ['QuestionAnsweringPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="question_answering") class QuestionAnsweringPipeline(BasePipeline): r"""Pipeline for token classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. id2label : a dict which maps label id to label str. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import QuestionAnsweringPipeline >>> from mindformers import AutoTokenizer, BertForQuestionAnswering, AutoConfig >>> input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] >>> tokenizer = AutoTokenizer.from_pretrained('qa_bert_base_uncased_squad') >>> qa_squad_config = AutoConfig.from_pretrained('qa_bert_base_uncased_squad') >>> model = BertForQuestionAnswering(qa_squad_config) >>> qa_pipeline = QuestionAnsweringPipeline(task='question_answering', ... model=model, ... tokenizer=tokenizer) >>> results = qa_pipeline(input_data) >>> print(results) [{'text': 'Berlin', 'score': 0.9941, 'start': 34, 'end': 40}] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['question_answering'].keys() def __init__(self, model, tokenizer, doc_stride=128, max_question_len=64, max_seq_len=384, top_k=1, n_best_size=20, max_answer_len=30, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") self.features = None self.examples = None self.basic_tokenizer = BasicTokenizer(do_lower_case=True) if doc_stride > max_seq_len: raise ValueError(f"`doc_stride` ({doc_stride}) is larger than `max_seq_len` ({max_seq_len})") if top_k < 1: raise ValueError(f"top_k parameter should be >= 1 (got {top_k})") if max_answer_len < 1: raise ValueError(f"max_answer_len parameter should be >= 1 (got {max_answer_len}") kwargs["doc_stride"] = doc_stride kwargs["max_question_len"] = max_question_len kwargs["max_seq_len"] = max_seq_len kwargs["top_k"] = top_k kwargs["n_best_size"] = n_best_size kwargs["max_answer_len"] = max_answer_len super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): """sanitize parameters for preprocess, forward, and postprocess.""" if 'batch_size' in pipeline_parameters: raise ValueError(f"The {self.__class__.__name__} does not support batch inference, please remove the " f"batch_size") preprocess_key_name = ['doc_stride', 'max_question_len', 'max_seq_len'] postprocess_key_name = ['top_k', 'n_best_size', 'max_answer_len'] preprocess_params = {k: v for k, v in pipeline_parameters.items() if k in preprocess_key_name} postprocess_params = {k: v for k, v in pipeline_parameters.items() if k in postprocess_key_name} return preprocess_params, {}, postprocess_params def preprocess(self, inputs, **preprocess_params): """ Preprocess of token classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output return_tensors ("ms"): the type of returned tensors Return: processed text. """ if not isinstance(inputs, str): raise ValueError("The type of inputs should be str.") if '-' not in inputs: raise ValueError("The inputs should contain context and question separeated by '-' syntax.") context_text, question_text = inputs.split("-") context_text = context_text.strip() question_text = question_text.strip() squad_example = SquadExample(None, question_text, context_text, None, None, None) features = convert_examples_to_features( examples=[squad_example], tokenizer=self.tokenizer, max_seq_len=preprocess_params['max_seq_len'], max_question_len=preprocess_params['max_question_len'], doc_stride=preprocess_params['doc_stride'], is_training=False ) self.features = features self.examples = [squad_example] return features def forward(self, model_inputs, **forward_params): """ Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. """ self.model.set_train(False) model_outputs = [] for feature in model_inputs: model_inputs = {"input_ids": Tensor(feature.input_ids, mstype.int32).expand_dims(0), "input_mask": Tensor(feature.input_mask, mstype.int32).expand_dims(0), "token_type_id": Tensor(feature.token_type_id, mstype.int32).expand_dims(0), "start_position": Tensor(feature.start_position, mstype.int32).expand_dims(0), "end_position": Tensor(feature.end_position, mstype.int32).expand_dims(0), "unique_id": Tensor(feature.unique_id, mstype.int32).expand_dims(0)} ids, start, end = self.network(**model_inputs) RawResult = collections.namedtuple("RawResult", ["unique_id", "start_logits", "end_logits"]) unique_id = int(ids[0]) start_logits = [float(x) for x in start[0]] end_logits = [float(x) for x in end[0]] model_outputs.append(RawResult(unique_id=unique_id, start_logits=start_logits, end_logits=end_logits)) return model_outputs def postprocess(self, model_outputs, **postprocess_params): """ Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results """ top_k = postprocess_params['top_k'] n_best_size = postprocess_params['n_best_size'] max_answer_len = postprocess_params['max_answer_len'] example_index_to_features = collections.defaultdict(list) for feature in self.features: example_index_to_features[feature.example_index].append(feature) unique_id_to_result = {} for result in model_outputs: unique_id_to_result[result.unique_id] = result for (example_index, example) in enumerate(self.examples): features = example_index_to_features[example_index] prelim_predictions = self._get_prelim_predictions(features, unique_id_to_result, max_answer_len, n_best_size) nbest = self._get_nbest(prelim_predictions, features, example, n_best_size) total_scores = [] best_non_null_entry = None for entry in nbest: total_scores.append(entry.start_logit + entry.end_logit) if not best_non_null_entry: if entry.text: best_non_null_entry = entry probs = self._compute_softmax(total_scores) results = [] for (i, entry) in enumerate(nbest): output = {} output["text"] = entry.text output["score"] = round(probs[i], 4) output["start"] = entry.start output["end"] = entry.end results.append(output) return results[:top_k] def _get_prelim_predictions(self, features, unique_id_to_result, max_answer_len, n_best_size): """get prelim predictions""" _PrelimPrediction = collections.namedtuple( "PrelimPrediction", ["feature_index", "start_index", "end_index", "start_logit", "end_logit"]) prelim_predictions = [] # keep track of the minimum score of null start+end of position 0 for (feature_index, feature) in enumerate(features): if feature.unique_id not in unique_id_to_result: continue result = unique_id_to_result[feature.unique_id] start_indexes = self._get_best_indexes(result.start_logits, n_best_size) end_indexes = self._get_best_indexes(result.end_logits, n_best_size) # if we could have irrelevant answers, get the min score of irrelevant for start_index in start_indexes: for end_index in end_indexes: # We could hypothetically create invalid predictions, e.g., predict # that the start of the span is in the question. We throw out all # invalid predictions. if start_index >= len(feature.tokens): continue if end_index >= len(feature.tokens): continue if start_index not in feature.token_to_orig_map: continue if end_index not in feature.token_to_orig_map: continue if not feature.token_is_max_context.get(start_index, False): continue if end_index < start_index: continue length = end_index - start_index + 1 if length > max_answer_len: continue prelim_predictions.append( _PrelimPrediction( feature_index=feature_index, start_index=start_index, end_index=end_index, start_logit=result.start_logits[start_index], end_logit=result.end_logits[end_index])) prelim_predictions = sorted( prelim_predictions, key=lambda x: (x.start_logit + x.end_logit), reverse=True) return prelim_predictions def _get_nbest(self, prelim_predictions, features, example, n_best_size): """get nbest predictions""" _NbestPrediction = collections.namedtuple( "NbestPrediction", ["text", "start_logit", "end_logit", "start", "end"]) seen_predictions = {} nbest = [] for pred in prelim_predictions: if len(nbest) >= n_best_size: break feature = features[pred.feature_index] if pred.start_index > 0: # this is a non-null prediction tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)] orig_doc_start = feature.token_to_orig_map[pred.start_index] orig_doc_end = feature.token_to_orig_map[pred.end_index] orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)] tok_text = " ".join(tok_tokens) # De-tokenize WordPieces that have been split off. tok_text = tok_text.replace(" ##", "") tok_text = tok_text.replace("##", "") # Clean whitespace tok_text = tok_text.strip() tok_text = " ".join(tok_text.split()) orig_text = " ".join(orig_tokens) final_text = self._get_final_text(tok_text, orig_text) char_start_index, char_end_index = self._get_answer_index(example.context_text, orig_doc_start, orig_doc_end) if final_text in seen_predictions: continue seen_predictions[final_text] = True else: final_text = "" seen_predictions[final_text] = True char_start_index, char_end_index = -1, -1 nbest.append( _NbestPrediction( text=final_text, start_logit=pred.start_logit, end_logit=pred.end_logit, start=char_start_index, end=char_end_index)) # In very rare edge cases we could have no valid predictions. So we # just create a nonce prediction in this case to avoid failure. if not nbest: nbest.append(_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0)) assert len(nbest) >= 1 return nbest def _get_answer_index(self, context_text, orig_doc_start, orig_doc_end): char_index = 0 for i, word in enumerate(context_text.split(" ")): if i == orig_doc_start: char_start_index = char_index if i == orig_doc_end: char_end_index = char_index + len(word) char_index += len(word) + 1 return char_start_index, char_end_index def _compute_softmax(self, scores): """Compute softmax probability over raw logits.""" if not scores: return [] max_score = None for score in scores: if max_score is None or score > max_score: max_score = score exp_scores = [] total_sum = 0.0 for score in scores: x = math.exp(score - max_score) exp_scores.append(x) total_sum += x probs = [] for score in exp_scores: probs.append(score / total_sum) return probs def _get_final_text(self, pred_text, orig_text): """Project the tokenized prediction back to the original text.""" def _strip_spaces(text): ns_chars = [] ns_to_s_map = collections.OrderedDict() for (i, c) in enumerate(text): if c == " ": continue ns_to_s_map[len(ns_chars)] = i ns_chars.append(c) ns_text = "".join(ns_chars) return (ns_text, ns_to_s_map) tok_text = " ".join(self.basic_tokenizer.tokenize(orig_text)) start_position = tok_text.find(pred_text) if start_position == -1: return orig_text end_position = start_position + len(pred_text) - 1 (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text) (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text) if len(orig_ns_text) != len(tok_ns_text): return orig_text tok_s_to_ns_map = {} for (i, tok_index) in six.iteritems(tok_ns_to_s_map): tok_s_to_ns_map[tok_index] = i orig_start_position = None if start_position in tok_s_to_ns_map: ns_start_position = tok_s_to_ns_map[start_position] if ns_start_position in orig_ns_to_s_map: orig_start_position = orig_ns_to_s_map[ns_start_position] if orig_start_position is None: return orig_text orig_end_position = None if end_position in tok_s_to_ns_map: ns_end_position = tok_s_to_ns_map[end_position] if ns_end_position in orig_ns_to_s_map: orig_end_position = orig_ns_to_s_map[ns_end_position] if orig_end_position is None: return orig_text output_text = orig_text[orig_start_position:(orig_end_position + 1)] return output_text def _get_best_indexes(self, logits, n_best_size): """Get the n-best logits from a list.""" index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True) best_indexes = [] for (i, score) in enumerate(index_and_score): if i >= n_best_size: break best_indexes.append(score[0]) return best_indexes pengcheng.mind-test/mindformers/pipeline/segment_anything_pipeline.py000066400000000000000000000646111471400134000266550ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Pipeline API.""" from typing import Optional, Union import cv2 import numpy as np import mindspore as ms from mindspore import ops from mindspore import Tensor, Model from mindformers.auto_class import AutoProcessor, AutoModel from mindformers.mindformer_book import MindFormerBook from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.sam import ( MaskData, area_from_rle, batch_iterator, batched_mask_to_box, box_xyxy_to_xywh, build_all_layer_point_grids, calculate_stability_score, coco_encode_rle, generate_crop_boxes, is_box_near_crop_edge, mask_to_rle, remove_small_regions, rle_to_mask, uncrop_boxes_xyxy, uncrop_masks, uncrop_points, box_area, nms ) from .base_pipeline import BasePipeline __all__ = ['SegmentAnythingPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="segment_anything") class SegmentAnythingPipeline(BasePipeline): r"""Pipeline for image segment Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import SegmentAnythingPipeline >>> from mindformers import ViTImageProcessor >>> processor = ViTImageProcessor(size=224) >>> classifier = ImageClassificationPipeline( ... model='vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> classifier(np.uint8(np.random.random((5, 3, 255, 255)))) [[{'score': 0.0016654134, 'label': 'matchstick'}, {'score': 0.0015071577, 'label': 'theater curtain'}, {'score': 0.0014839625, 'label': 'ocarina'}, {'score': 0.0014319294, 'label': 'abaya'}, {'score': 0.0014109017, 'label': 'bottlecap'}], ..., {'score': 0.0014109018, 'label': 'bottlecap'}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['segment_anything'].keys() def __init__(self, model: Union[str, BaseModel, Model], image_processor: Optional[BaseImageProcessor] = None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ImageClassificationForPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if image_processor is None: raise ValueError("ImageClassificationFoPipeline" " requires for a image_processor.") super().__init__(model.set_train(mode=False), image_processor=image_processor, **kwargs) self.reset_config(**kwargs) self.stack = ops.Stack(axis=-1) self.reset_image() def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {"seg_image": False} forward_params = {"seg_image": False} postprocess_params = {"seg_image": False} preprocess_list_0 = ["seg_image"] for item in preprocess_list_0: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) preprocess_params['reset_config'] = False preprocess_list_1 = ["points_per_side", "points_per_batch", "pred_iou_thresh", "stability_score_thresh", "stability_score_offset", "box_nms_thresh", "crop_n_layers", "crop_nms_thresh", "crop_overlap_ratio", "crop_n_points_downscale_factor", "point_grids", "min_mask_region_area", "output_mode"] for item in preprocess_list_1: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) preprocess_params['reset_config'] = True forward_list = ["seg_image", "multimask_output", "return_logits"] for item in forward_list: if item in pipeline_parameters: forward_params[item] = pipeline_parameters.get(item) post_list = ["seg_image"] for item in post_list: if item in pipeline_parameters: postprocess_params[item] = pipeline_parameters.get(item) return preprocess_params, forward_params, postprocess_params def preprocess(self, inputs, **preprocess_params): r"""The Preprocess For Task Args: inputs (dict): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. """ model_inputs = {} seg_image = preprocess_params["seg_image"] if not seg_image: image = inputs.get("image", None) if image is not None: input_image, original_size, input_size = self.preprocess_image(image) model_inputs["image"] = input_image model_inputs["features"] = None model_inputs["original_size"] = original_size model_inputs["input_size"] = input_size else: assert self.is_image_set, "image must be set when image not in inputs" original_size = self.original_size model_inputs["image"] = None model_inputs["features"] = self.features model_inputs["original_size"] = self.original_size model_inputs["input_size"] = self.input_size point_coords = inputs.get("points", None) point_labels = inputs.get("labels", None) boxes = inputs.get("boxes", None) masks = inputs.get("masks", None) point_coords, point_labels, boxes, masks = self.preprocess_prompts( original_size=original_size, point_coords=point_coords, point_labels=point_labels, boxes=boxes, masks=masks ) model_inputs["point_coords"] = point_coords model_inputs["point_labels"] = point_labels model_inputs["boxes"] = boxes model_inputs["mask_inputs"] = masks else: image = inputs.get("image", None) assert image is not None, "An image must be supported, when 'seg_image' is True." if isinstance(image, str): image = cv2.imread(image) image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) orig_size = image.shape[:2] reset_config = preprocess_params.get("reset_config", False) if reset_config: self.reset_config(**preprocess_params) crop_boxes, layer_idxs = generate_crop_boxes( orig_size, self.crop_n_layers, self.crop_overlap_ratio ) model_inputs["original_size"] = orig_size model_inputs["image_list"] = [] model_inputs["features_list"] = [] model_inputs["original_size_list"] = [] model_inputs["input_size_list"] = [] model_inputs["crop_boxes_list"] = [] model_inputs["points_for_image_list"] = [] model_inputs["point_coords_list"] = [] model_inputs["point_labels_list"] = [] for crop_box, layer_id in zip(crop_boxes, layer_idxs): x0, y0, x1, y1 = crop_box crop_image = image[y0:y1, x0:x1, :] crop_image_size = crop_image.shape[:2] self.set_image(crop_image) points_scale = np.array(crop_image_size)[None, ::-1] points_for_image = self.point_grids[layer_id] * points_scale point_coords, point_labels, _, _ = self.preprocess_prompts( original_size=crop_image_size, point_coords=points_for_image, multi_seg=True ) model_inputs["features_list"].append(self.features) model_inputs["original_size_list"].append(self.original_size) model_inputs["input_size_list"].append(self.input_size) model_inputs["crop_boxes_list"].append(crop_box) model_inputs["points_for_image_list"].append(points_for_image) model_inputs["point_coords_list"].append(point_coords) model_inputs["point_labels_list"].append(point_labels) return model_inputs def forward(self, model_inputs, **forward_params): r"""The Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ multimask_output = forward_params.get("multimask_output", True) return_logits = forward_params.get("return_logits", False) seg_image = forward_params["seg_image"] if not seg_image: masks, low_res_masks, iou_predictions = self.model( image=model_inputs["image"], features=model_inputs["features"], input_size=model_inputs["input_size"], original_size=model_inputs["original_size"], point_coords=model_inputs["point_coords"], point_labels=model_inputs["point_labels"], boxes=model_inputs["boxes"], mask_inputs=model_inputs["mask_inputs"], multimask_output=multimask_output, return_logits=return_logits ) return {"masks": masks, "low_res_masks": low_res_masks, "iou_predictions": iou_predictions} orig_size = model_inputs["original_size"] orig_h, orig_w = orig_size points_for_image_list = model_inputs["points_for_image_list"] point_coords_list = model_inputs["point_coords_list"] point_labels_list = model_inputs["point_labels_list"] crop_boxes_list = model_inputs["crop_boxes_list"] data = MaskData() for layer_id, features in enumerate(model_inputs["features_list"]): point_coords = point_coords_list[layer_id] point_labels = point_labels_list[layer_id] points_for_image = points_for_image_list[layer_id] crop_box = crop_boxes_list[layer_id] input_size = model_inputs["input_size_list"][layer_id] original_size = model_inputs["original_size_list"][layer_id] # Generate masks for this crop in batches data_crop = MaskData() for (points_for_image_batch, point_coords_batch, point_labels_batch) in \ batch_iterator(self.points_per_batch, points_for_image, point_coords, point_labels): masks, low_res_masks, iou_predictions = self.model( features=features, input_size=input_size, original_size=original_size, point_coords=point_coords_batch, point_labels=point_labels_batch, multimask_output=multimask_output, return_logits=True ) masks = masks.asnumpy() low_res_masks = low_res_masks.asnumpy() iou_predictions = iou_predictions.asnumpy() data_batch = MaskData( masks=masks.reshape((-1, masks.shape[-2], masks.shape[-1])), iou_preds=iou_predictions.reshape((-1,)), points=points_for_image_batch.repeat(masks.shape[1], axis=0) ) del masks # Filter by predicted IoU if self.pred_iou_thresh > 0.0: keep_mask = data_batch["iou_preds"] > self.pred_iou_thresh data_batch.filter(keep_mask) # Calculate stability score data_batch["stability_score"] = calculate_stability_score( data_batch["masks"], self.model.mask_threshold, self.stability_score_offset ) if self.stability_score_thresh > 0.0: keep_mask = data_batch["stability_score"] >= self.stability_score_thresh data_batch.filter(keep_mask) # Threshold masks and calculate boxes data_batch["masks"] = data_batch["masks"] > self.model.mask_threshold data_batch["boxes"] = batched_mask_to_box(data_batch["masks"]) # Filter boxes that touch crop boundaries keep_mask = ~is_box_near_crop_edge(data_batch["boxes"], crop_box, [0, 0, orig_w, orig_h]) if not keep_mask.all(): data_batch.filter(keep_mask) # Compress to RLE data_batch["masks"] = uncrop_masks(data_batch["masks"], crop_box, orig_h, orig_w) data_batch["rles"] = mask_to_rle(data_batch["masks"]) del data_batch["masks"] data_crop.cat(data_batch) # Remove duplicates within this crop. input_boxes = data_crop["boxes"] input_scores = data_crop["iou_preds"] keep_by_nms = nms(input_boxes, input_scores, iou_threshold=self.crop_nms_thresh) data_crop.filter(keep_by_nms) # Return to the original image frame data_crop["boxes"] = uncrop_boxes_xyxy(data_crop["boxes"], crop_box) data_crop["points"] = uncrop_points(data_crop["points"], crop_box) data_crop["crop_boxes"] = np.array([crop_box for _ in range(len(data_crop["rles"]))]) data.cat(data_crop) # Remove duplicate masks between crops if len(crop_boxes_list) > 1: # Prefer masks from smaller crops input_boxes = data["boxes"] input_scores = 1.0 / box_area(data["crop_boxes"]) keep_by_nms = nms(input_boxes, input_scores, iou_threshold=self.crop_nms_thresh) data.filter(keep_by_nms) return data def postprocess(self, model_outputs, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result. Return: classification results. """ seg_image = postprocess_params["seg_image"] if not seg_image: bs = model_outputs["masks"].shape[0] if bs == 1: for k, v in model_outputs.items(): model_outputs[k] = v[0].asnumpy() else: for k, v in model_outputs.items(): model_outputs[k] = v.asnumpy() return model_outputs # Filter small disconnected regions and holes in masks if self.min_mask_region_area > 0: model_outputs = self.postprocess_small_regions( model_outputs, self.min_mask_region_area, max(self.box_nms_thresh, self.crop_nms_thresh), ) # Encode masks if self.output_mode == "coco_rle": model_outputs["segmentations"] = [coco_encode_rle(rle) for rle in model_outputs["rles"]] elif self.output_mode == "binary_mask": model_outputs["segmentations"] = [rle_to_mask(rle) for rle in model_outputs["rles"]] else: model_outputs["segmentations"] = model_outputs["rles"] # Write mask records curr_anns = [] for idx in range(len(model_outputs["segmentations"])): ann = { "segmentation": model_outputs["segmentations"][idx], "area": area_from_rle(model_outputs["rles"][idx]), "bbox": box_xyxy_to_xywh(model_outputs["boxes"][idx]).tolist(), "predicted_iou": model_outputs["iou_preds"][idx].item(), "point_coords": [model_outputs["points"][idx].tolist()], "stability_score": model_outputs["stability_score"][idx].item(), "crop_box": box_xyxy_to_xywh(model_outputs["crop_boxes"][idx]).tolist(), } curr_anns.append(ann) return curr_anns def preprocess_image(self, image, image_format: str = "RGB") -> None: """ Calculates the image embeddings for the provided image, allowing masks to be predicted with the 'predict' method. Arguments: image (np.ndarray): The image for calculating masks. Expects an image in HWC uint8 format, with pixel values in [0, 255]. image_format (str): The color format of the image, in ['RGB', 'BGR']. """ if isinstance(image, str): image = cv2.imread(image) image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) assert image_format in [ "RGB", "BGR", ], f"image_format must be in ['RGB', 'BGR'], is {image_format}." if image_format != self.network.image_format: image = image[..., ::-1] # Transform the image to the form expected by the model original_size = image.shape[:2] # h, w input_image, input_size = self.image_processor(image) return input_image, original_size, input_size def preprocess_prompts(self, original_size, point_coords=None, point_labels=None, boxes=None, masks=None, multi_seg=False): """preprocess prompts""" if point_coords is not None: ndim = point_coords.ndim if ndim == 1: point_coords = point_coords[None, :] if ndim == 2: if point_labels is None: point_labels = np.ones(point_coords.shape[0], dtype=np.int32) point_coords = self.image_processor.transform.apply_coords(point_coords, original_size) point_coords = np.expand_dims(point_coords, axis=0) point_labels = np.expand_dims(point_labels, axis=0) if multi_seg: point_coords = point_coords.transpose(1, 0, 2) point_labels = point_labels.transpose() elif ndim == 3: if point_labels is None: point_labels = np.ones((point_coords.shape[0], point_coords.shape[1]), dtype=np.int32) point_coords = self.image_processor.transform.apply_coords_batch(point_coords, original_size) else: raise ValueError("points's ndim < 3.") point_coords = Tensor(point_coords, dtype=ms.float32) point_labels = Tensor(point_labels, dtype=ms.int32) if boxes is not None: ndim = boxes.ndim if ndim == 1: boxes = self.image_processor.transform.apply_boxes(boxes, original_size) boxes = np.expand_dims(boxes, axis=0) elif ndim == 2: boxes = self.image_processor.transform.apply_boxes_batch(boxes, original_size) else: raise ValueError("boxes's ndim must be 1 or 2.") boxes = Tensor(boxes, dtype=ms.float32) if masks is not None: masks = np.expand_dims(masks, axis=0) masks = Tensor(masks, dtype=ms.float32) return point_coords, point_labels, boxes, masks def set_image(self, image, image_format: str = "RGB"): self.reset_image() input_image, original_size, input_size = self.preprocess_image(image, image_format) self.features = self.network.image_encoder(input_image) self.original_size = original_size # h, w self.input_size = input_size # h, w self.is_image_set = True def reset_image(self) -> None: """Resets the currently set image.""" self.is_image_set = False self.features = None self.original_size = None self.input_size = None def get_image_embedding(self): """ Returns the image embeddings for the currently set image, with shape 1xCxHxW, where C is the embedding dimension and (H,W) are the embedding spatial dimension of SAM (typically C=256, H=W=64). """ if not self.is_image_set: raise RuntimeError( "An image must be set with .set_image(...) to generate an embedding." ) assert self.features is not None return self.features def reset_config(self, points_per_side=32, points_per_batch=64, pred_iou_thresh=0.88, stability_score_thresh=0.95, stability_score_offset=1.0, box_nms_thresh=0.7, crop_n_layers=0, crop_nms_thresh=0.7, crop_overlap_ratio=0.3413, crop_n_points_downscale_factor=1, point_grids=None, min_mask_region_area=0, output_mode="binary_mask", **kwargs) -> None: """reset config""" self.points_per_side = points_per_side or kwargs.get('points_per_side', 32) self.points_per_batch = points_per_batch or kwargs.get('points_per_batch', 64) self.pred_iou_thresh = pred_iou_thresh or kwargs.get('points_per_batch', 0.88) self.stability_score_thresh = stability_score_thresh or kwargs.get('stability_score_thresh', 0.95) self.stability_score_offset = stability_score_offset or kwargs.get('stability_score_offset', 1.0) self.box_nms_thresh = box_nms_thresh or kwargs.get('box_nms_thresh', 0.7) self.crop_n_layers = crop_n_layers or kwargs.get('crop_n_layers', 0) self.crop_nms_thresh = crop_nms_thresh or kwargs.get('crop_nms_thresh', 0.7) self.crop_overlap_ratio = crop_overlap_ratio or kwargs.get('crop_overlap_ratio', 0.3413) self.crop_n_points_downscale_factor = crop_n_points_downscale_factor or\ kwargs.get('crop_n_points_downscale_factor', 1) self.point_grids = point_grids or kwargs.get('point_grids', None) self.min_mask_region_area = min_mask_region_area or kwargs.get('min_mask_region_area', 0) self.output_mode = output_mode or kwargs.get('output_mode', "binary_mask") assert (self.points_per_side is None) != ( self.point_grids is None ), "Exactly one of points_per_side or point_grid must be provided." if self.points_per_side is not None: self.point_grids = build_all_layer_point_grids( self.points_per_side, self.crop_n_layers, self.crop_n_points_downscale_factor ) assert self.output_mode in [ "binary_mask", "uncompressed_rle", "coco_rle", ], f"Unknown output_mode {self.output_mode}." def postprocess_small_regions(self, mask_data: MaskData, min_area: int, nms_thresh: float) -> MaskData: """ Removes small disconnected regions and holes in masks, then reruns box NMS to remove any new duplicates. Edits mask_data in place. Requires open-cv as a dependency. """ if not mask_data["rles"]: return mask_data # Filter small disconnected regions and holes new_masks = [] scores = [] for rle in mask_data["rles"]: mask = rle_to_mask(rle) mask, changed = remove_small_regions(mask, min_area, mode="holes") unchanged = not changed mask, changed = remove_small_regions(mask, min_area, mode="islands") unchanged = unchanged and not changed new_masks.append(np.expand_dims(mask, axis=0)) # Give score=0 to changed masks and score=1 to unchanged masks # so NMS will prefer ones that didn't need postprocessing scores.append(float(unchanged)) # Recalculate boxes and remove any new duplicates masks = np.concatenate(new_masks, axis=0) boxes = batched_mask_to_box(masks) input_boxes = boxes.astype(np.float32) input_scores = np.array(scores) keep_by_nms = nms(input_boxes, input_scores, iou_threshold=nms_thresh) # Only recalculate RLEs for masks that have changed for i_mask in keep_by_nms: if scores[i_mask] == 0.0: mask_tensor = np.expand_dims(masks[i_mask], axis=0) mask_data["rles"][i_mask] = mask_to_rle(mask_tensor)[0] mask_data["boxes"][i_mask] = boxes[i_mask] # update res directly mask_data.filter(keep_by_nms) return mask_data pengcheng.mind-test/mindformers/pipeline/text_classification_pipeline.py000066400000000000000000000242671471400134000273540ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """TextClassificationPipeline""" import os.path import numpy as np from mindspore import ops, Tensor, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, Tokenizer from ..dataset.labels import labels __all__ = ['TextClassificationPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="text_classification") class TextClassificationPipeline(BasePipeline): """Pipeline for text classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import TextClassificationPipeline >>> from mindformers import AutoTokenizer, BertForMultipleChoice, AutoConfig >>> input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", ... "i don't know um do you do a lot of camping-I know exactly."] >>> tokenizer = AutoTokenizer.from_pretrained('txtcls_bert_base_uncased_mnli') >>> txtcls_mnli_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased_mnli') >>> model = BertForMultipleChoice(txtcls_mnli_config) >>> txtcls_pipeline = TextClassificationPipeline(task='text_classification', ... model=model, ... tokenizer=tokenizer, ... max_length=model.config.seq_length, ... padding="max_length") >>> results = txtcls_pipeline(input_data, top_k=1) >>> print(results) [[{'label': 'neutral', 'score': 0.9714198708534241}], [{'label': 'contradiction', 'score': 0.9967639446258545}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['text_classification'].keys() def __init__(self, model, tokenizer=None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): """sanitize parameters for preprocess, forward, and postprocess.""" if 'batch_size' in pipeline_parameters: raise ValueError(f"The {self.__class__.__name__} does not support batch inference, please remove the " f"batch_size") postprocess_params = {} forward_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length'] forward_kwargs = {} for item in forward_key_name: if item in pipeline_parameters: forward_kwargs[item] = pipeline_parameters.get(item) preprocess_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length', 'padding'] preprocess_params = {k: v for k, v in pipeline_parameters.items() if k in preprocess_key_name} if "top_k" in pipeline_parameters: postprocess_params["top_k"] = pipeline_parameters.get("top_k") if "dataset" in pipeline_parameters: postprocess_params["dataset"] = pipeline_parameters.get("dataset") if not labels.get(postprocess_params["dataset"]): raise ValueError(f"The dataset does not support {postprocess_params['dataset']}, " f"but only support {labels.keys()}") return preprocess_params, forward_kwargs, postprocess_params def inputs_process(self, inputs_zero, inputs_one): """ process of two sentences relationship classification Args: inputs_zero (str): the first sentence inputs_one (str): the second sentence Return: processed inputs, mask, token_type about two sentences """ len_inputs = len(inputs_zero["input_ids"]) inputs_zero_input = list(inputs_zero["input_ids"].asnumpy()) inputs_one_input = list(inputs_one["input_ids"].asnumpy()) inputs_zero_input = [x for x in inputs_zero_input if x != 0] inputs_one_input = [x for x in inputs_one_input if x != 0] token_type = [0] * len(inputs_zero_input) + [1] * (len(inputs_one_input) - 1) token_type = token_type + [0] * (len_inputs - len(token_type)) inputs = inputs_zero_input + inputs_one_input[1:] len_inputs_mask = len(inputs) inputs = inputs + [0] * (len_inputs - len(inputs)) mask = [1] * len_inputs_mask + [0] * (len_inputs - len_inputs_mask) return inputs, mask, token_type def preprocess(self, inputs, **preprocess_params): """ Preprocess of text classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed text. """ if not isinstance(inputs, str): raise ValueError("Inputs type must be str") expand_dims = ops.ExpandDims() if self.model.model_name == "gpt2_txtcls": tokens = self.tokenizer(inputs, return_tensors="ms", **preprocess_params) input_ids = tokens["input_ids"] attention_mask = tokens["attention_mask"] return {"input_ids": expand_dims(input_ids, 0), "attention_mask": expand_dims(attention_mask, 0)} if '-' not in inputs: raise ValueError("two texts of text pair should be split by -") inputs = inputs.split('-') max_length = preprocess_params.pop("max_length", 128) padding = preprocess_params.pop("padding", "max_length") inputs_zero = self.tokenizer(inputs[0], max_length=max_length, padding=padding, return_tensors="ms", **preprocess_params) inputs_one = self.tokenizer(inputs[1], max_length=max_length, padding=padding, return_tensors="ms", **preprocess_params) inputs, mask, token_type = self.inputs_process(inputs_zero, inputs_one) inputs_final = {} inputs_final["input_ids"] = Tensor.from_numpy(np.array(inputs, dtype=np.int32)) inputs_final["attention_mask"] = Tensor.from_numpy(np.array(mask, dtype=np.int32)) inputs_final["token_type_ids"] = Tensor.from_numpy(np.array(token_type, dtype=np.int32)) expand_dims = ops.ExpandDims() return {"input_ids": expand_dims(inputs_final["input_ids"], 0), "input_mask": expand_dims(inputs_final["attention_mask"], 0), "token_type_id": expand_dims(inputs_final["token_type_ids"], 0), "label_ids": None} def forward(self, model_inputs, **forward_params): """ Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. """ forward_params.pop("None", None) output_ids = self.network(**model_inputs) return output_ids def softmax(self, outputs): maxes = np.max(outputs, axis=-1, keepdims=True) shifted_exp = np.exp(outputs - maxes) return shifted_exp / shifted_exp.sum(axis=-1, keepdims=True) def postprocess(self, model_outputs, **postprocess_params): """ Postprocess Args: model_outputs (dict): outputs of forward process. Return: Classification results """ top_k = postprocess_params.pop("top_k", None) dataset = postprocess_params.pop("dataset", None) if dataset: id2label = {id: label for id, label in enumerate(labels.get(dataset))} else: id2label = {id: label for id, label in enumerate(labels.get("mnli"))} outputs = model_outputs[0] outputs = outputs.asnumpy() scores = self.softmax(outputs) dict_scores = [ {"label": id2label[i], "score": score.item()} for i, score in enumerate(scores) ] dict_scores.sort(key=lambda x: x["score"], reverse=True) if top_k is not None: dict_scores = dict_scores[:top_k] return [dict_scores] pengcheng.mind-test/mindformers/pipeline/text_generation_pipeline.py000066400000000000000000000140331471400134000265020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """TextGenerationPipeline""" import os.path from typing import Union, Optional import mindspore from mindspore import Tensor, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, BaseTokenizer __all__ = ['TextGenerationPipeline'] def _setup_support_list(support_model_list): support_list = [] for support_model in support_model_list: support_list.append(MindFormerBook.get_model_support_list().get(support_model)) return support_list @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="text_generation") class TextGenerationPipeline(BasePipeline): r"""Pipeline for Text Generation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and tokenizer's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.pipeline import TextGenerationPipeline >>> text_generate = TextGenerationPipeline("gpt2") >>> output = text_generate("I love Beijing, because ") """ _support_list = _setup_support_list(["gpt2", "glm"]) return_name = 'text_generation' def __init__(self, model: Union[str, BaseModel, Model], tokenizer: Optional[BaseTokenizer] = None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_keys = ['keys', 'add_special_tokens'] preprocess_params = {} for item in preprocess_keys: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) postprocess_params = {} forward_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length', 'seed'] forward_kwargs = {} for item in forward_key_name: if item in pipeline_parameters: forward_kwargs[item] = pipeline_parameters.get(item) return preprocess_params, forward_kwargs, postprocess_params def preprocess(self, inputs: Union[str, dict, Tensor], **preprocess_params): r"""The Preprocess For Translation Args: inputs (Union[str, dict, Tensor]): The text to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed text. """ add_special_tokens = preprocess_params.get('add_special_tokens', True) if isinstance(inputs, dict): keys = preprocess_params.get('keys', None) default_src_language_name = 'text' feature_name = keys.get('src_language', default_src_language_name) if keys else default_src_language_name inputs = inputs[feature_name] if isinstance(inputs, mindspore.Tensor): inputs = inputs.asnumpy().tolist() input_ids = self.tokenizer(inputs, return_tensors=None, add_special_tokens=add_special_tokens)["input_ids"] return {"input_ids": input_ids} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ forward_params.pop("None", None) input_ids = model_inputs["input_ids"] output_ids = self.network.generate(input_ids, **forward_params) return {"output_ids": output_ids} def postprocess(self, model_outputs: dict, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. Return: translation results. """ outputs = self.tokenizer.decode(model_outputs["output_ids"], skip_special_tokens=True) return [{self.return_name + '_text': outputs}] pengcheng.mind-test/mindformers/pipeline/token_classification_pipeline.py000066400000000000000000000236161471400134000275050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/lonePatient/daguan_2019_rank9/blob/master/pydatagrand/train/ner_utils.py # ============================================================================ """TokenClassificationPipeline""" import os.path import numpy as np from mindspore import ops, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, Tokenizer __all__ = ['TokenClassificationPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="token_classification") class TokenClassificationPipeline(BasePipeline): r"""Pipeline for token classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. id2label : a dict which maps label id to label str. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import TokenClassificationPipeline >>> from mindformers import AutoTokenizer, BertForTokenClassification, AutoConfig >>> from mindformers.dataset.labels import cluener_labels >>> id2label = {label_id: label for label_id, label in enumerate(cluener_labels)} >>> input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。"] >>> tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') >>> ner_dense_cluener_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese_cluener') >>> model = BertForTokenClassification(ner_dense_cluener_config) >>> tokcls_pipeline = TokenClassificationPipeline(task='token_classification', ... model=model, ... id2label=id2label, ... tokenizer=tokenizer, ... max_length=model.config.seq_length, ... padding="max_length") >>> results = tokcls_pipeline(input_data) >>> print(results) [[{'entity_group': 'address', 'start': 6, 'end': 8, 'score': 0.52329, 'word': '日内瓦'}, {'entity_group': 'name', 'start': 12, 'end': 25, 'score': 0.83922, 'word': 'freresoltramar'}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['token_classification'].keys() def __init__(self, model, id2label, tokenizer=None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") if id2label is None: raise ValueError(f"{self.__class__.__name__}" " requires for a dict which maps label id to label str.") self.id2label = id2label self.input_text = "" super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): """sanitize parameters for preprocess, forward, and postprocess.""" if 'batch_size' in pipeline_parameters: raise ValueError(f"The {self.__class__.__name__} does not support batch inference, please remove the " f"batch_size") postprocess_params = {'id2label'} preprocess_key_name = ['max_length', 'padding'] preprocess_params = {k: v for k, v in pipeline_parameters.items() if k in preprocess_key_name} postprocess_params = {k: v for k, v in pipeline_parameters.items() if k in postprocess_params} return preprocess_params, {}, postprocess_params def preprocess(self, inputs, **preprocess_params): """ Preprocess of token classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed text. """ if not isinstance(inputs, str): raise ValueError("Inputs type must be str") self.input_text = inputs max_length = preprocess_params.pop("max_length", 128) padding = preprocess_params.pop("padding", "max_length") inputs = self.tokenizer(inputs, max_length=max_length, padding=padding, return_tensors="ms", **preprocess_params) expand_dims = ops.ExpandDims() return {"input_ids": expand_dims(inputs["input_ids"], 0), "input_mask": expand_dims(inputs["attention_mask"], 0), "token_type_ids": expand_dims(inputs["token_type_ids"], 0)} def forward(self, model_inputs, **forward_params): """ Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. """ self.model.set_train(False) logits = self.network(**model_inputs) return {"logits": logits} def postprocess(self, model_outputs, **postprocess_params): """ Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results """ logits = model_outputs["logits"].asnumpy() maxes = np.max(logits, axis=-1, keepdims=True) shifted_exp = np.exp(logits - maxes) probs = shifted_exp / shifted_exp.sum(axis=-1, keepdims=True) batch_pred_ids = np.argmax(probs, axis=2).tolist() batch_best_scores = np.max(probs, axis=2).tolist() # remove CLS and SEP token pred_paths = [[self.id2label[id_] for id_ in pred_ids[1:-1]] for pred_ids in batch_pred_ids] best_scores = [best_scores[1:-1] for best_scores in batch_best_scores] total_result = [] for pred_path, best_score in zip(pred_paths, best_scores): single_result = [] pred_entities = self.get_entities_bios(pred_path) for pred_entity in pred_entities: entity_result = {} entity_result["entity_group"] = pred_entity[0] entity_result["start"] = pred_entity[1] entity_result["end"] = pred_entity[2] entity_result["score"] = sum(best_score[entity_result["start"]:entity_result["end"] + 1]) / \ (entity_result["end"] + 1 - entity_result["start"]) entity_result["score"] = round(entity_result["score"], 5) entity_result["word"] = self.input_text[entity_result["start"]:entity_result["end"] + 1] single_result.append(entity_result) total_result.append(single_result) return total_result def get_entities_bios(self, seq): """Gets entities from sequence. note: BIOS Args: seq (list): sequence of labels. Returns: list: list of (chunk_type, chunk_start, chunk_end). Example: # >>> seq = ['B-PER', 'I-PER', 'O', 'S-LOC'] # >>> get_entity_bios(seq) [['PER', 0, 1], ['LOC', 3, 3]] """ chunks = [] chunk = [-1, -1, -1] for indx, tag in enumerate(seq): if tag.startswith("S-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[2] = indx chunk[0] = tag.split('-')[1] chunks.append(chunk) chunk = [-1, -1, -1] if tag.startswith("B-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[0] = tag.split('-')[1] elif tag.startswith('I-') and chunk[1] != -1: entity_type = tag.split('-')[1] if entity_type == chunk[0]: chunk[2] = indx if indx == len(seq) - 1: chunks.append(chunk) else: if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] return chunks pengcheng.mind-test/mindformers/pipeline/translation_pipeline.py000066400000000000000000000131741471400134000256460ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """TranslationPipeline""" import os.path from typing import Union, Optional import mindspore from mindspore import Tensor, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, BaseTokenizer __all__ = ['TranslationPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="translation") class TranslationPipeline(BasePipeline): r"""Pipeline for Translation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and tokenizer's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.pipeline import TranslationPipeline >>> translator = TranslationPipeline("t5_small") >>> output = translator("abc") """ _support_list = MindFormerBook.get_model_support_list()['t5'] return_name = 'translation' def __init__(self, model: Union[str, BaseModel, Model], tokenizer: Optional[BaseTokenizer] = None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_keys = ['keys'] preprocess_params = {} for item in preprocess_keys: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) postprocess_params = {} forward_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length'] forward_kwargs = {} for item in forward_key_name: if item in pipeline_parameters: forward_kwargs[item] = pipeline_parameters.get(item) return preprocess_params, forward_kwargs, postprocess_params def preprocess(self, inputs: Union[str, dict, Tensor], **preprocess_params): r"""The Preprocess For Translation Args: inputs (Union[str, dict, Tensor]): The text to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed text. """ if isinstance(inputs, dict): keys = preprocess_params.get('keys', None) default_src_language_name = 'text' feature_name = keys.get('src_language', default_src_language_name) if keys else default_src_language_name inputs = inputs[feature_name] if isinstance(inputs, mindspore.Tensor): inputs = inputs.asnumpy().tolist() input_ids = self.tokenizer(inputs, return_tensors=None)["input_ids"] return {"input_ids": input_ids} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ forward_params.pop("None", None) input_ids = model_inputs["input_ids"] output_ids = self.network.generate(input_ids, **forward_params) return {"output_ids": output_ids} def postprocess(self, model_outputs: dict, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. Return: translation results. """ outputs = self.tokenizer.decode(model_outputs["output_ids"], skip_special_tokens=True) return [{self.return_name + '_text': outputs}] pengcheng.mind-test/mindformers/pipeline/zero_shot_image_classification_pipeline.py000066400000000000000000000216751471400134000315460ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ZeroShotImageClassificationPipeline""" from typing import Union, Optional from mindspore import Model from mindspore.ops import operations as P from mindformers.mindformer_book import MindFormerBook from mindformers.tools.image_tools import load_image from mindformers.models import BaseModel, BaseImageProcessor, Tokenizer from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.auto_class import AutoProcessor, AutoModel from .base_pipeline import BasePipeline from ..models import BaseTokenizer @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="zero_shot_image_classification") class ZeroShotImageClassificationPipeline(BasePipeline): r"""Pipeline For Zero Shot Image Classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer for text processing. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model, tokenizer, and image_processor's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.tools.image_tools import load_image >>> from mindformers.pipeline import ZeroShotImageClassificationPipeline >>> classifier = ZeroShotImageClassificationPipeline( ... model='clip_vit_b_32', ... candidate_labels=["sunflower", "tree", "dog", "cat", "toy"], ... hypothesis_template="This is a photo of {}." ... ) >>> img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." ... "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> classifier(img) [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, {'score': 2.396818e-06, 'label': 'cat'}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['zero_shot_image_classification'].keys() def __init__(self, model: Union[str, BaseModel, Model], tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" PretrainedTokenizer, but got {type(tokenizer)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ZeroShotImageClassificationPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError("ZeroShotImageClassificationPipeline" " requires for a tokenizer.") if image_processor is None: raise ValueError("ZeroShotImageClassificationPipeline" " requires for a image_processor.") super().__init__(model, tokenizer, image_processor, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {} postprocess_params = {} pre_list = ["candidate_labels", "hypothesis_template", "max_length", "padding", "return_tensors"] for item in pre_list: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) if "top_k" in pipeline_parameters: postprocess_params["top_k"] = pipeline_parameters.get("top_k") return preprocess_params, {}, postprocess_params def preprocess(self, inputs: dict, **preprocess_params): r"""Preprocess of ZeroShotImageClassificationPipeline Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. candidate_labels (List[str]): The candidate labels for classification. hypothesis_template (Optional[str]): Prompt for text input. max_length (Optional[int]): Max length of tokenizer's output padding (Optional[Union[False, "max_length"]]): Padding for max_length return_tensors (Optional["ms"]): The type of returned tensors Return: Processed data. Raises: ValueError: If candidate_labels or hypothesis_template is None. """ candidate_labels = preprocess_params.pop("candidate_labels", None) hypothesis_template = preprocess_params.pop("hypothesis_template", "a picture of {}.") max_length = preprocess_params.pop("max_length", 77) padding = preprocess_params.pop("padding", "max_length") return_tensors = preprocess_params.pop("return_tensors", "ms") if candidate_labels is None: raise ValueError("candidate_labels are supposed for" " ZeroShotImageClassificationPipeline, but got None.") if hypothesis_template is None: raise ValueError("hypothesis_template is supposed for" " ZeroShotImageClassificationPipeline, but got None.") if isinstance(inputs, dict): inputs = inputs['image'] if isinstance(inputs, str): inputs = load_image(inputs) image_processed = self.image_processor(inputs) sentences = [hypothesis_template.format(candidate_label) for candidate_label in candidate_labels] input_ids = self.tokenizer(sentences, max_length=max_length, padding=padding, return_tensors=return_tensors)["input_ids"] return {"image_processed": image_processed, "input_ids": input_ids, "candidate_labels": candidate_labels} def forward(self, model_inputs: dict, **forward_params): r"""Forward process Args: model_inputs (dict): Outputs of preprocess. Return: Probs dict. """ forward_params.pop("None", None) image_processed = model_inputs["image_processed"] input_ids = model_inputs["input_ids"] logits_per_image, _ = self.network(image_processed, input_ids) probs = P.Softmax()(logits_per_image).asnumpy() return {"probs": probs, "candidate_labels": model_inputs["candidate_labels"]} def postprocess(self, model_outputs: dict, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result Return: Classification results. """ top_k = postprocess_params.pop("top_k", None) labels = model_outputs['candidate_labels'] scores = model_outputs['probs'] outputs = [] for score in scores: sorted_res = sorted(zip(score, labels), key=lambda x: -x[0]) if top_k is not None: sorted_res = sorted_res[:min(top_k, len(labels))] outputs.append([{"score": score_item, "label": label} for score_item, label in sorted_res]) return outputs pengcheng.mind-test/mindformers/tools/000077500000000000000000000000001471400134000203765ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/.ipynb_checkpoints/000077500000000000000000000000001471400134000241675ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/.ipynb_checkpoints/logger-checkpoint.py000066400000000000000000000444021471400134000301510ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LOGGER Module""" import logging import logging.config import logging.handlers import os import stat import sys from functools import wraps from tempfile import TemporaryFile from typing import Dict, List, Tuple, Union from mindformers.tools.utils import get_num_nodes_devices, get_rank_info, \ convert_nodes_devices_input, generate_rank_list,\ check_in_modelarts, check_list, LOCAL_DEFAULT_PATH,\ get_output_root_path logger_list = [] LEVEL = ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL') MODELARTS_LOG_FILE_DIR = '/cache/ma-user-work/' LOCAL_DEFAULT_LOG_FILE_DIR = os.path.join( os.getenv("LOCAL_DEFAULT_PATH", LOCAL_DEFAULT_PATH), 'log') RANK_DIR_FORMATTER = 'rank_{}/' DEFAULT_FILEHANDLER_FORMAT = '[%(levelname)s] %(asctime)s ' \ '[%(pathname)s:%(lineno)d] %(funcName)s: %(message)s' DEFAULT_STDOUT_FORMAT = '%(asctime)s - %(name)s[%(filename)s:%(lineno)d] - %(levelname)s - %(message)s' DEFAULT_REDIRECT_FILE_NAME = 'mindspore.log' def judge_redirect(rank_id: int, rank_size: int, redirect_nodes: Union[List, Tuple, Dict[str, int], None] = None, redirect_devices: Union[List, Tuple, Dict[str, int], None] = None): """Determine if the stderr of this process needs to be redirected. Args: rank_id (int): Rank id. rank_size (int): Rank Size. redirect_nodes (list or tuple or dict or None): Node list. The nodes in the list will redirect stderr. redirect_devices (list or tuple or dict or None): Device list. The devices in the list will redirect stderr. Returns: prerequisite (bool): If true, stderr will redirect. """ is_redirect = True if rank_size > 1 and (redirect_nodes is not None or redirect_devices is not None): num_nodes, num_devices = get_num_nodes_devices(rank_size) redirect_nodes = convert_nodes_devices_input(redirect_nodes, num_nodes) redirect_devices = convert_nodes_devices_input(redirect_devices, num_devices) check_list('nodes', redirect_nodes, num_nodes) check_list('devices', redirect_devices, num_devices) rank_list = generate_rank_list(redirect_nodes, redirect_devices) if rank_id not in rank_list: is_redirect = False return is_redirect class StreamRedirector: """Stream Re-director for Log.""" def __init__(self, source_stream, target_stream): """Redirects the source stream to the target stream. Args: source_stream: Source stream. target_stream: Target stream. """ super(StreamRedirector, self).__init__() self.source_stream = source_stream self.target_stream = target_stream self.save_source_stream_fd = os.dup(self.source_stream.fileno()) def __call__(self, func): @wraps(func) def wrapper(*args, **kwargs): self.start() func(*args, **kwargs) self.stop() return wrapper def __enter__(self): self.start() def __exit__(self, exc_type, exc_val, exc_tb): self.stop() def start(self): """start.""" self.source_stream.flush() os.dup2(self.target_stream.fileno(), self.source_stream.fileno()) def stop(self): """stop.""" self.source_stream.flush() os.dup2(self.save_source_stream_fd, self.source_stream.fileno()) self.target_stream.flush() class AiLogFastStreamRedirect2File(StreamRedirector): """AiLogFastStreamRedirect2File.""" def __init__(self, source_stream=None, redirect_nodes: Union[List, Tuple, Dict[str, int], None] = None, redirect_devices: Union[List, Tuple, Dict[str, int], None] = None, **kwargs): """Redirect stream to file. Args: source_stream (file object or None): Streams that need to be redirected. Default: None, select stderr. redirect_nodes (list[int] or tuple[int] or optional): The computation nodes that will redirect stderr. Default is None: indicates that all nodes will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all redirect stderr. redirect_devices (list[int] or tuple[int] or optional): The computation devices that will redirect stderr. Default is None, indicates that all devices will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all redirect stderr. kwargs (dict): File-related parameters. file_save_dir (str): The folder where the files that save redirected stream are saved. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (str): Redirect file name. """ rank_id, rank_size = get_rank_info() self.is_redirect = judge_redirect(rank_id=rank_id, rank_size=rank_size, redirect_nodes=redirect_nodes, redirect_devices=redirect_devices) file_save_dir = kwargs.get('file_save_dir', '') append_rank_dir = kwargs.get('append_rank_dir', True) file_name = kwargs.get('file_name', '') if not file_save_dir: file_save_dir = os.path.join(get_output_root_path(), 'log') if append_rank_dir: rank_str = RANK_DIR_FORMATTER.format(rank_id) file_save_dir = os.path.join(file_save_dir, rank_str) if not file_name: file_name = DEFAULT_REDIRECT_FILE_NAME self.file_path = os.path.join(file_save_dir, file_name) self.file_save_dir = os.path.dirname(self.file_path) if source_stream is None: source_stream = sys.stderr target_stream = TemporaryFile(mode='w+') super(AiLogFastStreamRedirect2File, self).__init__(source_stream=source_stream, target_stream=target_stream) def start(self): if self.is_redirect: super(AiLogFastStreamRedirect2File, self).start() def stop(self): if self.is_redirect: self.target_stream.flush() if not os.path.exists(self.file_save_dir): os.makedirs(self.file_save_dir, exist_ok=True) self.target_stream.seek(0, 0) flags = os.O_WRONLY | os.O_CREAT modes = stat.S_IWUSR | stat.S_IRUSR with os.fdopen(os.open(self.file_path, flags, modes), 'w') as fp: for line in self.target_stream: fp.write(line) super(AiLogFastStreamRedirect2File, self).stop() self.target_stream.close() def judge_stdout(rank_id: int, rank_size: int, is_output: bool, nodes: Union[List, Tuple, Dict[str, int], None] = None, devices: Union[List, Tuple, Dict[str, int], None] = None) -> bool: """Determines if logs will be output. Args: rank_id (int): Rank id. rank_size (int): Rank size. is_output (int): If set to true, logs or others will be output. nodes (list or tuple or dict or None): Node list. The nodes in the list will output the log to stdout. devices (list or tuple or dict or None): Device list. The devices in the list or output the log to stdout. Returns: is_output (bool): If set to true, logs or others will be output or redirect. """ if is_output and rank_size > 1 and (nodes is not None or devices is not None): num_nodes, num_devices = get_num_nodes_devices(rank_size) stdout_nodes = convert_nodes_devices_input(nodes, num_nodes) stdout_devices = convert_nodes_devices_input(devices, num_devices) check_list('nodes', stdout_nodes, num_nodes) check_list('devices', stdout_devices, num_devices) rank_list = generate_rank_list(stdout_nodes, stdout_devices) if rank_id not in rank_list: is_output = False return is_output def validate_nodes_devices_input(var_name: str, var): """Check the list of nodes or devices. Args: var_name (str): Variable name. var: The name of the variable to be checked. Returns: None """ if not (var is None or isinstance(var, (list, tuple, dict))): raise TypeError('The value of {} can be None or a value of type tuple, ' 'list, or dict.'.format(var_name)) if isinstance(var, (list, tuple)): for item in var: if not isinstance(item, int): raise TypeError('The elements of a variable of type list or ' 'tuple must be of type int.') def validate_level(var_name: str, var): """Verify that the log level is correct. Args: var_name (str): Variable name. var: The name of variable to be checked. Returns: None """ if not isinstance(var, str): raise TypeError('The format of {} must be of type str.'.format(var_name)) if var not in LEVEL: raise ValueError('{}={} needs to be in {}'.format(var_name, var, LEVEL)) def validate_std_input_format(to_std: bool, stdout_nodes: Union[List, Tuple, None], stdout_devices: Union[List, Tuple, None], stdout_level: str): """Validate the input about stdout of the get_logger function.""" if not isinstance(to_std, bool): raise TypeError('The format of the to_std must be of type bool.') validate_nodes_devices_input('stdout_nodes', stdout_nodes) validate_nodes_devices_input('stdout_devices', stdout_devices) validate_level('stdout_level', stdout_level) def validate_file_input_format(file_level: Union[List[str], Tuple[str]], file_save_dir: str, append_rank_dir: str, file_name: Union[List[str], Tuple[str]]): """Validate the input about file of the get_logger function.""" if not isinstance(file_level, (tuple, list)): raise TypeError('The value of file_level should be list or a tuple.') for level in file_level: validate_level('level in file_level', level) if not len(file_level) == len(file_name): raise ValueError('The length of file_level and file_name should be equal.') if not isinstance(file_save_dir, str): raise TypeError('The value of file_save_dir should be a value of type str.') if not isinstance(append_rank_dir, bool): raise TypeError('The value of append_rank_dir should be a value of type bool.') if not isinstance(file_name, (tuple, list)): raise TypeError('The value of file_name should be list or a tuple.') for name in file_name: if not isinstance(name, str): raise TypeError('The value of name in file_name should be a value of type str.') def _convert_level(level: str) -> int: """Convert the format of the log to logging level. Args: level (str): User log level. Returns: level (str): Logging level. """ level_convert = { 'DEBUG': logging.DEBUG, 'INFO': logging.INFO, 'WARNING': logging.WARNING, 'ERROR': logging.ERROR, 'CRITICAL': logging.CRITICAL } level = level_convert.get(level, logging.INFO) return level def get_logger(logger_name: str = 'mindformers', **kwargs) -> logging.Logger: """Get the logger. Both computing centers and bare metal servers are available. Args: logger_name (str): Logger name. kwargs (dict): Other input. to_std (bool): If set to True, output the log to stdout. stdout_nodes (list[int] or tuple[int] or optional): The computation nodes that will output the log to stdout. default: None, indicates that all nodes will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all output logs to stdout. stdout_devices (list[int] or tuple[int] or optional): The computation devices that will output the log to stdout. default: None, indicates that all devices will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all output logs to stdout. stdout_level (str): The level of the log output to stdout. If the type is str, the options are DEBUG, INFO, WARNING, ERROR, CRITICAL. stdout_format (str): Log format. file_level (list[str] or tuple[str]): The level of the log output to file. eg: ['INFO', 'ERROR'] Indicates that the logger will output logs above the level INFO and ERROR in the list to the corresponding file. The length of the list needs to be the same as the length of file_name. file_save_dir (str): The folder where the log files are stored. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (list[str] or list[tuple]): Store a list of output file names. max_file_size (int): The maximum size of a single log file. Unit: MB. max_num_of_files (int): The maximum number of files to save. Returns: logger (logging.Logger): Logger. """ mf_logger = logging.getLogger(logger_name) if logger_name in logger_list: return mf_logger to_std = kwargs.get('to_std', True) stdout_nodes = kwargs.get('stdout_nodes', None) rank_id, rank_size = get_rank_info() def get_stdout_devices(): if os.getenv("STDOUT_DEVICES"): devices = os.getenv("STDOUT_DEVICES") if devices.startswith(("(", "[")) and devices.endswith((")", "]")): devices = devices[1:-1] devices = tuple(map(lambda x: int(x.strip()), devices.split(","))) elif check_in_modelarts(): devices = kwargs.get('stdout_devices', (min(rank_size - 1, 7),)) else: devices = kwargs.get('stdout_devices', None) return devices stdout_devices = get_stdout_devices() stdout_level = kwargs.get('stdout_level', 'INFO') stdout_format = kwargs.get('stdout_format', '') file_level = kwargs.get('file_level', ('INFO', 'ERROR')) file_save_dir = kwargs.get('file_save_dir', '') append_rank_dir = kwargs.get('append_rank_dir', True) file_name = kwargs.get('file_name', (f'info.log', 'error.log')) max_file_size = kwargs.get('max_file_size', 50) max_num_of_files = kwargs.get('max_num_of_files', 5) validate_std_input_format(to_std, stdout_nodes, stdout_devices, stdout_level) validate_file_input_format(file_level, file_save_dir, append_rank_dir, file_name) to_std = judge_stdout(rank_id=rank_id, rank_size=rank_size, is_output=to_std, nodes=stdout_nodes, devices=stdout_devices) if to_std: if not stdout_format: stdout_format = DEFAULT_STDOUT_FORMAT stream_handler = logging.StreamHandler(sys.stdout) stream_handler.setLevel(_convert_level(stdout_level)) stream_formatter = logging.Formatter(stdout_format) stream_handler.setFormatter(stream_formatter) mf_logger.addHandler(stream_handler) logging_level = [] for level in file_level: logging_level.append(_convert_level(level)) if not file_save_dir: file_save_dir = os.path.join(get_output_root_path(), 'log') rank_dir = RANK_DIR_FORMATTER if append_rank_dir: rank_str = rank_dir.format(rank_id) file_save_dir = os.path.join(file_save_dir, rank_str) file_path = [] for name in file_name: path = os.path.join(file_save_dir, name) path = os.path.realpath(path) base_dir = os.path.dirname(path) if not os.path.exists(base_dir): os.makedirs(base_dir, exist_ok=True) file_path.append(path) max_file_size = max_file_size * 1024 * 1024 file_formatter = logging.Formatter(DEFAULT_FILEHANDLER_FORMAT) for i, level in enumerate(file_level): file_handler = logging.handlers.RotatingFileHandler(filename=file_path[i], maxBytes=max_file_size, backupCount=max_num_of_files) file_handler.setLevel(level) file_handler.setFormatter(file_formatter) mf_logger.addHandler(file_handler) mf_logger.setLevel(_convert_level('INFO')) mf_logger.propagate = False logger_list.append(logger_name) return mf_logger class _LogActionOnce: """ A wrapper for modify the warning logging to an empty function. This is used when we want to only log once to avoid the repeated logging. Args: logger (logging): The logger object. """ is_logged = dict() def __init__(self, m_logger, key, no_warning=False): self.logger = m_logger self.key = key self.no_warning = no_warning def __call__(self, func): def wrapper(*args, **kwargs): if not hasattr(self.logger, 'warning'): return func(*args, **kwargs) old_func = self.logger.warning if self.no_warning or self.key in _LogActionOnce.is_logged: self.logger.warning = lambda x: x else: _LogActionOnce.is_logged[self.key] = True res = func(*args, **kwargs) if hasattr(self.logger, 'warning'): self.logger.warning = old_func return res return wrapper logger = get_logger() pengcheng.mind-test/mindformers/tools/.ipynb_checkpoints/utils-checkpoint.py000066400000000000000000000261511471400134000300330ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Utils For Tools.""" import json import os from typing import Dict, List, Tuple, Union from multiprocessing import Process try: import fcntl except ImportError: fcntl = None import numpy as np from mindspore import Tensor from mindspore import context PARALLEL_MODE = {'DATA_PARALLEL': context.ParallelMode.DATA_PARALLEL, 'SEMI_AUTO_PARALLEL': context.ParallelMode.SEMI_AUTO_PARALLEL, 'AUTO_PARALLEL': context.ParallelMode.AUTO_PARALLEL, 'HYBRID_PARALLEL': context.ParallelMode.HYBRID_PARALLEL, 'STAND_ALONE': context.ParallelMode.STAND_ALONE, 0: context.ParallelMode.DATA_PARALLEL, 1: context.ParallelMode.SEMI_AUTO_PARALLEL, 2: context.ParallelMode.AUTO_PARALLEL, 3: context.ParallelMode.HYBRID_PARALLEL} MODE = {'PYNATIVE_MODE': context.PYNATIVE_MODE, 'GRAPH_MODE': context.GRAPH_MODE, 0: context.GRAPH_MODE, 1: context.PYNATIVE_MODE} SAVE_WORK_PATH = '/cache/ma-user-work/rank_{}' MA_OUTPUT_ROOT = '/cache/ma-user-work' DEBUG_INFO_PATH = '/cache/debug' PROFILE_INFO_PATH = '/cache/profile' PLOG_PATH = '/root/ascend/log' LOCAL_DEFAULT_PATH = os.getenv("LOCAL_DEFAULT_PATH", './output') LAST_TRANSFORM_LOCK_PATH = "/tmp/last_transform_done.lock" RANK_FINISH_PATH = "/cache/rank_*_finish.txt" _PROTOCOL = 'obs' _PROTOCOL_S3 = 's3' def check_in_modelarts(): """Check if the training is on modelarts. Returns: (bool): If it is True, it means ModelArts environment. """ # 'KUBERNETES_PORT' in os.environ or \ return 'MA_LOG_DIR' in os.environ or \ 'MA_JOB_DIR' in os.environ or \ 'MA_LOCAL_LOG_PATH' in os.environ or \ 'S3_ACCESS_KEY_ID' in os.environ or \ 'S3_SECRET_ACCESS_KEY' in os.environ or \ 'BATCH_GROUP_NAME' in os.environ or \ 'MA_LOCAL_LOG_PATH' in os.environ class Validator: """validator for checking input parameters""" @staticmethod def check_type(arg_value, arg_type): """Check int.""" if not isinstance(arg_value, arg_type): raise TypeError('{} should be {} type, but get {}'.format(arg_value, arg_type, type(arg_value))) @staticmethod def is_obs_url(url): """Check obs url.""" return url.startswith(_PROTOCOL + '://') or url.startswith(_PROTOCOL_S3 + '://') def check_obs_url(url): """Check obs url.""" if not isinstance(url, str): raise TypeError('remote_save_url type should be a str, but get {}, ' 'please check your remote_save_url config'.format(type(url))) if not (url.startswith(_PROTOCOL + '://') or url.startswith(_PROTOCOL_S3 + '://')): raise TypeError('remote_save_url should be start with obs:// or s3://, ' 'but get {}, please check your remote_save_url config'.format(url)) return True def check_list(var_name: str, list_var: Union[Tuple, List], num: int): """Checks the legitimacy of elements within a node or device list. Args: var_name (str): The Name of variable need to check. list_var (tuple or list): Variables in list format. num (int): The number of nodes or devices. Returns: None """ for value in list_var: if value >= num: raise ValueError('The index of the {} needs to be less than the number of nodes {}.'.format(var_name, num)) def format_path(path): """Check path.""" return os.path.realpath(path) def sync_trans(f): """Asynchronous transport decorator.""" try: def wrapper(*args, **kwargs): pro = Process(target=f, args=args, kwargs=kwargs) pro.start() return pro return wrapper except Exception as e: raise e def get_output_root_path(): """get default output path in local/AICC.""" if check_in_modelarts(): return MA_OUTPUT_ROOT return LOCAL_DEFAULT_PATH def get_output_subpath(sub_class, rank_id=0, append_rank=True): """get output store path for sub output class.""" Validator.check_type(sub_class, str) root_path = get_output_root_path() directory = os.path.join(root_path, sub_class) if append_rank: directory = os.path.join(directory, 'rank_{}'.format(rank_id)) return format_path(directory) def set_remote_save_url(remote_save_url): check_obs_url(remote_save_url) os.environ.setdefault('REMOTE_SAVE_URL', remote_save_url) def get_remote_save_url(): return os.environ.get('REMOTE_SAVE_URL', None) def get_net_outputs(params): """Get network outputs.""" if isinstance(params, (tuple, list)): if isinstance(params[0], Tensor) and isinstance(params[0].asnumpy(), np.ndarray): params = params[0] elif isinstance(params, Tensor) and isinstance(params.asnumpy(), np.ndarray): params = np.mean(params.asnumpy()) return params def get_rank_info() -> Tuple[int, int]: """Get rank_info from environment variables. Returns: rank_id (int): Rank id. rand_size (int): The number of rank. """ rank_id = int(os.getenv('RANK_ID', '0')) rank_size = int(os.getenv('RANK_SIZE', '1')) return rank_id, rank_size def get_num_nodes_devices(rank_size: int) -> Tuple[int, int]: """Derive the number of nodes and devices based on rank_size. Args: rank_size (int): rank size. Returns: num_nodes (int): number of nodes. num_devices (int): number of devices. """ if rank_size in (2, 4, 8): num_nodes = 1 num_devices = rank_size else: num_nodes = rank_size // 8 num_devices = 8 return num_nodes, num_devices class Const: """Const.""" def __setattr__(self, key, value): if key in self.__dict__: raise PermissionError('Can not change const {0}.'.format(key)) if not key.isupper(): raise ValueError('Const name {0} is not all uppercase.'.format(key)) self.__dict__[key] = value def generate_rank_list(stdout_nodes: Union[List, Tuple], stdout_devices: Union[List, Tuple]): """Generate a list of the ranks to output the log. Args: stdout_nodes (list or tuple): The compute nodes that will output the log to stdout. stdout_devices (list or tuple): The compute devices that will output the log to stdout. Returns: rank_list (list): A list of the ranks to output the log to stdout. """ rank_list = [] for node in stdout_nodes: for device in stdout_devices: rank_list.append(8 * node + device) return rank_list def convert_nodes_devices_input(var: Union[List, Tuple, Dict[str, int], None], num: int) -> Union[List, Tuple]: """Convert node and device inputs to list format. Args: var (list[int] or tuple[int] or dict[str, int] or optional): The variables that need to be converted to the format. num (str): The number of nodes or devices. Returns: var (list[int] or tuple[int]): A list of nodes or devices. """ if var is None: var = tuple(range(num)) elif isinstance(var, dict): var = tuple(range(var['start'], var['end'])) return var def str2bool(b): """String convert to Bool.""" if b.lower() in ["false"]: output = False elif b.lower() in ["true"]: output = True else: raise Exception("Invalid Bool Value") return output def count_params(net): """Count number of parameters in the network Args: net (mindspore.nn.Cell): Mindspore network instance Returns: total_params (int): Total number of trainable params """ total_params = [np.prod(param.shape) for param in net.trainable_params()] return sum(total_params) // 1000000 def try_sync_file(file_name): """If the file is still downloading, we need to wait before the file finished downloading""" if fcntl: with open(file_name, 'r') as fp: fcntl.flock(fp.fileno(), fcntl.LOCK_EX) def is_version_le(current_version, base_version): """ return current_version <= base_version. Check whether the current version is lower than or equal to the base version. For example: for current_version: 1.8.1, base_version: 1.11.0, it return True. """ version_split_char = '.' if version_split_char not in base_version or version_split_char not in current_version: raise ValueError("The version string will contain the `.`." "For example, current_version 1.8.1, base_version: 1.11.0.") for x, y in zip(current_version.split(version_split_char), base_version.split(version_split_char)): if not x.isdigit() or not y.isdigit(): continue if int(x) != int(y): return int(x) <= int(y) return True def is_version_ge(current_version, base_version): """ return current_version >= base_version. Check whether the current version is higher than or equal to the base version. for current_version: 1.8.1, base_version: 1.11.0, it return False. """ version_split_char = '.' if version_split_char not in base_version or version_split_char not in current_version: raise ValueError("The version string will contain the `.`." "For example, current_version 1.8.1， base_version: 1.11.0.") for x, y in zip(current_version.split(version_split_char), base_version.split(version_split_char)): if not x.isdigit() or not y.isdigit(): continue if int(x) != int(y): return int(x) >= int(y) return True def parse_value(value): """ parse value from command line. handles with int, float, bool, string, list and dict. """ def isint(x): try: a = float(x) b = int(a) except (TypeError, ValueError): return False else: return a == b def isfloat(x): try: float(x) except (TypeError, ValueError): return False else: return True def isbool(x): return x in ["True", "False"] def isjson(x): try: json.loads(x) except json.decoder.JSONDecodeError: return False else: return True if isint(value): return int(value) if isfloat(value): return float(value) if isbool(value): return value == "True" if isjson(value): return json.loads(value) return value pengcheng.mind-test/mindformers/tools/__init__.py000066400000000000000000000020451471400134000225100ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Tools.""" from .cloud_adapter import * from .register import * from .logger import logger, StreamRedirector, AiLogFastStreamRedirect2File from .utils import PARALLEL_MODE, MODE, DEBUG_INFO_PATH, \ check_in_modelarts, str2bool, count_params, get_output_root_path, get_output_subpath __all__ = ['logger'] __all__.extend(cloud_adapter.__all__) __all__.extend(register.__all__) pengcheng.mind-test/mindformers/tools/__pycache__/000077500000000000000000000000001471400134000226065ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/__pycache__/__init__.cpython-39.pyc000066400000000000000000000011451471400134000267770ustar00rootroot00000000000000a >cf%@srdZddlTddlTddlmZmZmZddlmZmZm Z m Z m Z m Z m Z mZdgZeejeejdS)zMindFormers Tools.)*)loggerStreamRedirectorAiLogFastStreamRedirect2File) PARALLEL_MODEMODEDEBUG_INFO_PATHcheck_in_modelartsstr2bool count_paramsget_output_root_pathget_output_subpathrN)__doc__ cloud_adapterregisterrrrutilsrrrr r r r r __all__extendrrE/home/ma-user/work/mindformers-baichuan/mindformers/tools/__init__.pys( pengcheng.mind-test/mindformers/tools/__pycache__/download_tools.cpython-39.pyc000066400000000000000000000050141471400134000302660ustar00rootroot00000000000000a >cf @sdZddlZddlZddlZddlZddlmZddlmZz ddlZWn e yhdZe dYn0e GdddZ d d d Z dS) Zdownload_toolsN)tqdm)loggerz{The library fcntl is not found. This may cause the reading file failed when call the from_pretrained for different process.c@seZdZdZdZdS) StatusCodeN)__name__ __module__ __qualname____doc__succeedr r K/home/ma-user/work/mindformers-baichuan/mindformers/tools/download_tools.pyr!src CsLttdd}|ddkrs    pengcheng.mind-test/mindformers/tools/__pycache__/image_tools.cpython-39.pyc000066400000000000000000000022421471400134000275410ustar00rootroot00000000000000a >cf@s:dZddlZddlZddlZddlZddlZdddZdS)z Image tools Nc Cst|tr|ds|drztj|d|d}WnDttjjtjj fyv}zt d|d|WYd}~n d}~00t j |j}n(tj|rt j |}nt|dt|tjr|St|t j j std t j|}|d }|S) z load imagezhttps://zhttp://T)streamtimeoutzConnect error, please download .NzI is not a valid path. If URL, it must start with `http://` or `https://`.z]Input should be an url linking to an image, a local path, a Mindspore Tensor, or a PIL image.RGB) isinstancestr startswithrequestsget TimeoutErrorurllib3 exceptions MaxRetryError ProxyErrorConnectionErrorPILImageopenrawospathisfile ValueErrormsTensorImageOpsexif_transposeconvert)contentrresponseexcr"H/home/ma-user/work/mindformers-baichuan/mindformers/tools/image_tools.py load_images.   (    r$)r)__doc__rr r r mindsporerr$r"r"r"r#s pengcheng.mind-test/mindformers/tools/__pycache__/logger.cpython-39.pyc000066400000000000000000000365521471400134000265310ustar00rootroot00000000000000a >cfI @sdZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z mZmZddlmZmZmZmZmZmZmZmZgZdZdZejed ed Zd Z d Z!d Z"dZ#d-e$e$ee ee e%e$fdfee ee e%e$fdfdddZ&GdddZ'Gddde'Z(d.e$e$e)ee ee e%e$fdfee ee e%e$fdfe)dddZ*e%dddZ+e%dddZ,e)ee edfee edfe%ddd Z-ee e%ee%fe%e%ee e%ee%fd!d"d#Z.e%e$d$d%d&Z/d/e%ej0d(d)d*Z1Gd+d,d,Z2e1Z3dS)0z LOGGER ModuleNwraps) TemporaryFile)DictListTupleUnion)get_num_nodes_devices get_rank_infoconvert_nodes_devices_inputgenerate_rank_listcheck_in_modelarts check_listLOCAL_DEFAULT_PATHget_output_root_pathDEBUGINFOWARNINGERRORCRITICALz/cache/ma-user-work/rlogzrank_{}/zO[%(levelname)s] %(asctime)s [%(pathname)s:%(lineno)d] %(funcName)s: %(message)szM%(asctime)s - %(name)s[%(filename)s:%(lineno)d] - %(levelname)s - %(message)sz mindspore.logrank_id rank_sizeredirect_nodesredirect_devicescCsnd}|dkrj|dus|durjt|\}}t||}t||}td||td||t||}||vrjd}|S)aDetermine if the stderr of this process needs to be redirected. Args: rank_id (int): Rank id. rank_size (int): Rank Size. redirect_nodes (list or tuple or dict or None): Node list. The nodes in the list will redirect stderr. redirect_devices (list or tuple or dict or None): Device list. The devices in the list will redirect stderr. Returns: prerequisite (bool): If true, stderr will redirect. TNnodesdevicesFr r rr )rrrr is_redirect num_nodes num_devices rank_listr%C/home/ma-user/work/mindformers-baichuan/mindformers/tools/logger.pyjudge_redirect-s      r'csHeZdZdZfddZddZddZdd Zd d Zd d Z Z S)StreamRedirectorzStream Re-director for Log.cs0tt|||_||_t|j|_dS)zRedirects the source stream to the target stream. Args: source_stream: Source stream. target_stream: Target stream. N) superr(__init__ source_stream target_streamosdupfilenosave_source_stream_fd)selfr+r, __class__r%r&r*OszStreamRedirector.__init__cstfdd}|S)Ncs"|i|dSN)startstop)argskwargsfuncr1r%r&wrapper_sz*StreamRedirector.__call__..wrapperrr1r:r;r%r9r&__call__]szStreamRedirector.__call__cCs |dSr4)r5r1r%r%r& __enter__gszStreamRedirector.__enter__cCs |dSr4)r6)r1exc_typeexc_valexc_tbr%r%r&__exit__jszStreamRedirector.__exit__cCs&|jt|j|jdS)zstart.N)r+flushr-dup2r,r/r>r%r%r&r5ms zStreamRedirector.startcCs,|jt|j|j|jdS)zstop.N)r+rDr-rEr0r/r,r>r%r%r&r6rs zStreamRedirector.stop) __name__ __module__ __qualname____doc__r*r=r?rCr5r6 __classcell__r%r%r2r&r(Ls  r(csjeZdZdZd eeeeee fdfeeeeee fdfdfdd Z fddZ fdd Z Z S) AiLogFastStreamRedirect2FilezAiLogFastStreamRedirect2File.N)rrc  st\}}t||||d|_|dd}|dd}|dd} |sTtjtd}|rpt |} tj|| }| sxt } tj|| |_ tj |j |_ |durtj}td d } tt|j|| d dS) aRedirect stream to file. Args: source_stream (file object or None): Streams that need to be redirected. Default: None, select stderr. redirect_nodes (list[int] or tuple[int] or optional): The computation nodes that will redirect stderr. Default is None: indicates that all nodes will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all redirect stderr. redirect_devices (list[int] or tuple[int] or optional): The computation devices that will redirect stderr. Default is None, indicates that all devices will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all redirect stderr. kwargs (dict): File-related parameters. file_save_dir (str): The folder where the files that save redirected stream are saved. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (str): Redirect file name. r file_save_dirappend_rank_dirT file_namerNzw+)mode)r+r,)r r'r!getr-pathjoinrRANK_DIR_FORMATTERformatDEFAULT_REDIRECT_FILE_NAME file_pathdirnamerLsysstderrrr)rKr*) r1r+rrr8rrrLrNrOrank_strr,r2r%r&r*{s,      z%AiLogFastStreamRedirect2File.__init__cs|jrtt|dSr4)r!r)rKr5r>r2r%r&r5sz"AiLogFastStreamRedirect2File.startcs|jr|jtj|js.tj|jdd|jddtj tj B}t j t j B}tt|j||d&}|jD]}||qtWdn1s0Ytt||jdS)NTexist_okrw)r!r,rDr-rRexistsrLmakedirsseekO_WRONLYO_CREATstatS_IWUSRS_IRUSRfdopenopenrWwriter)rKr6close)r1flagsmodesfpliner2r%r&r6s    *z!AiLogFastStreamRedirect2File.stop)NNN)rFrGrHrIrrrrstrintr*r5r6rJr%r%r2r&rKys6 rK)rr is_outputrrreturnc Csn|rj|dkrj|dus|durjt|\}}t||}t||}td||td||t||} || vrjd}|S)a'Determines if logs will be output. Args: rank_id (int): Rank id. rank_size (int): Rank size. is_output (int): If set to true, logs or others will be output. nodes (list or tuple or dict or None): Node list. The nodes in the list will output the log to stdout. devices (list or tuple or dict or None): Device list. The devices in the list or output the log to stdout. Returns: is_output (bool): If set to true, logs or others will be output or redirect. rNrrFr ) rrrqrrr"r# stdout_nodesstdout_devicesr$r%r%r& judge_stdouts      ru)var_namecCsT|dus&t|tttfs&td|t|ttfrP|D]}t|ts8tdq8dS)zCheck the list of nodes or devices. Args: var_name (str): Variable name. var: The name of the variable to be checked. Returns: None NzDThe value of {} can be None or a value of type tuple, list, or dict.zEThe elements of a variable of type list or tuple must be of type int.) isinstancelisttupledict TypeErrorrUrp)rvvaritemr%r%r&validate_nodes_devices_inputs  r~cCs6t|tstd||tvr2td||tdS)zVerify that the log level is correct. Args: var_name (str): Variable name. var: The name of variable to be checked. Returns: None z%The format of {} must be of type str.z{}={} needs to be in {}N)rwror{rULEVEL ValueError)rvr|r%r%r&validate_levels rto_stdrsrt stdout_levelcCs4t|tstdtd|td|td|dS)z;Validate the input about stdout of the get_logger function.z.The format of the to_std must be of type bool.rsrtrN)rwboolr{r~rrr%r%r&validate_std_input_formats    r) file_levelrLrNrOcCst|ttfstd|D]}td|qt|t|ksBtdt|tsTtdt|tsftdt|ttfs|td|D]}t|tstdqdS) z9Validate the input about file of the get_logger function.z2The value of file_level should be list or a tuple.zlevel in file_levelz7The length of file_level and file_name should be equal.z9The value of file_save_dir should be a value of type str.z\}}tjj||||d}| |||||q| t dd|_ t||S)aGet the logger. Both computing centers and bare metal servers are available. Args: logger_name (str): Logger name. kwargs (dict): Other input. to_std (bool): If set to True, output the log to stdout. stdout_nodes (list[int] or tuple[int] or optional): The computation nodes that will output the log to stdout. default: None, indicates that all nodes will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all output logs to stdout. stdout_devices (list[int] or tuple[int] or optional): The computation devices that will output the log to stdout. default: None, indicates that all devices will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all output logs to stdout. stdout_level (str): The level of the log output to stdout. If the type is str, the options are DEBUG, INFO, WARNING, ERROR, CRITICAL. stdout_format (str): Log format. file_level (list[str] or tuple[str]): The level of the log output to file. eg: ['INFO', 'ERROR'] Indicates that the logger will output logs above the level INFO and ERROR in the list to the corresponding file. The length of the list needs to be the same as the length of file_name. file_save_dir (str): The folder where the log files are stored. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (list[str] or list[tuple]): Store a list of output file names. max_file_size (int): The maximum size of a single log file. Unit: MB. max_num_of_files (int): The maximum number of files to save. Returns: logger (logging.Logger): Logger. rTrsNcs~tdrNtd}|dr4|dr4|dd}ttdd|d}n,trnd t dd f}n d d}|S) NZSTDOUT_DEVICES)([))]rcSs t|Sr4)rpstripxr%r%r&qz8get_logger..get_stdout_devices..,rt) r-getenv startswithendswithrymapsplitr rQmin)rr8rr%r&get_stdout_devicesls    z&get_logger..get_stdout_devicesrr stdout_formatrMr)rrrLrNrOinfo.log error.log max_file_size2max_num_of_files)rrrqrrrr\i)filenamemaxBytes backupCountF)rr)!r getLogger logger_listrQr rrruDEFAULT_STDOUT_FORMAT StreamHandlerrYstdoutsetLevelr Formatter setFormatter addHandlerappendr-rRrSrrTrUrealpathrXr_r`DEFAULT_FILEHANDLER_FORMAT enumeratehandlersRotatingFileHandler propagate)rr8Z mf_loggerrrsrrrtrrrrLrNrOrrstream_handlerZstream_formatterZ logging_levelrZrank_dirr[rWrrRbase_dirZfile_formatteri file_handlerr%rr& get_loggerAs~"                          rc@s(eZdZdZeZdddZddZdS) _LogActionOncez A wrapper for modify the warning logging to an empty function. This is used when we want to only log once to avoid the repeated logging. Args: logger (logging): The logger object. FcCs||_||_||_dSr4)loggerkey no_warning)r1Zm_loggerrrr%r%r&r*sz_LogActionOnce.__init__csfdd}|S)Ncsttjds|i|Sjj}js4jtjvrBddj_n dtjj<|i|}tjdrp|j_|S)NwarningcSs|Sr4r%rr%r%r&rrz:_LogActionOnce.__call__..wrapper..T)hasattrrrrrr is_logged)r7r8Zold_funcresr9r%r&r;s   z(_LogActionOnce.__call__..wrapperr%r<r%r9r&r=sz_LogActionOnce.__call__N)F)rFrGrHrIrzrr*r=r%r%r%r&rs r)NN)NN)r)4rIrZlogging.configlogging.handlersr-rdrY functoolsrtempfilertypingrrrrmindformers.tools.utilsr r r r r rrrrrZMODELARTS_LOG_FILE_DIRrRrSrZLOCAL_DEFAULT_LOG_FILE_DIRrTrrrVrpror'r(rKrrur~rrrrLoggerrrrr%r%r%r&sb  (  -N !  z"pengcheng.mind-test/mindformers/tools/__pycache__/transform_ckpt.cpython-39.pyc000066400000000000000000000035411471400134000302760ustar00rootroot00000000000000a ?cf @sFdZddlZddlZddlZdddZedkrBeZej dddd ej d dd d ej d de d dej dde ddej dde dde Z ee j Z ee jZe jZe jZe jZeje jsJe jdede edeedeedeedeedeeeee eeddS)ztransform ckptNcCs|sdStj|s"J|dtj|r2|Stj|r|rZtj|d|d}ntj|d}tj|r~t|t|||SdS)zMerge strategy if strategy path is dir Args: startegy_path (str): The path of stategy. rank_id (int): The rank id of device. Returns: None or strategy path N not found!Zmerged_ckpt_strategy_z.ckptzmerged_ckpt_strategy.ckpt) ospathexistsisfileisdirjoinremovemsmerge_pipeline_strategys)Z startegy_pathrank_idZ merge_pathr K/home/ma-user/work/mindformers-baichuan/mindformers/tools/transform_ckpt.py get_strategys      r__main__z--src_ckpt_strategyzpath of src ckpt strategy)defaulthelpz--dst_ckpt_strategyzpath of dst ckpt strategyz--src_ckpt_dirzpath of src ckpt)rtyperz--dst_ckpt_dirzpath where to save dst ckptz--prefix checkpoint_z prefix of transformed checkpointrzsrc_ckpt_strategy: zdst_ckpt_strategy: zsrc_ckpt_dir: zdst_ckpt_dir: zprefix: z......Start transform......z......Transform succeed!......)N)__doc__rargparse mindsporer r__name__ArgumentParserparser add_argumentstr parse_argsargssrc_ckpt_strategydst_ckpt_strategyZ src_ckpt_dirZ dst_ckpt_dirprefixrrprinttransform_checkpointsr r r rsX   pengcheng.mind-test/mindformers/tools/__pycache__/utils.cpython-39.pyc000066400000000000000000000252471471400134000264110ustar00rootroot00000000000000a ?cf;, @s*dZddlZddlZddlmZmZmZmZddlm Z z ddl Z Wne yZdZ Yn0ddl Z ddlmZddlmZejjejjejjejjejjejjejjejjejjd ZejejejejdZd Zd Zd Zd Zd Ze ddZ!dZ"dZ#dZ$ddZ%GdddZ&ddZ'e(eeefe)dddZ*ddZ+ddZ,d d!Z-dEd#d$Z.d%d&Z/d'd(Z0d)d*Z1ee)e)fd+d,d-Z2e)ee)e)fd.d/d0Z3Gd1d2d2Z4eeefeeefd3d4d5Z5eeeee(e)fdfe)eeefd6d7d8Z6d9d:Z7d;d<Z8d=d>Z9d?d@Z:dAdBZ;dCdDZ.wrapperN) Exception)rArBerr@r sync_transzs  rEcCstr tStS)z&get default output path in local/AICC.)rMA_OUTPUT_ROOTrrrrrget_output_root_pathsrGTcCs@t|tt}tj||}|r8tj|d|}t|S)z+get output store path for sub output class.zrank_{}) rr"r0rGrr8joinrr:) sub_classrank_idZ append_rank root_path directoryrrrget_output_subpaths  rMcCst|tjd|dSNZREMOTE_SAVE_URL)r1rr setdefault)remote_save_urlrrrset_remote_save_urlsrQcCstjddSrN)rrgetrrrrget_remote_save_urlsrScCsft|ttfr:t|dtrbt|dtjrb|d}n(t|trbt|tjrbt|}|S)zGet network outputs.r)rtuplelistrasnumpynpndarraymean)paramsrrrget_net_outputss " r[)returncCs(ttdd}ttdd}||fS)zGet rank_info from environment variables. Returns: rank_id (int): Rank id. rand_size (int): The number of rank. RANK_ID0 RANK_SIZE1)intrgetenv)rJ rank_sizerrr get_rank_infosrd)rcr\cCs&|dvrd}|}n |d}d}||fS)zDerive the number of nodes and devices based on rank_size. Args: rank_size (int): rank size. Returns: num_nodes (int): number of nodes. num_devices (int): number of devices. )rrrfr)rc num_nodes num_devicesrrrget_num_nodes_devicess ric@seZdZdZddZdS)ConstzConst.cCs<||jvrtd||s.td|||j|<dS)NzCan not change const {0}.z$Const name {0} is not all uppercase.)__dict__PermissionErrorrisupperr5)selfkeyr6rrr __setattr__s  zConst.__setattr__N)r+r,r-r.rprrrrrjsrj) stdout_nodesstdout_devicescCs.g}|D] }|D]}|d||qq|S)alGenerate a list of the ranks to output the log. Args: stdout_nodes (list or tuple): The compute nodes that will output the log to stdout. stdout_devices (list or tuple): The compute devices that will output the log to stdout. Returns: rank_list (list): A list of the ranks to output the log to stdout. rf)append)rqrr rank_listnodedevicerrrgenerate_rank_lists rw)varr4r\cCs:|durtt|}n t|tr6tt|d|d}|S)aJConvert node and device inputs to list format. Args: var (list[int] or tuple[int] or dict[str, int] or optional): The variables that need to be converted to the format. num (str): The number of nodes or devices. Returns: var (list[int] or tuple[int]): A list of nodes or devices. Nr>end)rTrangerdict)rxr4rrrconvert_nodes_devices_inputs  r|cCs0|dvrd}n|dvr$d}ntd|S)zString convert to Bool.)falseF)trueTzInvalid Bool Value)lowerrC)boutputrrrstr2bools   rcCsdd|D}t|dS)zCount number of parameters in the network Args: net (mindspore.nn.Cell): Mindspore network instance Returns: total_params (int): Total number of trainable params cSsg|]}t|jqSr)rWprodshape).0paramrrr z count_params..i@B)trainable_paramssum)net total_paramsrrr count_params srcCsDtr@t|d"}t|tjWdn1s60YdS)zVIf the file is still downloading, we need to wait before the file finished downloadingrN)fcntlopenflockfilenoLOCK_EX) file_namefprrr try_sync_files rcCsvd}||vs||vrtdt||||D]>\}}|r2|sLq2t|t|kr2t|t|kSq2dS)z return current_version <= base_version. Check whether the current version is lower than or equal to the base version. For example: for current_version: 1.8.1, base_version: 1.11.0, it return True. .zaThe version string will contain the `.`.For example, current_version 1.8.1, base_version: 1.11.0.Tr5zipsplitisdigitraZcurrent_version base_versionZversion_split_charxyrrr is_version_lesrcCsvd}||vs||vrtdt||||D]>\}}|r2|sLq2t|t|kr2t|t|kSq2dS)z return current_version >= base_version. Check whether the current version is higher than or equal to the base version. for current_version: 1.8.1, base_version: 1.11.0, it return False. rucThe version string will contain the `.`.For example, current_version 1.8.1， base_version: 1.11.0.Trrrrr is_version_ge-srcCsfdd}dd}dd}dd}||r0t|S||r@t|S||rP|d kS||rbt|S|S) zj parse value from command line. handles with int, float, bool, string, list and dict. c Ss:zt|}t|}Wnttfy,YdS0||kSdS)NF)floatrarr5)rarrrrisintDs  zparse_value..isintc Ss.z t|Wnttfy$YdS0dSdSNFT)rrr5rrrrisfloatMs  zparse_value..isfloatcSs|dvS)N)TrueFalserrrrrisboolUszparse_value..isboolcSs0zt|Wntjjy&YdS0dSdSr)jsonloadsdecoderJSONDecodeErrorrrrrisjsonXs zparse_value..isjsonr)rarrr)r6rrrrrrr parse_value?s  r)rT)=r.rrtypingrrrrmultiprocessingrr ImportErrornumpyrW mindsporerr ParallelModer r r r r PARALLEL_MODErrMODEZSAVE_WORK_PATHrFDEBUG_INFO_PATHZPROFILE_INFO_PATHZ PLOG_PATHrbrZLAST_TRANSFORM_LOCK_PATHr&r'rrr1r0rar7r:rErGrMrQrSr[rdrirjrwr|rrrrrrrrrrsr              .  pengcheng.mind-test/mindformers/tools/cloud_adapter/000077500000000000000000000000001471400134000232045ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/cloud_adapter/.ipynb_checkpoints/000077500000000000000000000000001471400134000267755ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/cloud_adapter/.ipynb_checkpoints/cloud_adapter-checkpoint.py000066400000000000000000000227401471400134000343070ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Cloud Adapter.""" import os import time from mindspore.train.callback import Callback from ..logger import logger from ..utils import check_obs_url, check_in_modelarts, \ Validator, sync_trans if check_in_modelarts(): import moxing as mox __all__ = ['Local2ObsMonitor', 'Obs2Local', 'mox_adapter'] class Local2ObsMonitor(Callback): """File saved from local system to OBS server. Args: src_dir (str): Local system path, which means path of AI computing center platform. target_dir (str): OBS path starting with S3 or obs is used to save files. rank_id (int): the device's contents will be saved according to the actual rank_id. Default: None, means only the contents of the first device of each node are saved. upload_frequence (int): How often files are saved in AI computing center platform. Default: -1. keep_last (bool): Check whether files in the OBS are consistent with AI computing center platform. Default: True, means old file will be removed. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. """ def __init__(self, src_dir, target_dir, step_upload_frequence: int = 100, epoch_upload_frequence: int = -1, keep_last=True, retry=3, retry_time=5, log=logger): super(Local2ObsMonitor, self).__init__() self.src_dir = src_dir self.target_dir = target_dir self.step_upload_frequence = step_upload_frequence self.epoch_upload_frequence = epoch_upload_frequence self.keep_last = keep_last self.rank_id = int(os.getenv('RANK_ID', '0')) self.retry_time = retry_time self.retry = retry self.log = log self.cb_params = None self.pro = None self.on_modelarts = check_in_modelarts() if self.on_modelarts: check_obs_url(target_dir) def step_end(self, run_context): """upload files at the end of step.""" if not self.on_modelarts: return if self.step_upload_frequence <= 0: return self.cb_params = run_context.original_args() if self.cb_params.cur_step_num % self.step_upload_frequence == 0 and os.listdir(self.src_dir): self.log.info("Starting upload output file to obs!") self.upload() def epoch_end(self, run_context): """upload files at the end of epoch.""" if not self.on_modelarts: return if self.epoch_upload_frequence <= 0: return self.cb_params = run_context.original_args() if self.cb_params.cur_epoch_num % self.epoch_upload_frequence == 0 and os.listdir(self.src_dir): self.log.info("Starting upload output file to obs!") self.upload(True) print(f"finish uploading") def upload(self, last_epoch=False): """Upload Files to OBS.""" if self.pro: # wait for last upload done self.pro.join() if last_epoch: self.sync2obs_synchronize(self.src_dir, self.target_dir) else: self.pro = self.sync2obs(self.src_dir, self.target_dir) @sync_trans def sync2obs(self, src_dir, target_dir): """Asynchronous transfer to OBS.""" sub_dir_list = os.listdir(src_dir) for sub_dir in sub_dir_list: if 'checkpoint' in sub_dir: # for checkpoint, all processes upload its own checkpoint scr_ckpt_dir = os.path.join(src_dir, sub_dir, f"rank_{self.rank_id}") target_ckpt_dir = os.path.join(target_dir, sub_dir, f"rank_{self.rank_id}") if self.keep_last and mox.file.exists(target_ckpt_dir): mox.file.remove(target_ckpt_dir, recursive=True) mox_adapter(scr_ckpt_dir, target_ckpt_dir, self.retry, self.retry_time, self.log) elif self.rank_id % 8 == 0: # for other outputs, one rank upload all mox_adapter(os.path.join(src_dir, sub_dir), os.path.join(target_dir, sub_dir), self.retry, self.retry_time, self.log) def sync2obs_synchronize(self, src_dir, target_dir): """synchronous transfer to OBS.""" sub_dir_list = os.listdir(src_dir) for sub_dir in sub_dir_list: if 'checkpoint' in sub_dir: # for checkpoint, all processes upload its own checkpoint scr_ckpt_dir = os.path.join(src_dir, sub_dir, f"rank_{self.rank_id}") target_ckpt_dir = os.path.join(target_dir, sub_dir, f"rank_{self.rank_id}") if self.keep_last and mox.file.exists(target_ckpt_dir): mox.file.remove(target_ckpt_dir, recursive=True) mox_adapter(scr_ckpt_dir, target_ckpt_dir, self.retry, self.retry_time, self.log) elif self.rank_id % 8 == 0: # for other outputs, one rank upload all mox_adapter(os.path.join(src_dir, sub_dir), os.path.join(target_dir, sub_dir), self.retry, self.retry_time, self.log) class Obs2Local: """File saved from OBS server to local system of AI computing center platform. Args: rank_id (int): The obs's contents will be upload according to the actual rank_id value. Default: 0, means stored only one OBS file each node. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. """ def __init__(self, rank_id=0, retry=3, retry_time=5, log=logger): self.rank_id = int(rank_id) self.retry_time = retry_time self.retry = retry self.log = log def obs2local(self, obs_url, local_url, special_id=None): """Pull Obs Files to Local.""" obs_name = obs_url.split("/")[-1] mox_lock = os.path.join(local_url, "mox_copy_{}.lock".format(obs_name)) local_url = os.path.join(local_url, obs_name) if special_id is None: if self.rank_id % 8 == 0: mox_adapter(obs_url, local_url, self.retry, self.retry_time, self.log) try: os.mknod(mox_lock) except IOError: pass else: self.log.info("programming sleep for waiting download file from obs to local.") while True: if os.path.exists(mox_lock): break time.sleep(1) else: Validator.check_type(special_id, int) if self.rank_id == special_id: mox_adapter(obs_url, local_url, self.retry, self.retry_time, self.log) try: os.mknod(mox_lock) except IOError: pass else: self.log.info("programming sleep for waiting download file from obs to local.") while True: if os.path.exists(mox_lock): break time.sleep(1) return local_url def mox_adapter(src_dir, target_dir, retry=3, retry_time=5, log=logger): """File interaction with Moxing.""" success = False for i in range(retry + 1): start = time.time() try: mox.file.copy_parallel(src_url=src_dir, dst_url=target_dir, threads=0, is_processing=False) except (FileNotFoundError, RuntimeError) as e: log.info("%s, from %s download to %s failed, will retry(%d) again.", e, src_dir, target_dir, i) # sleep due to restriction of obs log.info("sleep time %d for waiting download file from obs.", retry_time) continue end = time.time() if Validator.is_obs_url(target_dir): if mox.file.exists(target_dir): success = True log.info("Pull/Push file %s success, cost time: %f", target_dir, end - start) break else: if os.path.exists(target_dir): success = True log.info("Pull/Push file %s success, cost time: %f", target_dir, end - start) break return success pengcheng.mind-test/mindformers/tools/cloud_adapter/.ipynb_checkpoints/cloud_monitor-checkpoint.py000066400000000000000000000072621471400134000343600ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Aicc monitor.""" import os import traceback import time from mindformers.tools.logger import get_logger from .cloud_adapter import mox_adapter from ..utils import DEBUG_INFO_PATH, PROFILE_INFO_PATH, PLOG_PATH, LAST_TRANSFORM_LOCK_PATH,\ get_output_root_path, get_remote_save_url, RANK_FINISH_PATH def cloud_monitor(log=get_logger()): """Aicc monitor for main function.""" def decorator(run_func): def wrapper(*args, **kwargs): local_id = int(os.getenv('RANK_ID', '0')) try: result = run_func(*args, **kwargs) except BaseException as exc: # pylint: disable=W0703 error = traceback.format_exc() log.error(error) raise exc finally: _last_transform(local_id, log) return result return wrapper return decorator def _last_transform(local_id, log=get_logger()): """Transform file when progress ending or except.""" local_node_id = local_id % 8 finish_file = RANK_FINISH_PATH.replace("*", str(local_node_id)) os.mknod(finish_file) world_size = int(os.getenv("RANK_SIZE", '1')) if world_size > 8: world_size = 8 if os.environ.get("SPECIAL_ID") and get_remote_save_url(): target_dir = get_remote_save_url() mox_adapter(src_dir=get_output_root_path(), target_dir=get_remote_save_url(), log=log) else: if local_id % 8 == 0 and get_remote_save_url(): for i in range(world_size): finish_file = RANK_FINISH_PATH.replace("*", str(i)) while not os.path.exists(finish_file): time.sleep(1) target_dir = get_remote_save_url() mox_adapter(src_dir=get_output_root_path(), target_dir=target_dir, log=log) if local_id % 8 == 0 and get_remote_save_url(): target_dir = get_remote_save_url() task_dir = os.path.join(target_dir, 'ascend-log') node = 'node_{}'.format(local_id) if os.path.exists(PLOG_PATH): mox_adapter(src_dir=PLOG_PATH, target_dir=os.path.join(task_dir, 'plog', node), log=log) if os.path.exists(DEBUG_INFO_PATH): mox_adapter(src_dir=DEBUG_INFO_PATH, target_dir=os.path.join(target_dir, 'debug_info', node), log=log) if os.path.exists(PROFILE_INFO_PATH): mox_adapter(src_dir=PROFILE_INFO_PATH, target_dir=os.path.join(target_dir, 'profile'), log=log) os.mknod(LAST_TRANSFORM_LOCK_PATH) elif get_remote_save_url(): log.info("Wait for the first card to complete the file and send it back to OBS: %s.", get_remote_save_url()) while True: if os.path.exists(LAST_TRANSFORM_LOCK_PATH): log.info("All files have been sent back to the OBS: %s," "and the process exits normally.", get_remote_save_url()) break def upload_log(): """Upload log according to runtime for FM.""" local_id = int(os.getenv('RANK_ID', '0')) _last_transform(local_id) pengcheng.mind-test/mindformers/tools/cloud_adapter/__init__.py000066400000000000000000000015121471400134000253140ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Cloud Adapter API.""" from .cloud_adapter import * from .cloud_monitor import cloud_monitor __all__ = ['cloud_monitor'] __all__.extend(cloud_adapter.__all__) pengcheng.mind-test/mindformers/tools/cloud_adapter/__pycache__/000077500000000000000000000000001471400134000254145ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/cloud_adapter/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005131471400134000316030ustar00rootroot00000000000000a ?cfJ@s.dZddlTddlmZdgZeejdS)zMindFormers Cloud Adapter API.)*) cloud_monitorrN)__doc__ cloud_adapterr__all__extendrrS/home/ma-user/work/mindformers-baichuan/mindformers/tools/cloud_adapter/__init__.pys pengcheng.mind-test/mindformers/tools/cloud_adapter/__pycache__/cloud_adapter.cpython-39.pyc000066400000000000000000000145041471400134000326570ustar00rootroot00000000000000a ?cfJ!@sdZddlZddlZddlmZddlmZddlmZmZm Z m Z erRddl Z gdZ Gdd d eZGd d d Zd d efddZdS)zCloud Adapter.N)Callback)logger) check_obs_urlcheck_in_modelarts Validator sync_trans)Local2ObsMonitor Obs2Local mox_adaptercsZeZdZdZdddddefeedfdd Zd d Zd d ZddZ e ddZ Z S)r aGFile saved from local system to OBS server. Args: src_dir (str): Local system path, which means path of AI computing center platform. target_dir (str): OBS path starting with S3 or obs is used to save files. rank_id (int): the device's contents will be saved according to the actual rank_id. Default: None, means only the contents of the first device of each node are saved. upload_frequence (int): How often files are saved in AI computing center platform. Default: -1. keep_last (bool): Check whether files in the OBS are consistent with AI computing center platform. Default: True, means old file will be removed. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. dT)step_upload_frequenceepoch_upload_frequencec svtt|||_||_||_||_||_tt dd|_ ||_ ||_ ||_d|_d|_t|_|jrrt|dS)NRANK_ID0)superr __init__src_dir target_dirrr keep_lastintosgetenvrank_id retry_timeretrylog cb_paramspror on_modelartsr) selfrrrrrrrr __class__X/home/ma-user/work/mindformers-baichuan/mindformers/tools/cloud_adapter/cloud_adapter.pyr3s zLocal2ObsMonitor.__init__cCsX|js dS|jdkrdS||_|jj|jdkrTt|jrT|j d| dS)z upload files at the end of step.Nr#Starting upload output file to obs!) r"r original_argsr cur_step_numrlistdirrrinfouploadr# run_contextr&r&r'step_endLs   zLocal2ObsMonitor.step_endcCsX|js dS|jdkrdS||_|jj|jdkrTt|jrT|j d| dS)z!upload files at the end of epoch.Nrr() r"rr)r cur_epoch_numrr+rrr,r-r.r&r&r' epoch_endWs   zLocal2ObsMonitor.epoch_endcCs&|jr|j||j|j|_dS)zUpload Files to OBS.N)r!joinsync2obsrr)r#r&r&r'r-bs zLocal2ObsMonitor.uploadcCst|}|D]}d|vrtj||d|j}tj||d|j}|jrltj|rltjj |ddt |||j |j |j q|jddkrt tj||tj|||j |j |j qdS)zAsynchronous transfer to OBS. checkpointrank_T) recursiverN)rr+pathr3rrmoxfileexistsremover rrr)r#rrZ sub_dir_listsub_dirZ scr_ckpt_dirZtarget_ckpt_dirr&r&r'r4is   zLocal2ObsMonitor.sync2obs) __name__ __module__ __qualname____doc__rrrr0r2r-rr4 __classcell__r&r&r$r'r s   r c@s,eZdZdZdddefddZd dd ZdS) r aFile saved from OBS server to local system of AI computing center platform. Args: rank_id (int): The obs's contents will be upload according to the actual rank_id value. Default: 0, means stored only one OBS file each node. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. rrrcCs t||_||_||_||_dS)N)rrrrr)r#rrrrr&r&r'rs zObs2Local.__init__NcCs(|dd}tj|d|}tj||}|dur|jddkrt|||j|j|j zt |Wqt y|Yq0n&|j dtj |rqtdqn|t|t|j|krt|||j|j|j zt |Wnt yYn0n,|j dtj |rq$tdq|S) zPull Obs Files to Local./r zmox_copy_{}.lockNr8rz>programming sleep for waiting download file from obs to local.)splitrr9r3formatrr rrrmknodIOErrorr,r<timesleepr check_typer)r#Zobs_urlZ local_urlZ special_idZobs_nameZmox_lockr&r&r' obs2locals4       zObs2Local.obs2local)N)r?r@rArBrrrMr&r&r&r'r |s r rrc Csd}t|dD]}t}ztjj||dddWnRttfy}z6|d|||||d|WYd}~qWYd}~n d}~00t} t |rtj |rd}|d || |qqt j |rd}|d || |qq|S) zFile interaction with Moxing.FrEr)Zsrc_urlZdst_urlthreadsZ is_processingz8%s, from %s download to %s failed, will retry(%d) again.z1sleep time %d for waiting download file from obs.NTz(Pull/Push file %s success, cost time: %f) rangerJr:r;Z copy_parallelFileNotFoundError RuntimeErrorr,rZ is_obs_urlr<rr9) rrrrrsuccessistarteendr&r&r'r s, "   r )rBrrJmindspore.train.callbackrrutilsrrrrmoxingr:__all__r r r r&r&r&r's  \6pengcheng.mind-test/mindformers/tools/cloud_adapter/__pycache__/cloud_monitor.cpython-39.pyc000066400000000000000000000044741471400134000327330ustar00rootroot00000000000000a ?cf @stdZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z efdd Zefd d Zd d ZdS)z Aicc monitor.N) get_logger) mox_adapter)DEBUG_INFO_PATHPROFILE_INFO_PATH PLOG_PATHLAST_TRANSFORM_LOCK_PATHget_output_root_pathget_remote_save_urlcsfdd}|S)zAicc monitor for main function.csfdd}|S)Nc s~ttdd}z\z|i|}Wn:ty^}z"t}||WYd}~n d}~00Wt|n t|0|S)NRANK_ID0)intosgetenv BaseException traceback format_excerror_last_transform)argskwargslocal_idresultexcr)logrun_funcX/home/ma-user/work/mindformers-baichuan/mindformers/tools/cloud_adapter/cloud_monitor.pywrappers z1cloud_monitor..decorator..wrapperr)rrr)rr decorators z cloud_monitor..decoratorr)rr!rr r cloud_monitors r"cCsBtjdr,tr,t}ttt|dn(|ddkrTtrTt}tt||d|ddkrtrt}tj|d}d|}tj t rtt tj|d||dtj t rtt tj|d||dtj t rtt tj|d |dt tn:tr>|d ttj tr|d tq>qd S) z.Transform file when progress ending or except.Z SPECIAL_ID)src_dir target_dirrrz ascend-logznode_{}Zplog debug_infoprofilezIWait for the first card to complete the file and send it back to OBS: %s.zLAll files have been sent back to the OBS: %s,and the process exits normally.N)renvirongetr rr pathjoinformatexistsrrrmknodr info)rrr$Ztask_dirnoderrrr.s8     rcCsttdd}t|dS)z'Upload log according to runtime for FM.r r N)rrrr)rrrr upload_logMsr1)__doc__rrmindformers.tools.loggerr cloud_adapterrutilsrrrr r r r"rr1rrrrs   pengcheng.mind-test/mindformers/tools/cloud_adapter/cloud_adapter.py000066400000000000000000000205121471400134000263640ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Cloud Adapter.""" import os import time from mindspore.train.callback import Callback from ..logger import logger from ..utils import check_obs_url, check_in_modelarts, \ Validator, sync_trans if check_in_modelarts(): import moxing as mox __all__ = ['Local2ObsMonitor', 'Obs2Local', 'mox_adapter'] class Local2ObsMonitor(Callback): """File saved from local system to OBS server. Args: src_dir (str): Local system path, which means path of AI computing center platform. target_dir (str): OBS path starting with S3 or obs is used to save files. rank_id (int): the device's contents will be saved according to the actual rank_id. Default: None, means only the contents of the first device of each node are saved. upload_frequence (int): How often files are saved in AI computing center platform. Default: -1. keep_last (bool): Check whether files in the OBS are consistent with AI computing center platform. Default: True, means old file will be removed. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. """ def __init__(self, src_dir, target_dir, step_upload_frequence: int = 100, epoch_upload_frequence: int = -1, keep_last=True, retry=3, retry_time=5, log=logger): super(Local2ObsMonitor, self).__init__() self.src_dir = src_dir self.target_dir = target_dir self.step_upload_frequence = step_upload_frequence self.epoch_upload_frequence = epoch_upload_frequence self.keep_last = keep_last self.rank_id = int(os.getenv('RANK_ID', '0')) self.retry_time = retry_time self.retry = retry self.log = log self.cb_params = None self.pro = None self.on_modelarts = check_in_modelarts() if self.on_modelarts: check_obs_url(target_dir) def step_end(self, run_context): """upload files at the end of step.""" if not self.on_modelarts: return if self.step_upload_frequence <= 0: return self.cb_params = run_context.original_args() if self.cb_params.cur_step_num % self.step_upload_frequence == 0 and os.listdir(self.src_dir): self.log.info("Starting upload output file to obs!") self.upload() def epoch_end(self, run_context): """upload files at the end of epoch.""" if not self.on_modelarts: return if self.epoch_upload_frequence <= 0: return self.cb_params = run_context.original_args() if self.cb_params.cur_epoch_num % self.epoch_upload_frequence == 0 and os.listdir(self.src_dir): self.log.info("Starting upload output file to obs!") self.upload() def upload(self): """Upload Files to OBS.""" if self.pro: # wait for last upload done self.pro.join() self.pro = self.sync2obs(self.src_dir, self.target_dir) @sync_trans def sync2obs(self, src_dir, target_dir): """Asynchronous transfer to OBS.""" sub_dir_list = os.listdir(src_dir) for sub_dir in sub_dir_list: if 'checkpoint' in sub_dir: # for checkpoint, all processes upload its own checkpoint scr_ckpt_dir = os.path.join(src_dir, sub_dir, f"rank_{self.rank_id}") target_ckpt_dir = os.path.join(target_dir, sub_dir, f"rank_{self.rank_id}") if self.keep_last and mox.file.exists(target_ckpt_dir): mox.file.remove(target_ckpt_dir, recursive=True) mox_adapter(scr_ckpt_dir, target_ckpt_dir, self.retry, self.retry_time, self.log) elif self.rank_id % 8 == 0: # for other outputs, one rank upload all mox_adapter(os.path.join(src_dir, sub_dir), os.path.join(target_dir, sub_dir), self.retry, self.retry_time, self.log) class Obs2Local: """File saved from OBS server to local system of AI computing center platform. Args: rank_id (int): The obs's contents will be upload according to the actual rank_id value. Default: 0, means stored only one OBS file each node. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. """ def __init__(self, rank_id=0, retry=3, retry_time=5, log=logger): self.rank_id = int(rank_id) self.retry_time = retry_time self.retry = retry self.log = log def obs2local(self, obs_url, local_url, special_id=None): """Pull Obs Files to Local.""" obs_name = obs_url.split("/")[-1] mox_lock = os.path.join(local_url, "mox_copy_{}.lock".format(obs_name)) local_url = os.path.join(local_url, obs_name) if special_id is None: if self.rank_id % 8 == 0: mox_adapter(obs_url, local_url, self.retry, self.retry_time, self.log) try: os.mknod(mox_lock) except IOError: pass else: self.log.info("programming sleep for waiting download file from obs to local.") while True: if os.path.exists(mox_lock): break time.sleep(1) else: Validator.check_type(special_id, int) if self.rank_id == special_id: mox_adapter(obs_url, local_url, self.retry, self.retry_time, self.log) try: os.mknod(mox_lock) except IOError: pass else: self.log.info("programming sleep for waiting download file from obs to local.") while True: if os.path.exists(mox_lock): break time.sleep(1) return local_url def mox_adapter(src_dir, target_dir, retry=3, retry_time=5, log=logger): """File interaction with Moxing.""" success = False for i in range(retry + 1): start = time.time() try: mox.file.copy_parallel(src_url=src_dir, dst_url=target_dir, threads=0, is_processing=False) except (FileNotFoundError, RuntimeError) as e: log.info("%s, from %s download to %s failed, will retry(%d) again.", e, src_dir, target_dir, i) # sleep due to restriction of obs log.info("sleep time %d for waiting download file from obs.", retry_time) continue end = time.time() if Validator.is_obs_url(target_dir): if mox.file.exists(target_dir): success = True log.info("Pull/Push file %s success, cost time: %f", target_dir, end - start) break else: if os.path.exists(target_dir): success = True log.info("Pull/Push file %s success, cost time: %f", target_dir, end - start) break return success pengcheng.mind-test/mindformers/tools/cloud_adapter/cloud_monitor.py000066400000000000000000000072621471400134000264420ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Aicc monitor.""" import os import traceback import time from mindformers.tools.logger import get_logger from .cloud_adapter import mox_adapter from ..utils import DEBUG_INFO_PATH, PROFILE_INFO_PATH, PLOG_PATH, LAST_TRANSFORM_LOCK_PATH,\ get_output_root_path, get_remote_save_url, RANK_FINISH_PATH def cloud_monitor(log=get_logger()): """Aicc monitor for main function.""" def decorator(run_func): def wrapper(*args, **kwargs): local_id = int(os.getenv('RANK_ID', '0')) try: result = run_func(*args, **kwargs) except BaseException as exc: # pylint: disable=W0703 error = traceback.format_exc() log.error(error) raise exc finally: _last_transform(local_id, log) return result return wrapper return decorator def _last_transform(local_id, log=get_logger()): """Transform file when progress ending or except.""" local_node_id = local_id % 8 finish_file = RANK_FINISH_PATH.replace("*", str(local_node_id)) os.mknod(finish_file) world_size = int(os.getenv("RANK_SIZE", '1')) if world_size > 8: world_size = 8 if os.environ.get("SPECIAL_ID") and get_remote_save_url(): target_dir = get_remote_save_url() mox_adapter(src_dir=get_output_root_path(), target_dir=get_remote_save_url(), log=log) else: if local_id % 8 == 0 and get_remote_save_url(): for i in range(world_size): finish_file = RANK_FINISH_PATH.replace("*", str(i)) while not os.path.exists(finish_file): time.sleep(1) target_dir = get_remote_save_url() mox_adapter(src_dir=get_output_root_path(), target_dir=target_dir, log=log) if local_id % 8 == 0 and get_remote_save_url(): target_dir = get_remote_save_url() task_dir = os.path.join(target_dir, 'ascend-log') node = 'node_{}'.format(local_id) if os.path.exists(PLOG_PATH): mox_adapter(src_dir=PLOG_PATH, target_dir=os.path.join(task_dir, 'plog', node), log=log) if os.path.exists(DEBUG_INFO_PATH): mox_adapter(src_dir=DEBUG_INFO_PATH, target_dir=os.path.join(target_dir, 'debug_info', node), log=log) if os.path.exists(PROFILE_INFO_PATH): mox_adapter(src_dir=PROFILE_INFO_PATH, target_dir=os.path.join(target_dir, 'profile'), log=log) os.mknod(LAST_TRANSFORM_LOCK_PATH) elif get_remote_save_url(): log.info("Wait for the first card to complete the file and send it back to OBS: %s.", get_remote_save_url()) while True: if os.path.exists(LAST_TRANSFORM_LOCK_PATH): log.info("All files have been sent back to the OBS: %s," "and the process exits normally.", get_remote_save_url()) break def upload_log(): """Upload log according to runtime for FM.""" local_id = int(os.getenv('RANK_ID', '0')) _last_transform(local_id) pengcheng.mind-test/mindformers/tools/dataset_preprocess/000077500000000000000000000000001471400134000242705ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/dataset_preprocess/bloom/000077500000000000000000000000001471400134000254005ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/dataset_preprocess/bloom/make_mindrecord.py000066400000000000000000000112321471400134000310740ustar00rootroot00000000000000# Copyright 2020 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """generate mindrecord script""" import os import argparse import collections import json import numpy as np from tqdm import tqdm from mindspore.mindrecord import FileWriter from mindformers import AutoTokenizer class AlpacaDatasetMaker: """ AlpacaDatasetMaker """ def __init__(self, input_dataset_file, output_dataset_file, seq_length, for_finetune): self.input_dataset_file = input_dataset_file self.output_dataset_file = output_dataset_file self.seq_length = seq_length self.for_finetune = for_finetune self.tokenizer = AutoTokenizer.from_pretrained("bloom_560m") self.eos_token_id = self.tokenizer.eos_token_id self.pad_token_id = self.tokenizer.pad_token_id self.prompt_with_input = ( "Below is an instruction that describes a task, paired with an input that provides further context. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n" ) self.prompt_without_input = ( "Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Response:\n" ) def make(self, num_of_prompts): """make mindrecord""" writer = FileWriter(self.output_dataset_file, 1) writer.add_schema( {"input_ids": {"type": "int64", "shape": [-1]}}, 'lm-schema') with open(self.input_dataset_file) as ds: dataset = json.load(ds) num_of_prompts = min(num_of_prompts, len(dataset)) if num_of_prompts > 0 else len(dataset) for i in tqdm(range(num_of_prompts)): prompt = dataset[i] prompt_ids = self.make_prompt_ids(prompt) features = collections.OrderedDict() features["input_ids"] = np.asarray(prompt_ids) writer.write_raw_data([features]) writer.commit() def make_prompt_ids(self, prompt): """make prompt dict into ids""" if 'input' in prompt: prompt_text = self.prompt_with_input.format_map(prompt) else: prompt_text = self.prompt_without_input.format_map(prompt) # question prompt_ids = self.tokenizer(prompt_text)["input_ids"] if self.for_finetune: prompt_ids = list(-np.array(prompt_ids, dtype=np.int32)) # answer output_text = prompt["output"] if output_text: output_ids = self.tokenizer(output_text)["input_ids"] else: output_ids = [] # Q+A ids = prompt_ids + output_ids if len(ids) > self.seq_length: ids = ids[:self.seq_length] self._add_eos_token_id(ids) self._add_pad_token_id(ids) return ids def _add_eos_token_id(self, ids): if len(ids) < self.seq_length: ids.append(self.eos_token_id) def _add_pad_token_id(self, ids): if len(ids) < self.seq_length: # pad eos instead of pad ids += [self.eos_token_id] * (self.seq_length - len(ids)) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--input_dataset_file", type=str, default="/home/work/czh/data/alpaca_data.json") parser.add_argument("--output_path", type=str, default="/home/work/czh/data/alpaca_2049/") parser.add_argument("--seq_length", type=int, default=2049) parser.add_argument("--N", type=int, default=-1) parser.add_argument("--for_finetune", type=int, default=1) args = parser.parse_args() if args.output_path and not os.path.exists(args.output_path): os.makedirs(args.output_path, exist_ok=True) args.output_dataset_file = os.path.join( args.output_path, "dataset.mindrecord") maker = AlpacaDatasetMaker( args.input_dataset_file, args.output_dataset_file, args.seq_length, args.for_finetune) maker.make(args.N) pengcheng.mind-test/mindformers/tools/dataset_preprocess/glm/000077500000000000000000000000001471400134000250475ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/dataset_preprocess/glm/adgen_dataset.py000066400000000000000000000241541471400134000302120ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Process of ADGEN dataset.""" import argparse import json import numpy as np from tqdm import tqdm from mindspore.mindrecord import FileWriter from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer from mindformers.tools.logger import logger prompt_column = "content" response_column = "summary" history_column = None prefix = "" def get_masks(input_ids, bos_token_id=130004): """Get attention mask.""" seq_length = input_ids.shape[0] mask = bos_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(input_ids, mask) # 要求input_ids中有且仅有一个bos_token_id context_lengths = np.argwhere(mask)[:, -1] attention_mask = np.tril(np.ones((seq_length, seq_length), dtype=np.float32)) for _, context_length in enumerate(context_lengths): attention_mask[:, :context_length] = 1 attention_mask = np.logical_not(attention_mask.astype(np.bool_)) attention_mask = attention_mask.astype(np.float32) attention_mask = np.expand_dims(attention_mask, 0) return attention_mask def get_position_ids(input_ids, mask_positions, use_gmasks=None, bos_token_id=130004, position_encoding_2d=True): """Get position ids.""" seq_length = input_ids.shape[0] if use_gmasks is None: use_gmasks = [False] mask = bos_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(input_ids, mask) # 要求input_ids中有且仅有一个bos_token_id context_lengths = np.argwhere(mask)[:, -1] if position_encoding_2d: position_ids = np.arange(seq_length, dtype=np.int64) for i, context_length in enumerate(context_lengths): position_ids[context_length:] = mask_positions[i] block_position_ids = [np.concatenate(( np.zeros(context_length, dtype=np.int64), np.arange(seq_length - context_length, dtype=np.int64) + 1 )) for context_length in context_lengths] block_position_ids = np.stack(block_position_ids, axis=0).squeeze() position_ids = np.stack((position_ids, block_position_ids), axis=0) else: position_ids = np.arange(seq_length, dtype=np.int64) for i, context_length in enumerate(context_lengths): if not use_gmasks[i]: position_ids[context_length:] = mask_positions[i] return position_ids def create_position_ids(input_ids, gmask_token_id=130001): """Get position ids.""" seq_length = input_ids.shape[0] seqs = input_ids # 要求input_ids中, 每行有且仅有一个gMASK use_gmasks = gmask_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(seqs, use_gmasks) mask_positions = np.argwhere(mask)[:, -1] position_ids = get_position_ids(input_ids, mask_positions=mask_positions, use_gmasks=use_gmasks) return position_ids def preprocess_function(input_file, vocab_file, output_file, num_splits, max_source_length, max_target_length, mode='train', ignore_pad_token_for_loss=True): """ Process data file to mindrecord data file. Args: input_file (str): Original data set file. vocab_file (str): Vocab file path. output_file (str): Output MindRecord file. num_splits (str): The MindRecord file will be split into the number of partition. max_source_length (int): The max sequence length. max_target_length (int): The max label sequence length. mode (str): Dataset use to train or test. Default: 'train'. ignore_pad_token_for_loss (bool): Whether ignore pad token for loss, default: True. Returns: MindDataset object """ if input_file is None: raise ValueError("Please enter a valid dataset path") if vocab_file is None: raise ValueError("Please enter a valid vocab file path") max_seq_length = max_source_length + max_target_length tokenizer = ChatGLMTokenizer(vocab_file=vocab_file) writer = FileWriter(output_file, num_splits, overwrite=True) if mode == 'train': data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]}, "position_ids": {"type": "int32", "shape": [2, max_seq_length]}, "attention_mask": {"type": "int32", "shape": [1, max_seq_length, max_seq_length]} } else: data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]}} writer.add_schema(data_schema, "lm-schema") total_written = 0 file = open(input_file, 'r') lines = file.readlines() examples = {} content_list = [] summary_list = [] for line in lines: content_list.append(json.loads(line)[prompt_column]) summary_list.append(json.loads(line)[response_column]) examples[prompt_column] = content_list examples[response_column] = summary_list model_inputs = {} for i in tqdm(range(len(examples[prompt_column])), ascii=True, ncols=120): if examples[prompt_column][i] and examples[response_column][i]: query, answer = examples[prompt_column][i], examples[response_column][i] if history_column is None: prompt = query else: prompt = "" history = examples[history_column][i] for turn_idx, (old_query, response) in enumerate(history): prompt += "[Round {}]\n问：{}\n答：{}\n".format(turn_idx, old_query, response) prompt += "[Round {}]\n问：{}\n答：".format(len(history), query) prompt = prefix + prompt if mode == 'train': prompt_ids = tokenizer.encode(text=prompt, add_special_tokens=False) answer_ids = tokenizer.encode(text=answer, add_special_tokens=False) if len(prompt_ids) > max_source_length - 1: prompt_ids = prompt_ids[: max_source_length - 1] if len(answer_ids) > max_target_length - 2: answer_ids = answer_ids[: max_target_length - 2] input_ids = tokenizer.build_inputs_with_special_tokens(prompt_ids, answer_ids) context_length = input_ids.index(tokenizer.bos_token_id) mask_position = context_length - 1 label = [-100] * context_length + input_ids[mask_position + 2:] # +1 for logits shift pad_len = max_seq_length - len(input_ids) input_ids = input_ids + [tokenizer.pad_token_id] * pad_len label = label + [tokenizer.pad_token_id] * (pad_len + 1) # +1 for logits shift if ignore_pad_token_for_loss: label = [(l if l != tokenizer.pad_token_id else -100) for l in label] position_ids = create_position_ids(np.array(input_ids)) attention_mask = get_masks(np.array(input_ids)) model_inputs["position_ids"] = np.array(position_ids) model_inputs["attention_mask"] = np.array(attention_mask) else: if len(prompt) > max_source_length - 2: prompt = prompt[: max_source_length - 2] if len(answer) > max_target_length - 2: answer = answer[: max_target_length - 2] input_ids = tokenizer.encode(text=prompt, add_special_tokens=True) label = tokenizer.encode(text=answer, add_special_tokens=True) pad_len = max_source_length - len(input_ids) input_ids = input_ids + [tokenizer.pad_token_id] * pad_len model_inputs["input_ids"] = np.array(input_ids) model_inputs["labels"] = np.array(label) writer.write_raw_data([model_inputs]) total_written += 1 writer.commit() logger.info(f"Wrote {total_written} total instances") if __name__ == "__main__": parser = argparse.ArgumentParser(description="OPT convert script") parser.add_argument('--input_file', type=str, required=True, default='./AdvertiseGen/train.json', help="The number of layers of the model to be converted.") parser.add_argument("--vocab_file", type=str, default='ice_text.model', required=True, help="The torch checkpoint path.") parser.add_argument("--output_file", type=str, required=True, default="adgen_train.mindrecord", help="The output mindspore checkpoint path.") parser.add_argument("--max_source_length", type=int, default=64, help="The max input sequence length") parser.add_argument("--max_target_length", type=int, default=64, help="The max label sequence length") parser.add_argument("--mode", type=str, default="train", help="Process train or eval dataset.") opt = parser.parse_args() preprocess_function( input_file=opt.input_file, vocab_file=opt.vocab_file, output_file=opt.output_file, num_splits=1, max_source_length=opt.max_source_length, max_target_length=opt.max_target_length, mode=opt.mode, ignore_pad_token_for_loss=True ) pengcheng.mind-test/mindformers/tools/dataset_preprocess/gpt2/000077500000000000000000000000001471400134000251445ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/dataset_preprocess/gpt2/txtcls_dataset_to_mindrecord.py000066400000000000000000000130411471400134000334530ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """convert dataset to mindrecord""" from __future__ import absolute_import from __future__ import division from __future__ import print_function import argparse import collections import numpy as np from mindspore.mindrecord import FileWriter from mindformers.tools.logger import logger from mindformers import AutoTokenizer, MindFormerBook def create_instance(ds_name, tokenizer, text, max_length): """A single sample instance for LM task.""" if ds_name == 'cola': _, label, _, sentence = text.strip().split("\t") elif ds_name == 'ag_news': label, title, description = text.strip().split("\",\"") sentence = title + ". " + description label = label.strip("\"") sentence = sentence.strip("\"") label = int(label) - 1 elif ds_name == 'imdb': sentence = text.strip()[:-9] label = text.strip()[-8:] sentence = sentence.strip("\"") label = 1 if label == 'positive' else 0 else: sentence, label = text.strip().split("\t") output = tokenizer(sentence, padding='max_length', max_length=max_length) output['labels'] = label return output def write_instance_to_file(writer, instance): """write the instance to file""" input_ids = instance["input_ids"] attention_mask = instance["attention_mask"] label = instance["labels"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["attention_mask"] = np.asarray(attention_mask) features["labels"] = np.asarray(label) writer.write_raw_data([features]) return features def main(): dataset_support_list = ['cola', 'sst_2', 'ag_news', 'imdb'] tokenizer_support_list = list(MindFormerBook.get_tokenizer_url_support_list().keys()) parser = argparse.ArgumentParser() parser.add_argument("--dataset_name", type=str, required=True, help='Dataset name. Now only supports [cola, sst_2, ag_news, imdb]. ', choices=dataset_support_list) parser.add_argument("--tokenizer_type", type=str, default="gpt2", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ", choices=tokenizer_support_list) parser.add_argument("--data_columns", type=list, default=["input_ids", "labels", "attention_mask"], help="The data columns which should be saved in mindrecord. This can refer used yaml file. ") parser.add_argument("--input_file", type=str, required=True, help='Input raw text file. ') parser.add_argument("--output_file", type=str, required=True, help='Output MindRecord file. ') parser.add_argument("--max_length", type=int, default=1024, help='Maximum sequence length. ') parser.add_argument("--header", type=bool, default=True, help='Has header or not. ') args = parser.parse_args() tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) input_file = args.input_file logger.info("***** Reading from input files *****") logger.info("Input File: %s", input_file) output_file = args.output_file logger.info("***** Writing to output files *****") logger.info("Output File: %s", output_file) writer = FileWriter(output_file) data_schema = {"input_ids": {"type": "int64", "shape": [-1]}, "attention_mask": {"type": "int64", "shape": [-1]}, "labels": {"type": "int64", "shape": [-1]} } data_columns = args.data_columns need_del_keys = set(data_columns) - set(data_schema.keys()) for need_del_key in need_del_keys: del data_schema[need_del_key] writer.add_schema(data_schema) total_written = 0 total_read = 0 logger.info("***** Reading from %s *****", input_file) with open(input_file, "r") as f: if args.header: f.readline() while True: line = f.readline() if not line: break total_read += 1 if total_read % 500 == 0: logger.info("%d ...", total_read) output = create_instance(args.dataset_name, tokenizer, line, args.max_length) features = write_instance_to_file(writer, instance=output) total_written += 1 if total_written <= 20: logger.info("***** Example *****") logger.info("input tokens: %s", tokenizer.decode(output["input_ids"])) logger.info("label: %s", output["labels"]) for feature_name in features.keys(): feature = features[feature_name] logger.info("%s: %s", feature_name, feature) writer.commit() logger.info("Wrote %d total instances", total_written) if __name__ == "__main__": main() pengcheng.mind-test/mindformers/tools/dataset_preprocess/gpt2/wikitext2_data_process.py000066400000000000000000000204101471400134000321740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Data process for wikitext-2 Dataset The generated dataset has output columns: ["input_ids", "attention_mask", "labels"], and the data type of three columns is int64. Columns： input_ids: the tokenized inputs, Tensor of shape :math:`(batch, seq_length)`. attention_mask: the mask indicating whether each position is a valid input and is not the added prompt, Tensor of shape :math:`(batch, seq_length)`. labels: same as input_ids, Tensor of shape :math:`(batch, seq_length)`. About wikitext-2 Dataset: The wikitext-2 language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies. You can unzip the dataset files into the following structure: .. code-block:: . └── wikitext-2 ├── wiki.test.tokens ├── wiki.train.tokens └── wiki.valid.tokens """ from __future__ import absolute_import from __future__ import division from __future__ import print_function import argparse import collections import logging import re import numpy as np from mindspore.mindrecord import FileWriter from mindformers.auto_class import AutoTokenizer def wikitext_clean(string): """ string clean """ # contractions string = string.replace("s '", "s'") string = re.sub(r"/' [0-9]/", r"/'[0-9]/", string) # number separators string = string.replace(" @-@ ", "-") string = string.replace(" @,@ ", ",") string = string.replace(" @.@ ", ".") # punctuation string = string.replace(" : ", ": ") string = string.replace(" ; ", "; ") string = string.replace(" . ", ". ") string = string.replace(" .", ".") string = string.replace(" ! ", "! ") string = string.replace(" ? ", "? ") string = string.replace(" , ", ", ") # double brackets string = re.sub(r"\(\s*([^\)]*?)\s*\)", r"(\1)", string) string = re.sub(r"\[\s*([^\]]*?)\s*\]", r"[\1]", string) string = re.sub(r"{\s*([^}]*?)\s*}", r"{\1}", string) string = re.sub(r"\"\s*([^\"]*?)\s*\"", r'"\1"', string) string = re.sub(r"'\s*([^']*?)\s*'", r"'\1'", string) # miscellaneous string = string.replace("= = = =", "====") string = string.replace("= = =", "===") string = string.replace("= =", "==") string = string.replace(" " + chr(176) + " ", chr(176)) string = string.replace(" \n", "\n") string = string.replace("\n ", "\n") string = string.replace(" N ", " 1 ") string = string.replace(" 's", "'s") return string def preprocess_data(input_file): """ preprocess data """ dataset_valid = [] passage = [] count = 0 with open(input_file, 'r', encoding='utf-8') as f: for line in f: line = line.strip() if line: if line.startswith('=') and line.endswith('=') and passage: dataset_valid.append(passage) count += 1 passage = [] elif line.startswith('=') and line.endswith('='): continue else: passage.append(line) print('read {} file finished!\n total count = {}'.format(input_file, count)) res = [] for line in dataset_valid: text = "" for sentence in line: sentence = wikitext_clean(sentence) text = text + " " + sentence text = text.strip() res.append(text) return res def create_instance(tokenizer, sentence, ids, max_length=None): """A single sample instance for LM task.""" pair_ids = None if len(sentence) == 2: pair_ids = tokenizer.encode(sentence[1]) output = tokenizer.prepare_for_model(ids=ids, pair_ids=pair_ids, add_special_tokens=False, max_length=max_length, padding='max_length', truncate_direction="LEFT", return_overflowing_tokens=False, return_attention_mask=True) return output def write_instance_to_file(writer, instance): """write the instance to file""" input_ids = instance["input_ids"] attention_mask = instance["attention_mask"] labels = instance["input_ids"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["attention_mask"] = np.asarray(attention_mask) features["labels"] = np.asarray(labels) writer.write_raw_data([features]) return features def main(): parser = argparse.ArgumentParser() parser.add_argument("--input_file", type=str, default="../wikitext-2/wiki.valid.tokens", help='Input raw text file. ') parser.add_argument("--output_file", type=str, default="../wikitext2_processed/wikitext-2.mindrecord", help='Output MindRecord file. ') parser.add_argument("--num_splits", type=int, default=1, help="The MindRecord file will be split into the number of partition. ") parser.add_argument("--max_length", type=int, default=1024, help="Maximum sequence length. ") parser.add_argument("--tokenizer_type", type=str, default="gpt2", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ") parser.add_argument("--data_columns", type=list, default=["input_ids", "attention_mask"], help="The data columns which should be saved in mindrecord. This can refer used yaml file. ") args = parser.parse_args() tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) tokenizer.pad_token = tokenizer.eos_token tokenizer.add_tokens([]) input_file = args.input_file logging.info("***** Reading from input files *****") logging.info("Input File: %s", input_file) output_file = args.output_file logging.info("***** Writing to output files *****") logging.info("Output File: %s", output_file) writer = FileWriter(output_file, args.num_splits) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "attention_mask": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]} } data_columns = args.data_columns need_del_keys = set(data_columns) - set(data_schema.keys()) for need_del_key in need_del_keys: del data_schema[need_del_key] writer.add_schema(data_schema, "lm-schema") dataset_valid = preprocess_data(args.input_file) total_written = 0 logging.info("***** Reading from %s *****", input_file) text_total = "\n".join(dataset_valid) # the logic of \n is copied from modelzoo sentence = text_total.strip().split("\t") block_size = args.max_length total_ids = tokenizer.encode(sentence[0]) total_length = len(total_ids) total_length = (total_length // block_size) * block_size print("total_length", total_length) for i in range(total_length // block_size): ids = total_ids[block_size*i:block_size*(i+1)] output = create_instance(tokenizer, sentence, ids, args.max_length) write_instance_to_file(writer, instance=output) total_written += 1 writer.commit() logging.info("Wrote %d total instances", total_written) if __name__ == "__main__": logging.basicConfig(level=logging.INFO) main() pengcheng.mind-test/mindformers/tools/dataset_preprocess/llama/000077500000000000000000000000001471400134000253565ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/dataset_preprocess/llama/alpaca_converter.py000066400000000000000000000037521471400134000312470ustar00rootroot00000000000000""" fastchat stanford alpaca data convert tools. """ import argparse import json import pathlib # Prompt from stanford alpaca's training script PROMPT_DICT = { "prompt_input": ( "Below is an instruction that describes a task, paired with an input that provides further context. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:" ), "prompt_no_input": ( "Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Response:" ), } def main(args_param): data_path = pathlib.Path(args_param.data_path) with data_path.open() as f: data = json.load(f) prompt_input, prompt_no_input = ( PROMPT_DICT["prompt_input"], PROMPT_DICT["prompt_no_input"], ) sources = [ prompt_input.format_map(example) if example.get("input", "") != "" else prompt_no_input.format_map(example) for example in data ] targets = [example["output"] for example in data] new_data = [] cnt = 1 for s, t in zip(sources, targets): new_data.append( { "id": str(cnt), "conversations": [ { "from": "human", "value": s, }, { "from": "gpt", "value": t, }, ], } ) cnt += 1 json.dump(new_data, open(args_param.output_path, "w"), indent=2) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--data_path", type=str, default="alpaca-data.json") parser.add_argument( "--output_path", type=str, default="alpaca-data-conversation.json" ) args = parser.parse_args() main(args) pengcheng.mind-test/mindformers/tools/dataset_preprocess/llama/llama_preprocess.py000066400000000000000000000221161471400134000312650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ transform wikitext-2, wikitext-103, lambada, openwebtext dataset to mindrecord. """ import argparse import json import os import re import numpy as np # fschat >= 0.2.13 from fastchat.conversation import get_conv_template from mindspore.mindrecord import FileWriter from mindformers.models.llama.llama_tokenizer import LlamaTokenizer IGNORE_TOKEN_ID = -100 def chunks(lst, n): """ yield n sized chunks from list""" for i in range(0, len(lst), n): yield lst[i:i+n] def package_file(it, n): """ package multiple files""" stop = False while not stop: batch = [] for _ in range(n): try: batch.append(next(it)) except StopIteration: stop = True if not batch: break yield batch def clean_wikitext(string): """ cleaning wikitext dataset""" # contractions string = string.replace("s '", "s'") string = re.sub(r"/' [0-9]/", r"/'[0-9]/", string) # number separators string = string.replace(" @-@ ", "-") string = string.replace(" @,@ ", ",") string = string.replace(" @.@ ", ".") # punctuation string = string.replace(" : ", ": ") string = string.replace(" ; ", "; ") string = string.replace(" . ", ". ") string = string.replace(" ! ", "! ") string = string.replace(" ? ", "? ") string = string.replace(" , ", ", ") # double brackets string = re.sub(r"\(\s*([^\)]*?)\s*\)", r"(\1)", string) string = re.sub(r"\[\s*([^\]]*?)\s*\]", r"[\1]", string) string = re.sub(r"{\s*([^}]*?)\s*}", r"{\1}", string) string = re.sub(r"\"\s*([^\"]*?)\s*\"", r'"\1"', string) string = re.sub(r"'\s*([^']*?)\s*'", r"'\1'", string) # miscellaneous string = string.replace("= = = =", "====") string = string.replace("= = =", "===") string = string.replace("= =", "==") string = string.replace(" "+chr(176)+" ", chr(176)) string = string.replace(" \n", "\n") string = string.replace("\n ", "\n") string = string.replace(" N ", " 1 ") string = string.replace(" 's", "'s") return string def preprocess(sources, tokenizer, seq_length): """conversation preprocess.""" conv = get_conv_template("vicuna_v1.1").copy() roles = {"human": conv.roles[0], "gpt": conv.roles[1]} # Apply prompt templates conversations = [] for i, source in enumerate(sources): if roles[source[0]["from"]] != conv.roles[0]: # Skip the first one if it is not from human source = source[1:] conv.messages = [] for j, sentence in enumerate(source): role = roles[sentence["from"]] assert role == conv.roles[j % 2], f"{i}" conv.append_message(role, sentence["value"]) conversations.append(conv.get_prompt()) sep = conv.sep + conv.roles[1] + ": " # Tokenize conversations input_ids = [] targets = [] # attention_mask = [] for conversation in conversations: rounds = conversation.split(conv.sep2) ids = [tokenizer.bos_token_id] mask = [1] for i, rou in enumerate(rounds): if rou == "": break conv_out = tokenizer(rou) ids.extend(conv_out['input_ids'][1:]) mask.extend(conv_out['attention_mask'][1:]) d = {'input_ids': ids, 'attention_mask': mask} # pylint: disable=W0212 d = tokenizer._pad(d, max_length=seq_length, padding_strategy='max_length') input_ids.append(d['input_ids']) # attention_mask.append(d['attention_mask']) target = np.array(d['input_ids']) total_len = int(np.not_equal(target, tokenizer.pad_token_id).sum()) cur_len = 1 target[:cur_len] = IGNORE_TOKEN_ID for i, rou in enumerate(rounds): if rou == "": break parts = rou.split(sep) if len(parts) != 2: break parts[0] += sep round_len = len(tokenizer(rou)['input_ids']) - 1 instruction_len = len(tokenizer(parts[0])['input_ids']) - 3 target[cur_len : cur_len + instruction_len] = IGNORE_TOKEN_ID cur_len += round_len target[cur_len:] = IGNORE_TOKEN_ID if cur_len < seq_length: if cur_len != total_len: target[:] = IGNORE_TOKEN_ID targets.append(target.tolist()) input_ids = np.array(input_ids) targets = np.array(targets) return dict( input_ids=input_ids, labels=targets, ) class SupervisedDataset: """Dataset for supervised fine-tuning.""" def __init__(self, raw_data, tokenizer, seq_length): super(SupervisedDataset, self).__init__() sources = [example["conversations"] for example in raw_data] data_dict = preprocess(sources, tokenizer, seq_length) self.input_ids = data_dict["input_ids"] self.labels = data_dict["labels"] def __len__(self): return len(self.input_ids) def __getitem__(self, i): return dict( input_ids=self.input_ids[i], labels=self.labels[i] ) def tokenize_wiki(tokenizer, file_path, seq_length, repeat): """tokenize wikitext-2/wikitext-103 dataset""" content = [] with open(file_path, 'r', encoding='utf-8') as f: for para in clean_wikitext(f.read()).split("\n\n"): if para and para.strip().startswith('=') is False: content += tokenizer(para)['input_ids'] content_out = [] for _ in range(repeat): content_out.extend(content) content = content_out for chunk in chunks(content, seq_length): sample = {} if len(chunk) == seq_length: sample['input_ids'] = np.array(chunk, dtype=np.int32) yield sample def tokenize_qa(tokenizer, file_path, seq_length): raw_data = json.load(open(file_path, "r")) dataset_cls = SupervisedDataset(raw_data, tokenizer, seq_length) for i in range(len(dataset_cls)): yield dataset_cls[i] if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('--dataset_type', type=str, default='wiki') parser.add_argument('--input_glob', type=str, default='/mnt/luolan/wikitext-2/wiki.train.tokens') parser.add_argument('--output_file', type=str, default='./dataset/wiki2048/wiki2048') parser.add_argument('--tokenizer', type=str, default='llama', choices=['llama']) parser.add_argument('--model_file', type=str, default='/mnt/luolan/llama/tokenizer.model') parser.add_argument('--file_partition', type=int, default=1) parser.add_argument('--repeat', type=int, default=1) parser.add_argument('--seq_length', type=int, default=2048) args = parser.parse_args() out_dir, out_file = os.path.split(os.path.abspath(args.output_file)) if not os.path.exists(out_dir): os.mkdir(out_dir) if args.dataset_type == 'wiki': schema = {'input_ids': {"type": "int32", "shape": [-1]},} elif args.dataset_type == 'qa': schema = {'input_ids': {"type": "int32", "shape": [-1]}, 'labels': {"type": "int32", "shape": [-1]}} writer = FileWriter(file_name=args.output_file, shard_num=args.file_partition) writer.add_schema(schema, args.dataset_type) writer.open_and_set_header() # Start to load tokenizer if not os.path.exists(args.model_file): raise FileNotFoundError(f"file {args.model_file} do not exists.") transforms_count = 0 word_tokenizer = LlamaTokenizer(vocab_file=args.model_file) if hasattr(word_tokenizer, 'add_bos_token'): word_tokenizer.add_bos_token = True if hasattr(word_tokenizer, 'add_eos_token'): word_tokenizer.add_eos_token = True if args.dataset_type == 'wiki': for x in tokenize_wiki(word_tokenizer, args.input_glob, args.seq_length + 1, args.repeat): transforms_count += 1 writer.write_raw_data([x]) print("Transformed {} records.".format(transforms_count)) elif args.dataset_type == 'qa': for x in tokenize_qa(word_tokenizer, args.input_glob, args.seq_length + 1): transforms_count += 1 writer.write_raw_data([x]) print("Transformed {} records.".format(transforms_count)) else: raise ValueError( "Not support dataset type: {}".format(args.dataset_type)) writer.commit() out_file = args.output_file if args.file_partition > 1: out_file += '0' print("Transform finished, output files refer: {}".format(out_file)) pengcheng.mind-test/mindformers/tools/dataset_preprocess/llama/squad_data_process.py000066400000000000000000000136031471400134000315770ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Data process for SQuAD Dataset""" import argparse import json import collections import copy import logging import pathlib import numpy as np from mindspore.mindrecord import FileWriter from mindformers.auto_class import AutoTokenizer IGNORE_TOKEN_ID = -100 def write_instance_to_file(writer, instance): """write the instance to file""" input_ids = instance["input_ids"] label = instance["labels"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["labels"] = np.asarray(label) writer.write_raw_data([features]) return features def main(): parser = argparse.ArgumentParser() parser.add_argument("--input_file", type=str, default="./squad/train-v1.1.json", required=False, help='Input raw json file. ') parser.add_argument("--output_file", type=str, default="./squad/recode_train.mindrecord", required=False, help='Output MindRecord file. ') parser.add_argument("--mode", type=str, default="train", help='Set Data for train or eval.') parser.add_argument("--max_length", type=int, default=2048, help='Maximum sequence length. ') parser.add_argument("--tokenizer_type", type=str, default="llama_7b", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ") args = parser.parse_args() logging.info("***** Reading from input files *****") logging.info("Input File: %s", args.input_file) tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) input_file = pathlib.Path(args.input_file) with input_file.open() as f: file = json.load(f) sources = [] targets = [] for data in file["data"]: for paragraph in data["paragraphs"]: passage = paragraph["context"] query = paragraph["qas"][0]["question"] answer = paragraph["qas"][0]["answers"][0]["text"] input_str = f"Read the passage and answer the question below.\n\n### Instruction:\n{passage}\n\n### Input:\n{query}\n\n### Response:" sources.append(input_str) targets.append(answer) logging.info("***** Writing to output files *****") logging.info("Output File: %s", args.output_file) writer = FileWriter(args.output_file, 1) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]} } writer.add_schema(data_schema, "lm-schema") total_written = 0 # for eval if args.mode == "eval": if hasattr(tokenizer, 'add_bos_token'): tokenizer.add_bos_token = True if hasattr(tokenizer, 'add_eos_token'): tokenizer.add_eos_token = False for prompt, answer in zip(sources, targets): total_written += 1 input_ids = tokenizer.encode(prompt, add_special_tokens=True) if len(input_ids) >= args.max_length: input_ids = input_ids[:args.max_length] else: input_ids = np.pad(input_ids, (0, args.max_length - len(input_ids)), 'constant', constant_values=(tokenizer.pad_token_id, tokenizer.pad_token_id)) input_ids = np.array(input_ids).reshape(1, -1) label_id = tokenizer.encode(answer, add_special_tokens=False) label_id = np.pad(label_id, (0, args.max_length - len(label_id)), 'constant', constant_values=(tokenizer.pad_token_id, tokenizer.pad_token_id)) label_id = np.array(label_id).reshape(1, -1) instance = {"input_ids": input_ids, "labels": label_id} write_instance_to_file(writer, instance=instance) # for train/finetune elif args.mode == "train": if hasattr(tokenizer, 'add_bos_token'): tokenizer.add_bos_token = True if hasattr(tokenizer, 'add_eos_token'): tokenizer.add_eos_token = True for prompt, answer in zip(sources, targets): total_written += 1 concated_qa = prompt + answer input_ids = tokenizer.encode(concated_qa, add_special_tokens=True) input_ids = np.array(input_ids) prompt_ids = tokenizer.encode(prompt, add_special_tokens=False) prompt_ids = np.array(prompt_ids) prompt_length = len(prompt_ids) concat_length = len(input_ids) pad_length = args.max_length + 1 - concat_length input_ids_new = np.pad(input_ids, (0, pad_length), 'constant', constant_values=(tokenizer.pad_token_id, tokenizer.pad_token_id)) label_id_new = copy.deepcopy(input_ids_new) label_id_new[:prompt_length] = IGNORE_TOKEN_ID label_id_new[-pad_length:] = IGNORE_TOKEN_ID instance = {"input_ids": input_ids_new, "labels": label_id_new} write_instance_to_file(writer, instance=instance) else: logging.error("No mode named %s, please set mode as train/eval.", args.mode) writer.commit() logging.info("Wrote %d total instances", total_written) if __name__ == "__main__": logging.basicConfig(level=logging.INFO) main() pengcheng.mind-test/mindformers/tools/dataset_preprocess/pangualpha/000077500000000000000000000000001471400134000264105ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/dataset_preprocess/pangualpha/cmrc2018_data_process.py000066400000000000000000000252651471400134000327620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Data process for CMRC2018 Dataset""" import re import json import argparse import collections import logging import numpy as np from mindspore.mindrecord import FileWriter from mindformers.auto_class import AutoTokenizer np.random.seed(666) def gen_prompt(prompts_, n_shot, len_ori, max_length=1024): if n_shot == 0: res = "" else: ids = np.random.choice(len(prompts_), n_shot).tolist() res = "" for i in ids: if len(res) + len(prompts_[i]) < max_length - len_ori - 10: res = res + prompts_[i] return res def cut_sent(para): para = re.sub(r'([。！？\?])([^”’])', r"\1\n\2", para) para = re.sub(r'([。！？\?][”’])([^，。！？\?])', r'\1\n\2', para) para = para.rstrip() # del extra \n return para.split("\n") class Cmrc2018Dataset: """dataset process""" def __init__(self, cmrc2018_train_json, cmrc2018_dev_json): with open(cmrc2018_train_json, "r", encoding="utf-8") as f: self.train_data = json.load(f)["data"] with open(cmrc2018_dev_json, "r", encoding="utf-8") as f: self.dev_data = json.load(f)["data"] def gen_prompt_from_train(self, seg=False): """gen prompt from train dataset""" data_list = self.train_data index = 0 prompts_ = [] if seg: for data in data_list: context = data["paragraphs"][0]["context"] context_splits = cut_sent(context) qas = data["paragraphs"][0]["qas"] for qa in qas: index += 1 q = qa["question"] a = qa["answers"][0]["text"] for sent in context_splits: if a in sent: prompt = f"阅读文章：{sent}\n问：{q}\n答：{a}\n" prompts_.append(prompt) prompt_max_length = 80 prompts_ = [x for x in prompts_ if len(x) < prompt_max_length] else: for data in data_list: context = data["paragraphs"][0]["context"] qas = data["paragraphs"][0]["qas"] for qa in qas: index += 1 q = qa["question"] a = qa["answers"][0]["text"] prompt = f"阅读文章：{context}\n问：{q}\n答：{a}\n" prompts_.append(prompt) prompt_max_length = 400 prompts_ = [x for x in prompts_ if len(x) < prompt_max_length] return prompts_ def get_data_allshot(self, n_content=50, n_q=1, max_length=1024): """gen data for all shots""" prompts_seg = self.gen_prompt_from_train(seg=True) prompts_full = self.gen_prompt_from_train(seg=False) input_prompts = [] answers = [] data_list = self.dev_data for data in data_list[:n_content]: context_ = data["paragraphs"][0]["context"] qas = data["paragraphs"][0]["qas"] for qa in qas[:n_q]: q = qa["question"] a = qa["answers"][0]["text"] input_str0 = f"阅读文章：{context_}\n问：{q}\n答：" demo0 = "" demo1 = gen_prompt(prompts_full, n_shot=1, len_ori=len(input_str0), max_length=max_length) demo2 = gen_prompt(prompts_seg, n_shot=5, len_ori=len(input_str0), max_length=max_length) input_prompts.append([demo0 + input_str0, demo1 + input_str0, demo2 + input_str0]) answers.append(a) return input_prompts, answers def get_data(self, n_content=50, n_q=1, n_shot=0): """get finally data""" prompts_seg = self.gen_prompt_from_train(seg=True) prompts_full = self.gen_prompt_from_train(seg=False) input_prompts = [] answers = [] data_list = self.dev_data for data in data_list[:n_content]: context_ = data["paragraphs"][0]["context"] qas = data["paragraphs"][0]["qas"] for qa in qas[:n_q]: q = qa["question"] a = qa["answers"][0]["text"] input_str0 = f"阅读文章：{context_}\n问：{q}\n答：" if n_shot == 0: demo = "" elif n_shot == 1: demo = gen_prompt(prompts_full, n_shot=1, len_ori=len(input_str0)) else: demo = gen_prompt(prompts_seg, n_shot=n_shot, len_ori=len(input_str0)) input_prompts.append(demo + input_str0) answers.append(a) return input_prompts, answers def write_instance_to_file(writer, instance, need_del_keys): """write the instance to file""" input_ids = instance["input_ids"] attention_mask = instance["attention_mask"] label = instance["labels"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["attention_mask"] = np.asarray(attention_mask) features["labels"] = np.asarray(label) for need_del_key in need_del_keys: del features[need_del_key] writer.write_raw_data([features]) return features def main(): parser = argparse.ArgumentParser() parser.add_argument("--train_file", type=str, default="./cmrc2018_public/train.json", required=True, help='Input raw train file. ') parser.add_argument("--dev_file", type=str, default="./cmrc2018_public/dev.json", required=True, help='Input raw dev file. ') parser.add_argument("--output_file", type=str, default="./cmrc2018.mindrecord", required=True, help='Output MindRecord file. ') parser.add_argument("--num_splits", type=int, default=1, help='The MindRecord file will be split into the number of partition. ') parser.add_argument("--max_length", type=int, default=1024, help='Maximum sequence length. ') parser.add_argument("--n_shot", type=str, default="zero_shot", choices=['zero_shot', 'one_shot', 'five_shot'], help="N shot learning of cmrc2018. ") parser.add_argument("--tokenizer_type", type=str, default="pangualpha_2_6b", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ") parser.add_argument("--n_content", type=int, default=500, help="How many contents are sampled. ") parser.add_argument("--n_q", type=int, default=100, help="How many questions per content as most are sampled. ") parser.add_argument("--mindrecord_schema", type=str, default="pangualpha_cmrc2018", help="The name of mindrecord_schema. ") parser.add_argument("--data_columns", type=list, default=["input_ids", "labels"], help="The data columns which should be saved in mindrecord. This can refer used yaml file. " "Please note that 'labels' does input model.construct, it only used at train.evaluate. ") args = parser.parse_args() dev_file = args.dev_file logging.info("***** Reading from input files *****") logging.info("Input File: %s", dev_file) tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) shot_dict = {'zero_shot': 0, 'one_shot': 1, 'five_shot': 2} dataset_cmrc2018 = Cmrc2018Dataset(cmrc2018_train_json=args.train_file, cmrc2018_dev_json=args.dev_file) prompts, answers = dataset_cmrc2018.get_data_allshot( n_content=args.n_content, n_q=args.n_q, max_length=args.max_length) output_file = args.output_file logging.info("***** Writing to output files *****") logging.info("Output File: %s", args.output_file) writer = FileWriter(output_file, args.num_splits) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "attention_mask": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]} } data_columns = args.data_columns need_del_keys = set(data_columns) - set(data_schema.keys()) for need_del_key in need_del_keys: del data_schema[need_del_key] writer.add_schema(data_schema, args.mindrecord_schema) total_written = 0 for prompt, answer in zip(prompts, answers): prompt = prompt[shot_dict[args.n_shot]] input_ids = tokenizer.encode(prompt, add_special_tokens=False) # this logic refers the open source # https://gitee.com/foundation-models/tk-models/blob/master/models/pangu_alpha/cmrc2018/src/evaluate_main.py if len(input_ids) >= args.max_length - 4: truncated_length = (args.max_length // 100 - 1) * 100 input_ids = input_ids[-truncated_length:] attention_mask = [1] * len(input_ids) pad_len = args.max_length - len(input_ids) input_ids += [tokenizer.pad_token_id] * pad_len attention_mask += [0] * pad_len input_ids = np.array(input_ids).reshape(1, -1) attention_mask = np.array(attention_mask).reshape(1, -1) label_id = tokenizer.encode(answer, add_special_tokens=False) label_id = np.array(label_id).reshape(1, -1) instance = {"input_ids": input_ids, "attention_mask": attention_mask, "labels": label_id} features = write_instance_to_file(writer, instance=instance, need_del_keys=need_del_keys) total_written += 1 display_num = 20 # how many example to display if total_written <= display_num: logging.info("***** Example *****") logging.info("input tokens: %s", tokenizer.decode(instance["input_ids"])) logging.info("label tokens: %s", tokenizer.decode(instance["labels"])) for feature_name in features.keys(): feature = features[feature_name] logging.info("%s: %s", feature_name, feature) writer.commit() logging.info("Wrote %d total instances", total_written) if __name__ == "__main__": # logging.basicConfig(level=logging.INFO) main() pengcheng.mind-test/mindformers/tools/dataset_preprocess/pangualpha/tnews_data_process.py000066400000000000000000000246721471400134000326640ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Data process for TNEWS Dataset. The generated dataset has output columns: ["input_ids", "labels", "attention_mask"], and the data type of three columns is int32. Columns： input_ids: the tokenized inputs, shape is [num_labels, seq_length] labels: the index of the true label, shape is [1,] attention_mask: the mask indicating whether each position is a valid input and is not the added prompt, shape is [num_labels, seq_length] About TNEWS Dataset: TNEWS is a 15-class(finance, technology, sports, etc.) short news text classification dataset. Each piece of data has three attributes, which are classification ID, classification name, and news string (only including the title). You can unzip the dataset files into the following structure: .. code-block:: . └── tnews_public ├── train,json ├── test.json ├── test1.0.json ├── dev.json ├── labels.json └── README.txt """ import itertools import json import argparse import collections import logging import numpy as np from mindspore.mindrecord import FileWriter from mindformers.auto_class import AutoTokenizer EN2ZH_LABEL_MAP = {"news_story": "故事", "news_culture": "文化", "news_entertainment": "娱乐", "news_sports": "体育", "news_finance": "财经", "news_house": "房产", "news_car": "汽车", "news_edu": "教育", "news_tech": "科技", "news_military": "军事", "news_travel": "旅行", "news_world": "世界", "news_stock": "股票", "news_agriculture": "农业", "news_game": "游戏"} NUM_LABELS = 4 def get_en2zh_label(label_path): """get label mapping""" en2zh_labels = {} print('loading label file ') # labels_ch.json is the results that translated from original labels with open(label_path, "r", encoding="utf-8") as f: lines = f.readlines() for line in lines: line_json = json.loads(line) if "label_desc_ch" in line_json: en_label = line_json['label_desc'] zh_label = line_json['label_desc_ch'] en2zh_labels[en_label] = zh_label else: break if en2zh_labels == {}: en2zh_labels = EN2ZH_LABEL_MAP return en2zh_labels def load_tnews_example_for_shot(data_path, en2zh_labels, num_sample=6, np_rng=None, max_len=150, prompt="这是关于{}的文章：{}"): """get prompt for n shot""" with open(data_path, 'r', encoding="utf-8") as fid: data = [json.loads(x) for x in fid.readlines()] if np_rng is None: np_rng = np.random.default_rng(0) # select sample with balanced labels label_key = lambda x: x[1] sample_groupbyed = itertools.groupby( sorted([(x, en2zh_labels[y['label_desc']]) for x, y in enumerate(data)], key=label_key), key=label_key) group_index = [np.array([z[0] for z in y]) for x, y in sample_groupbyed] for x in group_index: np_rng.shuffle(x) # in-place nums = (num_sample - 1) // len(group_index) + 1 group_index_concated = np.concatenate([x[:nums] for x in group_index]) np_rng.shuffle(group_index_concated) selected_index = group_index_concated[:num_sample] examples = [] for x in selected_index: sentence = data[x]['sentence'] example_formated = prompt.format(en2zh_labels[data[x]['label_desc']], sentence)[:max_len] examples.append(example_formated) ret = { 'zero_shot': '', 'one_shot': examples[0] + '\n', 'few_shot': ('\n'.join(examples)) + '\n', } return ret def get_data_generate(data_path, en2zh_labels, tokenizer, shot_examples, pad_token="", seq_length=1024, np_rng=None, task="one_shot", prompt="这是关于{}的文章：{}"): """gen data instances""" with open(data_path, 'r', encoding="utf-8") as fid: data = [json.loads(x) for x in fid.readlines()] label_list = sorted({en2zh_labels[x['label_desc']] for x in data}) id_to_label = dict(enumerate(label_list)) print('All test case num ', len(data)) example = shot_examples[task] pad_id = tokenizer.encoder[pad_token] for instance in data: true_label = en2zh_labels[instance['label_desc']] tmp0 = sorted(list(set(id_to_label.values()) - {true_label})) fake_label = [tmp0[x] for x in np_rng.permutation(len(tmp0))[:NUM_LABELS-1]] # [:3] instance_tf_label = [true_label] + fake_label instance_tf_label = [instance_tf_label[x] for x in np_rng.permutation(len(instance_tf_label))] # shuffle input_ids_list = [] mask_list = [] label_list = [] instance_res = {} for label_i in instance_tf_label: prompt_with_shot_example = "{}" + prompt tmp0 = tokenizer.tokenize(prompt_with_shot_example.format(example, label_i, "")) tmp1 = prompt_with_shot_example.format(example, label_i, instance['sentence']) input_ids = tokenizer.encode(tmp1)[:seq_length] mask = np.zeros(seq_length) mask[len(tmp0):len(input_ids)] = 1 input_ids = np.pad(input_ids, ((0, seq_length - len(input_ids)),), 'constant', constant_values=(0, pad_id)) input_ids_list.append(input_ids) mask_list.append(mask) label_list.append(label_i) true_label = [x for x, y in enumerate(label_list) if y == true_label][0] instance_res["input_ids"] = input_ids_list instance_res["labels"] = true_label instance_res["attention_mask"] = mask_list yield instance_res def write_instance_to_file(writer, instance, need_del_keys): """write the instance to file""" input_ids = instance["input_ids"] labels = instance["labels"] attention_mask = instance["attention_mask"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["labels"] = np.asarray(labels) features["attention_mask"] = np.asarray(attention_mask) for need_del_key in need_del_keys: del features[need_del_key] writer.write_raw_data([features]) return features def main(): parser = argparse.ArgumentParser() parser.add_argument("--input_file", type=str, default="./tnews_public/dev.json", required=True, help="Input raw text file. ") parser.add_argument("--label_file", type=str, default="./tnews_public/labels.json", required=True, help="Input label file. ") parser.add_argument("--output_file", type=str, default="./tnews.mindrecord", required=True, help="Output MindRecord file which ends with '.mindrecord' ") parser.add_argument("--num_splits", type=int, default=1, help="The MindRecord file will be split into the number of partition. ") parser.add_argument("--max_length", type=int, default=1024, help="Maximum sequence length. ") parser.add_argument("--tokenizer_type", type=str, default="pangualpha_2_6b", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ") parser.add_argument("--mindrecord_schema", type=str, default="pangualpha_tnews", help="The name of mindrecord_schema. ") parser.add_argument("--data_columns", type=list, default=["input_ids", "labels", "attention_mask"], help="The data columns which should be saved in mindrecord. This can refer used yaml file. ") parser.add_argument("--n_shot", type=str, default="one_shot", choices=['zero_shot', 'one_shot', 'five_shot'], help="N shot learning of t-news. ") parser.add_argument("--prompt_text", type=str, default="这是关于{}的文章：{}", help="Input String format. ") args = parser.parse_args() np_rng = np.random.default_rng(seed=2) input_file = args.input_file logging.info("***** Reading from input files *****") logging.info("Input File: %s", input_file) tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) en2zh_labels = get_en2zh_label(args.label_file) shot_to_example = load_tnews_example_for_shot(args.input_file, en2zh_labels, np_rng=np_rng, prompt=args.prompt_text) data_generate = get_data_generate(args.input_file, en2zh_labels=en2zh_labels, tokenizer=tokenizer, shot_examples=shot_to_example, seq_length=args.max_length, np_rng=np_rng, task=args.n_shot, prompt=args.prompt_text) output_file = args.output_file logging.info("***** Writing to output files *****") logging.info("Output File: %s", args.output_file) writer = FileWriter(output_file, args.num_splits) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]}, "attention_mask": {"type": "int32", "shape": [-1]} } data_columns = args.data_columns need_del_keys = set(data_columns) - set(data_schema.keys()) for need_del_key in need_del_keys: del data_schema[need_del_key] writer.add_schema(data_schema, args.mindrecord_schema) total_written = 0 for instance in data_generate: write_instance_to_file(writer, instance=instance, need_del_keys=need_del_keys) total_written += 1 writer.commit() logging.info("Wrote %d total instances", total_written) if __name__ == "__main__": # logging.basicConfig(level=logging.INFO) main() pengcheng.mind-test/mindformers/tools/download_tools.py000066400000000000000000000067421471400134000240100ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ '''download_tools''' import time import os import requests import urllib3 from tqdm import tqdm from mindformers.tools.logger import logger try: import fcntl except ImportError: fcntl = None logger.warning("The library fcntl is not found. This may cause the reading file failed " "when call the from_pretrained for different process.") urllib3.disable_warnings() class StatusCode: '''StatusCode''' succeed = 200 def download_with_progress_bar(url, filepath, chunk_size=1024, timeout=4): """download_with_progress_bar""" local_id = int(os.getenv("RANK_ID", "0")) if local_id % 8 != 0: logger.info("Wait for the first card to download file. ") while True: if os.path.exists(filepath+".lock"): return True if not os.path.exists(filepath): os.makedirs(os.path.dirname(filepath), exist_ok=True) header = { "Accept-Encoding": "identity", "User-agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:65.0) Gecko/20100101 Firefox/65.0' } start = time.time() try: response = requests.get(url, verify=False, stream=True, timeout=timeout) except (TimeoutError, urllib3.exceptions.MaxRetryError, requests.exceptions.ProxyError, requests.exceptions.ConnectionError): logger.error("Connect error, please download %s to %s.", url, filepath) return False content_size = response.headers.get('content-length') if content_size is None: response_json = response.json() download_url = response_json.get("data").get("download_url") if download_url: response = requests.get(download_url, verify=False, stream=True, timeout=timeout, headers=header) content_size = int(response.headers.get('content-length')) else: logger.error("Download url parsing failed from json file, please download %s to %s.", url, filepath) return False else: content_size = int(content_size) size = 0 if response.status_code == StatusCode.succeed: logger.info('Start download %s', filepath) with open(filepath, 'wb') as file: if fcntl: fcntl.flock(file.fileno(), fcntl.LOCK_EX) with tqdm(total=content_size, desc='Downloading', leave=True, ncols=100, unit='B', unit_scale=True) as pbar: for data in response.iter_content(chunk_size=chunk_size): file.write(data) size += len(data) pbar.update(1024) end = time.time() logger.info('Download completed!,times: %.2fs', (end - start)) if not os.path.exists(filepath+".lock"): os.mknod(filepath+".lock") return True logger.error("%s is unconnected!", url) return False pengcheng.mind-test/mindformers/tools/download_tools_multithread.py000066400000000000000000000141431471400134000264040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ '''download_tools_multithread''' import os import time from threading import Thread, Lock import requests import urllib3 from tqdm import tqdm from logger import logger try: import fcntl except ImportError: fcntl = None logger.warning("The library fcntl is not found. This may cause the reading file failed " "when call the from_pretrained for different process.") urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) def download_chunk(url, start, end, file, pbar, lock, chunk_size=1024): '''Download a chunk of a file from the given URL and write it to the provided file object. Args: url (str): The URL to download from. start (int): The starting byte position of the chunk to download. end (int): The ending byte position of the chunk to download. file (file): The file object to write the downloaded chunk to. pbar (tqdm.tqdm): The progress bar object to update the download progress. lock (threading.Lock): The lock object to synchronize writing to the file. chunk_size (int, optional): The size of each chunk to download in bytes. Defaults to 1024. Returns: None. Raises: None. ''' headers = {'Range': 'bytes=%d-%d' % (start, end)} response = requests.get(url, headers=headers, verify=False, stream=True) if response.status_code == requests.codes.partial_content: with lock: file.seek(start) for data in response.iter_content(chunk_size=chunk_size): file.write(data) pbar.update(chunk_size) else: logger.error("%s is unconnected!", url) def download_with_progress_bar(url, filepath, num_threads=1, timeout=4): """Downloads a file from the given URL with multi-threading support, resuming from breakpoints, and displays a progress bar to show the progress of the download. Args: url (str): The URL to download the file from. filepath (str): The path where the downloaded file will be saved. num_threads (int, optional): The number of threads to use for the download. Default is 8. timeout (int, optional): The connection timeout in seconds. Default is 4 seconds. Returns: bool: Returns True if the download is successful; otherwise, False. """ if not os.path.exists(filepath): os.makedirs(os.path.dirname(filepath), exist_ok=True) start_time = time.time() try: response = requests.head(url, verify=False, stream=True, timeout=timeout) content_size = int(response.headers['content-length']) etag = response.headers.get('etag') except (TimeoutError, urllib3.exceptions.MaxRetryError, requests.exceptions.ProxyError, requests.exceptions.ConnectionError): logger.error("Connect error, please download %s to %s.", url, filepath) return False if content_size is None: response_json = response.json() download_url = response_json.get("data").get("download_url") if download_url: header = { "Accept-Encoding": "identity", "User-agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:65.0) Gecko/20100101 Firefox/65.0' } try: response = requests.head(download_url, verify=False, stream=True, timeout=timeout, headers=header) content_size = int(response.headers.get('content-length')) etag = response.headers.get('etag') except (TimeoutError, urllib3.exceptions.MaxRetryError, requests.exceptions.ProxyError, requests.exceptions.ConnectionError): logger.error("Connect error, please download %s to %s.", url, filepath) return False else: logger.error("Download url parsing failed from json file, please download %s to %s.", url, filepath) return False start = 0 if os.path.exists(filepath): # check file size and etag file_size = os.path.getsize(filepath) if file_size == content_size and etag == response.headers.get('etag'): logger.info('File already exists: %s', filepath) return True logger.info('File is not complete, resuming download: %s', filepath) start = file_size with open(filepath, 'ab') as file: if fcntl: fcntl.flock(file.fileno(), fcntl.LOCK_EX) chunk_size = (content_size - start) // num_threads with tqdm(total=content_size, initial=start, unit='B', unit_scale=True, desc=filepath.split('/')[-1]) as pbar: threads = [] for i in range(num_threads): if i < num_threads - 1: end = start + chunk_size - 1 else: end = content_size - 1 t = Thread(target=download_chunk, args=(url, start, end, file, pbar, Lock(), 1024)) threads.append(t) t.start() start += chunk_size for t in threads: t.join() # check downloaded file size and etag file_size = os.path.getsize(filepath) if file_size == content_size and etag == response.headers.get('etag'): end_time = time.time() logger.info('Download finished: %s times: %.2fs', filepath, (end_time - start_time)) return True logger.error('Download failed or interrupted: %s', filepath) return False pengcheng.mind-test/mindformers/tools/export.py000066400000000000000000000200061471400134000222670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """export mindir.""" import argparse import glob import os.path import numpy as np import mindspore as ms import mindspore.common.dtype as mstype from mindformers import AutoModel from mindformers.tools.register import MindFormerConfig def get_glm_prefill_model_input(batch_size, seq_length): """get glm model input tuple.""" x = ms.Tensor(np.ones([batch_size, seq_length]).astype(np.int32)) position_ids = ms.Tensor(np.ones([batch_size, 2, seq_length]).astype(np.int32)) attention_mask = ms.Tensor(np.ones([batch_size, 1, seq_length, seq_length]).astype(np.int32)) return x, position_ids, attention_mask def get_llm_common_prefill_model_input(batch_size, seq_length): """get llama model input tuple.""" x = ms.Tensor(np.ones([batch_size, seq_length]).astype(np.int32)) return x def get_bloom_inc_model_input(batch_size, seq_length, prefill): """get bloom kv cache model input tuple.""" if not prefill: seq_length = 1 init_reset = not prefill input_ids = ms.Tensor(np.ones((batch_size, seq_length)), mstype.int32) input_position = ms.Tensor([127] * batch_size, mstype.int32) init_reset = ms.Tensor([init_reset], mstype.bool_) batch_valid_length = ms.Tensor([128] * batch_size, mstype.int32) return input_ids, input_position, None, None, None, None, init_reset, batch_valid_length def get_llama_inc_model_input(batch_size, seq_length, prefill): """get llama kv cache model input tuple.""" if not prefill: seq_length = 1 init_reset = not prefill input_ids = ms.Tensor(np.ones((batch_size, seq_length)), mstype.int32) input_position = ms.Tensor([127] * batch_size, mstype.int32) init_reset = ms.Tensor([init_reset], mstype.bool_) batch_valid_length = ms.Tensor([128] * batch_size, mstype.int32) return input_ids, None, input_position, None, None, None, init_reset, batch_valid_length def get_glm_inc_model_input(batch_size, seq_length, prefill): """get glm kv cache model input tuple.""" if prefill: input_ids = ms.Tensor(np.ones((batch_size, seq_length)), mstype.int32) position_ids = ms.Tensor(np.ones([batch_size, 2, seq_length]).astype(np.int32)) attention_mask = ms.Tensor(np.ones([batch_size, 1, seq_length, seq_length]).astype(np.int32)) else: input_ids = ms.Tensor(np.ones((batch_size, 1)), mstype.int32) position_ids = ms.Tensor(np.ones([batch_size, 2, 1]).astype(np.int32)) attention_mask = ms.Tensor(np.ones([batch_size, 1, 1, seq_length]).astype(np.int32)) input_position = ms.Tensor([127] * batch_size, mstype.int32) init_reset = not prefill init_reset = ms.Tensor([init_reset], mstype.bool_) batch_valid_length = ms.Tensor([128] * batch_size, mstype.int32) return input_ids, position_ids, attention_mask, input_position, None, None, init_reset, batch_valid_length PREFILL_MODEL_INPUT_MAP = { "bloom": get_llm_common_prefill_model_input, "llama": get_llm_common_prefill_model_input, "glm": get_glm_prefill_model_input, } INCREMENT_MODEL_INPUT_MAP = { "bloom": get_bloom_inc_model_input, "llama": get_llama_inc_model_input, "glm": get_glm_inc_model_input } def export_single_model(config, batch_size, model_type: str = 'MINDIR', model_dir=None): """ export no kvcache model. Args: config: config from yaml. batch_size(int): batch size model_type: model type """ model_name = config.trainer.model_name if model_dir: model = AutoModel.from_pretrained(model_dir) else: model = AutoModel.from_pretrained(model_name) model.set_train(False) inputs = () for item in PREFILL_MODEL_INPUT_MAP: if model_name.startswith(item): inputs = PREFILL_MODEL_INPUT_MAP[item](batch_size, config.infer.infer_seq_length) break if not inputs: raise NotImplementedError(f"model {model_name} not implemented.") suffix = model_type.lower() filename = config.infer.prefill_model_path.rstrip("." + suffix) model.add_flags_recursive(is_first_iteration=True) ms.export(model, *inputs, file_name=filename, file_format=model_type.upper()) def export_inc_model(config, batch_size, model_type: str = 'MINDIR', model_dir=None): """ export kvcache model. Args: config: config from yaml. batch_size(int): batch size model_type: model type """ model_name = config.trainer.model_name if model_dir: model = AutoModel.from_pretrained(model_dir) else: model = AutoModel.from_pretrained(model_name) model.set_train(False) func = None for item in INCREMENT_MODEL_INPUT_MAP: if model_name.startswith(item): func = INCREMENT_MODEL_INPUT_MAP[item] break if not func: raise NotImplementedError(f"model {model_name} not implemented.") suffix = model_type.lower() # export prefill filename = config.infer.prefill_model_path.rstrip("." + suffix) inputs = func(batch_size, config.infer.infer_seq_length, True) model.add_flags_recursive(is_first_iteration=True) ms.export(model, *inputs, file_name=filename, file_format=model_type.upper()) print("Prefill model exported.") # export inc filename = config.infer.increment_model_path.rstrip("." + suffix) inputs = func(batch_size, config.infer.infer_seq_length, False) model.add_flags_recursive(is_first_iteration=False) ms.export(model, *inputs, file_name=filename, file_format="MINDIR") print("Increment model exported.") def main(args_): """export mindir.""" ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=args_.device_id) if args_.config_path is None and args_.model_dir is None: raise ValueError(f"config_path or model_dir must exist one.But get none.") if args_.config_path: config = MindFormerConfig(args_.config_path) model_dir = None else: model_dir = os.path.realpath(args_.model_dir) run_config = glob.glob(model_dir + "/*.yaml") if len(run_config) > 1: raise ValueError(f"{model_dir} contains multi config files: {run_config}.") config = MindFormerConfig(run_config[0]) if config.infer is None: raise KeyError(f"infer config not in {args_.config_path}.") if not config.infer.prefill_model_path: raise ValueError(f"prefill_model_path in {args_.config_path} is empty.") if not config.infer.increment_model_path: export_single_model(config, args_.batch_size, args_.model_type, model_dir) else: export_inc_model(config, args_.batch_size, args_.model_type, model_dir) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument( '--config_path', default=None, type=str, help='YAML config files') parser.add_argument( '--model_dir', default=None, type=str, help='model config and ckpt file dir path') parser.add_argument( '--batch_size', default=1, type=int, help='batch size of model.') parser.add_argument( '--model_type', default="MINDIR", type=str, help='model type of exported model.') parser.add_argument( '--device_id', default=0, type=int, help='device id to export mindir.') args = parser.parse_args() main(args) pengcheng.mind-test/mindformers/tools/hccl_tools.py000066400000000000000000000142521471400134000231050ustar00rootroot00000000000000# Copyright 2020 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """generate hccl config file script""" import json import os import socket import sys from argparse import ArgumentParser from typing import Dict, Any def parse_args(): """ parse args . Args: Returns: args. Examples: >>> parse_args() """ parser = ArgumentParser(description="mindspore distributed training launch " "helper utility that will generate hccl" " config file") parser.add_argument("--device_num", type=str, default="[0,8)", help="The number of the Ascend accelerators used. please note that the Ascend accelerators" "used must be continuous, such [0,4) means using four chips " "0，1，2，3; [0,1) means using chip 0; In the most Ascend system, " "the first four chips belong to one group, and the last four chips belong to another one." "Only full chips are allowed to cross-group such as [0,8), other cross-group such as [3,6)" "are prohibited.") parser.add_argument("--visible_devices", type=str, default="0,1,2,3,4,5,6,7", help="The visible devices according to the software system. " "Usually used in the virtual system or docker container " "that makes the device_id dismatch logic_id. --device_num uses logic_id. " "For example \"4,5,6,7\" means the system has 4 logic chips " "which are actually the last 4 chips in hardware " "while `--device_num` could only be set to \"[0, 4)\" instead of \"[4, 8)\"") parser.add_argument("--server_ip", type=str, default="", help="Set the server_ip manually, to avoid errors in auto detection.") args = parser.parse_args() return args def get_host_ip(): """ get host ip """ ip = None try: hostname = socket.gethostname() ip = socket.gethostbyname(hostname) except EOFError: pass return ip def main(): print("start", __file__) args = parse_args() # visible_devices visible_devices = args.visible_devices.split(',') print('visible_devices:{}'.format(visible_devices)) # server_id ip = get_host_ip() if args.server_ip: server_id = args.server_ip elif ip: server_id = ip else: raise ValueError("please input server ip!") print('server_id:{}'.format(server_id)) # device_num first_num = int(args.device_num[1]) last_num = int(args.device_num[3]) if first_num < 0 or last_num > 8: raise ValueError("device num {} must be in range [0,8] !".format(args.device_num)) if first_num > last_num: raise ValueError("First num {} of device num {} must less than last num {} !".format(first_num, args.device_num, last_num)) if first_num < 4 < last_num: if first_num == 0 and last_num == 8: pass else: raise ValueError("device num {} must be in the same group of [0,4] or [4,8] !".format(args.device_num)) device_num_list = list(range(first_num, last_num)) print("device_num_list:", device_num_list) assert len(visible_devices) >= len(device_num_list) # construct hccn_table device_ips: Dict[Any, Any] = {} try: for device_id in device_num_list: ret = os.popen("hccn_tool -i %d -ip -g" % device_id).readlines() device_ips[str(device_id)] = ret[0].split(":")[1].replace('\n', '') except IndexError: print("Failed to call hccn_tool, try to read /etc/hccn.conf instead") try: with open('/etc/hccn.conf', 'r') as fin: for hccn_item in fin.readlines(): if hccn_item.strip().startswith('address_'): device_id, device_ip = hccn_item.split('=') device_id = device_id.split('_')[1] device_ips[device_id] = device_ip.strip() except OSError: print("Failed to read /etc/hccn.conf") raise SystemError("Failed to find information for hccl") hccn_table = {'version': '1.0', 'server_count': '1', 'server_list': []} device_list = [] rank_id = 0 for instance_id in device_num_list: device_id = visible_devices[instance_id] device_ip = device_ips[device_id] device = {'device_id': device_id, 'device_ip': device_ip, 'rank_id': str(rank_id)} print('rank_id:{}, device_id:{}, device_ip:{}'.format(rank_id, device_id, device_ip)) rank_id += 1 device_list.append(device) hccn_table['server_list'].append({ 'server_id': server_id, 'device': device_list, 'host_nic_ip': 'reserve' }) hccn_table['status'] = 'completed' # save hccn_table to file table_path = os.getcwd() table_fn = os.path.join(table_path, 'hccl_{}p_{}_{}.json'.format(len(device_num_list), "".join(map(str, device_num_list)), server_id)) with open(table_fn, 'w') as table_fp: json.dump(hccn_table, table_fp, indent=4) sys.stdout.flush() print("Completed: hccl file was save in :", table_fn) if __name__ == "__main__": main() pengcheng.mind-test/mindformers/tools/image_tools.py000066400000000000000000000036051471400134000232560ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ ''' Image tools ''' import os import requests import urllib3 import PIL import mindspore as ms def load_image(content, timeout=4): """load image""" if isinstance(content, str): if content.startswith("https://") or content.startswith("http://"): try: response = requests.get(content, stream=True, timeout=timeout) except (TimeoutError, urllib3.exceptions.MaxRetryError, requests.exceptions.ProxyError) as exc: raise ConnectionError(f"Connect error, please download {content}.") from exc content = PIL.Image.open(response.raw) elif os.path.isfile(content): content = PIL.Image.open(content) else: raise ValueError( f"{content} is not a valid path. If URL, it must start with `http://` or `https://`." ) if isinstance(content, ms.Tensor): return content if not isinstance(content, PIL.Image.Image): raise ValueError( "Input should be an url linking to an image," " a local path, a Mindspore Tensor, or a PIL image." ) content = PIL.ImageOps.exif_transpose(content) content = content.convert("RGB") return content pengcheng.mind-test/mindformers/tools/logger.py000066400000000000000000000444021471400134000222330ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LOGGER Module""" import logging import logging.config import logging.handlers import os import stat import sys from functools import wraps from tempfile import TemporaryFile from typing import Dict, List, Tuple, Union from mindformers.tools.utils import get_num_nodes_devices, get_rank_info, \ convert_nodes_devices_input, generate_rank_list,\ check_in_modelarts, check_list, LOCAL_DEFAULT_PATH,\ get_output_root_path logger_list = [] LEVEL = ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL') MODELARTS_LOG_FILE_DIR = '/cache/ma-user-work/' LOCAL_DEFAULT_LOG_FILE_DIR = os.path.join( os.getenv("LOCAL_DEFAULT_PATH", LOCAL_DEFAULT_PATH), 'log') RANK_DIR_FORMATTER = 'rank_{}/' DEFAULT_FILEHANDLER_FORMAT = '[%(levelname)s] %(asctime)s ' \ '[%(pathname)s:%(lineno)d] %(funcName)s: %(message)s' DEFAULT_STDOUT_FORMAT = '%(asctime)s - %(name)s[%(filename)s:%(lineno)d] - %(levelname)s - %(message)s' DEFAULT_REDIRECT_FILE_NAME = 'mindspore.log' def judge_redirect(rank_id: int, rank_size: int, redirect_nodes: Union[List, Tuple, Dict[str, int], None] = None, redirect_devices: Union[List, Tuple, Dict[str, int], None] = None): """Determine if the stderr of this process needs to be redirected. Args: rank_id (int): Rank id. rank_size (int): Rank Size. redirect_nodes (list or tuple or dict or None): Node list. The nodes in the list will redirect stderr. redirect_devices (list or tuple or dict or None): Device list. The devices in the list will redirect stderr. Returns: prerequisite (bool): If true, stderr will redirect. """ is_redirect = True if rank_size > 1 and (redirect_nodes is not None or redirect_devices is not None): num_nodes, num_devices = get_num_nodes_devices(rank_size) redirect_nodes = convert_nodes_devices_input(redirect_nodes, num_nodes) redirect_devices = convert_nodes_devices_input(redirect_devices, num_devices) check_list('nodes', redirect_nodes, num_nodes) check_list('devices', redirect_devices, num_devices) rank_list = generate_rank_list(redirect_nodes, redirect_devices) if rank_id not in rank_list: is_redirect = False return is_redirect class StreamRedirector: """Stream Re-director for Log.""" def __init__(self, source_stream, target_stream): """Redirects the source stream to the target stream. Args: source_stream: Source stream. target_stream: Target stream. """ super(StreamRedirector, self).__init__() self.source_stream = source_stream self.target_stream = target_stream self.save_source_stream_fd = os.dup(self.source_stream.fileno()) def __call__(self, func): @wraps(func) def wrapper(*args, **kwargs): self.start() func(*args, **kwargs) self.stop() return wrapper def __enter__(self): self.start() def __exit__(self, exc_type, exc_val, exc_tb): self.stop() def start(self): """start.""" self.source_stream.flush() os.dup2(self.target_stream.fileno(), self.source_stream.fileno()) def stop(self): """stop.""" self.source_stream.flush() os.dup2(self.save_source_stream_fd, self.source_stream.fileno()) self.target_stream.flush() class AiLogFastStreamRedirect2File(StreamRedirector): """AiLogFastStreamRedirect2File.""" def __init__(self, source_stream=None, redirect_nodes: Union[List, Tuple, Dict[str, int], None] = None, redirect_devices: Union[List, Tuple, Dict[str, int], None] = None, **kwargs): """Redirect stream to file. Args: source_stream (file object or None): Streams that need to be redirected. Default: None, select stderr. redirect_nodes (list[int] or tuple[int] or optional): The computation nodes that will redirect stderr. Default is None: indicates that all nodes will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all redirect stderr. redirect_devices (list[int] or tuple[int] or optional): The computation devices that will redirect stderr. Default is None, indicates that all devices will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all redirect stderr. kwargs (dict): File-related parameters. file_save_dir (str): The folder where the files that save redirected stream are saved. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (str): Redirect file name. """ rank_id, rank_size = get_rank_info() self.is_redirect = judge_redirect(rank_id=rank_id, rank_size=rank_size, redirect_nodes=redirect_nodes, redirect_devices=redirect_devices) file_save_dir = kwargs.get('file_save_dir', '') append_rank_dir = kwargs.get('append_rank_dir', True) file_name = kwargs.get('file_name', '') if not file_save_dir: file_save_dir = os.path.join(get_output_root_path(), 'log') if append_rank_dir: rank_str = RANK_DIR_FORMATTER.format(rank_id) file_save_dir = os.path.join(file_save_dir, rank_str) if not file_name: file_name = DEFAULT_REDIRECT_FILE_NAME self.file_path = os.path.join(file_save_dir, file_name) self.file_save_dir = os.path.dirname(self.file_path) if source_stream is None: source_stream = sys.stderr target_stream = TemporaryFile(mode='w+') super(AiLogFastStreamRedirect2File, self).__init__(source_stream=source_stream, target_stream=target_stream) def start(self): if self.is_redirect: super(AiLogFastStreamRedirect2File, self).start() def stop(self): if self.is_redirect: self.target_stream.flush() if not os.path.exists(self.file_save_dir): os.makedirs(self.file_save_dir, exist_ok=True) self.target_stream.seek(0, 0) flags = os.O_WRONLY | os.O_CREAT modes = stat.S_IWUSR | stat.S_IRUSR with os.fdopen(os.open(self.file_path, flags, modes), 'w') as fp: for line in self.target_stream: fp.write(line) super(AiLogFastStreamRedirect2File, self).stop() self.target_stream.close() def judge_stdout(rank_id: int, rank_size: int, is_output: bool, nodes: Union[List, Tuple, Dict[str, int], None] = None, devices: Union[List, Tuple, Dict[str, int], None] = None) -> bool: """Determines if logs will be output. Args: rank_id (int): Rank id. rank_size (int): Rank size. is_output (int): If set to true, logs or others will be output. nodes (list or tuple or dict or None): Node list. The nodes in the list will output the log to stdout. devices (list or tuple or dict or None): Device list. The devices in the list or output the log to stdout. Returns: is_output (bool): If set to true, logs or others will be output or redirect. """ if is_output and rank_size > 1 and (nodes is not None or devices is not None): num_nodes, num_devices = get_num_nodes_devices(rank_size) stdout_nodes = convert_nodes_devices_input(nodes, num_nodes) stdout_devices = convert_nodes_devices_input(devices, num_devices) check_list('nodes', stdout_nodes, num_nodes) check_list('devices', stdout_devices, num_devices) rank_list = generate_rank_list(stdout_nodes, stdout_devices) if rank_id not in rank_list: is_output = False return is_output def validate_nodes_devices_input(var_name: str, var): """Check the list of nodes or devices. Args: var_name (str): Variable name. var: The name of the variable to be checked. Returns: None """ if not (var is None or isinstance(var, (list, tuple, dict))): raise TypeError('The value of {} can be None or a value of type tuple, ' 'list, or dict.'.format(var_name)) if isinstance(var, (list, tuple)): for item in var: if not isinstance(item, int): raise TypeError('The elements of a variable of type list or ' 'tuple must be of type int.') def validate_level(var_name: str, var): """Verify that the log level is correct. Args: var_name (str): Variable name. var: The name of variable to be checked. Returns: None """ if not isinstance(var, str): raise TypeError('The format of {} must be of type str.'.format(var_name)) if var not in LEVEL: raise ValueError('{}={} needs to be in {}'.format(var_name, var, LEVEL)) def validate_std_input_format(to_std: bool, stdout_nodes: Union[List, Tuple, None], stdout_devices: Union[List, Tuple, None], stdout_level: str): """Validate the input about stdout of the get_logger function.""" if not isinstance(to_std, bool): raise TypeError('The format of the to_std must be of type bool.') validate_nodes_devices_input('stdout_nodes', stdout_nodes) validate_nodes_devices_input('stdout_devices', stdout_devices) validate_level('stdout_level', stdout_level) def validate_file_input_format(file_level: Union[List[str], Tuple[str]], file_save_dir: str, append_rank_dir: str, file_name: Union[List[str], Tuple[str]]): """Validate the input about file of the get_logger function.""" if not isinstance(file_level, (tuple, list)): raise TypeError('The value of file_level should be list or a tuple.') for level in file_level: validate_level('level in file_level', level) if not len(file_level) == len(file_name): raise ValueError('The length of file_level and file_name should be equal.') if not isinstance(file_save_dir, str): raise TypeError('The value of file_save_dir should be a value of type str.') if not isinstance(append_rank_dir, bool): raise TypeError('The value of append_rank_dir should be a value of type bool.') if not isinstance(file_name, (tuple, list)): raise TypeError('The value of file_name should be list or a tuple.') for name in file_name: if not isinstance(name, str): raise TypeError('The value of name in file_name should be a value of type str.') def _convert_level(level: str) -> int: """Convert the format of the log to logging level. Args: level (str): User log level. Returns: level (str): Logging level. """ level_convert = { 'DEBUG': logging.DEBUG, 'INFO': logging.INFO, 'WARNING': logging.WARNING, 'ERROR': logging.ERROR, 'CRITICAL': logging.CRITICAL } level = level_convert.get(level, logging.INFO) return level def get_logger(logger_name: str = 'mindformers', **kwargs) -> logging.Logger: """Get the logger. Both computing centers and bare metal servers are available. Args: logger_name (str): Logger name. kwargs (dict): Other input. to_std (bool): If set to True, output the log to stdout. stdout_nodes (list[int] or tuple[int] or optional): The computation nodes that will output the log to stdout. default: None, indicates that all nodes will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all output logs to stdout. stdout_devices (list[int] or tuple[int] or optional): The computation devices that will output the log to stdout. default: None, indicates that all devices will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all output logs to stdout. stdout_level (str): The level of the log output to stdout. If the type is str, the options are DEBUG, INFO, WARNING, ERROR, CRITICAL. stdout_format (str): Log format. file_level (list[str] or tuple[str]): The level of the log output to file. eg: ['INFO', 'ERROR'] Indicates that the logger will output logs above the level INFO and ERROR in the list to the corresponding file. The length of the list needs to be the same as the length of file_name. file_save_dir (str): The folder where the log files are stored. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (list[str] or list[tuple]): Store a list of output file names. max_file_size (int): The maximum size of a single log file. Unit: MB. max_num_of_files (int): The maximum number of files to save. Returns: logger (logging.Logger): Logger. """ mf_logger = logging.getLogger(logger_name) if logger_name in logger_list: return mf_logger to_std = kwargs.get('to_std', True) stdout_nodes = kwargs.get('stdout_nodes', None) rank_id, rank_size = get_rank_info() def get_stdout_devices(): if os.getenv("STDOUT_DEVICES"): devices = os.getenv("STDOUT_DEVICES") if devices.startswith(("(", "[")) and devices.endswith((")", "]")): devices = devices[1:-1] devices = tuple(map(lambda x: int(x.strip()), devices.split(","))) elif check_in_modelarts(): devices = kwargs.get('stdout_devices', (min(rank_size - 1, 7),)) else: devices = kwargs.get('stdout_devices', None) return devices stdout_devices = get_stdout_devices() stdout_level = kwargs.get('stdout_level', 'INFO') stdout_format = kwargs.get('stdout_format', '') file_level = kwargs.get('file_level', ('INFO', 'ERROR')) file_save_dir = kwargs.get('file_save_dir', '') append_rank_dir = kwargs.get('append_rank_dir', True) file_name = kwargs.get('file_name', (f'info.log', 'error.log')) max_file_size = kwargs.get('max_file_size', 50) max_num_of_files = kwargs.get('max_num_of_files', 5) validate_std_input_format(to_std, stdout_nodes, stdout_devices, stdout_level) validate_file_input_format(file_level, file_save_dir, append_rank_dir, file_name) to_std = judge_stdout(rank_id=rank_id, rank_size=rank_size, is_output=to_std, nodes=stdout_nodes, devices=stdout_devices) if to_std: if not stdout_format: stdout_format = DEFAULT_STDOUT_FORMAT stream_handler = logging.StreamHandler(sys.stdout) stream_handler.setLevel(_convert_level(stdout_level)) stream_formatter = logging.Formatter(stdout_format) stream_handler.setFormatter(stream_formatter) mf_logger.addHandler(stream_handler) logging_level = [] for level in file_level: logging_level.append(_convert_level(level)) if not file_save_dir: file_save_dir = os.path.join(get_output_root_path(), 'log') rank_dir = RANK_DIR_FORMATTER if append_rank_dir: rank_str = rank_dir.format(rank_id) file_save_dir = os.path.join(file_save_dir, rank_str) file_path = [] for name in file_name: path = os.path.join(file_save_dir, name) path = os.path.realpath(path) base_dir = os.path.dirname(path) if not os.path.exists(base_dir): os.makedirs(base_dir, exist_ok=True) file_path.append(path) max_file_size = max_file_size * 1024 * 1024 file_formatter = logging.Formatter(DEFAULT_FILEHANDLER_FORMAT) for i, level in enumerate(file_level): file_handler = logging.handlers.RotatingFileHandler(filename=file_path[i], maxBytes=max_file_size, backupCount=max_num_of_files) file_handler.setLevel(level) file_handler.setFormatter(file_formatter) mf_logger.addHandler(file_handler) mf_logger.setLevel(_convert_level('INFO')) mf_logger.propagate = False logger_list.append(logger_name) return mf_logger class _LogActionOnce: """ A wrapper for modify the warning logging to an empty function. This is used when we want to only log once to avoid the repeated logging. Args: logger (logging): The logger object. """ is_logged = dict() def __init__(self, m_logger, key, no_warning=False): self.logger = m_logger self.key = key self.no_warning = no_warning def __call__(self, func): def wrapper(*args, **kwargs): if not hasattr(self.logger, 'warning'): return func(*args, **kwargs) old_func = self.logger.warning if self.no_warning or self.key in _LogActionOnce.is_logged: self.logger.warning = lambda x: x else: _LogActionOnce.is_logged[self.key] = True res = func(*args, **kwargs) if hasattr(self.logger, 'warning'): self.logger.warning = old_func return res return wrapper logger = get_logger() pengcheng.mind-test/mindformers/tools/merge_hccl.py000066400000000000000000000044141471400134000230430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """generate hccl config file script""" import os import sys import json from argparse import ArgumentParser def parse_args(): """ parse args . Args: Returns: args. Examples: >>> parse_args() """ parser = ArgumentParser(description="Merge several hccl config json files" "of single server into one config file of the whole cluster") parser.add_argument("file_list", type=str, nargs="+", help="Hccl file lists") arg = parser.parse_args() return arg if __name__ == "__main__": args = parse_args() print(args.file_list) server_count = 0 json_list = [] for f_name in args.file_list: with open(f_name) as f: f_json = json.load(f) json_list.append(f_json) server_count += int(f_json['server_count']) hccl_table = {'version': '1.0', 'server_count': f'{server_count}', 'server_list': []} rank_id = 0 for j in json_list: server_list = j['server_list'] for server in server_list: for device in server['device']: device['rank_id'] = str(rank_id) rank_id += 1 hccl_table['server_list'].extend(server_list) hccl_table['status'] = 'completed' table_path = os.getcwd() table_name = os.path.join(table_path, 'hccl_{}s_{}p.json'.format(server_count, rank_id)) with open(table_name, 'w') as table_fp: json.dump(hccl_table, table_fp, indent=4) sys.stdout.flush() print("Completed: hccl file was save in :", table_name) pengcheng.mind-test/mindformers/tools/register/000077500000000000000000000000001471400134000222225ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/register/__init__.py000066400000000000000000000016111471400134000243320ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Register Tools.""" from .config import MindFormerConfig, ActionDict from .register import MindFormerRegister, MindFormerModuleType __all__ = ['MindFormerRegister', 'MindFormerModuleType', 'ActionDict', 'MindFormerConfig'] pengcheng.mind-test/mindformers/tools/register/__pycache__/000077500000000000000000000000001471400134000244325ustar00rootroot00000000000000pengcheng.mind-test/mindformers/tools/register/__pycache__/__init__.cpython-39.pyc000066400000000000000000000006241471400134000306240ustar00rootroot00000000000000a ?cf@s0dZddlmZmZddlmZmZgdZdS)zMindFormers Register Tools.)MindFormerConfig ActionDict)MindFormerRegisterMindFormerModuleType)rrrrN)__doc__configrrregisterrr__all__r r N/home/ma-user/work/mindformers-baichuan/mindformers/tools/register/__init__.pyspengcheng.mind-test/mindformers/tools/register/__pycache__/config.cpython-39.pyc000066400000000000000000000232211471400134000303300ustar00rootroot00000000000000a ?cf+@sdZddlZddlZddlZddlmZddlmZddlZdZGddde Z Gdd d eZ ej efd d Z dejefd d ZddZdS)z& Transformer-Config dict parse module N)Action) OrderedDictZ base_configcspeZdZdZfddZddZddZdd Zdd d Zd dZ e ddZ e dddZ e ddZ ZS)MindFormerConfiga A Config class is inherit from dict. Config class can parse arguments from a config file of yaml or a dict. Args: args (list) : config filenames kwargs (dict) : config dictionary list Example: test.yaml: a:1 >>> cfg = MindFormerConfig('./test.yaml') >>> cfg.a 1 >>> cfg = MindFormerConfig(**dict(a=1, b=dict(c=[0,1]))) >>> cfg.b {'c': [0, 1]} csptt|i}|D]6}t|tr|ds8|drt|}||q|dur`||t||dS)NyamlZyml) superr__init__ isinstancestrendswith _file2dictupdate _dict2config)selfargskwargscfg_dictargZraw_dict __class__L/home/ma-user/work/mindformers-baichuan/mindformers/tools/register/config.pyr1s    zMindFormerConfig.__init__cCs||vr dS||S)zGet a object attr by its `key` Args: key (str) : the name of object attr. Returns: attr of object that name is `key` Nrrkeyrrr __getattr__Bs zMindFormerConfig.__getattr__cCs |||<dS)zSet a object value `key` with `value` Args: key (str) : The name of object attr. value : the `value` need to set to the target object attr. Nr)rrvaluerrr __setattr__PszMindFormerConfig.__setattr__cCs ||=dS)zkDelete a object attr by its `key`. Args: key (str) : The name of object attr. Nrrrrr __delattr__YszMindFormerConfig.__delattr__Nc Cs:t}|D]&}|t||t|||q|S)zDeep copy operation on arbitrary MindFormerConfig objects. Args: memo (dict) : Objects that already copied. Returns: MindFormerConfig : The deep copy of the given MindFormerConfig object. )rkeysrcopydeepcopyr)rmemoconfigrrrr __deepcopy__as  zMindFormerConfig.__deepcopy__c Csxi}|D]N\}}|}|d}|ddD]}||t||}q.|d}|||<q t||}t||dS)a:Merge options into config file. Args: options (dict): dict of configs to merge from Examples: >>> options = {'model.arch': 'simmim'} >>> cfg = MindFormerConfig(**dict(model=dict(backbone=dict(type='vit')))) >>> cfg.merge_from_dict(options) .N)itemssplit setdefaultr_merge_a_into_br ) roptionsZoption_cfg_dictfull_keyrdkey_listZsub_key merge_dictrrrmerge_from_dictos     z MindFormerConfig.merge_from_dictcCsN|}|D]8\}}t|tr@||vr@t|||||<q|||<q|S)a1Merge dict ``a`` into dict ``b`` Values in ``a`` will overwrite ``b`` Args: a (dict) : The source dict to be merged into b. b (dict) : The origin dict to be fetch keys from ``a``. Returns: dict: The modified dict of ``b`` using ``a``. )rr%rdictrr()abkvrrrr(s  z MindFormerConfig._merge_a_into_bc Cs|durtd|tj|}t|dd}t|tjd}Wdn1sR0Yt |vrtj |}| t }t |t r|n|g}t }|D]"}ttj||}||qt} |D]} | | qt|| } | }|S)zdConvert config file to dictionary. Args: filename (str) : config file. NzThis {} cannot be empty.zutf-8)encoding) yaml_loader) NameErrorformatospathrealpathopenordered_yaml_loadrZ FullLoader BASE_CONFIGdirnamepoprlistrr joinappendr/r r() filenamefilepathfprZcfg_dirZbase_filenamesZ cfg_dict_list base_filenameZ cfg_dict_itemZ base_cfg_dictcfgrrrr s2 ,      zMindFormerConfig._file2dictcCsXt|trT|D]@\}}t|trFt}t|||t||q||||<qdS)zConvert dictionary to config. Args: config : Config object dic (dict) : dictionary Returns: Exceptions: N)rr/r%r __setitem__r )r!dicrrZ sub_configrrrr s  zMindFormerConfig._dict2config)N)N)__name__ __module__ __qualname____doc__rrrrr"r. staticmethodr(r r __classcell__rrrrrs     $rc@s>eZdZdZeddZeddZeddZd d d ZdS) ActionDicta Argparse action to split an option into KEY=VALUE from on the first = and append to dictionary. List options can be passed as comma separated values. i.e. 'KEY=Val1,Val2,Val3' or with explicit brackets i.e. 'KEY=[Val1,Val2,Val3]'. cCsVz t|WStyYn0z t|WSty:Yn0|dvrR|jdkS|S)z9convert string val to int or float or bool or do nothing.)TRUEFALSErQ)int ValueErrorfloatupper)valrrr_parse_int_float_bools      z ActionDict._parse_int_float_boolcCs|d|dkr(|d|dks,Jt|}t|D]L\}}|d|}|dkr<|d|dkr<|d|dkr<|}qq<|S)zfind the position of next comma in the string. note: '(' and ')' or '[' and']' must appear in pairs or not exist. ()[]N,)countlen enumerate)Zval_strendidxcharprerrrfind_next_commas zActionDict.find_next_commacCs|ddd}d}|dr<|dr>> ActionDict._parse_value_iter('1,2,3') [1,2,3] >>> ActionDict._parse_value_iter('[1,2,3]') [1,2,3] >>> ActionDict._parse_value_iter('(1,2,3)') (1,2,3) >>> ActionDict._parse_value_iter('[1,[1,2],(1,2,3)') [1, [1, 2], (1, 2, 3)] z'" FrYrZTr$r[r\r]rN) stripreplace startswithr rPrXr_re_parse_value_iterrBtuple)rWis_tuplevaluesZ len_of_valZ comma_idxelerrrrls(    zActionDict._parse_value_iterNc Cs@i}|D]$}|jddd\}}||||<qt||j|dS)N=rh)maxsplit)r&rlsetattrdest) rparser namespacero option_stringr) key_valuerrrrr__call__1s zActionDict.__call__)N) rJrKrLrMrNrXrerlryrrrrrPs   0rPcs:Gddd|}fdd}|tjjj|t||S)zLoad Yaml File in Orderedly.c@s eZdZdS)z(ordered_yaml_load..OrderedLoaderNrJrKrLrrrr OrderedLoader<sr{cs||||SN)Zflatten_mappingZconstruct_pairs)loadernodeobject_pairs_hookrr_construct_mapping?s z-ordered_yaml_load.._construct_mapping)Zadd_constructorrresolver BaseResolverDEFAULT_MAPPING_TAGload)streamr5rr{rrrrr<9s r<cKs:Gddd|}dd}|||tj|||fi|S)z$Dump Dict to Yaml File in Orderedly.c@s eZdZdS)z(ordered_yaml_dump..OrderedDumperNrzrrrr OrderedDumperLsrcSs|tjjj|Sr|)Zrepresent_mappingrrrrr%)Zdumperdatarrr_dict_representerOsz,ordered_yaml_dump.._dict_representer)Zadd_representerrdump)rrZ yaml_dumperrrrrrrrordered_yaml_dumpIs rcCs&td}|jddtddd|S)zh Parse arguments from `yaml or yml` config file. Returns: object: argparse object. zTransformer Config.z-cz--configrgz(Enter the path of the model config file.)typedefaulthelp)argparseArgumentParser add_argumentr parse_args)rurrrrXs r)rMr8rrr collectionsrrr=r/rrPZ SafeLoaderr<Z SafeDumperrrrrrrs"  ;d  pengcheng.mind-test/mindformers/tools/register/__pycache__/register.cpython-39.pyc000066400000000000000000000131121471400134000307050ustar00rootroot00000000000000a ?cf@s,dZddlZGdddZGdddZdS)z' Class Register Module For MindFormers.Nc@seZdZdZddZdZdZdZdZdZ d Z d Z d Z d Z d ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdS)MindFormerModuleTypez%Class module type for vision pretraincCsdSNselfrrN/home/ma-user/work/mindformers-baichuan/mindformers/tools/register/register.py__init__szMindFormerModuleType.__init__trainerpipeline processor tokenizerdatasetZ mask_policyZdataset_loaderZdataset_sampler transformsencodermodelsmodules transformerZ base_layercoreheadlosslr optimizercontextcallbackwrappermetricconfigtoolsZfeature_extractorN)__name__ __module__ __qualname____doc__rZTRAINERZPIPELINEZ PROCESSORZ TOKENIZERZDATASETZ MASK_POLICYZDATASET_LOADERZDATASET_SAMPLERZ TRANSFORMSZENCODERZMODELSMODULESZ TRANSFORMERZ BASE_LAYERZCOREZHEADZLOSSLRZ OPTIMIZERCONTEXTZCALLBACKZWRAPPERZMETRICCONFIGTOOLSZFEATURE_EXTRACTORrrrrrs8rc@seZdZdZddZiZeejdfddZ eejdfddZ edd d Z edd d Z eejdfd dZ eejdfddZdS)MindFormerRegisterz+ Module class factory for ring-mo. cCsdSrrrrrrr=szMindFormerRegister.__init__Ncsfdd}|S)zRegister class into registry Args: module_type (ModuleType) : module type name, default ModuleType.GENERAL alias (str) : class alias Returns: wrapper cs>dur n|j}jvr,||ij<n|j|<|S)zRegister-Class with wrapper function. Args: register_class : class need to register Returns: wrapper of register_class Nrregistry)register_class class_namealiascls module_typerrrNs  z,MindFormerRegister.register..wrapperr)r.r/r-rrr,rregisterBs zMindFormerRegister.registercCs>|dur |n|j}||jvr,||i|j|<n||j||<|S)zRegister class with type name. Args: register_class : class need to register module_type : module type name, default ModuleType.COMMON alias : class name Nr()r.r*r/r-r+rrr register_cls`s  zMindFormerRegister.register_clscCs,|s||jvS||jvo&||j|v}|S)zDetermine whether class name is in the current type group. Args: module_type : Module type class_name : class name Returns: True/False )r)get)r.r/r+ registeredrrris_existps  zMindFormerRegister.is_existcCsD|||std|||s.td||j||}|S)zGet class Args: module_type : Module type class_name : class name Returns: register_class DCan't find class type {} class name {} in class registryz!Can't find class. class type = {})r4 ValueErrorformatr)r2)r.r/r+r*rrrget_clss zMindFormerRegister.get_clsc Cs,t|tstdt|d|vr4td||t|tsX|dusXtdt||}|dur|D]&\}}||vr|||qp|||<qp| d}t|t r| ||}n t |r|}ntdt|z|fi|WSty&} z"t| d|j| WYd} ~ n d} ~ 00dS)a%Get instance. Args: cfg (dict) : Config dict. It should at least contain the key "type". module_type : module type default_args (dict, optional) : Default initialization arguments. Returns: obj : The constructed object. z Cfg must be a Config, but got {}typezA`cfg` or `default_args` must contain the key "type",but got {} {}Nz-default_args must be a dict or Nonebut got {}r5{}: {}) isinstancedict TypeErrorr7r9KeyErrorcopyitems setdefaultpopstrr8inspectisclassr6 Exceptionr) r.cfgr/ default_argsargskvobj_typeobj_clserrrget_instance_from_cfgsD      z(MindFormerRegister.get_instance_from_cfgc Ks|durtdt|tr(|||}n t|r8|}ntdt|z|fi|WSty}z"t|d|j |WYd}~n d}~00dS)zGet instance. Args: module_type : module type class_name : class type Returns: object : The constructed object NzClass name cannot be None.zECan't find class type {} class name {} in class registry.r:) r6r;rCr8rDrEr7r9rFr)r.r/r+kwargsrMrNrrr get_instances   zMindFormerRegister.get_instance)N)N)rrr r!rr) classmethodrr&r0r1r4r8rOrQrrrrr'8s  .r')r!rDrr'rrrrs"pengcheng.mind-test/mindformers/tools/register/config.py000066400000000000000000000254351471400134000240520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Transformer-Config dict parse module """ import os import copy import argparse from argparse import Action from collections import OrderedDict import yaml BASE_CONFIG = 'base_config' class MindFormerConfig(dict): """ A Config class is inherit from dict. Config class can parse arguments from a config file of yaml or a dict. Args: args (list) : config filenames kwargs (dict) : config dictionary list Example: test.yaml: a:1 >>> cfg = MindFormerConfig('./test.yaml') >>> cfg.a 1 >>> cfg = MindFormerConfig(**dict(a=1, b=dict(c=[0,1]))) >>> cfg.b {'c': [0, 1]} """ def __init__(self, *args, **kwargs): super(MindFormerConfig, self).__init__() cfg_dict = {} # load from file for arg in args: if isinstance(arg, str): if arg.endswith('yaml') or arg.endswith('yml'): raw_dict = MindFormerConfig._file2dict(arg) cfg_dict.update(raw_dict) # load dictionary configs if kwargs is not None: cfg_dict.update(kwargs) MindFormerConfig._dict2config(self, cfg_dict) def __getattr__(self, key): """Get a object attr by its `key` Args: key (str) : the name of object attr. Returns: attr of object that name is `key` """ if key not in self: return None return self[key] def __setattr__(self, key, value): """Set a object value `key` with `value` Args: key (str) : The name of object attr. value : the `value` need to set to the target object attr. """ self[key] = value def __delattr__(self, key): """Delete a object attr by its `key`. Args: key (str) : The name of object attr. """ del self[key] def __deepcopy__(self, memo=None): """Deep copy operation on arbitrary MindFormerConfig objects. Args: memo (dict) : Objects that already copied. Returns: MindFormerConfig : The deep copy of the given MindFormerConfig object. """ config = MindFormerConfig() for key in self.keys(): config.__setattr__(copy.deepcopy(key, memo), copy.deepcopy(self.__getattr__(key), memo)) return config def merge_from_dict(self, options): """Merge options into config file. Args: options (dict): dict of configs to merge from Examples: >>> options = {'model.arch': 'simmim'} >>> cfg = MindFormerConfig(**dict(model=dict(backbone=dict(type='vit')))) >>> cfg.merge_from_dict(options) """ option_cfg_dict = {} for full_key, value in options.items(): d = option_cfg_dict key_list = full_key.split('.') for sub_key in key_list[:-1]: d.setdefault(sub_key, MindFormerConfig()) d = d[sub_key] sub_key = key_list[-1] d[sub_key] = value merge_dict = MindFormerConfig._merge_a_into_b(option_cfg_dict, self) MindFormerConfig._dict2config(self, merge_dict) @staticmethod def _merge_a_into_b(a, b): """Merge dict ``a`` into dict ``b`` Values in ``a`` will overwrite ``b`` Args: a (dict) : The source dict to be merged into b. b (dict) : The origin dict to be fetch keys from ``a``. Returns: dict: The modified dict of ``b`` using ``a``. """ b = b.copy() for k, v in a.items(): if isinstance(v, dict) and k in b: b[k] = MindFormerConfig._merge_a_into_b(v, b[k]) else: b[k] = v return b @staticmethod def _file2dict(filename=None): """Convert config file to dictionary. Args: filename (str) : config file. """ if filename is None: raise NameError('This {} cannot be empty.'.format(filename)) filepath = os.path.realpath(filename) with open(filepath, encoding='utf-8') as fp: cfg_dict = ordered_yaml_load(fp, yaml_loader=yaml.FullLoader) # Load base config file. if BASE_CONFIG in cfg_dict: cfg_dir = os.path.dirname(filename) base_filenames = cfg_dict.pop(BASE_CONFIG) base_filenames = base_filenames if isinstance( base_filenames, list) else [base_filenames] cfg_dict_list = list() for base_filename in base_filenames: cfg_dict_item = MindFormerConfig._file2dict( os.path.join(cfg_dir, base_filename)) cfg_dict_list.append(cfg_dict_item) base_cfg_dict = dict() for cfg in cfg_dict_list: base_cfg_dict.update(cfg) # Merge config base_cfg_dict = MindFormerConfig._merge_a_into_b(cfg_dict, base_cfg_dict) cfg_dict = base_cfg_dict return cfg_dict @staticmethod def _dict2config(config, dic): """Convert dictionary to config. Args: config : Config object dic (dict) : dictionary Returns: Exceptions: """ if isinstance(dic, dict): for key, value in dic.items(): if isinstance(value, dict): sub_config = MindFormerConfig() dict.__setitem__(config, key, sub_config) MindFormerConfig._dict2config(sub_config, value) else: config[key] = dic[key] class ActionDict(Action): """ Argparse action to split an option into KEY=VALUE from on the first = and append to dictionary. List options can be passed as comma separated values. i.e. 'KEY=Val1,Val2,Val3' or with explicit brackets i.e. 'KEY=[Val1,Val2,Val3]'. """ @staticmethod def _parse_int_float_bool(val): """convert string val to int or float or bool or do nothing.""" try: return int(val) except ValueError: pass try: return float(val) except ValueError: pass if val.upper() in ['TRUE', 'FALSE']: return val.upper == 'TRUE' return val @staticmethod def find_next_comma(val_str): """find the position of next comma in the string. note: '(' and ')' or '[' and']' must appear in pairs or not exist. """ assert (val_str.count('(') == val_str.count(')')) and \ (val_str.count('[') == val_str.count(']')) end = len(val_str) for idx, char in enumerate(val_str): pre = val_str[:idx] if ((char == ',') and (pre.count('(') == pre.count(')')) and (pre.count('[') == pre.count(']'))): end = idx break return end @staticmethod def _parse_value_iter(val): """Convert string format as list or tuple to python list object or tuple object. Args: val (str) : Value String Returns: list or tuple Examples: >>> ActionDict._parse_value_iter('1,2,3') [1,2,3] >>> ActionDict._parse_value_iter('[1,2,3]') [1,2,3] >>> ActionDict._parse_value_iter('(1,2,3)') (1,2,3) >>> ActionDict._parse_value_iter('[1,[1,2],(1,2,3)') [1, [1, 2], (1, 2, 3)] """ # strip ' and " and delete whitespace val = val.strip('\'\"').replace(" ", "") is_tuple = False if val.startswith('(') and val.endswith(')'): is_tuple = True # remove start '(' and end ')' val = val[1:-1] elif val.startswith('[') and val.endswith(']'): # remove start '[' and end ']' val = val[1:-1] elif ',' not in val: return ActionDict._parse_int_float_bool(val) values = [] len_of_val = len(val) while len_of_val > 0: comma_idx = ActionDict.find_next_comma(val) ele = ActionDict._parse_value_iter(val[:comma_idx]) values.append(ele) val = val[comma_idx + 1:] len_of_val = len(val) if is_tuple: values = tuple(values) return values def __call__(self, parser, namespace, values, option_string=None): options = {} for key_value in values: key, value = key_value.split('=', maxsplit=1) options[key] = self._parse_value_iter(value) setattr(namespace, self.dest, options) def ordered_yaml_load(stream, yaml_loader=yaml.SafeLoader, object_pairs_hook=OrderedDict): """Load Yaml File in Orderedly.""" class OrderedLoader(yaml_loader): pass def _construct_mapping(loader, node): loader.flatten_mapping(node) return object_pairs_hook(loader.construct_pairs(node)) OrderedLoader.add_constructor( yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping) return yaml.load(stream, OrderedLoader) def ordered_yaml_dump(data, stream=None, yaml_dumper=yaml.SafeDumper, object_pairs_hook=OrderedDict, **kwargs): """Dump Dict to Yaml File in Orderedly.""" class OrderedDumper(yaml_dumper): pass def _dict_representer(dumper, data): return dumper.represent_mapping( yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, data.items()) OrderedDumper.add_representer(object_pairs_hook, _dict_representer) return yaml.dump(data, stream, OrderedDumper, **kwargs) def parse_args(): """ Parse arguments from `yaml or yml` config file. Returns: object: argparse object. """ parser = argparse.ArgumentParser("Transformer Config.") parser.add_argument('-c', '--config', type=str, default="", help='Enter the path of the model config file.') return parser.parse_args() pengcheng.mind-test/mindformers/tools/register/register.py000066400000000000000000000156201471400134000244240ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project # https://gitee.com/mindspore/vision/blob/master/mindvision/engine/class_factory.py # ============================================================================ """ Class Register Module For MindFormers.""" import inspect class MindFormerModuleType: """Class module type for vision pretrain""" def __init__(self): pass TRAINER = 'trainer' PIPELINE = 'pipeline' PROCESSOR = 'processor' TOKENIZER = 'tokenizer' DATASET = 'dataset' MASK_POLICY = 'mask_policy' DATASET_LOADER = 'dataset_loader' DATASET_SAMPLER = 'dataset_sampler' TRANSFORMS = 'transforms' ENCODER = 'encoder' MODELS = 'models' MODULES = 'modules' TRANSFORMER = 'transformer' BASE_LAYER = 'base_layer' CORE = 'core' HEAD = 'head' LOSS = 'loss' LR = 'lr' OPTIMIZER = 'optimizer' CONTEXT = 'context' CALLBACK = 'callback' WRAPPER = 'wrapper' METRIC = 'metric' CONFIG = 'config' TOOLS = 'tools' FEATURE_EXTRACTOR = 'feature_extractor' class MindFormerRegister: """ Module class factory for ring-mo. """ def __init__(self): pass registry = {} @classmethod def register(cls, module_type=MindFormerModuleType.TOOLS, alias=None): """Register class into registry Args: module_type (ModuleType) : module type name, default ModuleType.GENERAL alias (str) : class alias Returns: wrapper """ def wrapper(register_class): """Register-Class with wrapper function. Args: register_class : class need to register Returns: wrapper of register_class """ class_name = alias if alias is not None else register_class.__name__ if module_type not in cls.registry: cls.registry[module_type] = {class_name: register_class} else: cls.registry[module_type][class_name] = register_class return register_class return wrapper @classmethod def register_cls(cls, register_class, module_type=MindFormerModuleType.TOOLS, alias=None): """Register class with type name. Args: register_class : class need to register module_type : module type name, default ModuleType.COMMON alias : class name """ class_name = alias if alias is not None else register_class.__name__ if module_type not in cls.registry: cls.registry[module_type] = {class_name: register_class} else: cls.registry[module_type][class_name] = register_class return register_class @classmethod def is_exist(cls, module_type, class_name=None): """Determine whether class name is in the current type group. Args: module_type : Module type class_name : class name Returns: True/False """ if not class_name: return module_type in cls.registry registered = module_type in cls.registry and class_name in cls.registry.get(module_type) return registered @classmethod def get_cls(cls, module_type, class_name=None): """Get class Args: module_type : Module type class_name : class name Returns: register_class """ if not cls.is_exist(module_type, class_name): raise ValueError("Can't find class type {} class name {} \ in class registry".format(module_type, class_name)) if not class_name: raise ValueError( "Can't find class. class type = {}".format(class_name)) register_class = cls.registry.get(module_type).get(class_name) return register_class @classmethod def get_instance_from_cfg(cls, cfg, module_type=MindFormerModuleType.TOOLS, default_args=None): """Get instance. Args: cfg (dict) : Config dict. It should at least contain the key "type". module_type : module type default_args (dict, optional) : Default initialization arguments. Returns: obj : The constructed object. """ if not isinstance(cfg, dict): raise TypeError( "Cfg must be a Config, but got {}".format(type(cfg)) ) if 'type' not in cfg: raise KeyError( '`cfg` or `default_args` must contain the key "type",' 'but got {}\n{}'.format(cfg, default_args) ) if not (isinstance(default_args, dict) or default_args is None): raise TypeError( 'default_args must be a dict or None' 'but got {}'.format(type(default_args))) args = cfg.copy() if default_args is not None: for k, v in default_args.items(): if k not in args: args.setdefault(k, v) else: args[k] = v obj_type = args.pop('type') if isinstance(obj_type, str): obj_cls = cls.get_cls(module_type, obj_type) elif inspect.isclass(obj_type): obj_cls = obj_type else: raise ValueError("Can't find class type {} class name {} \ in class registry".format(type, obj_type)) try: return obj_cls(**args) except Exception as e: raise type(e)('{}: {}'.format(obj_cls.__name__, e)) @classmethod def get_instance(cls, module_type=MindFormerModuleType.TOOLS, class_name=None, **kwargs): """Get instance. Args: module_type : module type class_name : class type Returns: object : The constructed object """ if class_name is None: raise ValueError("Class name cannot be None.") if isinstance(class_name, str): obj_cls = cls.get_cls(module_type, class_name) elif inspect.isclass(class_name): obj_cls = class_name else: raise ValueError("Can't find class type {} class name {} \ in class registry.".format(type, class_name)) try: return obj_cls(**kwargs) except Exception as e: raise type(e)('{}: {}'.format(obj_cls.__name__, e)) pengcheng.mind-test/mindformers/tools/transform_ckpt.py000066400000000000000000000062221471400134000240060ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """transform ckpt""" import os import argparse import mindspore as ms def get_strategy(startegy_path, rank_id=None): """Merge strategy if strategy path is dir Args: startegy_path (str): The path of stategy. rank_id (int): The rank id of device. Returns: None or strategy path """ if not startegy_path: return None assert os.path.exists(startegy_path), f'{startegy_path} not found!' if os.path.isfile(startegy_path): return startegy_path if os.path.isdir(startegy_path): if rank_id: merge_path = os.path.join(startegy_path, f'merged_ckpt_strategy_{rank_id}.ckpt') else: merge_path = os.path.join(startegy_path, f'merged_ckpt_strategy.ckpt') if os.path.exists(merge_path): os.remove(merge_path) ms.merge_pipeline_strategys(startegy_path, merge_path) return merge_path return None if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('--src_ckpt_strategy', default="", help='path of src ckpt strategy') parser.add_argument('--dst_ckpt_strategy', default="", help='path of dst ckpt strategy') parser.add_argument('--src_ckpt_dir', default="", type=str, help='path of src ckpt') parser.add_argument('--dst_ckpt_dir', default="", type=str, help='path where to save dst ckpt') parser.add_argument('--prefix', default='checkpoint_', type=str, help='prefix of transformed checkpoint') args = parser.parse_args() src_ckpt_strategy = get_strategy(args.src_ckpt_strategy) dst_ckpt_strategy = get_strategy(args.dst_ckpt_strategy) src_ckpt_dir = args.src_ckpt_dir dst_ckpt_dir = args.dst_ckpt_dir prefix = args.prefix assert os.path.exists(args.src_ckpt_dir), f'{args.src_ckpt_dir} not found!' print(f"src_ckpt_strategy: {src_ckpt_strategy}") print(f"dst_ckpt_strategy: {dst_ckpt_strategy}") print(f"src_ckpt_dir: {src_ckpt_dir}") print(f"dst_ckpt_dir: {dst_ckpt_dir}") print(f"prefix: {prefix}") print("......Start transform......") ms.transform_checkpoints(src_ckpt_dir, dst_ckpt_dir, prefix, src_ckpt_strategy, dst_ckpt_strategy) print("......Transform succeed!......") pengcheng.mind-test/mindformers/tools/utils.py000066400000000000000000000261511471400134000221150ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Utils For Tools.""" import json import os from typing import Dict, List, Tuple, Union from multiprocessing import Process try: import fcntl except ImportError: fcntl = None import numpy as np from mindspore import Tensor from mindspore import context PARALLEL_MODE = {'DATA_PARALLEL': context.ParallelMode.DATA_PARALLEL, 'SEMI_AUTO_PARALLEL': context.ParallelMode.SEMI_AUTO_PARALLEL, 'AUTO_PARALLEL': context.ParallelMode.AUTO_PARALLEL, 'HYBRID_PARALLEL': context.ParallelMode.HYBRID_PARALLEL, 'STAND_ALONE': context.ParallelMode.STAND_ALONE, 0: context.ParallelMode.DATA_PARALLEL, 1: context.ParallelMode.SEMI_AUTO_PARALLEL, 2: context.ParallelMode.AUTO_PARALLEL, 3: context.ParallelMode.HYBRID_PARALLEL} MODE = {'PYNATIVE_MODE': context.PYNATIVE_MODE, 'GRAPH_MODE': context.GRAPH_MODE, 0: context.GRAPH_MODE, 1: context.PYNATIVE_MODE} SAVE_WORK_PATH = '/cache/ma-user-work/rank_{}' MA_OUTPUT_ROOT = '/cache/ma-user-work' DEBUG_INFO_PATH = '/cache/debug' PROFILE_INFO_PATH = '/cache/profile' PLOG_PATH = '/root/ascend/log' LOCAL_DEFAULT_PATH = os.getenv("LOCAL_DEFAULT_PATH", './output') LAST_TRANSFORM_LOCK_PATH = "/tmp/last_transform_done.lock" RANK_FINISH_PATH = "/cache/rank_*_finish.txt" _PROTOCOL = 'obs' _PROTOCOL_S3 = 's3' def check_in_modelarts(): """Check if the training is on modelarts. Returns: (bool): If it is True, it means ModelArts environment. """ # 'KUBERNETES_PORT' in os.environ or \ return 'MA_LOG_DIR' in os.environ or \ 'MA_JOB_DIR' in os.environ or \ 'MA_LOCAL_LOG_PATH' in os.environ or \ 'S3_ACCESS_KEY_ID' in os.environ or \ 'S3_SECRET_ACCESS_KEY' in os.environ or \ 'BATCH_GROUP_NAME' in os.environ or \ 'MA_LOCAL_LOG_PATH' in os.environ class Validator: """validator for checking input parameters""" @staticmethod def check_type(arg_value, arg_type): """Check int.""" if not isinstance(arg_value, arg_type): raise TypeError('{} should be {} type, but get {}'.format(arg_value, arg_type, type(arg_value))) @staticmethod def is_obs_url(url): """Check obs url.""" return url.startswith(_PROTOCOL + '://') or url.startswith(_PROTOCOL_S3 + '://') def check_obs_url(url): """Check obs url.""" if not isinstance(url, str): raise TypeError('remote_save_url type should be a str, but get {}, ' 'please check your remote_save_url config'.format(type(url))) if not (url.startswith(_PROTOCOL + '://') or url.startswith(_PROTOCOL_S3 + '://')): raise TypeError('remote_save_url should be start with obs:// or s3://, ' 'but get {}, please check your remote_save_url config'.format(url)) return True def check_list(var_name: str, list_var: Union[Tuple, List], num: int): """Checks the legitimacy of elements within a node or device list. Args: var_name (str): The Name of variable need to check. list_var (tuple or list): Variables in list format. num (int): The number of nodes or devices. Returns: None """ for value in list_var: if value >= num: raise ValueError('The index of the {} needs to be less than the number of nodes {}.'.format(var_name, num)) def format_path(path): """Check path.""" return os.path.realpath(path) def sync_trans(f): """Asynchronous transport decorator.""" try: def wrapper(*args, **kwargs): pro = Process(target=f, args=args, kwargs=kwargs) pro.start() return pro return wrapper except Exception as e: raise e def get_output_root_path(): """get default output path in local/AICC.""" if check_in_modelarts(): return MA_OUTPUT_ROOT return LOCAL_DEFAULT_PATH def get_output_subpath(sub_class, rank_id=0, append_rank=True): """get output store path for sub output class.""" Validator.check_type(sub_class, str) root_path = get_output_root_path() directory = os.path.join(root_path, sub_class) if append_rank: directory = os.path.join(directory, 'rank_{}'.format(rank_id)) return format_path(directory) def set_remote_save_url(remote_save_url): check_obs_url(remote_save_url) os.environ.setdefault('REMOTE_SAVE_URL', remote_save_url) def get_remote_save_url(): return os.environ.get('REMOTE_SAVE_URL', None) def get_net_outputs(params): """Get network outputs.""" if isinstance(params, (tuple, list)): if isinstance(params[0], Tensor) and isinstance(params[0].asnumpy(), np.ndarray): params = params[0] elif isinstance(params, Tensor) and isinstance(params.asnumpy(), np.ndarray): params = np.mean(params.asnumpy()) return params def get_rank_info() -> Tuple[int, int]: """Get rank_info from environment variables. Returns: rank_id (int): Rank id. rand_size (int): The number of rank. """ rank_id = int(os.getenv('RANK_ID', '0')) rank_size = int(os.getenv('RANK_SIZE', '1')) return rank_id, rank_size def get_num_nodes_devices(rank_size: int) -> Tuple[int, int]: """Derive the number of nodes and devices based on rank_size. Args: rank_size (int): rank size. Returns: num_nodes (int): number of nodes. num_devices (int): number of devices. """ if rank_size in (2, 4, 8): num_nodes = 1 num_devices = rank_size else: num_nodes = rank_size // 8 num_devices = 8 return num_nodes, num_devices class Const: """Const.""" def __setattr__(self, key, value): if key in self.__dict__: raise PermissionError('Can not change const {0}.'.format(key)) if not key.isupper(): raise ValueError('Const name {0} is not all uppercase.'.format(key)) self.__dict__[key] = value def generate_rank_list(stdout_nodes: Union[List, Tuple], stdout_devices: Union[List, Tuple]): """Generate a list of the ranks to output the log. Args: stdout_nodes (list or tuple): The compute nodes that will output the log to stdout. stdout_devices (list or tuple): The compute devices that will output the log to stdout. Returns: rank_list (list): A list of the ranks to output the log to stdout. """ rank_list = [] for node in stdout_nodes: for device in stdout_devices: rank_list.append(8 * node + device) return rank_list def convert_nodes_devices_input(var: Union[List, Tuple, Dict[str, int], None], num: int) -> Union[List, Tuple]: """Convert node and device inputs to list format. Args: var (list[int] or tuple[int] or dict[str, int] or optional): The variables that need to be converted to the format. num (str): The number of nodes or devices. Returns: var (list[int] or tuple[int]): A list of nodes or devices. """ if var is None: var = tuple(range(num)) elif isinstance(var, dict): var = tuple(range(var['start'], var['end'])) return var def str2bool(b): """String convert to Bool.""" if b.lower() in ["false"]: output = False elif b.lower() in ["true"]: output = True else: raise Exception("Invalid Bool Value") return output def count_params(net): """Count number of parameters in the network Args: net (mindspore.nn.Cell): Mindspore network instance Returns: total_params (int): Total number of trainable params """ total_params = [np.prod(param.shape) for param in net.trainable_params()] return sum(total_params) // 1000000 def try_sync_file(file_name): """If the file is still downloading, we need to wait before the file finished downloading""" if fcntl: with open(file_name, 'r') as fp: fcntl.flock(fp.fileno(), fcntl.LOCK_EX) def is_version_le(current_version, base_version): """ return current_version <= base_version. Check whether the current version is lower than or equal to the base version. For example: for current_version: 1.8.1, base_version: 1.11.0, it return True. """ version_split_char = '.' if version_split_char not in base_version or version_split_char not in current_version: raise ValueError("The version string will contain the `.`." "For example, current_version 1.8.1, base_version: 1.11.0.") for x, y in zip(current_version.split(version_split_char), base_version.split(version_split_char)): if not x.isdigit() or not y.isdigit(): continue if int(x) != int(y): return int(x) <= int(y) return True def is_version_ge(current_version, base_version): """ return current_version >= base_version. Check whether the current version is higher than or equal to the base version. for current_version: 1.8.1, base_version: 1.11.0, it return False. """ version_split_char = '.' if version_split_char not in base_version or version_split_char not in current_version: raise ValueError("The version string will contain the `.`." "For example, current_version 1.8.1， base_version: 1.11.0.") for x, y in zip(current_version.split(version_split_char), base_version.split(version_split_char)): if not x.isdigit() or not y.isdigit(): continue if int(x) != int(y): return int(x) >= int(y) return True def parse_value(value): """ parse value from command line. handles with int, float, bool, string, list and dict. """ def isint(x): try: a = float(x) b = int(a) except (TypeError, ValueError): return False else: return a == b def isfloat(x): try: float(x) except (TypeError, ValueError): return False else: return True def isbool(x): return x in ["True", "False"] def isjson(x): try: json.loads(x) except json.decoder.JSONDecodeError: return False else: return True if isint(value): return int(value) if isfloat(value): return float(value) if isbool(value): return value == "True" if isjson(value): return json.loads(value) return value pengcheng.mind-test/mindformers/trainer/000077500000000000000000000000001471400134000207025ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/.ipynb_checkpoints/000077500000000000000000000000001471400134000244735ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/.ipynb_checkpoints/base_trainer-checkpoint.py000066400000000000000000001266151471400134000316430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Base Trainer.""" import math import os import shutil import time from pprint import pprint from functools import partial from typing import Optional, Union, List import numpy as np from PIL.Image import Image import mindspore as ms from mindspore import Tensor, DatasetHelper from mindspore.train.model import Model from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn.wrap.cell_wrapper import _VirtualDatasetCell from mindspore.nn import TrainOneStepCell, Optimizer, Cell, \ PipelineCell, MicroBatchInterleaved from mindspore.common import initializer as init from mindformers.mindformer_book import MindFormerBook from mindformers.core import build_lr, build_optim, build_callback, build_metric from mindformers.core.callback.callback import EvalCallBack from mindformers.core.parallel_config import build_parallel_config from mindformers.dataset import build_dataset, check_dataset_config, BaseDataset from mindformers.models import build_model, build_processor, build_tokenizer, \ BaseModel, BaseTokenizer, BaseImageProcessor from mindformers.pipeline import pipeline from mindformers.wrapper import build_wrapper from mindformers.tools.register import MindFormerConfig from mindformers.tools.logger import logger from mindformers.tools.utils import count_params from mindformers.auto_class import AutoModel from .config_args import ConfigArguments from .training_args import TrainingArguments from .utils import check_runner_config, transform_and_load_checkpoint, load_resume_context_from_checkpoint, load_merged_checkpoint from .optimizer_grouped_parameters import get_optimizer_grouped_parameters, get_mup_optimizer_grouped_parameters from .utils import set_seed, check_train_data_loader_type, \ check_eval_data_loader_type, check_optimizer_and_lr_type, check_wrapper_config SUPPORT_TASKS = MindFormerBook().get_trainer_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_PIPELINE_INPUT_DATA = MindFormerBook().get_pipeline_support_input_data_list() CURRENT_PROJECT_PATH = MindFormerBook().get_project_path() DEFAULT_CONFIG_DIR = 'configs' def download_ckpt_from_obs(remote_url, local_url, rank): """ Download the dataset from the obs. src_data_url (Str): should be the dataset path in the obs tgt_data_path (Str): the local dataset path rank (Int): the current rank id """ import moxing as mox # print(f"begin download ckpt from {obs_ckpt_url}") # if not os.path.exists(os.path.join(cache_url, f"rank_{rank}")): # os.makedirs(os.path.join(cache_url, f"rank_{rank}"), exist_ok=True) # if not os.path.exists(cache_url): # os.makedirs(cache_url, exist_ok=True) time.sleep(rank % 4) try: mox.file.copy(src_url=remote_url, dst_url=local_url) except Exception as error: print(f"Error: {error}, download again...") time.sleep(10) mox.file.copy(src_url=remote_url, dst_url=local_url) print(f"Rank {rank} download ckpt {remote_url} succeed!", flush=True) class BaseTrainer: r"""Base Task Trainer. Args: task (str): The task name supported. model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, task: str = None, model_name: str = None): if model_name is None: model_name = "model name unspecified." if task is None: task = "task name unspecified." logger.info("Now Running Task is: %s, Model is: %s", task, model_name) self.model_name = model_name self.task = task self.config = None self.default_task_config = None self.train_dataset = None self.eval_dataset = None self.network = None self.optimizer = None self.image_processor = None self.audio_processor = None self.tokenizer = None self.callbacks = None self.eval_callbacks = None self.model_wrapper = None self.compute_metrics = None self.kwargs = None self.pipeline_task = None if not os.path.exists(os.path.join('.', DEFAULT_CONFIG_DIR)): configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) if os.path.exists(os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR)): mindformers_configs_directory = os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR) # python 3.7 版本不支持dirs_exist_ok入参, python 3.8及以上版本支持 try: # adapt to python 3.8+ # pylint: disable=E1123 shutil.copytree(mindformers_configs_directory, configs_directory, dirs_exist_ok=True) except TypeError: try: # adapt to python 3.7 to avoid dirs_exist_ok=False shutil.copytree(mindformers_configs_directory, configs_directory) except FileExistsError: pass if task not in SUPPORT_TASKS.keys(): logger.warning("Input task name is not in the supported list or unspecified.") if task in SUPPORT_TASKS.keys() and model_name not in SUPPORT_TASKS.get(task).keys(): logger.warning("Input model name is not in the supported list or unspecified.") logger.warning("See the list of supported task and model name: %s", SUPPORT_TASKS) logger.warning("The default model config: %s will now be used for the %s task ", SUPPORT_TASKS.get(self.task).get("common"), task) self.model_name = "common" def set_config(self, config: Optional[Union[dict, str, ConfigArguments, TrainingArguments]] = None, is_full_config: bool = False): """Set the task config for task trainer.""" if config is not None: if is_full_config: self.config = config else: self.setup_task_config() self.config = self._merge_config(config) else: self.setup_task_config() self.config = self.default_task_config build_parallel_config(self.config) self._check_global_batch_size_for_auto_parallel() return self.config def setup_task_config(self): """Setup the default task config.""" task_config = None if self.task in SUPPORT_TASKS.keys() and self.model_name in SUPPORT_TASKS.get(self.task).keys(): task_config = MindFormerConfig(SUPPORT_TASKS.get(self.task).get(self.model_name)) if isinstance(task_config, MindFormerConfig): self.default_task_config = task_config else: logger.warning("If the default config arguments is not specified," "you must define the required model or wrapper, optimizer, and so on" "in the train or evaluate or predict attribute function.") def _check_global_batch_size_for_auto_parallel(self): """Check global batch size in auto parallel mode.""" batch_size = self.config.runner_config.batch_size dp = self.config.parallel_config.data_parallel micro_batch_num = self.config.parallel_config.micro_batch_num micro_batch_interleave_num = self.config.micro_batch_interleave_num parallel_mode = ms.get_auto_parallel_context("parallel_mode") full_batch = ms.get_auto_parallel_context("full_batch") pp = self.get_pipeline_stages() if parallel_mode in ["semi_auto_parallel", "auto_parallel"]: if full_batch: if pp > 1: logger.info("Pipeline parallel was opened: pipeline_stages = %s, full batch is True, " "global batch size will be changed: " "global_batch_size = " "batch_size * data_parallel * micro_batch_num * micro_batch_interleave_num " "= %s * %s * %s * %s = %s).", pp, batch_size, dp, micro_batch_num, micro_batch_interleave_num, batch_size * dp * micro_batch_num * micro_batch_interleave_num) self.config.runner_config.batch_size = \ batch_size * dp * micro_batch_num * micro_batch_interleave_num self._reset_wrapper_for_pipeline_parallel() else: logger.info("The current parallel mode is %s, full batch is True," "so global batch size will be changed: " "global_batch_size = batch_size * data_parallel * micro_batch_interleave_num " "= %s * %s * %s = %s", parallel_mode, batch_size, dp, micro_batch_interleave_num, batch_size * dp * micro_batch_interleave_num) self.config.runner_config.batch_size = batch_size * dp * micro_batch_interleave_num else: if pp > 1: logger.info("Pipeline parallel was opened: pipeline_stages = %s, full batch is False, " "batch size per card will be changed: " "per_batch_size = batch_size * micro_batch_num * micro_batch_interleave_num " "= %s * %s * %s = %s).", pp, batch_size, micro_batch_num, micro_batch_interleave_num, batch_size * micro_batch_num * micro_batch_interleave_num) logger.info( "global_batch_size = per_batch_size * device_num " "= %s * %s = %s", batch_size * micro_batch_num * micro_batch_interleave_num, int(os.getenv('RANK_SIZE', '1')), batch_size * micro_batch_num * micro_batch_interleave_num * int(os.getenv('RANK_SIZE', '1'))) self.config.runner_config.batch_size = \ batch_size * micro_batch_num * micro_batch_interleave_num self._reset_wrapper_for_pipeline_parallel() else: logger.info("The current parallel mode is %s, batch size per card will not be changed: " "batch_size_per_card = %s", parallel_mode, batch_size) logger.info( "global_batch_size = batch_size_per_card * device_num " "= %s * %s = %s", batch_size, int(os.getenv('RANK_SIZE', '1')), batch_size * int(os.getenv('RANK_SIZE', '1'))) self.config.parallel_config.data_parallel = 1 self.config.parallel_config.model_parallel = 1 self.config.parallel_config.pipeline_stage = 1 self.config.parallel_config.optimizer_shard = False self.config.parallel_config.micro_batch_num = 1 logger.info("parallel_config will be change to default config: %s.", self.config.parallel_config) def _reset_wrapper_for_pipeline_parallel(self): """Reset wrapper when pipeline parallel.""" if self.config.runner_wrapper is not None: self.config.runner_wrapper.type = "MFPipelineWithLossScaleCell" \ if self.config.runner_wrapper.type != "MFPipelineWithLossScaleCell" else self.config.runner_wrapper.type self.config.runner_wrapper.micro_batch_num = self.config.parallel_config.micro_batch_num logger.warning( "When using the pipeline parallel mode, " "the MFPipelineWithLossScaleCell class is used by default.") else: logger.info( "When using the pipeline parallel mode, " "because the wrapper class is not specified, " "MindSpore's built-in PipelineCell is used by default") logger.info("PipelineWrapper under evaluate or predict mode will not take effect.") def _reset_dataset_batch_size(self): """Reset dataset batch size according to the global batch size of runner config.""" check_dataset_config(self.config) def _merge_config(self, args): """Merge config from default task config.""" if self.default_task_config is None: logger.warning("default task config is None, you will not be able to merge config parameters.") return args if isinstance(args, dict): self.default_task_config.merge_from_dict(args) elif isinstance(args, str): if not (os.path.realpath(args) and os.path.exists(args)): raise FileNotFoundError(f"config path must be exist, but get {args}.") if not args.endswith(('.yaml', '.yml')): raise ValueError(f"config file must be end with .yaml or .yml, but get {args}") self.default_task_config = MindFormerConfig(args) elif isinstance(args, ConfigArguments): if hasattr(args, 'train_dataset'): check_train_data_loader_type(args, self.default_task_config) if hasattr(args, 'eval_dataset'): check_eval_data_loader_type(args, self.default_task_config) if hasattr(args, 'optimizer'): check_optimizer_and_lr_type(args, self.default_task_config) if hasattr(args, 'runner_wrapper'): check_wrapper_config(args, self.default_task_config) self.default_task_config.merge_from_dict(args.__dict__) elif isinstance(args, TrainingArguments): args.convert_args_to_mindformers_config(self.default_task_config) else: logger.warning( "The type of config parameter to merge is not supported, " "currently supported types are [dict, config_path(str), ConfigArguments, TrainingArguments], " "but get %s", type(args)) return self.default_task_config def create_train_dataset(self, default_args: dict = None): """Create the train dataset for training.""" logger.info(".........Build Dataset From Config..........") self._reset_dataset_batch_size() train_dataset = build_dataset(self.config.train_dataset_task, default_args=default_args) return train_dataset def create_eval_dataset(self, default_args: dict = None): """Create the eval dataset for evaluate.""" logger.info(".........Build Dataset From Config..........") self._reset_dataset_batch_size() # reduce batch size on eval mode, for that micro_batch will not take effect on eval. parallel_mode = ms.get_auto_parallel_context("parallel_mode") if parallel_mode in ["semi_auto_parallel", "auto_parallel"]: self.config.eval_dataset.batch_size = \ self.config.eval_dataset.batch_size // self.config.parallel_config.micro_batch_num self.config.eval_dataset.batch_size = \ self.config.eval_dataset.batch_size // self.config.micro_batch_interleave_num eval_dataset = build_dataset(self.config.eval_dataset_task, default_args=default_args) return eval_dataset def create_network(self, default_args: dict = None): """Create the network for task trainer.""" logger.info(".........Build Network From Config..........") eval_network = build_model(self.config.model, default_args=default_args) network = eval_network micro_batch_interleave_num = self.config.micro_batch_interleave_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = MicroBatchInterleaved(network, micro_batch_interleave_num) return network, eval_network def create_pipeline_network(self, default_args: dict = None): """Create the network of pipeline parallel for task trainer.""" logger.info(".........Build Pipeline Network From Config..........") eval_network = build_model(self.config.model, default_args=default_args) network = eval_network micro_batch_interleave_num = self.config.micro_batch_interleave_num micro_batch_num = self.config.parallel_config.micro_batch_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = PipelineCell(MicroBatchInterleaved(network, micro_batch_interleave_num), micro_size=micro_batch_num) else: network = PipelineCell(network, micro_size=micro_batch_num) network = _VirtualDatasetCell(network) return network, eval_network def create_image_processor(self, default_args: dict = None): """Create the image processor for predict.""" logger.info(".........Build Image Processor From Config..........") self.image_processor = build_processor( self.config.processor.image_processor, default_args=default_args) return self.image_processor def create_tokenizer(self, default_args: dict = None): """Create the tokenizer for task trainer.""" logger.info(".........Build Text Tokenizer From Config..........") self.tokenizer = build_tokenizer( self.config.processor.tokenizer, default_args=default_args) return self.tokenizer def create_optimizer_scheduler(self, network, layer_scale=False): """Create the optimizer for training.""" logger.info(".........Build Optimizer From Config..........") # learning rate scale for multi-nodes training learning_scale = self.config.lr_scale scale_factor = self.config.lr_scale_factor # build learning rate schedule lr_schedule = self.create_lr_scheduler(learning_scale, scale_factor) weight_decay = self.config.optimizer.weight_decay if self.config.optimizer.weight_decay else 0. layer_decay = self.config.layer_decay if self.config.layer_decay else 1.0 if self.config.model.model_config.mup_width_base is not None: ratio = self.config.model.model_config.hidden_size / self.config.model.model_config.mup_width_base group_params = get_mup_optimizer_grouped_parameters(network, weight_decay, ratio, self.config.lr_schedule) else: group_params = get_optimizer_grouped_parameters(network, weight_decay, lr_schedule, layer_scale=layer_scale, layer_decay=layer_decay) if lr_schedule is not None: self.optimizer = build_optim( self.config.optimizer, default_args={"params": group_params, "learning_rate": lr_schedule}) else: if self.config.optimizer.learning_rate is None: raise ValueError("learning_rate must be input") self.config.optimizer.learning_rate = self.learning_rate_scale( self.config.optimizer.learning_rate, scale_factor) \ if learning_scale and scale_factor is not None else self.config.optimizer.learning_rate self.optimizer = build_optim( self.config.optimizer, default_args={"params": group_params}) return self.optimizer def create_lr_scheduler(self, learning_scale: bool = False, scale_factor: int = 256): """Create the learning rate scheduler.""" logger.info(".........Build LR Schedule From Config..........") train_data_size = self.get_train_data_size() if self.config.lr_schedule: warmup_epochs = self.config.lr_schedule.pop("warmup_epochs", None) warmup_ratio = self.config.lr_schedule.pop("warmup_ratio", None) # WSD decay_ratio = self.config.lr_schedule.pop("decay_ratio", None) if not self.config.runner_config.sink_mode: total_steps = int(self.config.runner_config.epochs * train_data_size) else: total_steps = int(self.config.runner_config.epochs * self.config.runner_config.sink_size) if warmup_epochs is not None and warmup_ratio is not None: logger.warning("warmup_epochs and warmup_ratio are set simultaneously," "warmup_ratio takes precedence.") warmup_epochs = None if warmup_epochs is not None: logger.warning("warmup_epochs was set in lr_schedule," "it will multiply the data size to represent the warmup steps") self.config.lr_schedule.warmup_steps = int(warmup_epochs * train_data_size) self.config.lr_schedule.total_steps = total_steps \ if self.config.lr_schedule.total_steps is None or self.config.lr_schedule.total_steps == -1 \ else int(self.config.lr_schedule.total_steps) if warmup_ratio is not None: self.config.lr_schedule.warmup_steps = int(self.config.lr_schedule.total_steps * warmup_ratio) if decay_ratio is not None: #WSD self.config.lr_schedule.decay_last_steps = int(self.config.lr_schedule.total_steps * decay_ratio) self.config.lr_schedule.learning_rate = self.learning_rate_scale( self.config.lr_schedule.learning_rate, scale_factor) \ if learning_scale and scale_factor is not None else self.config.lr_schedule.learning_rate lr_schedule = build_lr(self.config.lr_schedule) return lr_schedule def create_model_wrapper(self, network, optimizer): """Create the model wrapper for training.""" logger.info(".........Build Model Wrapper for Train From Config..........") model_wrapper = build_wrapper(self.config.runner_wrapper, default_args={"network": network, "optimizer": optimizer, "parallel_config": self.config.parallel_config}) return model_wrapper def create_callbacks(self, default_args: dict = None): """Create the callback list for training.""" logger.info(".........Build Callbacks for Train From Config..........") self.callbacks = [] if self.config.profile: self.callbacks.append(self.config.profile_cb) self.callbacks.extend(build_callback(self.config.callbacks, default_args=default_args)) return self.callbacks def create_eval_callbacks(self, default_args: dict = None): """Create the eval callback list for training.""" logger.info(".........Build Callbacks for Evaluate From Config..........") self.eval_callbacks = [] self.eval_callbacks.extend(build_callback(self.config.eval_callbacks, default_args=default_args)) return self.eval_callbacks def create_metrics(self, metric_name: str = None): """Create Metrics For Evaluate or Fit.""" if metric_name is None: metric_name = self.model_name + "_metric" self.compute_metrics = {metric_name: build_metric(self.config.metric)} return self.compute_metrics def count_parameters(self): """Count network parameters number.""" if self.network is not None: logger.info("Network Parameters: %s M.", str(count_params(self.network))) elif self.model_wrapper is not None: logger.info("Network Parameters: %s M.", str(count_params(self.model_wrapper.network))) else: logger.warning("Network is None, parameters incalculable.") def set_seed(self, seed: int = None): """Set seed for training.""" if seed is None: if self.config.seed is None: raise ValueError("seed is not set in config, it is None.") set_seed(self.config.seed) else: set_seed(seed) def set_train_dataset(self, dataset): """Set the attribute of train dataset.""" if dataset is None: raise ValueError("Train dataset is None") self.train_dataset = dataset def set_eval_dataset(self, dataset): """Set the attribute of eval dataset .""" if dataset is None: raise ValueError("Eval dataset is None") self.eval_dataset = dataset def set_network(self, network, is_train: bool = True): """Set the attribute of network.""" if network is None: raise ValueError("network is None") if isinstance(network, (Cell, BaseModel)): network.set_train(is_train) self.network = network def set_model_wrapper(self, model_wrapper): """Set the attribute of model_wrapper.""" if model_wrapper is None: raise ValueError("model wrapper is None") self.model_wrapper = model_wrapper def get_train_data_size(self): """Get train dataset size.""" if self.train_dataset is None: raise NotImplementedError("train dataset is None") return self.train_dataset.get_dataset_size() def get_eval_data_size(self): """Get eval dataset size.""" if self.eval_dataset is None: raise NotImplementedError("train dataset is None") return self.eval_dataset.get_dataset_size() def get_pipeline_stages(self): """Get pipeline stages for task trainer.""" pipeline_stages = ms.get_auto_parallel_context("pipeline_stages") return pipeline_stages def learning_rate_scale(self, base_learning_rate: float = 0., scale_factor: Optional[Union[float, int]] = 256.): """Scale learning rate for training.""" if not isinstance(base_learning_rate, float): raise ValueError(f"learning rate must be float type, but get {type(base_learning_rate)}") if not isinstance(scale_factor, (float, int)): raise ValueError(f"scale_factor must be float or int type, but get {type(scale_factor)}") device_num = int(os.getenv("RANK_SIZE", "1")) per_device_batch_size = self.config.train_dataset.batch_size learning_rate = (base_learning_rate * device_num * per_device_batch_size) / scale_factor return learning_rate def training_process( self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, optimizer: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): """Train or Fine-tune for BaseTrainer in MindFormers.""" self.kwargs = kwargs is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) if config.resume_training and config.load_checkpoint: logger.info(".............Start load resume context from checkpoint..................") rank = int(os.getenv("RANK_ID", "0")) if config.resume_merged_ckpt: logger.info(".............Start load merged checkpoint..................") obs_ckpt_url = os.path.join(config.load_checkpoint, config.restore_ckpt_name_prefix) local_url = f"/cache/checkpoint/{config.restore_ckpt_name_prefix}" config.load_checkpoint = local_url if rank%8==0: download_ckpt_from_obs(obs_ckpt_url, local_url, rank) with open('/cache/checkpoint/download_successed.txt', 'w')as fl: fl.write("") else: while not os.path.isfile('/cache/checkpoint/download_successed.txt'): time.sleep(3) else: download_rank = rank%config.parallel.optimizer_weight_shard_size if config.reuse_ckpt else rank obs_ckpt_url = os.path.join(config.load_checkpoint, f"rank_{download_rank}", f"{config.restore_ckpt_name_prefix.replace('*', str(download_rank))}") local_url = f"/cache/checkpoint/rank_{rank}.ckpt" download_ckpt_from_obs(obs_ckpt_url, local_url, rank) config.load_checkpoint = local_url load_resume_context_from_checkpoint(config) # build dataset logger.info(".........Build Dataset For Train..........") if dataset is None: dataset = self.create_train_dataset() self.set_train_dataset(dataset) check_runner_config(config, dataset) config.runner_config.epochs += config.runner_config.initial_epoch if config.runner_config.sink_mode: epoch_num = math.ceil((config.runner_config.epochs - config.runner_config.initial_epoch) * config.runner_config.sink_size / dataset.get_dataset_size()) # pylint: disable=W0212 dataset._dataset_helper = DatasetHelper(dataset, config.runner_config.sink_mode, config.runner_config.sink_size, epoch_num) # build network logger.info(".........Build Net For Train..........") eval_network = None if network is None and wrapper is None and \ self.model_wrapper is None and self.network is None: # If neither network nor wrapper exists, create a network if self.get_pipeline_stages() > 1: network, eval_network = self.create_pipeline_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) else: network, eval_network = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) elif network is None and wrapper is None and self.network is not None: logger.info(".........Using The Existing Network For Train:: %s", self.network.__class__.__name__) network = self.network if network is not None: self.set_network(network, is_train=True) if eval_network is None: eval_network = network if wrapper is not None: self.set_model_wrapper(wrapper) self.count_parameters() # build optimizer logger.info(".........Build Optimizer For Train..........") if optimizer is None and wrapper is None and self.model_wrapper is None: optimizer = self.create_optimizer_scheduler(network, layer_scale=config.layer_scale) # build model wrapper if wrapper is None and self.model_wrapper is None: logger.info(".........Build Running Wrapper From Config For Train..........") wrapper = self.create_model_wrapper(network, optimizer) elif wrapper is None and self.model_wrapper is not None: logger.info(".........Using The Existing Model Wrapper: %s", self.model_wrapper.__class__.__name__) wrapper = self.model_wrapper # build callback logger.info(".........Build Callbacks For Train..........") if callbacks is None: callbacks = self.create_callbacks(default_args={ "learning_rate": optimizer.learning_rate if optimizer else wrapper.optimizer.learning_rate, "origin_epochs": config.runner_config.origin_epochs, "dataset_size": config.data_size, "micro_batch_interleave_num": config.micro_batch_interleave_num, "micro_batch_num": config.parallel_config.micro_batch_num, "initial_epoch": config.runner_config.initial_epoch, "global_batch_size": self.config.runner_config.batch_size, "device_num": int(os.getenv('RANK_SIZE', '1'))}) # define compute metrics for evaluate in training compute_metrics = None if config.do_eval: compute_metrics = self.create_metrics() # define Model and begin training logger.info(".........Starting Init Train Model..........") if wrapper is not None: model = Model(wrapper, metrics=compute_metrics, eval_network=eval_network) else: model = Model(network, optimizer=optimizer, metrics=compute_metrics, eval_network=eval_network) # resume checkpoint if config.load_checkpoint or config.only_save_strategy: if config.resume_training: logger.info(".............Start resume training from checkpoint..................") if config.resume_merged_ckpt: load_merged_checkpoint(config, model, network, dataset, optimizer=optimizer) else: transform_and_load_checkpoint(config, model, network, dataset, optimizer=optimizer) else: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path transform_and_load_checkpoint(config, model, network, dataset) # build evaluate in training if config.do_eval: logger.info(".........Build Evaluate in Training Callback..........") eval_dataset = kwargs.get('eval_dataset', None) if eval_dataset is None: eval_dataset = self.create_eval_dataset() eval_callback = EvalCallBack( partial( self._evaluate_in_training, model=model, eval_dataset=eval_dataset, ), step_interval=config.eval_step_interval if config.eval_step_interval else 100, epoch_interval=config.eval_epoch_interval if config.eval_epoch_interval else -1, ) callbacks.append(eval_callback) logger.info(".........Starting Training Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) logger.info(".........Model Compiling, Please Wait a Moment...........") model.train(config.runner_config.epochs, dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode, sink_size=config.runner_config.sink_size, initial_epoch=config.runner_config.initial_epoch) logger.info(".........Training Over!.............") def evaluate_process( self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): """Evaluate for BaseTrainer in MindFormers.""" metric_name = kwargs.get("metric_name") is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) # build dataset logger.info(".........Build Dataset For Evaluate..........") if dataset is None: dataset = self.create_eval_dataset() self.set_eval_dataset(dataset) logger.info("Create evaluate dataset finish, dataset size:%d", dataset.get_dataset_size()) # build network eval_network = None if network is None and self.network is None: network, eval_network = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) elif network is None and self.network is not None: logger.info(".........Using The Existing Network For Evaluate: %s", self.network.__class__.__name__) network = self.network self.set_network(network, is_train=False) if eval_network is None: eval_network = network self.count_parameters() # build metric logger.info(".........Build Compute Metrics For Evaluate..........") if compute_metrics is None: compute_metrics = self.create_metrics(metric_name=metric_name) # build callback logger.info(".........Build Callbacks For Evaluate..........") if callbacks is None: callbacks = self.create_eval_callbacks() logger.info(".........Starting Init Evaluate Model..........") model = Model(network, metrics=compute_metrics, eval_network=eval_network) if config.load_checkpoint or config.only_save_strategy: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path logger.info(".............Start load checkpoint for eval..................") transform_and_load_checkpoint(config, model, network, dataset, do_eval=True) logger.info(".........Starting Evaluate Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) output = model.eval(dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode) logger.info('%s = %s', metric_name, str(output)) logger.info(".........Evaluate Over!.............") def predict_process(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, task: str = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseImageProcessor] = None, **kwargs): """Predict for BaseTrainer in MindFormers.""" if not self.pipeline_task: is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) if ms.context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: if task not in ["translation", "text_generation"]: raise SystemExit("Currently distributed predict only support translation and text_generation. " "Process exit!") if config.parallel_config.pipeline_stage > 1: raise SystemExit("Currently distributed predict dose not support pipeline parallel. " "Process exit!") # build network if network is None: network, _ = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) self.set_network(network, is_train=False) network.model_name = kwargs.get("model_name", None) self.count_parameters() if tokenizer is None and config.processor.tokenizer: tokenizer = build_tokenizer(config.processor.tokenizer) if image_processor is None and config.processor.image_processor: image_processor = build_processor(config.processor.image_processor) if audio_processor is None and config.processor.audio_processor: audio_processor = build_processor(config.processor.audio_processor) model = Model(network) if config.load_checkpoint or config.only_save_strategy: if ms.context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: seq_length = config.model.model_config.seq_length infer_data = Tensor(shape=(1, seq_length), dtype=ms.int32, init=init.One()) transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True) else: transform_and_load_checkpoint(config, model, network, None, do_predict=True) self.pipeline_task = pipeline( task=task, model=network, tokenizer=tokenizer, image_processor=image_processor, audio_processor=audio_processor, **kwargs ) top_k = kwargs.pop("top_k", None) if top_k is None: if config.top_k is not None: top_k = config.top_k elif config.model and config.model.model_config.top_k: top_k = config.model.model_config.top_k else: top_k = 1 save_file = kwargs.pop("save_file", None) if save_file is None: if config.save_file is not None: save_file = config.save_file else: save_file = f"{task}_result.txt" if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) output_results = self.pipeline_task(input_data, top_k=top_k) output_info = [] for one_output in output_results: if isinstance(one_output, dict) and "info" in one_output: output_info.append(one_output["info"]) else: output_info.append(one_output) with open(save_file, 'w') as file: file.write(str(output_info)) file.close() logger.info("output result is: %s", str(output_info)) logger.info("output result is saved at: %s", save_file) logger.info(".........Predict Over!.............") return output_results def _evaluate_in_training(self, model: Model, eval_dataset: BaseDataset): origin_phase = model.eval_network.phase model.eval_network.set_train(False) output = model.eval( eval_dataset, dataset_sink_mode=self.config.runner_config.sink_mode ) model.eval_network.set_train(origin_phase) return output optimizer_grouped_parameters-checkpoint.py000066400000000000000000000173541471400134000351170ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/.ipynb_checkpoints# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Grouped parameters of optimizer.""" import json from functools import partial from typing import Optional import copy from mindspore.nn import Cell from mindspore.nn.learning_rate_schedule import LearningRateSchedule from mindformers.models import BaseModel from mindformers.core.lr import LearningRateWiseLayer from mindformers.core import build_lr from mindformers.tools import logger from .utils import check_keywords_in_name def get_optimizer_grouped_parameters(model: Optional[BaseModel] = None, weight_decay: float = 0.0, dynamic_lr_schedule: Optional[LearningRateSchedule] = None, layer_scale: bool = False, layer_decay: float = 1.0): """Get grouped parameters of the network for training.""" if not isinstance(model, (Cell, BaseModel)): raise TypeError(f"model type should be BaseModel, but get {type(model)}") skip_params = {} skip_keywords = {} if hasattr(model, 'no_weight_decay'): skip_params = model.no_weight_decay() logger.info('No weight decay: %s', skip_params) if hasattr(model, 'no_weight_decay_keywords'): skip_keywords = model.no_weight_decay_keywords() logger.info('No weight decay keywords: %s', skip_keywords) decay_parameters_names = [ param.name for param in model.trainable_params() if not (len(param.shape) == 1 or param.name.endswith(".bias") or (param.name in skip_params) or check_keywords_in_name(param.name, skip_keywords)) ] get_layer_id_func = None scales_list = [] if dynamic_lr_schedule is not None: if layer_scale: if model.__class__.__name__ == 'SwinForImageClassification': depths = model.config.depths num_layers = sum(depths) get_layer_id_func = partial(get_swin_layer, num_layers=num_layers + 2, depths=depths) scales_list = list(layer_decay ** i for i in reversed(range(num_layers + 2))) elif model.__class__.__name__ == 'ViTForImageClassification': num_layers = model.config.depth get_layer_id_func = partial(get_vit_layer, num_layers=num_layers + 2) scales_list = list(layer_decay ** i for i in reversed(range(num_layers + 2))) else: logger.warning("Now only support VitModel and SwinModel," "if use dynamic_lr_schedule and layer_scale, they will be reset and invalid.") layer_scale = False dynamic_lr_schedule = None else: logger.warning("dynamic_lr_schedule will be reset and invalid when layer_scale is False.") dynamic_lr_schedule = None parameter_group_names = {} parameter_group_vars = {} scale = 1. for param in model.trainable_params(): if param.name in decay_parameters_names: group_name = 'decay' else: group_name = 'no_decay' weight_decay = 0. if get_layer_id_func is not None: layer_id = get_layer_id_func(param.name) group_name = "layer_%d_%s" % (layer_id, group_name) if scales_list is not None: scale = scales_list[layer_id] if group_name not in parameter_group_names: parameter_group_names[group_name] = { "weight_decay": weight_decay, "params": [], } parameter_group_vars[group_name] = { "weight_decay": weight_decay, "params": [], } if isinstance(dynamic_lr_schedule, LearningRateSchedule): if layer_scale: parameter_group_vars[group_name]["lr"] = LearningRateWiseLayer(dynamic_lr_schedule, scale) else: parameter_group_vars[group_name]["lr"] = dynamic_lr_schedule parameter_group_vars[group_name]["params"].append(param) parameter_group_names[group_name]["params"].append(param.name) param_groups = json.dumps(parameter_group_names, indent=2) logger.info("Param groups = %s", param_groups) return list(parameter_group_vars.values()) def get_mup_optimizer_grouped_parameters(model: Optional[BaseModel] = None, weight_decay: float = 0.0, mup_ratio: float = 0.0, lr_schedule_config = None): """Get grouped parameters of the network for training.""" if not isinstance(model, (Cell, BaseModel)): raise TypeError(f"model type should be BaseModel, but get {type(model)}") def find_in_name(name, key_list): for k in key_list: if name.find(k) >= 0: return True return False middle_weight_key_list = [ "attention.wo", "attention.wq", "attention.wk", "attention.wv", "feed_forward", "lm_head" ] parameter_group_vars = {} middle_weight_group = {"params": []} # mlp and attention for param in model.trainable_params(): if find_in_name(param.name, middle_weight_key_list): middle_weight_group["params"].append(param) middle_lr_config = copy.deepcopy(lr_schedule_config) middle_lr_config.learning_rate = middle_lr_config.learning_rate / mup_ratio middle_lr_config.lr_middle = middle_lr_config.lr_middle / mup_ratio middle_lr_config.lr_end = middle_lr_config.lr_end / mup_ratio middle_lr = build_lr(middle_lr_config) middle_weight_group["lr"] = middle_lr middle_weight_group["weight_decay"] = weight_decay parameter_group_vars["middle_output"] = middle_weight_group norm_key_list = [ "norm", "embedding" ] norm_weight_group = {"params": []} for param in model.trainable_params(): if find_in_name(param.name, norm_key_list): norm_weight_group["params"].append(param) norm_weight_group["weight_decay"] = 0.0 parameter_group_vars["norm_key"] = norm_weight_group return list(parameter_group_vars.values()) def get_vit_layer(name, num_layers): """get vit layer""" if name.endswith(("cls_tokens", "mask_tokens", "pos_embed")): layer_num = 0 elif name.startswith("vit.patch_embed"): layer_num = 0 elif name.startswith("vit.blocks"): layer_id = int(name.split('.')[2]) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num def get_swin_layer(name, num_layers, depths): """get swin layer""" if name in ("encoder.mask_token",): layer_num = 0 elif name.startswith("encoder.patch_embed"): layer_num = 0 elif name.startswith("encoder.layers"): layer_id = int(name.split('.')[2]) block_id = name.split('.')[4] if block_id in ('reduction', 'norm'): layer_num = sum(depths[:layer_id + 1]) else: layer_id = sum(depths[:layer_id]) + int(block_id) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num pengcheng.mind-test/mindformers/trainer/.ipynb_checkpoints/trainer-checkpoint.py000066400000000000000000001504651471400134000306510ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Trainer API For Import.""" import os import shutil from collections import OrderedDict from typing import List, Optional, Union import numpy as np from PIL.Image import Image import mindspore as ms from mindspore import Tensor from mindspore.common import set_seed from mindspore import load_checkpoint, load_param_into_net from mindspore.nn import TrainOneStepCell, Optimizer, Cell, \ PipelineCell, MicroBatchInterleaved from mindspore.nn.wrap.cell_wrapper import _VirtualDatasetCell from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.dataset.engine.datasets import BatchDataset, RepeatDataset from mindformers.core.parallel_config import build_parallel_config from mindformers.dataset import build_dataset, build_dataset_loader, \ check_dataset_config, BaseDataset from mindformers.mindformer_book import MindFormerBook from mindformers.models import build_model, BaseModel, BaseImageProcessor, \ BaseTokenizer, BaseAudioProcessor from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerConfig from mindformers.tools.register.config import ordered_yaml_dump from .build_trainer import build_trainer from .config_args import ConfigArguments from .training_args import TrainingArguments from .utils import check_train_data_loader_type, check_eval_data_loader_type, \ check_optimizer_and_lr_type, check_wrapper_config, config2dict __all__ = ['Trainer'] SUPPORT_TASKS = MindFormerBook().get_trainer_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_PIPELINE_INPUT_DATA = MindFormerBook().get_pipeline_support_input_data_list() CURRENT_PROJECT_PATH = MindFormerBook().get_project_path() DEFAULT_CHECKPOINT_DIR = 'checkpoint' DEFAULT_CONFIG_DIR = 'configs' class Trainer: r""" Trainer package to train\evaluate\predict class. The trainer interface is used to quickly start training, evaluation and predict for integrated tasks. It also allows users to customize the model, optimizer, dataset, tokenizer, processor, train_one_step, callback, and metric. Args: args (Optional[Union[str, dict, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It support yaml path or config dict or ConfigArguments class. Default: None. task (str): The task name supported. Please refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: 'general'. model (Optional[Union[str, Cell, BaseModel]]): The network for trainer. It support model name supported or BaseModel or MindSpore Cell class. Supported model name can refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: None. model_name (Optional[Union[str]]): The model name supported. Default: None. train_dataset (Optional[Union[str, BaseDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. eval_dataset (Optional[Union[str, BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for text preprocessing. It support BaseTokenizer class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. audio_processor (Optional[BaseAudioProcessor]): The processor for audio preprocessing. It support BaseAudioProcessor class. Default: None. optimizers (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. eval_callbacks (Optional[Union[Callback, List[Callback]]]): The evaluate callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. save_config (bool): Save current the config of task. Default: False. Raises: KeyError: If 'task' or 'model' not in supported trainer. Examples: >>> from mindformers import Trainer >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index] ... ... def __len__(self): ... return len(self._data) >>> #1) input task name and model name to init trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train') >>> #2) input config to init trainer >>> from mindformers.trainer.config_args import ConfigArguments, OptimizerConfig, \ ... RunnerConfig, LRConfig, WrapperConfig >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindspore.train.callback import LossMonitor >>> runner_config = RunnerConfig(epochs=10, batch_size=2, image_size=224) >>> lr_schedule_config = LRConfig(lr_type='WarmUpLR', learning_rate=0.001, warmup_steps=10) >>> optim_config = OptimizerConfig(optim_type='Adam', beta1=0.009, learning_rate=lr_schedule_config) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper_config = WrapperConfig(wrapper_type='TrainOneStepWithLossScaleCell', scale_sense=loss_scale) >>> dataset = GeneratorDataset(source=MyDataLoader(), column_names='image') >>> dataset = dataset.batch(batch_size=2) >>> config = ConfigArguments(seed=2022, runner_config=runner_config, ... optimizer=optim_config, runner_wrapper=wrapper_config) >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... args=config, train_dataset=dataset) >>> #3) input instance to init trainer >>> from mindformers.models import ViTForImageClassification >>> vit_model_with_loss = ViTForImageClassification() >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=vit_model_with_loss.trainable_params()) >>> loss_cb = LossMonitor(per_print_times=2) >>> callbacks = [loss_cb] >>> task_trainer = Trainer(task='image_classification', ... model=vit_model_with_loss, ... args=config, ... optimizers=optimizer, ... train_dataset=dataset, ... callbacks=callbacks) """ def __init__(self, args: Optional[Union[str, dict, ConfigArguments, TrainingArguments]] = None, task: Optional[str] = 'general', model: Optional[Union[str, Cell, BaseModel]] = None, model_name: Optional[Union[str]] = None, train_dataset: Optional[Union[str, BaseDataset]] = None, eval_dataset: Optional[Union[str, BaseDataset]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, optimizers: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, pet_method: Optional[str] = '', callbacks: Optional[Union[Callback, List[Callback]]] = None, eval_callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, save_config: bool = False, **kwargs): self.task = task self.model = model self.model_name = model_name self.pet_method = pet_method self.train_dataset = train_dataset self.eval_dataset = eval_dataset self.optimizers = optimizers self.wrapper = wrapper self.tokenizer = tokenizer self.image_processor = image_processor self.audio_processor = audio_processor self.callbacks = callbacks self.eval_callbacks = eval_callbacks self.compute_metrics = compute_metrics self.default_checkpoint_name_or_path = None self.configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) self.kwargs = kwargs if not os.path.exists(os.path.join('.', DEFAULT_CONFIG_DIR)): configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) if os.path.exists(os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR)): mindformers_configs_directory = os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR) # python 3.7 版本不支持dirs_exist_ok入参, python 3.8及以上版本支持 try: # adapt to python 3.8+ # pylint: disable=E1123 shutil.copytree(mindformers_configs_directory, configs_directory, dirs_exist_ok=True) except TypeError: try: # adapt to python 3.7 shutil.copytree(mindformers_configs_directory, configs_directory) except FileExistsError: pass if wrapper is not None: if model is not None: logger.warning( 'wrapper has existed, input model invalid, it should be include in wrapper.') if optimizers is not None: logger.warning( 'wrapper has existed, input optimizers invalid, it should be include in wrapper.') assert task in SUPPORT_TASKS.keys(), \ f"task name must be in {SUPPORT_TASKS.keys()}, but get {task}." if isinstance(model, str): model = model + '_{}'.format(pet_method) if pet_method else model assert model in SUPPORT_MODEL_NAMES, \ f"model must be in {SUPPORT_MODEL_NAMES} when model's type is string, but get {model}." self.model_name = model self.model = None if isinstance(self.model, (Cell, BaseModel)): logger.info("The model instance has been entered, " "and the model will not be created from model_config") if pet_method: logger.warning("pet_method is not valid when a model instance is passed in." "Currently, only part of the model in MindFormers is supported." "Please pass in the model keyword") self.is_model_instance = True else: self.is_model_instance = False if self.is_model_instance and self.model_name is None: logger.warning( "Recognizing that a model instance is sent and model_name is None,") logger.warning( "it is recommended to select a model configuration that corresponds " "to the support of MindFormers based on the instance model and set model_name.") logger.warning( "Otherwise, they will default to a general configuration." "You are advised to pass instances such as optimizers, metric, tokenizer, and processor") self.model_name = 'common' task_config = MindFormerConfig(SUPPORT_TASKS.get(self.task).get(self.model_name)) if self.model_name == "common": if self.model is not None: task_config.trainer.model_name = self.model.__class__.__name__ if self.wrapper is not None: task_config.trainer.model_name = self.wrapper.network.__class__.__name__ if args is None: self.config = task_config else: if isinstance(args, dict): task_config.merge_from_dict(args) elif isinstance(args, str): assert os.path.realpath(args) and os.path.exists(args), \ f"config path must be exist, but get {args}." assert args.endswith(('.yaml', '.yml')), \ f"config file must be end with .yaml or .yml, but get {args}" task_config = MindFormerConfig(args) elif isinstance(args, ConfigArguments): if hasattr(args, 'train_dataset'): check_train_data_loader_type(args, task_config) if hasattr(args, 'eval_dataset'): check_eval_data_loader_type(args, task_config) if hasattr(args, 'optimizer'): check_optimizer_and_lr_type(args, task_config) if hasattr(args, 'runner_wrapper'): check_wrapper_config(args, task_config) task_config.merge_from_dict(args.__dict__) elif isinstance(args, TrainingArguments): logger.warning( "When using the TrainingArguments class, " "its arguments will override the default config configuration.") args.convert_args_to_mindformers_config(task_config) self.config = task_config if save_config: self.save_config_to_yaml(self.config) logger.info("save running config success of %s_new.", task_config.trainer.model_name.lower()) # check dataset config if isinstance(train_dataset, str): assert os.path.exists(train_dataset), \ f"train dataset path must be exist, but get {train_dataset}." self.config.train_dataset.data_loader.dataset_dir = train_dataset self.train_dataset = None if isinstance(eval_dataset, str): assert os.path.exists(eval_dataset), \ f"eval dataset path must be exist, but get {eval_dataset}." self.config.eval_dataset.data_loader.dataset_dir = eval_dataset self.eval_dataset = None if tokenizer is not None: if self.config.train_dataset is not None: self.config.train_dataset.tokenizer = tokenizer if self.config.eval_dataset is not None: self.config.eval_dataset.tokenizer = tokenizer check_dataset_config(self.config) # build parallel config self.rank_id = int(os.getenv("RANK_ID", "0")) self.device_num = int(os.getenv("RANK_SIZE", "1")) self.config.rank_id = self.rank_id self.config.device_num = self.device_num self.is_set_parallel_config = False self.is_set_moe_config = False self.is_set_recompute_config = False # set seed if self.config.seed and \ ms.context.get_auto_parallel_context("parallel_mode") \ not in ["semi_auto_parallel", "auto_parallel"]: set_seed(self.config.seed) np.random.seed(self.config.seed) # set output directory os.environ.setdefault("LOCAL_DEFAULT_PATH", self.config.output_dir) # build task trainer self.trainer = build_trainer(self.config.trainer) if self.trainer is None: raise ModuleNotFoundError("config must be contain 'trainer' key, but get None.") def train(self, train_checkpoint: Optional[Union[str, bool]] = False, resume_training: Optional[bool] = None, auto_trans_ckpt: Optional[bool] = None, do_eval: bool = False, **kwargs): r"""Train task for Trainer. This function is used to train or fine-tune the network. Args: train_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It supports real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. resume_training (bool): Whether to perform resume training. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if resume_or_finetune_from_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/eval') >>> # 1) default train task to reproduce model. >>> task_trainer.train() >>> # 2) eval network when train task to reproduce model. >>> task_trainer.train(do_eval=True) >>> # 3) resume train task to auto load the last checkpoint, if training break after 10 epochs. >>> task_trainer.train(train_checkpoint=True, resume_training=True) >>> # 4) resume train task according to checkpoint path, if training break after 10 epochs. >>> task_trainer.train( ... resume_or_finetune_from_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... resume_training=True) """ if train_checkpoint is not None and \ not isinstance(train_checkpoint, (bool, str)): raise TypeError(f"resume_or_finetune_from_checkpoint must be one of [None, string, bool], " f"but get {train_checkpoint}") if train_checkpoint is False: train_checkpoint = None do_eval = do_eval or self.config.do_eval if do_eval: if self.eval_dataset is None: logger.info("do_eval is enabled, building eval_dataset from config.") self.eval_dataset = build_dataset(self.config.eval_dataset_task) if self.eval_dataset is None: raise ValueError(f"if do_eval is true, eval_dataset must be input, " f"the task {self.task} is not support eval now.") # open do_eval for trainer config self.config.do_eval = True if train_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(train_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = train_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None if resume_training is not None: self.config.resume_training = resume_training if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=True) self.trainer.train( config=self.config, network=self.model, dataset=self.train_dataset, optimizer=self.optimizers, eval_dataset=self.eval_dataset if do_eval else None, wrapper=self.wrapper, callbacks=self.callbacks, is_full_config=True, **kwargs) def finetune(self, finetune_checkpoint: Optional[Union[str, bool]] = False, resume_training: Optional[bool] = None, auto_trans_ckpt: Optional[bool] = None, do_eval: bool = False, **kwargs): r"""Finetune task for Trainer. This function is used to fine-tune the network. Args: finetune_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. if resume_training is true, this checkpoint will be used to restore training of the network. Default: False. resume_training (bool): Whether to perform resume training. Default: False auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if load_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='text_generation', ... model='gpt2', ... pet_method='lora', ... train_dataset='./train', ... eval_dataset='./eval') >>> # 1) default finetune task. >>> task_trainer.finetune() >>> # 2) eval network when finetune model. >>> task_trainer.finetune(do_eval=True) >>> # 3) The last weight in the output directory is automatically loaded to resume training. >>> task_trainer.finetune(finetune_checkpoint=True, resume_training=True) >>> # 4) The last weight in the output directory is automatically loaded for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint=True) >>> # 5) Specify weights to fine-tune. >>> task_trainer.finetune(finetune_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') >>> # 6) Automatically load preset weights for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint='gpt2') """ if finetune_checkpoint is not None and \ not isinstance(finetune_checkpoint, (bool, str)): raise TypeError(f"load_checkpoint must be one of [None, string, bool], " f"but get {finetune_checkpoint}") if finetune_checkpoint is False: finetune_checkpoint = None do_eval = do_eval or self.config.do_eval if do_eval: if self.eval_dataset is None: logger.info("do_eval is enabled, building eval_dataset from config.") self.eval_dataset = build_dataset(self.config.eval_dataset_task) if self.eval_dataset is None: raise ValueError(f"if do_eval is true, eval_dataset must be input, " f"the task {self.task} is not support eval now.") # open do_eval for trainer config self.config.do_eval = True if finetune_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(finetune_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = finetune_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be fine-tune are specified" "by the finetune_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None self.config.model.model_config.checkpoint_name_or_path = None if resume_training is not None: self.config.resume_training = resume_training if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=True) self.trainer.train( config=self.config, network=self.model, dataset=self.train_dataset, optimizer=self.optimizers, eval_dataset=self.eval_dataset if do_eval else None, wrapper=self.wrapper, callbacks=self.callbacks, is_full_config=True, **kwargs) def evaluate(self, eval_checkpoint: Optional[Union[str, bool]] = False, auto_trans_ckpt: Optional[bool] = None, **kwargs): r"""Evaluate task for Trainer. This function is used to evaluate the network. Args: eval_checkpoint (Optional[Union[str, bool]]): Used to evaluate the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model Raises: TypeError: if eval_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... eval_dataset='data/imagenet/train') >>> # 1) default evaluate task to test model. >>> task_trainer.evaluate() >>> # 2) evaluate task to auto load the last checkpoint. >>> task_trainer.evaluate(eval_checkpoint=True) >>> # 3) evaluate task according to checkpoint path. >>> task_trainer.evaluate(eval_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') """ if eval_checkpoint is not None and not isinstance(eval_checkpoint, (bool, str)): raise TypeError(f"eval_checkpoint must be one of [None, string, bool], " f"but get {eval_checkpoint}") if eval_checkpoint is False: eval_checkpoint = None if eval_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(eval_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = eval_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be evaluated are specified" "by the eval_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=False) self.trainer.evaluate( config=self.config, network=self.model, dataset=self.eval_dataset, callbacks=self.eval_callbacks, is_full_config=True, **kwargs) def predict(self, predict_checkpoint: Optional[Union[str, bool]] = None, auto_trans_ckpt: Optional[bool] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, **kwargs): r"""Predict task for Trainer. This function is used to predict the network. Args: predict_checkpoint (Optional[Union[str, bool]]): Used to predict the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. Return: predict result (dict). Raises: TypeError: if predict_checkpoint is not bool or str type. TypeError: if input_data is not Tensor or np.ndarray or Image or str or list. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16') >>> input_data = "./sunflower.png" >>> # 1) predict task to auto load the last checkpoint. >>> task_trainer.predict(predict_checkpoint=True, input_data=input_data) >>> # 2) predict task according to checkpoint path. >>> task_trainer.predict(predict_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... input_data=input_data) >>> # 3) download and auto load the checkpoint on obs and predict. >>> task_trainer.predict(input_data=input_data) """ if predict_checkpoint is not None and not isinstance(predict_checkpoint, (bool, str)): raise TypeError(f"predict_checkpoint must be one of [None, string, bool], " f"but get {predict_checkpoint}") if self.task not in SUPPORT_PIPELINES.keys(): raise NotImplementedError(f"The {self.task} not support predict, " f"now this tasks {SUPPORT_PIPELINES.keys()} is support predict.") if predict_checkpoint is False: predict_checkpoint = None if predict_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(predict_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = predict_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be evaluated are specified" "by the eval_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if input_data is None: input_data = build_dataset_loader(self.config.eval_dataset.data_loader) logger.info("dataset by config is used as input_data.") assert isinstance(input_data, (GeneratorDataset, BaseDataset, RepeatDataset, BatchDataset, Tensor, np.ndarray, Image, str, list)), \ "Input data's type must be one of [GeneratorDataset," \ " str, ms.Tensor, np.ndarray, PIL.Image.Image]" if self.is_model_instance: self.reset_model_instance(is_train=False) kwargs['model_name'] = self.model_name output_result = self.trainer.predict( config=self.config, input_data=input_data, network=self.model, image_processor=self.image_processor, audio_processor=self.audio_processor, tokenizer=self.tokenizer, is_full_config=True, **kwargs) return output_result def build_network(self, input_checkpoint: Optional[Union[str, bool]] = None, is_train: bool = True): """build network for trainer.""" if self.model is None and self.task != 'general': logger.info("...........Start Init Network..........") self.model = build_model(self.config.model) # set running mode if self.model is not None: self.model.set_train(is_train) else: logger.warning("network will be create in %s task trainer class.", self.task) if self.is_model_instance and input_checkpoint: self._load_model_checkpoint() def set_parallel_config( self, data_parallel=1, model_parallel=1, expert_parallel=1, pipeline_stage=1, micro_batch_interleave_num=1, micro_batch_num=1, use_seq_parallel=False, optimizer_shard=False, gradient_aggregation_group=4, vocab_emb_dp=True): r""" set_parallel_config for the setting global data parallel, model parallel and fusion group. The parallel configure setting for Trainer. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. use_seq_parallel (bool): Whether to enable sequence parallel. Default False. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. micro_batch_interleave_num (int): split num of batch size. Default: 1. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_parallel_config(data_parallel=2, model_parallel=2) """ self.config.parallel_config.data_parallel = data_parallel self.config.parallel_config.model_parallel = model_parallel self.config.parallel_config.expert_parallel = expert_parallel self.config.parallel_config.pipeline_stage = pipeline_stage self.config.parallel_config.use_seq_parallel = use_seq_parallel self.config.parallel_config.optimizer_shard = optimizer_shard self.config.parallel_config.micro_batch_num = micro_batch_num self.config.parallel_config.vocab_emb_dp = vocab_emb_dp self.config.parallel_config.gradient_aggregation_group = gradient_aggregation_group self.config.micro_batch_interleave_num = micro_batch_interleave_num self.is_set_parallel_config = True def set_recompute_config(self, recompute=False, parallel_optimizer_comm_recompute=False, select_recompute=False, mp_comm_recompute=True, recompute_slice_activation=False): r"""Set recompute config. TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. select_recompute (bool): Only Enable recomputation of the attention layer or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_recompute_config(recompute=True) """ self.config.recompute_config.recompute = recompute self.config.recompute_config.select_recompute = select_recompute self.config.recompute_config.parallel_optimizer_comm_recompute = parallel_optimizer_comm_recompute self.config.recompute_config.mp_comm_recompute = mp_comm_recompute self.config.recompute_config.recompute_slice_activation = recompute_slice_activation self.is_set_recompute_config = True def set_moe_config(self, expert_num=1, capacity_factor=1.1, aux_loss_factor=0.05, num_experts_chosen=1, expert_group_size=None, group_wise_a2a=False, comp_comm_parallel=False, comp_comm_parallel_degree=2): r"""The configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_moe_config(expert_num=2, capacity_factor=1.2, aux_loss_factor=0.001) """ self.config.moe_config.expert_num = expert_num self.config.moe_config.capacity_factor = capacity_factor self.config.moe_config.aux_loss_factor = aux_loss_factor self.config.moe_config.num_experts_chosen = num_experts_chosen self.config.moe_config.expert_group_size = expert_group_size self.config.moe_config.group_wise_a2a = group_wise_a2a self.config.moe_config.comp_comm_parallel = comp_comm_parallel self.config.moe_config.comp_comm_parallel_degree = comp_comm_parallel_degree self.is_set_moe_config = True def reset_model_instance(self, is_train=True): """Reset model instance for new model config.""" if True not in [self.is_set_parallel_config, self.is_set_moe_config, self.is_set_recompute_config]: return if self.is_set_parallel_config: logger.info("The incoming model will be configured in parallel.") if self.is_set_recompute_config: logger.info("The incoming model will be configured in recompute.") if self.is_set_moe_config: logger.info("The incoming model will be configured in moe.") if not isinstance(self.model, BaseModel): raise NotImplementedError("Currently only the integrated model structure in MindFormers is supported.") build_parallel_config(self.config) model_config = self.model.config model_config.parallel_config = self.config.parallel_config model_config.moe_config = self.config.moe_config self.model.__init__(model_config) self.model.set_train(is_train) if not is_train: return network = self.model micro_batch_interleave_num = self.config.micro_batch_interleave_num if ms.get_auto_parallel_context("pipeline_stages") > 1: micro_batch_num = self.config.parallel_config.micro_batch_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = PipelineCell(MicroBatchInterleaved(self.model, micro_batch_interleave_num), micro_size=micro_batch_num) else: network = PipelineCell(self.model, micro_size=micro_batch_num) network = _VirtualDatasetCell(network) else: if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = MicroBatchInterleaved(self.model, micro_batch_interleave_num) del self.model self.model = network def get_train_dataloader(self): """get train dataloader of mindspore.""" return build_dataset_loader(self.config.train_dataset.data_loader) def get_eval_dataloader(self): """get eval dataloader of mindspore.""" return build_dataset_loader(self.config.eval_dataset.data_loader) def get_last_checkpoint(self): """get last checkpoint for resuming or finetune.""" output_folder = self.config.output_dir checkpoint_dir = os.path.join( output_folder, DEFAULT_CHECKPOINT_DIR, 'rank_{}'.format(self.rank_id)) output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] if not output_checkpoint_path: return None output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) return os.path.join(checkpoint_dir, output_checkpoint_path[-1]) def save_config_to_yaml(self, config: dict = None): """save now config file to yaml file.""" if config is None: config = self.config model_name = self.config.trainer.model_name config_dict = _reset_config_for_save(config) config_dir = os.path.join( self.configs_directory, model_name.lower() + '_new') if not os.path.exists(config_dir): os.makedirs(config_dir, exist_ok=True) run_yaml_path = os.path.join(config_dir, 'run_{}.yaml'.format(model_name.lower())) _save_config_to_yaml(run_yaml_path, config_dict) def _load_model_checkpoint(self): """Load model checkpoint to network.""" checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if checkpoint_name_or_path is None: logger.warning("checkpoint_name_or_path is None, not load input checkpoint.") elif isinstance(checkpoint_name_or_path, str): is_exist_path = os.path.exists(checkpoint_name_or_path) is_checkpoint_name = checkpoint_name_or_path in SUPPORT_MODEL_NAMES if is_exist_path: logger.info("now input valid checkpoint path, it will load to network.") checkpoint_dict = load_checkpoint(checkpoint_name_or_path) not_load_params = load_param_into_net(self.model, checkpoint_dict) logger.info("not load parameters is: %s", str(not_load_params)) elif is_checkpoint_name: logger.info("now input valid checkpoint name, it will load to network.") if isinstance(self.model, (Cell, BaseModel)): self.model.load_checkpoint(self.config.model.model_config) else: logger.warning("model must be BaseModel or Cell type, but get %s", type(self.model)) else: logger.warning("input checkpoint args is invalid, " "it must be valid and real checkpoint path or a valid checkpoint name," "but get %s", checkpoint_name_or_path) else: raise TypeError(f"checkpoint_name_or_path type error, " f"it should be one of [None, str], " f"but get {type(checkpoint_name_or_path)}") def _check_checkpoint_config(self, checkpoint: Optional[Union[str, bool]] = None): """check checkpoint config.""" if checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = self.get_last_checkpoint() elif isinstance(checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = checkpoint else: if self.default_checkpoint_name_or_path is not None: self.config.model.model_config.checkpoint_name_or_path = self.default_checkpoint_name_or_path def _save_config_to_yaml(save_file_path: str = None, save_config: dict = None): r"""Save Config to Yaml File. Args: save_file_path (str): The real path to save yaml file. Default: None. save_config (dict): The task config. Default: None. """ if save_config is None: save_config = {} with open(save_file_path, 'w', encoding='utf-8') as file_pointer: file_pointer.write( ordered_yaml_dump( save_config, default_flow_style=False, allow_unicode=True, sort_keys=False)) def _reset_config_for_save(config: dict = None): r"""Reset Config According to Yaml File Number. Args: config (dict): The task config. Default: None. model_name (str): The model name to save. Default: 'common'. """ if config is None: config = {} config = config.copy() config_dict = OrderedDict() if config.get('model') is not None: model_config = config2dict(config.pop('model')) config_dict.setdefault('model', model_config) if config.get('processor') is not None: processor_config = config2dict(config.pop('processor')) config_dict.setdefault('processor', processor_config) if config.get('train_dataset_task') is not None and config.get('train_dataset') is not None: train_dataset_config = config2dict(config.pop('train_dataset')) train_dataset_task_config = config2dict(config.pop('train_dataset_task')) config_dict.setdefault('train_dataset', train_dataset_config) config_dict.setdefault('train_dataset_task', train_dataset_task_config) if config.get('eval_dataset_task') is not None and config.get('eval_dataset') is not None: eval_dataset_config = config2dict(config.pop('eval_dataset')) eval_dataset_task_config = config2dict(config.pop('eval_dataset_task')) config_dict.setdefault('train_dataset', eval_dataset_config) config_dict.setdefault('train_dataset_task', eval_dataset_task_config) if config.get('context') is not None: context_config = config2dict(config.pop('context')) parallel_context_config = config2dict(config.pop('parallel')) moe_conifg = config2dict(config.pop('moe_config')) recompute_config = config2dict(config.pop('recompute_config')) parallel_config = config2dict(config.pop('parallel_config')) config_dict.setdefault('context', context_config) config_dict.setdefault('parallel', parallel_context_config) config_dict.setdefault('moe_conifg', moe_conifg) config_dict.setdefault('recompute_config', recompute_config) config_dict.setdefault('parallel_config', parallel_config) if config.get('runner_config') is not None: runner_config = config2dict(config.pop('runner_config')) config_dict.setdefault('runner_config', runner_config) if config.get('runner_wrapper') is not None: wrapper_config = config2dict(config.pop('runner_wrapper')) config_dict.setdefault('runner_wrapper', wrapper_config) if config.get('optimizer') is not None: optim_config = config2dict(config.pop('optimizer')) config_dict.setdefault('optimizer', optim_config) if config.get('lr_schedule') is not None: lr_config = config2dict(config.pop('lr_schedule')) config_dict.setdefault('lr_schedule', lr_config) if config.get('callbacks') is not None: cb_config = config2dict(config.pop('callbacks')) config_dict.setdefault('callbacks', cb_config) run_config = config2dict(config) for key, value in run_config.items(): config_dict.setdefault(key, value) return config_dict pengcheng.mind-test/mindformers/trainer/.ipynb_checkpoints/utils-checkpoint.py000066400000000000000000000625201471400134000303370ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Trainer Utils.""" import os import time import random from glob import glob from enum import Enum import numpy as np import mindspore as ms from mindspore.communication.management import get_rank, get_group_size from mindspore import context, load_checkpoint, load_param_into_net from mindspore import set_seed as ms_set_seed from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerConfig from mindformers.tools.utils import check_in_modelarts, get_output_root_path from mindformers.tools.transform_ckpt import get_strategy if check_in_modelarts(): import moxing as mox ELAPSE_TIME = 7200 class BaseEnum(str, Enum): """ Base Enum for MindFormers. """ @classmethod def _missing_(cls, value): """Enum with more explicit error message for missing values.""" raise ValueError( f"{value} is not a valid {cls.__name__}, please select one of {list(cls._value2member_map_.keys())}" ) class SaveIntervalStrategy(BaseEnum): """ Stores the acceptable string identifiers for save checkpoint monitor. """ NO = "no" STEPS = "steps" SECONDS = "seconds" class LRType(BaseEnum): """ Stores the acceptable string identifiers for learning rate schedule. """ # will be support item for future. LINEAR = "linear" COSINE = "cosine" COSINE_WITH_RESTARTS = "cosine_with_restarts" POLYNOMIAL = "polynomial" CONSTANT_WITH_WARMUP = "constant_with_warmup" class OptimizerType(BaseEnum): """ Stores the acceptable string identifiers for optimizers. """ # supported item for test, will be delete in the future. ADAMWEIGHTDECAY = 'AdamWeightDecay' # will be support item for future. ADAMW = "adamw" ADAM = "adam" SGD = "sgd" ADAGRAD = "adagrad" ADAFACTOR = "adafactor" class WrapperType(BaseEnum): """ Stores the acceptable string identifiers for training wrapper. """ # will be support item for future. MFWRAPPER = 'mf_wrapper' TRAINONESTEP = 'wrapper' TRAINONESTEPWITHLOSSSCALE = 'loss_scale_wrapper' def set_seed(seed: int = 0): """ Helper function for reproducible behavior to set the seed in `random`, `numpy`, `MindSpore`. Args: seed (`int`): The seed to set. """ random.seed(seed) np.random.seed(seed) ms_set_seed(seed) def check_keywords_in_name(name, keywords=()): """ Check keywords in name. """ isin = False for keyword in keywords: if keyword in name: isin = True return isin def check_runner_config(config, dataset): """ Check runner config. """ data_size = dataset.get_dataset_size() new_epochs = config.runner_config.epochs config.runner_config.origin_epochs = new_epochs if config.runner_config.initial_epoch is None: config.runner_config.initial_epoch = 0 if config.runner_config.sink_mode: if config.runner_config.sink_size != -1: if config.runner_config.sink_size <= 0: raise ValueError("per epoch size must be more than 0 or equal to -1, " f"but get {config.runner_config.sink_size}") if data_size < config.runner_config.sink_size: logger.warning("The data size %s (get from dataset.get_dataset_size()) is smaller " "than the sink_size %s (get from config.runner_config.sink_size), " "you should set the config.runner_config.sink_size to %s", data_size, config.runner_config.sink_size, data_size) config.runner_config.epochs = int((data_size / config.runner_config.sink_size) * new_epochs) else: config.runner_config.sink_size = data_size else: logger.warning("Sink mode is False, per epoch size is invalid, it will reset -1.") config.runner_config.sink_size = -1 config.data_size = data_size logger.info("Will be Training epochs:%d, sink_size:%d", config.runner_config.origin_epochs, config.runner_config.sink_size) logger.info("Create training dataset finish, dataset size:%d", data_size) def check_train_data_loader_type(new_config, old_config): """Check train data loader config type.""" if new_config.train_dataset is None: return None if new_config.train_dataset.get('data_loader') is None: return None train_data_loader_type = new_config.train_dataset.get('data_loader').get('type') if old_config.train_dataset is not None and train_data_loader_type is not None: default_train_data_loader_type = old_config.train_dataset.data_loader.type if train_data_loader_type != default_train_data_loader_type: logger.warning("train dataset's data_loader type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", train_data_loader_type) old_config.train_dataset.data_loader = {} return None def check_eval_data_loader_type(new_config, old_config): """Check eval data loader config type.""" if new_config.eval_dataset is None: return None if new_config.eval_dataset.get('data_loader') is None: return None eval_data_loader_type = new_config.eval_dataset.get('data_loader').get('type') if old_config.eval_dataset is not None and eval_data_loader_type is not None: default_eval_data_loader_type = old_config.eval_dataset.data_loader.type if eval_data_loader_type != default_eval_data_loader_type: logger.warning("eval dataset's data_loader type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", eval_data_loader_type) old_config.eval_dataset.data_loader = {} return None def check_optimizer_and_lr_type(new_config, old_config): """Check optimizer and lr schedule config type.""" optimizer_type = new_config.optimizer.get('type') if old_config.optimizer is not None and optimizer_type is not None: default_optimizer_type = old_config.optimizer.type if optimizer_type != default_optimizer_type: logger.warning( "optimizer type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually except (params).", optimizer_type) old_config.optimizer = {} if hasattr(new_config.optimizer, 'learning_rate'): lr_type = new_config.optimizer.learning_rate.get('type') if old_config.lr_schedule is not None and lr_type is not None: default_lr_type = old_config.lr_schedule.type if lr_type != default_lr_type: logger.warning( "lr schedule type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", lr_type) old_config.lr_schedule = None def check_wrapper_config(new_config, old_config): """Check wrapper config.""" wrapper_type = new_config.runner_wrapper.get('type') if old_config.runner_wrapper is not None and wrapper_type is not None: default_wrapper_type = old_config.runner_wrapper.type if wrapper_type != default_wrapper_type: logger.warning( "wrapper type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", wrapper_type) old_config.runner_wrapper = {} def config2dict(config): """MindFormerConfig Type Convert to Dict.""" if not isinstance(config, (dict, MindFormerConfig)): return config new_dict = {} for key, value in config.items(): if isinstance(value, MindFormerConfig): value = config2dict(value) new_dict.setdefault(key, value) return new_dict def load_distributed_checkpoint(config, specify_prefix=None): """Load Checkpoint in Parallel Mode.""" checkpoint_dir = config.load_checkpoint if os.path.isdir(checkpoint_dir): logger.info( "When distributed loads are sliced weights," "load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*}," "The directory structure is as follows: **checkpoint_root_dir/checkpoint/rank_{0-*}/**.ckpt") distribute_checkpoint_dir = os.path.join( checkpoint_dir, "rank_{}".format(int(os.getenv("RANK_ID", "0")))) distribute_checkpoint_path = get_last_checkpoint(distribute_checkpoint_dir) elif os.path.isfile(checkpoint_dir): logger.info("Your load_checkpoint is file, it will be load in network.") distribute_checkpoint_path = checkpoint_dir else: raise FileNotFoundError(f"{checkpoint_dir} is not found.") checkpoint_dict = load_checkpoint(distribute_checkpoint_path, specify_prefix=specify_prefix) logger.info("Distribute load is success.") return checkpoint_dict def load_resume_context_from_checkpoint(config): """resume training, load training info from checkpoint to config""" if not os.path.realpath(config.load_checkpoint) or \ not os.path.exists(config.load_checkpoint): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") if os.path.isdir(config.load_checkpoint): resume_dict = load_distributed_checkpoint(config, ["loss_scale", "epoch_num"]) if not config.runner_config.sink_mode: raise ValueError("When distributed loads are sliced weights, sink_mode must be set True.") else: resume_dict = load_checkpoint(config.load_checkpoint, specify_prefix=["loss_scale", "epoch_num"]) if "epoch_num" in resume_dict: config.runner_config.initial_epoch = int(resume_dict["epoch_num"]) else: config.runner_config.initial_epoch = 0 for callback in config.callbacks: if "type" in callback and callback["type"] == "CheckpointMointor": if config.runner_wrapper.scale_sense is not None and "loss_scale" in resume_dict: config.runner_wrapper.scale_sense.loss_scale_value = resume_dict["loss_scale"] break def transform_and_load_checkpoint(config, model, network, dataset, optimizer=None, do_eval=False, do_predict=False): """ load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt """ if not config.only_save_strategy and (not os.path.realpath(config.load_checkpoint) or not os.path.exists(config.load_checkpoint)): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: # 1. build net if parallel mode is auto_parallel build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) if config.auto_trans_ckpt: # 2. get strategy dst_ckpt_strategy = get_dst_strategy(config) # 3. transform checkpoint if needed transform_ckpt(config, dst_ckpt_strategy=dst_ckpt_strategy) # 4. load ckpt load_ckpt(config, network, optimizer=optimizer) def load_merged_checkpoint(config, model, network, dataset, optimizer=None, do_eval=False, do_predict=False): """ load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt """ if not os.path.realpath(config.load_checkpoint or not os.path.exists(config.load_checkpoint)): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") # 1. load merged ckpt load_ckpt(config, network, optimizer=optimizer) if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: # 2. build net if parallel mode is auto_parallel build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) def build_model(config, model, dataset, do_eval=False, do_predict=False): """build model, generate strategy file""" if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: if not config.runner_config.sink_mode: raise ValueError("When distributed loads are sliced weights, sink_mode must be set True.") if do_eval: model.infer_predict_layout(*next(dataset.create_tuple_iterator())) elif do_predict: model.infer_predict_layout(dataset) else: if config.runner_config.epochs > 1 and config.runner_config.sink_size == 1: raise ValueError(f"When distributed loads are sliced weights, it does not support" f"epochs = {config.runner_config.epochs} > 1 and " f"sink_size = {config.runner_config.sink_size} = 1," f"sink_size must be more than 1") logger.info(".........Building model.........") model.build(train_dataset=dataset, epoch=config.runner_config.epochs, sink_size=config.runner_config.sink_size) if config.only_save_strategy: raise SystemExit("Only_save_strategy is True, model.compile() finished, system exit! ") elif config.only_save_strategy: raise SystemExit("only_save_strategy is True, " "but stand_alone and data_parallel mode do not have strategy file, system exit! ") def get_dst_strategy(config): """get strategy""" rank_id = int(os.getenv('RANK_ID', '0')) world_size = int(os.getenv('RANK_SIZE', '1')) dst_strategy_path = None if world_size == 1: return dst_strategy_path if check_in_modelarts(): # local send all strategy file to obs obs_save_dir = os.path.join(config.remote_save_url, "strategy") mox.file.make_dirs(obs_save_dir) local_strategy_path = config.parallel.strategy_ckpt_save_file obs_strategy_path = os.path.join(obs_save_dir, os.path.basename(local_strategy_path)) mox.file.copy(local_strategy_path, obs_strategy_path) # obs send all strategy to each node logger.info(".........Collecting strategy.........") local_strategy_dir = os.path.join(get_output_root_path(), "strategy") if rank_id % 8 == 0: wait_collect_all_strategy(local_strategy_dir, world_size, obs_save_dir) else: wait_collect_all_strategy(local_strategy_dir, world_size) logger.info(".........All strtegy as follow.........") local_strategy_paths = glob(os.path.join(local_strategy_dir, "*_rank_*.ckpt")) local_strategy_paths.sort() for local_strategy_path in local_strategy_paths: logger.info("strategy: %s", local_strategy_path) logger.info(".........Collecting %d strategy.........", len(local_strategy_paths)) # merge strategy if pipeline_stage > 1 if config.parallel_config.pipeline_stage > 1: if rank_id % 8 == 0: logger.info(".........Merging strategy.........") merged_strategy_path = get_strategy(local_strategy_dir) merged_strategy_name = os.path.basename(merged_strategy_path) obs_merged_strategy_path = os.path.join(obs_save_dir, merged_strategy_name) mox.file.copy(merged_strategy_path, obs_merged_strategy_path) logger.info("Save %s to %s", merged_strategy_path, obs_merged_strategy_path) logger.info(".........Merging succeed.........") dst_strategy_path = merged_strategy_path else: dst_strategy_path = None else: dst_strategy_path = local_strategy_path else: logger.info(".........Collecting strategy.........") local_strategy_dir = os.path.join(get_output_root_path(), "strategy") if world_size <= 8: wait_collect_all_strategy(local_strategy_dir, world_size) logger.info(".........All strtegy as follow.........") local_strategy_paths = glob(os.path.join(local_strategy_dir, "*_rank_*.ckpt")) local_strategy_paths.sort() for local_strategy_path in local_strategy_paths: logger.info("strategy: %s", local_strategy_path) logger.info(".........Collecting %d strategy.........", len(local_strategy_paths)) # merge strategy if pipeline_stage > 1 if config.parallel_config.pipeline_stage > 1: if rank_id % 8 == 0: logger.info(".........Merging strategy.........") merged_strategy_path = get_strategy(local_strategy_dir) logger.info(".........Merging succeed.........") dst_strategy_path = merged_strategy_path else: dst_strategy_path = None else: dst_strategy_path = local_strategy_paths[0] else: logger.warning("Can't collecting all strategy, device num > 8!") config.auto_trans_ckpt = False dst_strategy_path = None return dst_strategy_path def transform_ckpt(config, dst_ckpt_strategy=None): """auto transform ckpt""" rank_id = get_rank() if config.use_parallel else 0 world_size = get_group_size() if config.use_parallel else 1 transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") os.makedirs(transformed_ckpt_dir, exist_ok=True) if rank_id % 8 == 0: logger.info(".........Transforming ckpt.........") src_ckpt_strategy = get_strategy(config.src_strategy_path_or_dir) logger.info("Src ckpt strategy: %s", src_ckpt_strategy) logger.info("Src ckpt: %s", config.load_checkpoint) logger.info("Dst ckpt strategy: %s", dst_ckpt_strategy) logger.info("Dst ckpt: %s", transformed_ckpt_dir) ms.transform_checkpoints(config.load_checkpoint, transformed_ckpt_dir, 'checkpoint_', src_ckpt_strategy, dst_ckpt_strategy) logger.info(".........Transform succeed!.........") transform_succeed_txt = os.path.join(transformed_ckpt_dir, f'transform_succeed_rank_{rank_id}.txt') f = open(transform_succeed_txt, 'w') f.close() if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") mox.file.make_dirs(transformed_ckpt_dir_obs) transform_succeed_txt_obs = os.path.join(transformed_ckpt_dir_obs, f'transform_succeed_rank_{rank_id}.txt') mox.file.copy(transform_succeed_txt, transform_succeed_txt_obs) wait_transform(config, rank_id, world_size) if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") rank_transformed_ckpt_dir_obs = os.path.join(transformed_ckpt_dir_obs, f'rank_{rank_id}') rank_transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint", f'rank_{rank_id}') mox.file.copy_parallel(rank_transformed_ckpt_dir, rank_transformed_ckpt_dir_obs) print(f"Rank {rank_id}: Save {rank_transformed_ckpt_dir} to {rank_transformed_ckpt_dir_obs}") config.load_checkpoint = transformed_ckpt_dir def wait_transform(config, rank_id, world_size): """wait all node transform over""" print(f'Rank {rank_id}: Waiting transforming ckpt......') time_out = False if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") transform_succeed_txts_obs = [os.path.join(transformed_ckpt_dir_obs, f'transform_succeed_rank_{int(i/8)*8}.txt') for i in range(0, world_size, 8)] total_num = len(transform_succeed_txts_obs) all_node_transform_succeed = False start_time = time.time() while not all_node_transform_succeed: node_transform_succeed_num = 0 for transform_succeed_txt in transform_succeed_txts_obs: if mox.file.exists(transform_succeed_txt): node_transform_succeed_num += 1 else: break if node_transform_succeed_num == total_num: all_node_transform_succeed = True else: elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: logger.warning("Automatic weight transform is timed out!") time_out = True break time.sleep(5) else: transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") transform_succeed_txt = os.path.join(transformed_ckpt_dir, f'transform_succeed_rank_0.txt') start_time = time.time() while not os.path.exists(transform_succeed_txt): elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: time_out = True break time.sleep(5) print(f'Rank {rank_id}: Transform succeed!') if time_out: return False return True def wait_collect_all_strategy(strategy_dir, total_num, obs_strategy_dir=None): """wait all strategy collect over""" start_time = time.time() time_out = False while True: if obs_strategy_dir: obs_strategy_paths = mox.file.glob(os.path.join(obs_strategy_dir, "*.ckpt")) if len(obs_strategy_paths) < total_num: time.sleep(5) continue for obs_strategy_path in obs_strategy_paths: local_strategy_path = os.path.join(strategy_dir, os.path.basename(obs_strategy_path)) mox.file.copy(obs_strategy_path, local_strategy_path) local_strategy_paths = glob(os.path.join(strategy_dir, "*_rank_*.ckpt")) if len(local_strategy_paths) == total_num: break else: elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: logger.warning("Collection of all strategy timed out!") time_out = True break time.sleep(5) if time_out: return False return True def load_ckpt(config, network, optimizer=None): """load checkpoint""" # load checkpoint params into dict logger.info(".............Start load checkpoint from checkpoint..................") if os.path.isdir(config.load_checkpoint): checkpoint_dict = load_distributed_checkpoint(config) else: checkpoint_dict = load_checkpoint(config.load_checkpoint) # load params into net not_load_network_params = load_param_into_net(network, checkpoint_dict) logger.info("Network parameters are not loaded: %s", str(not_load_network_params)) if optimizer: not_load_optim_params = load_param_into_net(optimizer, checkpoint_dict) logger.info("Optimizer parameters are not loaded: %s", str(not_load_optim_params)) def get_last_checkpoint(checkpoint_dir): """get last checkpoint for resuming or finetune.""" if not os.path.isdir(checkpoint_dir): raise NotADirectoryError( f"{checkpoint_dir} is not a real directory," "When distributed loads are sliced weights," "load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*}," "The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/checkpoint/**.ckpt") output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] if not output_checkpoint_path: return None output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) return os.path.join(checkpoint_dir, output_checkpoint_path[-1]) pengcheng.mind-test/mindformers/trainer/__init__.py000066400000000000000000000036761471400134000230270ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Trainer API.""" from .config_args import * from .image_classification import * from .masked_image_modeling import * from .masked_language_modeling import * from .general_task_trainer import * from .contrastive_language_image_pretrain import * from .image_to_text_retrieval import * from .image_to_text_generation import * from .translation import * from .text_classfication import * from .token_classification import * from .question_answering import * from .causal_language_modeling import * from .trainer import Trainer from .training_args import TrainingArguments from .base_trainer import BaseTrainer from .build_trainer import build_trainer __all__ = ['BaseTrainer', 'Trainer', 'TrainingArguments'] __all__.extend(config_args.__all__) __all__.extend(image_classification.__all__) __all__.extend(masked_image_modeling.__all__) __all__.extend(masked_language_modeling.__all__) __all__.extend(general_task_trainer.__all__) __all__.extend(contrastive_language_image_pretrain.__all__) __all__.extend(image_to_text_retrieval.__all__) __all__.extend(image_to_text_generation.__all__) __all__.extend(translation.__all__) __all__.extend(text_classfication.__all__) __all__.extend(token_classification.__all__) __all__.extend(question_answering.__all__) __all__.extend(causal_language_modeling.__all__) pengcheng.mind-test/mindformers/trainer/__pycache__/000077500000000000000000000000001471400134000231125ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/__pycache__/__init__.cpython-39.pyc000066400000000000000000000020151471400134000273000ustar00rootroot00000000000000a >cf@sDdZddlTddlTddlTddlTddlTddlTddlTddlTddl Tddl Tddl Tddl Tddl TddlmZddlmZddlmZddlmZgdZeejeejeejeejeejeejeejeejee jee jee jee jee jdS) zMindFormers Trainer API.)*)Trainer)TrainingArguments) BaseTrainer) build_trainer)rrrN)__doc__ config_argsimage_classificationmasked_image_modelingZmasked_language_modelingZgeneral_task_trainer#contrastive_language_image_pretrainimage_to_text_retrievalimage_to_text_generation translationZtext_classficationtoken_classificationquestion_answeringZcausal_language_modelingtrainerrZ training_argsrZ base_trainerrr__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/trainer/__init__.pys>                pengcheng.mind-test/mindformers/trainer/__pycache__/base_trainer.cpython-39.pyc000066400000000000000000000640771471400134000302170ustar00rootroot00000000000000a of@s4dZddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl Z ddlmZddlZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZm Z m!Z!dd l"m#Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:ddl;mZ>ddl?m@Z@ddlAmBZBddlCmDZDddlEmFZFddlGmHZHddlImJZJddlKmLZLmMZMmNZNddlOmPZPmQZQddlKmRZRmSZSmTZTmUZUmVZVe&WZXe&YZZe&[Z\e&]Z^e&_Z`d Zad!d"ZbGd#d$d$ZcdS)%z Base Trainer.N)pprint)partial)OptionalUnionList)Image)Tensor DatasetHelper)Model)Callback)GeneratorDataset)_VirtualDatasetCell)TrainOneStepCell OptimizerCell PipelineCellMicroBatchInterleaved) initializer)MindFormerBook)build_lr build_optimbuild_callback build_metric) EvalCallBack)build_parallel_config) build_datasetcheck_dataset_config BaseDataset) build_modelbuild_processorbuild_tokenizer BaseModel BaseTokenizerBaseImageProcessor)pipeline) build_wrapper)MindFormerConfig)logger) count_params) AutoModel)ConfigArguments)TrainingArguments)check_runner_configtransform_and_load_checkpoint#load_resume_context_from_checkpoint) get_optimizer_grouped_parameters$get_mup_optimizer_grouped_parameters)set_seedcheck_train_data_loader_typecheck_eval_data_loader_typecheck_optimizer_and_lr_typecheck_wrapper_configconfigsc Csddl}t|dz|jj||dWnNtyx}z6td|dtd|jj||dWYd}~n d}~00td|d |d d d dS) z Download the dataset from the obs. src_data_url (Str): should be the dataset path in the obs tgt_data_path (Str): the local dataset path rank (Int): the current rank id rN)src_urldst_urlzError: z, download again... zRank z download ckpt z succeed!T)flush)moxingtimesleepfilecopy Exceptionprint)Z remote_url local_urlrankmoxerrorrHK/home/ma-user/work/mindformers-baichuan/mindformers/trainer/base_trainer.pydownload_ckpt_from_obs>s &rJc @seZdZdZdVeedddZdWeeeee e fe ddd Z d d Z d d ZddZddZddZdXedddZdYedddZdZedddZd[edddZd\edddZd]eddd Zd^d!d"Zd_e ed$d%d&Zd'd(Zd`edd)d*Zdaedd+d,Zdbed-d.d/Zd0d1Zdced2d3d4Z d5d6Z!d7d8Z"dde d:d;d<Z#d=d>Z$d?d@Z%dAdBZ&dCdDZ'dee(eee(efdGdHdIZ)dfeeee*e e feee+e,feee-e.fee/ee0eee1e2e1fdJdKdLZ3dgeeee*e e feee+e,feee-e.feee1e2e1feeee4fdMdNdOZ5dheeee*e e feee.e6e7j8e9ee:feeee+e,fee;ee<ee<dPdQdRZ=e>e-dSdTdUZ?dS)i BaseTraineraBase Task Trainer. Args: task (str): The task name supported. model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N)task model_namec Cs|dur d}|durd}td||||_||_d|_d|_d|_d|_d|_d|_ d|_ d|_ d|_ d|_ d|_d|_d|_d|_d|_tjtjdts(tjdt}tjtjttr(tjtt}ztj||ddWn:ty&zt||Wnty Yn0Yn0|tvr@td|tvr|t |vrtdtd ttd t |j d |d |_dS) Nzmodel name unspecified.ztask name unspecified.z%Now Running Task is: %s, Model is: %s.T) dirs_exist_okzThe default model config: %s will now be used for the %s task common)!r'inforMrLconfigdefault_task_config train_dataset eval_datasetnetwork optimizerimage_processoraudio_processor tokenizer callbackseval_callbacks model_wrappercompute_metricskwargs pipeline_taskospathexistsjoinDEFAULT_CONFIG_DIRCURRENT_PROJECT_PATHshutilcopytree TypeErrorFileExistsError SUPPORT_TASKSkeyswarningget)selfrLrMZconfigs_directoryZmindformers_configs_directoryrHrHrI__init___sT  "  zBaseTrainer.__init__F)rRis_full_configcCsR|dur*|r||_q:||||_n||j|_t|j||jS)z%Set the task config for task trainer.N)rRsetup_task_config _merge_configrSr*_check_global_batch_size_for_auto_parallel)rorRrqrHrHrI set_configs zBaseTrainer.set_configcCs`d}|jtvr@|jt|jvr@tt|j|j}t|trR||_n t ddS)zSetup the default task config.NzIf the default config arguments is not specified,you must define the required model or wrapper, optimizer, and so onin the train or evaluate or predict attribute function.) rLrkrlrMrnr& isinstancerSr'rm)ro task_configrHrHrIrrs $ zBaseTrainer.setup_task_configc Cs|jjj}|jjj}|jjj}|jj}td}td}| }|dvrH|r|dkrt d||||||||||||||jj_| n.t d|||||||||||jj_n||dkrt d|||||||t d|||t td d |||t td d ||||jj_| n|t d ||t d |t td d |t td d d|jj_d|jj_d|jj_d |jj_d|jj_t d|jjdS)z.Check global batch size in auto parallel mode. parallel_mode full_batchsemi_auto_parallel auto_parallelr*zPipeline parallel was opened: pipeline_stages = %s, full batch is True, global batch size will be changed: global_batch_size = batch_size * data_parallel * micro_batch_num * micro_batch_interleave_num = %s * %s * %s * %s = %s).zThe current parallel mode is %s, full batch is True,so global batch size will be changed: global_batch_size = batch_size * data_parallel * micro_batch_interleave_num = %s * %s * %s = %szPipeline parallel was opened: pipeline_stages = %s, full batch is False, batch size per card will be changed: per_batch_size = batch_size * micro_batch_num * micro_batch_interleave_num = %s * %s * %s = %s).z>global_batch_size = per_batch_size * device_num = %s * %s = %s RANK_SIZE1zbThe current parallel mode is %s, batch size per card will not be changed: batch_size_per_card = %szCglobal_batch_size = batch_size_per_card * device_num = %s * %s = %sFz5parallel_config will be change to default config: %s.N)rR runner_config batch_sizeparallel_config data_parallelmicro_batch_nummicro_batch_interleave_nummsget_auto_parallel_contextget_pipeline_stagesr'rQ$_reset_wrapper_for_pipeline_parallelintragetenvmodel_parallelpipeline_stageoptimizer_shard)rordprrrxrypprHrHrIrtsh                   z6BaseTrainer._check_global_batch_size_for_auto_parallelcCsb|jjdurJ|jjjdkrdn|jjj|jj_|jjj|jj_tdn tdtddS)z%Reset wrapper when pipeline parallel.NZMFPipelineWithLossScaleCellz`When using the pipeline parallel mode, the MFPipelineWithLossScaleCell class is used by default.zWhen using the pipeline parallel mode, because the wrapper class is not specified, MindSpore's built-in PipelineCell is used by defaultzDPipelineWrapper under evaluate or predict mode will not take effect.)rRrunner_wrappertyperrr'rmrQrorHrHrIrs  z0BaseTrainer._reset_wrapper_for_pipeline_parallelcCst|jdS)zMReset dataset batch size according to the global batch size of runner config.N)rrRrrHrHrI_reset_dataset_batch_sizesz%BaseTrainer._reset_dataset_batch_sizecCs(|jdurtd|St|tr0|j|nt|trtj |rRtj |sbt d|d| dszt d|t||_nt|trt|drt||jt|drt||jt|d rt||jt|d rt||j|j|jn*t|tr||jntd t||jS) z&Merge config from default task config.NzMdefault task config is None, you will not be able to merge config parameters.z#config path must be exist, but get rN)z.yamlz.ymlz4config file must be end with .yaml or .yml, but get rTrUrWrzThe type of config parameter to merge is not supported, currently supported types are [dict, config_path(str), ConfigArguments, TrainingArguments], but get %s)rSr'rmrvdictmerge_from_dictstrrarbrealpathrcFileNotFoundErrorendswith ValueErrorr&r+hasattrr3r4r5r6__dict__r,"convert_args_to_mindformers_configr)roargsrHrHrIrss8                zBaseTrainer._merge_config default_argscCs&td|t|jj|d}|S)z&Create the train dataset for training.,.........Build Dataset From Config..........r)r'rQrrrRtrain_dataset_task)rorrTrHrHrIcreate_train_dataset)s z BaseTrainer.create_train_datasetcCsjtd|td}|dvrV|jjj|jjj |jj_|jjj|jj |jj_t |jj |d}|S)z%Create the eval dataset for evaluate.rrxrzr) r'rQrrrrRrUrrrrreval_dataset_task)rorrxrUrHrHrIcreate_eval_dataset0s  zBaseTrainer.create_eval_datasetcCsJtdt|jj|d}|}|jj}|dkrBtdt||}||fS)z$Create the network for task trainer.z,.........Build Network From Config..........rr*Nmicro_batch_interleave_num > 1, the double copy parallel feature is turned on.)r'rQrrRmodelrr)ror eval_networkrVrrHrHrIcreate_network>s   zBaseTrainer.create_networkcCsrtdt|jj|d}|}|jj}|jjj}|dkrVtdtt |||d}n t||d}t |}||fS)z9Create the network of pipeline parallel for task trainer.z5.........Build Pipeline Network From Config..........rr*r) micro_size) r'rQrrRrrrrrrr )rorrrVrrrHrHrIcreate_pipeline_networkIs     z#BaseTrainer.create_pipeline_networkcCs$tdt|jjj|d|_|jS)z'Create the image processor for predict.z4.........Build Image Processor From Config..........r)r'rQrrR processorrXrorrHrHrIcreate_image_processorYs   z"BaseTrainer.create_image_processorcCs$tdt|jjj|d|_|jS)z&Create the tokenizer for task trainer.z3.........Build Text Tokenizer From Config..........r)r'rQr rRrrZrrHrHrIcreate_tokenizer`s   zBaseTrainer.create_tokenizerc Cs"td|jj}|jj}|||}|jjjr:|jjjnd}|jjrN|jjnd}|jj j j dur|jj j j |jj j j }t ||||jj} nt|||||d} |durt|jj| |dd|_nZ|jjjdurtd|r|dur||jjj|n|jjj|jj_t|jjd | id|_|jS) z"Create the optimizer for training.z..........Build Optimizer From Config..........g?N) layer_scale layer_decay)params learning_raterzlearning_rate must be inputr)r'rQrRlr_scaleZlr_scale_factorcreate_lr_schedulerrW weight_decayrr model_configmup_width_base hidden_sizer1 lr_scheduler0rrrlearning_rate_scale) rorVrlearning_scale scale_factorrrrratio group_paramsrHrHrIcreate_optimizer_schedulergsR    z&BaseTrainer.create_optimizer_scheduler)rrc Cs~td|}|jjrn|jjdd}|jjdd}|jjdd}|jjjsjt|jjj |}nt|jjj |jjj }|dur|durt dd}|durt dt|||jj_ |jjj dus|jjj dkr|n t|jjj |jj_ |durt|jjj ||jj_ |dur|_t:|||||j6rtd |d!d }|d ur|?}t@tA|jB||d"|jCr|jCnd#|jDr|jDnd$d%}|E|td&tt ddd'd(kr@tF|td)|jG|jj|||jj|jj|jjd*td+d S),z2Train or Fine-tune for BaseTrainer in MindFormers.rqFzH.............Start load resume context from checkpoint..................RANK_ID0rank_*z/cache/checkpoint/rank_z.ckptz*.........Build Dataset For Train..........Nz&.........Build Net For Train..........r*r moe_configrz2.........Using The Existing Network For Train:: %sTrz,.........Build Optimizer For Train..........)rz>.........Build Running Wrapper From Config For Train..........z-.........Using The Existing Model Wrapper: %sz,.........Build Callbacks For Train..........r}r~)r origin_epochs dataset_sizerr initial_epochglobal_batch_sizerz,.........Starting Init Train Model..........metricsr)rWrrzD.............Start resume training from checkpoint..................)rWz6.........Build Evaluate in Training Callback..........rUrrUdr) step_intervalepoch_intervalz*.........Starting Training Model..........rz9.........Model Compiling, Please Wait a Moment...........)r[dataset_sink_moderrz$.........Training Over!.............)Hr_rnruresume_trainingload_checkpointr'rQrrarrbrdrestore_ckpt_name_prefixreplacerrJr/rrr-rrmathceilrrrrr _dataset_helperr]rVrrrrr __class____name__rrrrrrrrrWr data_sizerrrRrdo_evalrr only_save_strategyr.SUPPORT_MODEL_NAMESr)from_pretrained default_checkpoint_download_pathrrr_evaluate_in_trainingeval_step_intervaleval_epoch_intervalrrtrain)rorRrVrrWrr[r_rqrEZ obs_ckpt_urlrD epoch_numrr^rrU eval_callbackrHrHrItraining_process$s                                       zBaseTrainer.training_process)rRrVrr[r^c Ks|d}|dd}|||}td|dur<|}||td|d} |dur|jdur|j|j |j dd\}} n*|dur|jdurtd |jj j |j}|j |dd | dur|} |td |dur|j|d }td |dur|}tdt||| d} |js6|jrn|jtvrRt|jj|_tdt|| ||ddtdttddddkrt|| j|||jjd} td|t | tddS)z(Evaluate for BaseTrainer in MindFormers.rrqFz-.........Build Dataset For Evaluate..........Nz/Create evaluate dataset finish, dataset size:%drrz4.........Using The Existing Network For Evaluate: %srz5.........Build Compute Metrics For Evaluate..........rz/.........Build Callbacks For Evaluate..........z/.........Starting Init Evaluate Model..........rz=.............Start load checkpoint for eval..................T)r z*.........Starting Evaluate Model..........rrrr)r[rz%s = %sz$.........Evaluate Over!.............)!rnrur'rQrrrrVrrrrr rrrrr rr r r)rrr.rrarrevalrrr) rorRrVrr[r^r_rrqrroutputrHrHrIevaluate_processsZ               zBaseTrainer.evaluate_process)rR input_datarLrVrZrXrYcKs|js||dd} ||| }tjddvrT|dvr@td|jjdkrTtd|d urv|j |j|j d d \}} |j |dd |d d |_ | |d ur|jjrt|jj}|d ur|jjrt|jj}|d ur|jjrt|jj}t|} |js|jr`tjddvrN|jjj} td| ftjtd} t|| || ddnt|| |d ddtf|||||d||_| dd }|d ur|j!d ur|j!}n$|jr|jjj!r|jjj!}nd}| dd }|d ur|j"d ur|j"}n |d}t#t$%ddddkr t&||j||d}g}|D]6}t'|t(r`d|vr`|)|dn |)|q6t*|d}|+t,|Wd n1s0Y|-t./dt,|t./d|t./d|S)z'Predict for BaseTrainer in MindFormers.rqFrx)r{r|hybrid_parallel) translationtext_generationzYCurrently distributed predict only support translation and text_generation. Process exit!r*zOCurrently distributed predict dose not support pipeline parallel. Process exit!NrrrrM)shapedtypeinitT) do_predict)rLrrZrXrYtop_k save_filez _result.txtrrrr)r"rQwzoutput result is: %szoutput result is saved at: %sz#.........Predict Over!.............)0r`rnrurcontextr SystemExitrrrrrrMrrrZr rXrrYr rr rr seq_lengthrint32r Oner.r$rr"r#rrarrrvrropenwritercloser'rQ)rorRrrLrVrZrXrYr_rq_rr'Z infer_datar"r#Zoutput_results output_infoZ one_outputr@rHrHrIpredict_processs                    .  zBaseTrainer.predict_processrcCs8|jj}|jd|j||jjjd}|j||S)NF)r)rphaserrrRrr)rorrU origin_phaserrHrHrIrXs   z!BaseTrainer._evaluate_in_training)NN)NF)N)N)N)N)N)N)F)Fr)N)N)N)N)T)rr)NNNNNN)NNNNN)NNNNNNN)@r  __module__ __qualname____doc__rrprrrr+r,boolrurrrtrrrsrrrrrrrrrrrrrrr2rrrrrrrrrr&rr!rr rrr rrsetrrnpndarrayrlistr"r#r/r rrHrHrHrIrKVs6  A!  (+      A  drK)dr4rrargr>r functoolsrtypingrrrnumpyr7 PIL.Imager mindsporerrr mindspore.train.modelr mindspore.trainr mindspore.datasetr mindspore.nn.wrap.cell_wrapperr mindspore.nnrrrrrmindspore.commonrr mindformers.mindformer_bookrmindformers.corerrrrZ"mindformers.core.callback.callbackr mindformers.core.parallel_configrmindformers.datasetrrrmindformers.modelsrrr r!r"r#mindformers.pipeliner$mindformers.wrapperr%mindformers.tools.registerr&mindformers.tools.loggerr'mindformers.tools.utilsr(mindformers.auto_classr) config_argsr+ training_argsr,utilsr-r.r/Zoptimizer_grouped_parametersr0r1r2r3r4r5r6get_trainer_support_task_listrkget_model_name_support_listr get_pipeline_support_task_listZSUPPORT_PIPELINES$get_pipeline_support_input_data_listZSUPPORT_PIPELINE_INPUT_DATAget_project_pathrfrerJrKrHrHrHrIsT                         pengcheng.mind-test/mindformers/trainer/__pycache__/build_trainer.cpython-39.pyc000066400000000000000000000054561471400134000304000ustar00rootroot00000000000000a >cfL@stdZddlmZmZmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZd eeeeddd Zd d ZedS) zBuild Trainer API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) ImageClassificationTrainer"ZeroShotImageClassificationTrainerMaskedImageModelingTrainerMaskedLanguageModelingTrainerImageToTextRetrievalTrainerTranslationTrainerTokenClassificationTrainerTextClassificationTrainer'ContrastiveLanguageImagePretrainTrainerQuestionAnsweringTrainerGeneralTaskTrainerNtrainer)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)a|Build trainer API. Instantiate the task trainer from MindFormerRegister's registry. Args: config (dict): The task trainer's config. Default: None. default_args (dict): The default argument of trainer API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'trainer'. class_name (str): The class name of task trainer API. Default: None. Return: The function instance of task trainer API. Examples: >>> from mindformers import build_trainer >>> trainer_config = {'type': 'image_classification', 'model_name': 'vit'} >>> # 1) use config dict to build trainer >>> cls_trainer_config = build_trainer(trainer_config) >>> # 2) use class name to build trainer >>> cls_trainer_class_name = build_trainer(class_name='image_classification', model_name='vit') N)r) isinstancedictrrget_instance_from_cfgrTRAINER get_instance)rrrrkwargsrL/home/ma-user/work/mindformers-baichuan/mindformers/trainer/build_trainer.py build_trainersrcCstjttjddtjttjddtjttjddtjttjddtjttjddtjt tjddtjt tjddtjt tjd dtjt tjd dtjt tjd dtjttjd dd S)z( register MindFomrers builtin LR class. image_classification)raliaszero_shot_image_classificationmasked_image_modeling fill_masktoken_classificationtext_classification#contrastive_language_image_pretrain translationquestion_answeringimage_to_text_retrievalgeneralN)r register_clsrrrrrr r r rr rr rrrrrregister_mf_trainer8sTr,)NNrN)__doc__mindformers.tools.registerrrrrrrr r r r r rrrrstrrr,rrrrs4 !.pengcheng.mind-test/mindformers/trainer/__pycache__/config_args.cpython-39.pyc000066400000000000000000000232521471400134000300300ustar00rootroot00000000000000a >cfl5@sFdZddlZddlmZmZddlmZddlZddlm Z ddl m Z m Z gdZ eGdd d ZeGd d d eZeGd d d eZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZdS) zMindFormers' Config API.N)OptionalUnion) dataclass)CheckpointMointor)MindFormerRegisterMindFormerModuleType) BaseArgsConfig RunnerConfig DatasetConfigDataLoaderConfigConfigArguments ContextConfig CloudConfigCheckpointConfigParallelContextConfigOptimizerConfigLRConfig WrapperConfigc@seZdZdZgZddZdS)rzBase Argument config.cKsd|dur`|D]N\}}||jvs8Jd|jd||durBqt|trR|j}|||qdS)Nz+The Config Class support input argument is z , but get )items_support_kwargs isinstancer__dict__ __setattr__)selfkwargskeyvaluerJ/home/ma-user/work/mindformers-baichuan/mindformers/trainer/config_args.py__init__$s  zBaseArgsConfig.__init__N)__name__ __module__ __qualname____doc__rrrrrrrsrcsZeZdZdZgdZddeeddddfee ee fe ee e d fd d Z Z S) r zContext Config.)modeprecompile_only device_target device_id save_graphssave_graphs_path enable_dumpauto_tune_modesave_dump_pathenable_reduce_precisionvariable_memory_max_sizeenable_profilingprofiling_optionsenable_auto_mixed_precisionenable_graph_kernelreserve_class_name_in_scope check_bpropmax_device_memoryprint_file_path enable_sparsemax_call_depthenv_config_pathgraph_kernel_flagssave_compile_cacheruntime_num_threadsload_compile_cachegrad_for_scalarpynative_synchronizemempool_block_sizerAscend DEVICE_ID0F.)r$r&r'r(r)c s&tt|jf|||||d|dS)N)r$r'r&r(r))superr r)rr$r&r'r(r)r __class__rrr>szContextConfig.__init__)r r!r"r#rintosgetenvrrstrboolr __classcell__rrrFrr 0s r csFeZdZdZgdZdeedddfeee dfdd Z Z S) rzParallel Context Config.) device_num global_rankgradients_meangradient_fp32_sync parallel_modeauto_parallel_search_mode search_modeparameter_broadcaststrategy_ckpt_load_filestrategy_ckpt_save_file full_batchenable_parallel_optimizerenable_alltoallall_reduce_fusion_configpipeline_stagesgrad_accumulation_stepparallel_optimizer_config comm_fusion STAND_ALONE RANK_SIZE1FrRrNrPc s"tt|jf|||d|dS)Nrc)rErr)rrRrNrPrrFrrrVs zParallelContextConfig.__init__) r r!r"r#rrHrIrJrKrLrrMrrrFrrJs rcs8eZdZdZgdZd eeeeedfdd ZZ S) rzCloud Config For ModelArts.)obs_path root_pathrank_idupload_frequence keep_lastretry retry_timeN/cacheFrdrerfrgrhc s&tt|jf|||||d|dS)Nrm)rErr)rrdrerfrgrhrrFrrrgszCloudConfig.__init__)NrkNrlF) r r!r"r#rrKrHrLrrMrrrFrr_src s<eZdZdZgdZdeeeeeeedfdd ZZS)r z'MindFormers' config when running model.) epochs batch_size sink_mode sink_size initial_epochhas_trained_epocheshas_trained_steps image_size num_classesrqN)rnrorprqrrrsrtc s*tt|jf|||||||d|dS)N)rnrorprqrrrtrs)rEr r) rrnrorprqrrrsrtrrFrrr~szRunnerConfig.__init__)NNNNNNN) r r!r"r#rrHrLrrMrrrFrr tsr c s@eZdZdZeejZd e e e e e e e dfdd Z Z S) rz$MindFormers' save checkpoint config. mindformersNrlTF)prefix directorysave_checkpoint_stepskeep_checkpoint_maxintegrated_save async_save saved_networkc s*tt|jf|||||||d|dS)N)rxryr~rzr{r|r})rErr) rrxryrzr{r|r}r~rrFrrrszCheckpointConfig.__init__)rwNrlrlTFN)r r!r"r#inspectgetfullargspecrargsrrKrHrLrrMrrrFrrs$ rcs0eZdZdZgdZdedfdd ZZS)rz+MindFormers' learning rate schedule config.)typemax_lrmin_lr decay_steps decay_ratepowerend_learning_rate warmup_stepsN)lr_typec sJ|dur,tjtj|d}|jt|jt t |j fd|i|dSN module_type class_namer) rget_clsrLRrextendrrrrErr)rrr lr_schedulerFrrrs zLRConfig.__init__)Nr r!r"r#rrKrrMrrrFrrsrcs>eZdZdZgdZdeeeee fdfdd Z Z S)rzMindFormers' optimizer config.) r learning_ratebeta1beta2epsepsilon weight_decay loss_scalemomentumN) optim_typerc sL|dur,tjtj|d}|jt|jt t |j f||d|dS)Nr)rr) rrr OPTIMIZERrrrrrrErr)rrrr optimizerrFrrrszOptimizerConfig.__init__)NN) r r!r"r#rrKrrrfloatrrMrrrFrrsrcs0eZdZdZgdZdedfdd ZZS)rzMindFormers' wrapper config.)rsens scale_senseN) wrapper_typec sJ|dur,tjtj|d}|jt|jt t |j fd|i|dSr) rrrWRAPPERrrrrrrErr)rrrwrapperrFrrrs zWrapperConfig.__init__)NrrrrFrrsrcs2eZdZdZgdZdeedfdd ZZS)r z MindFormers' data loader config.)r dataset_dir num_samplesnum_parallel_workersshufflesampler extensionsclass_indexing language_pairdecode num_shardsshard_idcachedecrypttaskusagetest_set valid_set padded_sample num_paddedN)dataloader_typerc sL|dur,tjtj|d}|jt|jt t |j f||d|dS)N)r)rr) rrrDATASET_LOADERrrrrrrEr r)rrrr dataloaderrFrrrszDataLoaderConfig.__init__)NNrrrrFrr sr c s|eZdZdZgdZd eeeefeee e feee e feee e fe e e eee e e fdfdd ZZS) r zMindFormers' dataset config.) data_loader input_columnsoutput_columns column_orderdrop_remainderrepeatrorur per_batch_mappython_multiprocessing max_rowsizeroffloadNTrl)rrrrrrroruc s,tt|jf||||||||d| dS)N)rrorurrrrr)rEr r) rrrrrrrrorurrFrrrszDatasetConfig.__init__)NNNNTrlNN)r r!r"r#rrrdictrrKlistrLrHtuplerrMrrrFrr s r cseZdZdZgdZd eeeeeeee e e fee e e fee e e fee e e fee e e fee e e fee e e fee e e fdfd d Z Z S) r zMindFormers' config arguments.) output_dirprofile auto_tunefilepath_prefixautotune_per_step train_dataset eval_datasetZpredict_dataset runner_configrrsave_checkpoint cloud_configseedrunner_wrapper./outputF ./autotune N)rrrrrrrrrrrrrrcs0tt|j||||||||| | | | | |ddS)N)rrrrrrrrrrrrrr)rEr r)rrrrrrrrrrrrrrrrFrrrs zConfigArguments.__init__)rFFrrNNNNNNNNN)r r!r"r#rrKrLrHrrrrrrMrrrFrr s2r )r#rItypingrr dataclassesrrmindformers.core.callbackrmindformers.tools.registerrr__all__rr rrr rrrrr r r rrrrs>  pengcheng.mind-test/mindformers/trainer/__pycache__/optimizer_grouped_parameters.cpython-39.pyc000066400000000000000000000120351471400134000335360ustar00rootroot00000000000000a of@sdZddlZddlmZddlmZddlZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZd d lmZdee eee eedddZdee eedddZddZddZdS)z Grouped parameters of optimizer.N)partial)Optional)Cell)LearningRateSchedule) BaseModel)LearningRateWiseLayer)build_lr)logger)check_keywords_in_nameF?)model weight_decaydynamic_lr_schedule layer_scale layer_decaycs^t|ttfs tdt|iit|drF|tdt|drd| tdfdd| D}d}g}|durR|rD|j j d kr|j j}t|} tt| d |d }tfd d tt| d D}n\|j j dkr0|j j} tt| d d}tfdd tt| d D}ntdd}d}ntdd}i} i} d} | D]} | j|vr|d}nd}d}|dur|| j}d||f}|dur||} || vr|gd| |<|gd| |<t|tr|rt|| | |d<n || |d<| |d| | |d| jqftj| d d}td|t| S)3Get grouped parameters of the network for training.(model type should be BaseModel, but get no_weight_decayzNo weight decay: %sno_weight_decay_keywordszNo weight decay keywords: %scsBg|]:}t|jdks|jds|jvst|js|jqS)r z.bias)lenshapenameendswithr ).0param) skip_keywords skip_params[/home/ma-user/work/mindformers-baichuan/mindformers/trainer/optimizer_grouped_parameters.py 1s    z4get_optimizer_grouped_parameters..NZSwinForImageClassification) num_layersdepthsc3s|]}|VqdSNrrirrr Az3get_optimizer_grouped_parameters..ZViTForImageClassification)r#c3s|]}|VqdSr%rr&r(rr r)Er*zsNow only support VitModel and SwinModel,if use dynamic_lr_schedule and layer_scale, they will be reset and invalid.FzHdynamic_lr_schedule will be reset and invalid when layer_scale is False.r decayZno_decayr z layer_%d_%s)rparamslrr,)indentzParam groups = %s) isinstancerr TypeErrortypehasattrrr infortrainable_params __class____name__configr$sumrget_swin_layerlistreversedrangedepth get_vit_layerwarningrrrappendjsondumpsvalues)rrrrrZdecay_parameters_namesZget_layer_id_funcZ scales_listr$r#Zparameter_group_namesparameter_group_varsscaler group_namelayer_id param_groupsr)rrrr get_optimizer_grouped_parameters sx       $$              rI)rr mup_ratioc Cst|ttfs tdt|dd}gd}i}dgi}|D]}||j|rD|d|qDt |} | j || _ | j || _ t | } | |d<||d<||d<d d g} dgi} |D]}||j| r| d|qd | d<| |d <t |S) rrcSs"|D]}||dkrdSqdS)NrTF)find)rkey_listkrrr find_in_name~sz:get_mup_optimizer_grouped_parameters..find_in_name)z attention.woz attention.wqz attention.wkz attention.wv feed_forwardlm_headr,r-r middle_outputnorm embeddingr Znorm_key)r/rrr0r1r4rr@copydeepcopy learning_ratelr_endrr:rC) rrrJZlr_schedule_configrNZmiddle_weight_key_listrDZmiddle_weight_grouprZmiddle_lr_configZ middle_lrZ norm_key_listZnorm_weight_grouprrr $get_mup_optimizer_grouped_parametersvs4        rXcCsR|drd}n>|dr d}n.|drFt|dd}|d}n|d}|S)z get vit layer)Z cls_tokens mask_tokens pos_embedrzvit.patch_embedz vit.blocks.r"r )r startswithintsplit)rr# layer_numrGrrr r>s    r>cCs|dvrd}n|drd}nr|drt|dd}|dd}|dvrft|d |d }qt|d |t|}|d }n|d }|S) zget swin layer)zencoder.mask_tokenrzencoder.patch_embedzencoder.layersr[r") reductionrRNr )r\r]r^r8)rr#r$r_rGblock_idrrr r9s   r9)Nr NFr )Nr r N)__doc__rA functoolsrtypingrrT mindspore.nnr#mindspore.nn.learning_rate_schedulermindformers.modelsrZmindformers.core.lrrmindformers.corermindformers.toolsr utilsr floatboolrIrXr>r9rrrr s<          V 3pengcheng.mind-test/mindformers/trainer/__pycache__/trainer.cpython-39.pyc000066400000000000000000001127661471400134000272240ustar00rootroot00000000000000a >cf5@sdZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl Zddl mZddlmZddl mZmZdd lmZmZmZmZmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m8Z8ddl9m:Z:ddl;mZ>m?Z?m@Z@mAZAmBZBdgZCe+DZEe+FZGe+HZIe+JZKe+LZMdZNdZOGdddZPd$eQeRddd ZSd%eRd!d"d#ZTdS)&zTrainer API For Import.N) OrderedDict)ListOptionalUnion)Image)Tensor)set_seed)load_checkpointload_param_into_net)TrainOneStepCell OptimizerCell PipelineCellMicroBatchInterleaved)_VirtualDatasetCell)Callback)GeneratorDataset) BatchDataset RepeatDataset)build_parallel_config) build_datasetbuild_dataset_loadercheck_dataset_config BaseDataset)MindFormerBook) build_model BaseModelBaseImageProcessor BaseTokenizerBaseAudioProcessor)logger)MindFormerConfig)ordered_yaml_dump) build_trainer)ConfigArguments)TrainingArguments)check_train_data_loader_typecheck_eval_data_loader_typecheck_optimizer_and_lr_typecheck_wrapper_config config2dictTrainer checkpointconfigsc@seZdZdZd4eeeeee feeeeee e feeeeeee feeee fee eeeeeeeeeeeeeeefeeeeefeeeefedddZd5eeeefeeeeed d d Zd6eeeefeeeeed d dZd7eeeefeedddZd8eeeefeeeeeeejeeefdddZ d9eeeefedddZ!d:ddZ"d;ddZ#ded,d-d.Z)d/d0Z*d?eeeefd1d2d3Z+dS)@r,a Trainer package to train\evaluate\predict class. The trainer interface is used to quickly start training, evaluation and predict for integrated tasks. It also allows users to customize the model, optimizer, dataset, tokenizer, processor, train_one_step, callback, and metric. Args: args (Optional[Union[str, dict, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It support yaml path or config dict or ConfigArguments class. Default: None. task (str): The task name supported. Please refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: 'general'. model (Optional[Union[str, Cell, BaseModel]]): The network for trainer. It support model name supported or BaseModel or MindSpore Cell class. Supported model name can refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: None. model_name (Optional[Union[str]]): The model name supported. Default: None. train_dataset (Optional[Union[str, BaseDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. eval_dataset (Optional[Union[str, BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for text preprocessing. It support BaseTokenizer class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. audio_processor (Optional[BaseAudioProcessor]): The processor for audio preprocessing. It support BaseAudioProcessor class. Default: None. optimizers (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. eval_callbacks (Optional[Union[Callback, List[Callback]]]): The evaluate callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. save_config (bool): Save current the config of task. Default: False. Raises: KeyError: If 'task' or 'model' not in supported trainer. Examples: >>> from mindformers import Trainer >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index] ... ... def __len__(self): ... return len(self._data) >>> #1) input task name and model name to init trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train') >>> #2) input config to init trainer >>> from mindformers.trainer.config_args import ConfigArguments, OptimizerConfig, \ ... RunnerConfig, LRConfig, WrapperConfig >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindspore.train.callback import LossMonitor >>> runner_config = RunnerConfig(epochs=10, batch_size=2, image_size=224) >>> lr_schedule_config = LRConfig(lr_type='WarmUpLR', learning_rate=0.001, warmup_steps=10) >>> optim_config = OptimizerConfig(optim_type='Adam', beta1=0.009, learning_rate=lr_schedule_config) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper_config = WrapperConfig(wrapper_type='TrainOneStepWithLossScaleCell', scale_sense=loss_scale) >>> dataset = GeneratorDataset(source=MyDataLoader(), column_names='image') >>> dataset = dataset.batch(batch_size=2) >>> config = ConfigArguments(seed=2022, runner_config=runner_config, ... optimizer=optim_config, runner_wrapper=wrapper_config) >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... args=config, train_dataset=dataset) >>> #3) input instance to init trainer >>> from mindformers.models import ViTForImageClassification >>> vit_model_with_loss = ViTForImageClassification() >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=vit_model_with_loss.trainable_params()) >>> loss_cb = LossMonitor(per_print_times=2) >>> callbacks = [loss_cb] >>> task_trainer = Trainer(task='image_classification', ... model=vit_model_with_loss, ... args=config, ... optimizers=optimizer, ... train_dataset=dataset, ... callbacks=callbacks) NgeneralF)argstaskmodel model_name train_dataset eval_dataset tokenizerimage_processoraudio_processor optimizerswrapper pet_method callbackseval_callbackscompute_metrics save_configc Ks>||_||_||_| |_||_||_| |_| |_||_||_ | |_ | |_ ||_ ||_ d|_tjdt|_||_tjtjdts tjdt}tjtjttr tjtt}ztj||ddWn:ty zt||WntyYn0Yn0| dur>|dur*td| dur>td|tvsfJdtd|dt|t r| r|d!| n|}|t"vsJd t"d |d||_d|_t|jt#t$frt%d | rtd d|_&nd |_&|j&r(|jdur(tdtdtdd|_t't(|j(|j}|jdkr|jdurf|jj)j*|j+_|jdur|jj,j)j*|j+_|dur||_-n t|t.r|/|nt|t r tj0|rtj|sJd|d|1dsJd|t'|}nt|t2r|t3|dr,t4||t3|drBt5||t3|drXt6||t3|drnt7|||/|j8n t|t9rtd|:|||_-|r|;|j-t%d|j+j<t|t rtj|sJd|d||j-jj=_>d|_t|t r@tj|s.Jd|d||j-jj=_>d|_|durz|j-jdurb||j-j_|j-jdurz||j-j_t?|j-t@tAdd|_Bt@tAdd |_C|jB|j-_B|jC|j-_Cd |_Dd |_Ed |_F|j-jGrtHjIJd!d"vrtK|j-jGtLjMG|j-jGtjNOd#|j-jPtQ|j-j+|_+|j+dur:tRd$dS)%N.T) dirs_exist_okzJwrapper has existed, input model invalid, it should be include in wrapper.zOwrapper has existed, input optimizers invalid, it should be include in wrapper.ztask name must be in z , but get z_{}zmodel must be in z& when model's type is string, but get zXThe model instance has been entered, and the model will not be created from model_configzpet_method is not valid when a model instance is passed in.Currently, only part of the model in MindFormers is supported.Please pass in the model keywordFzARecognizing that a model instance is sent and model_name is None,zit is recommended to select a model configuration that corresponds to the support of MindFormers based on the instance model and set model_name.zOtherwise, they will default to a general configuration.You are advised to pass instances such as optimizers, metric, tokenizer, and processorcommonz#config path must be exist, but get )z.yamlz.ymlz4config file must be end with .yaml or .yml, but get r5r6 optimizerrunner_wrapperzeWhen using the TrainingArguments class, its arguments will override the default config configuration.z&save running config success of %s_new.z*train dataset path must be exist, but get z)eval dataset path must be exist, but get RANK_ID0 RANK_SIZE1 parallel_mode)semi_auto_parallel auto_parallelLOCAL_DEFAULT_PATHz3config must be contain 'trainer' key, but get None.)Sr2r3r4r<r5r6r:r;r7r8r9r=r>r?default_checkpoint_name_or_pathospathjoinDEFAULT_CONFIG_DIRconfigs_directorykwargsexistsCURRENT_PROJECT_PATHshutilcopytree TypeErrorFileExistsErrorr warning SUPPORT_TASKSkeys isinstancestrformatSUPPORT_MODEL_NAMESr rinfois_model_instancer!get __class____name__trainernetworkconfigdictmerge_from_dictrealpathendswithr%hasattrr'r(r)r*__dict__r&"convert_args_to_mindformers_configsave_config_to_yamllower data_loader dataset_dirrintgetenvrank_id device_numis_set_parallel_configis_set_moe_configis_set_recompute_configseedmscontextget_auto_parallel_contextrnprandomenviron setdefault output_dirr$ModuleNotFoundError)selfr1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rTrSmindformers_configs_directory task_configrF/home/ma-user/work/mindformers-baichuan/mindformers/trainer/trainer.py__init__s                                              zTrainer.__init__)train_checkpointresume_trainingauto_trans_ckptdo_evalc KsX|dur$t|ttfs$td||dur0d}|p:|jj}|r|jdurbtdt |jj |_|jdur~t d|j dd|j_|durd|jj j_||j_n:t|trd|jj j_||j_n|jj jj|_d|jj j_|dur||j_|dur||j_|jr|jdd|jjf|j|j |j|j|r<|jnd|j|jdd |dS) aPTrain task for Trainer. This function is used to train or fine-tune the network. Args: train_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It supports real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. resume_training (bool): Whether to perform resume training. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if resume_or_finetune_from_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/eval') >>> # 1) default train task to reproduce model. >>> task_trainer.train() >>> # 2) eval network when train task to reproduce model. >>> task_trainer.train(do_eval=True) >>> # 3) resume train task to auto load the last checkpoint, if training break after 10 epochs. >>> task_trainer.train(train_checkpoint=True, resume_training=True) >>> # 4) resume train task according to checkpoint path, if training break after 10 epochs. >>> task_trainer.train( ... resume_or_finetune_from_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... resume_training=True) NzPresume_or_finetune_from_checkpoint must be one of [None, string, bool], but get F6do_eval is enabled, building eval_dataset from config.9if do_eval is true, eval_dataset must be input, the task  is not support eval now.Tis_trainrirhdatasetrDr6r;r=is_full_config)r^boolr_rYrirr6r rbreval_dataset_task ValueErrorr2r3 model_configcheckpoint_name_or_pathget_last_checkpointr rNrrrcreset_model_instancergtrainr5r:r;r=)rrrrrrTrrrrWsT&             z Trainer.train)finetune_checkpointrrrc Ks|dur$t|ttfs$td||dur0d}|p:|jj}|r|jdurbtdt |jj |_|jdur~t d|j dd|j_|durd|jj j_||j_nxt|trd|jj j_||j_nX|jj jj|_|r |jrtdn|jj jj|j_d|jj j_nd|j_d|jj j_|dur2||j_|durD||j_|jrX|jdd |jjf|j|j |j|j|r||jnd|j|jdd |dS) atFinetune task for Trainer. This function is used to fine-tune the network. Args: finetune_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. if resume_training is true, this checkpoint will be used to restore training of the network. Default: False. resume_training (bool): Whether to perform resume training. Default: False auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if load_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='text_generation', ... model='gpt2', ... pet_method='lora', ... train_dataset='./train', ... eval_dataset='./eval') >>> # 1) default finetune task. >>> task_trainer.finetune() >>> # 2) eval network when finetune model. >>> task_trainer.finetune(do_eval=True) >>> # 3) The last weight in the output directory is automatically loaded to resume training. >>> task_trainer.finetune(finetune_checkpoint=True, resume_training=True) >>> # 4) The last weight in the output directory is automatically loaded for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint=True) >>> # 5) Specify weights to fine-tune. >>> task_trainer.finetune(finetune_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') >>> # 6) Automatically load preset weights for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint='gpt2') Nz=load_checkpoint must be one of [None, string, bool], but get FrrrTzWhen a model instance is identified,the weights that are currently proposed to be fine-tune are specifiedby the finetune_checkpoint argument or a model instance with the model weights already loaded is importedrr)r^rr_rYrirr6r rbrrrr2r3rrrr rNrcr[rrrrgrr5r:r;r=)rrrrrrTrrrfinetunesd+              zTrainer.finetune)eval_checkpointrcKs|dur$t|ttfs$td||dur0d}|durRd|jjj_||j_ njt|trrd|jjj_||j_ nJ|jjjj|_ |r|j rt dn|jjjj|j_ d|jjj_nd|j_ |dur||j_|j r|jdd|jjf|j|j|j|jdd|dS)aEvaluate task for Trainer. This function is used to evaluate the network. Args: eval_checkpoint (Optional[Union[str, bool]]): Used to evaluate the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model Raises: TypeError: if eval_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... eval_dataset='data/imagenet/train') >>> # 1) default evaluate task to test model. >>> task_trainer.evaluate() >>> # 2) evaluate task to auto load the last checkpoint. >>> task_trainer.evaluate(eval_checkpoint=True) >>> # 3) evaluate task according to checkpoint path. >>> task_trainer.evaluate(eval_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') Nz=eval_checkpoint must be one of [None, string, bool], but get FTWhen a model instance is identified,the weights that are currently proposed to be evaluated are specifiedby the eval_checkpoint argument or a model instance with the model weights already loaded is importedr)rirhrr=r)r^rr_rYrir3rrrr rNrcr r[rrrgevaluater6r>)rrrrTrrrr s@     zTrainer.evaluate)predict_checkpointr input_datac Ks|dur$t|ttfs$td||jtvrNtd|jdtd|durZd}|dur|d|jj j _ | |j_ njt|trd|jj j _ ||j_ nJ|jj j j |_|r|jrtdn|jj j j |j_ d|jj j _ nd|j_ |dur||j_|durt|jjj}td t|ttttttjtttf s@Jd |jrT|j dd |j!|d <|j"j#f|j||j |j$|j%|j&dd |}|S)azPredict task for Trainer. This function is used to predict the network. Args: predict_checkpoint (Optional[Union[str, bool]]): Used to predict the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. Return: predict result (dict). Raises: TypeError: if predict_checkpoint is not bool or str type. TypeError: if input_data is not Tensor or np.ndarray or Image or str or list. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16') >>> input_data = "./sunflower.png" >>> # 1) predict task to auto load the last checkpoint. >>> task_trainer.predict(predict_checkpoint=True, input_data=input_data) >>> # 2) predict task according to checkpoint path. >>> task_trainer.predict(predict_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... input_data=input_data) >>> # 3) download and auto load the checkpoint on obs and predict. >>> task_trainer.predict(input_data=input_data) Nz@predict_checkpoint must be one of [None, string, bool], but get zThe z% not support predict, now this tasks z is support predict.FTrz(dataset by config is used as input_data.z`Input data's type must be one of [GeneratorDataset, str, ms.Tensor, np.ndarray, PIL.Image.Image]rr4)rirrhr8r9r7r)'r^rr_rYr2SUPPORT_PIPELINESr]NotImplementedErrorrir3rrrr rNrcr r[rrr6rsrbrrrrrrndarrayrlistrr4rgpredictr8r9r7)rrrrrTZ output_resultrrrrOs`%           zTrainer.predictT)input_checkpointrcCsh|jdur,|jdkr,tdt|jj|_|jdurD|j|ntd|j|jrd|rd| dS)zbuild network for trainer.Nr/z'...........Start Init Network..........z0network will be create in %s task trainer class.) r3r2r rbrri set_trainr[rc_load_model_checkpoint)rrrrrr build_networks   zTrainer.build_networkr#c Csl||jj_||jj_||jj_||jj_||jj_||jj_||jj_| |jj_ | |jj_ ||j_ d|_ dS)an set_parallel_config for the setting global data parallel, model parallel and fusion group. The parallel configure setting for Trainer. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. use_seq_parallel (bool): Whether to enable sequence parallel. Default False. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. micro_batch_interleave_num (int): split num of batch size. Default: 1. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_parallel_config(data_parallel=2, model_parallel=2) TN) riparallel_config data_parallelmodel_parallelexpert_parallelpipeline_stageuse_seq_paralleloptimizer_shardmicro_batch_num vocab_emb_dpgradient_aggregation_groupmicro_batch_interleave_numry) rrrrrrrrrrrrrrset_parallel_configs         zTrainer.set_parallel_configcCs<||jj_||jj_||jj_||jj_||jj_d|_dS)a>Set recompute config. TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. select_recompute (bool): Only Enable recomputation of the attention layer or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_recompute_config(recompute=True) TN)rirecompute_config recomputeselect_recompute!parallel_optimizer_comm_recomputemp_comm_recomputerecompute_slice_activationr{)rrrrrrrrrset_recompute_configs      zTrainer.set_recompute_config皙?皙?c CsZ||jj_||jj_||jj_||jj_||jj_||jj_||jj_||jj_ d|_ dS)aThe configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_moe_config(expert_num=2, capacity_factor=1.2, aux_loss_factor=0.001) TN) ri moe_config expert_numcapacity_factoraux_loss_factornum_experts_chosenexpert_group_sizegroup_wise_a2acomp_comm_parallelcomp_comm_parallel_degreerz) rrrrrrrrrrrrset_moe_configs&        zTrainer.set_moe_configcCs8d|j|j|jfvrdS|jr(td|jr8td|jrHtdt|jts\tdt |j |jj }|j j |_ |j j |_ |j ||j||sdS|j}|j j}tddkr |j j j}|dkrtd tt|j||d }nt|j|d }t|}n |dkr*td t|j|}|`||_dS) z*Reset model instance for new model config.TNz2The incoming model will be configured in parallel.z3The incoming model will be configured in recompute.z-The incoming model will be configured in moe.zJCurrently only the integrated model structure in MindFormers is supported.pipeline_stagesr#zNmicro_batch_interleave_num > 1, the double copy parallel feature is turned on.) micro_size)ryrzr{r rbr^r3rrrrirrrrrr}rrrrr)rrrrhrrrrrr7sD                zTrainer.reset_model_instancecCst|jjjS)z"get train dataloader of mindspore.)rrir5rsrrrrget_train_dataloaderbszTrainer.get_train_dataloadercCst|jjjS)z!get eval dataloader of mindspore.)rrir6rsrrrrget_eval_dataloaderfszTrainer.get_eval_dataloadercsb|jj}tj|td|jddtD}|sos z/Trainer.get_last_checkpoint..Ncstjtj|S)N)rOrPgetmtimerQ)xcheckpoint_dirrrvz-Trainer.get_last_checkpoint..)key) rirrOrPrQDEFAULT_CHECKPOINT_DIRr`rwlistdirsorted)rZ output_folderoutput_checkpoint_pathrrrrjs zTrainer.get_last_checkpointricCsx|dur|j}|jjj}t|}tj|j|d}tj |sRtj |ddtj|d |}t ||dS)z"save now config file to yaml file.N_newT)exist_okz run_{}.yaml) rirgr4_reset_config_for_saverOrPrQrSrrrUmakedirsr`_save_config_to_yaml)rrir4 config_dictZ config_dirZ run_yaml_pathrrrrqys  zTrainer.save_config_to_yamlcCs|jjjj}|dur tdnt|trtj |}|t v}|rrt dt |}t|j|}t dt|q|rt dt|jttfr|j |jjjqtdt|jqtd|ntdt|dS) z!Load model checkpoint to network.Nz;checkpoint_name_or_path is None, not load input checkpoint.z9now input valid checkpoint path, it will load to network.znot load parameters is: %sz9now input valid checkpoint name, it will load to network.z0model must be BaseModel or Cell type, but get %szqinput checkpoint args is invalid, it must be valid and real checkpoint path or a valid checkpoint name,but get %szMcheckpoint_name_or_path type error, it should be one of [None, str], but get )rir3rrr r[r^r_rOrPrUrarbr r r rtyperY)rrZ is_exist_pathZis_checkpoint_namecheckpoint_dictZnot_load_paramsrrrrs,       zTrainer._load_model_checkpoint)r-cCsN|dur||jjj_n0t|tr2||jjj_n|jdurJ|j|jjj_dS)zcheck checkpoint config.TN)rrir3rrr^r_rN)rr-rrr_check_checkpoint_configs   z Trainer._check_checkpoint_config)Nr/NNNNNNNNNr0NNNF)FNNF)FNNF)FN)NNN)NT) r#r#r#r#r#r#FFrT)FFFTF)r#rrr#NFFr)T)N)N),rf __module__ __qualname____doc__rrr_rjr%r&r rrrrrr r rrsetrrrrrrrrrrrrrrrrrrrrrqrrrrrrr,=sj  2 S c"D  ^ ,  1 +)save_file_pathr@c CsT|dur i}t|ddd&}|t|ddddWdn1sF0YdS)zSave Config to Yaml File. Args: save_file_path (str): The real path to save yaml file. Default: None. save_config (dict): The task config. Default: None. Nwzutf-8)encodingFT)default_flow_style allow_unicode sort_keys)openwriter")rr@ file_pointerrrrrsrrcCs|dur i}|}t}|ddurBt|d}|d||ddurjt|d}|d||ddur|ddurt|d}t|d}|d||d||ddur|ddurt|d}t|d}|d||d||ddurt|d}t|d } t|d } t|d } t|d } |d||d | |d | |d | |d | |ddurt|d} |d| |ddurt|d}|d||ddurt|d}|d||ddurHt|d}|d||ddurrt|d}|d|t|}|D]\}}|||q|S)zReset Config According to Yaml File Number. Args: config (dict): The task config. Default: None. model_name (str): The model name to save. Default: 'common'. Nr3 processortrain_dataset_taskr5rr6r~parallelrrr moe_conifg runner_configrErD lr_scheduler=)copyrrdr+popritems)rirrZprocessor_configZtrain_dataset_configZtrain_dataset_task_configZeval_dataset_configZeval_dataset_task_configcontext_configZparallel_context_configrrrrZwrapper_configZ optim_configZ lr_configZ cb_configZ run_configrvaluerrrrsd                 r)NN)N)UrrOrW collectionsrtypingrrrnumpyrZ PIL.Imager mindsporer}rmindspore.commonrr r mindspore.nnr r r rrmindspore.nn.wrap.cell_wrapperrZmindspore.trainrmindspore.datasetrZ!mindspore.dataset.engine.datasetsrr mindformers.core.parallel_configrmindformers.datasetrrrrmindformers.mindformer_bookrmindformers.modelsrrrrrmindformers.tools.loggerr mindformers.tools.registerr!Z!mindformers.tools.register.configr"r$ config_argsr% training_argsr&utilsr'r(r)r*r+__all__get_trainer_support_task_listr\get_model_name_support_listraget_pipeline_support_task_listr$get_pipeline_support_input_data_listSUPPORT_PIPELINE_INPUT_DATAget_project_pathrVrrRr,r_rjrrrrrrsV                    wpengcheng.mind-test/mindformers/trainer/__pycache__/training_args.cpython-39.pyc000066400000000000000000000220041471400134000303700ustar00rootroot00000000000000a >cf9@s~dZddlZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z mZmZd d Zd d ZeGd ddZdS)z'Default Training Arguments for Trainer.N)OptionalUnion) dataclassfield)MindFormerConfig)logger)LRType OptimizerTypeSaveIntervalStrategycCs|durt|trdSdS)z,check task config for adapting hugging-face.NTF) isinstancer)Z check_configr L/home/ma-user/work/mindformers-baichuan/mindformers/trainer/training_args.py_check_task_configsrcCs|dur |S|S)z/check training arguments for adapt MindFormers.Nr ) ori_value new_valuer r r_check_training_args#src@seZdZUdZedddidZeed<edddidZe ed<eddd idZ e ed <eddd idZ e ed <eddd idZ e ed<edddidZ e ed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed <eddd!idZeed"<eddd#idZeed$<eddd%idZeeefed&<eddd'idZeeefed(<ed)dd*idZeed+<eddd,idZ ee!efed-<eddd.idZ"e ed/<eddd0idZ#eed1<eddd2idZ$eed3<eddd4idZ%e&eed5<eddd6idZ'eed7<eddd8idZ(e&eed9<eddd:idZ)e&eed;<edddidZ+e ed?<eddd@idZ,e edA<edddBidZ-eedC<edddDidZ.eedE<ee/0dFdGZ1ee/0dHdIZ2ee/0dJdIZ3dKdLZ4e5dMdNZ6e5dOdPZ7e5dQdRZ8d`e9dSdTdUZ:dVdWZ;dXdYZd^d_Z?dS)aTrainingArgumentsz TrainingArguments is the subset of the arguments we use in our default config **which is relate to the training in MindSpore**. NhelpzQThe output directory where the model predictions and checkpoints will be written.)defaultmetadata output_dirzFThe use_parallel is used to enable distribute parallel of the network. use_parallelz7The profile is used to enable profiling of the network.profilezFThe only_save_strategy is used to enable save strategy of the network.only_save_strategyz9The sink_mode is used to enable data sink of the network. sink_modezUThe sink_size is used to enable data sink number per step for training or evaluation. sink_sizezCGlobal batch size per GPU/TPU core/CPU for training and evaluation. batch_sizez-Batch size per GPU/TPU core/CPU for training.per_device_train_batch_sizez3Batch size m per GPU/TPU core/CPU for evaluation.per_device_eval_batch_sizez$The initial learning rate for AdamW. learning_ratez(Weight decay for AdamW if we apply some. weight_decayzBeta1 for AdamW optimizer adam_beta1zBeta2 for AdamW optimizer adam_beta2zEpsilon for AdamW optimizer. adam_epsilonzMax gradient norm. max_grad_normz+Total number of training epochs to perform.num_train_epochszThe lr scheduler type to use.lr_scheduler_typezThe optimizer type to use.optimrz Linear warmup over warmup_steps. warmup_stepsz$The checkpoint save strategy to use. save_strategyz Whether to merge and save the split Tensor in the automatic parallel scenario. Integrated save function is only supported in automatic parallel scene, not supportedin manual parallel.integrated_savez&Save checkpoint every X updates steps. save_stepsz(Save checkpoint every X updates seconds. save_secondszxLimit the total amount of checkpoints. Deletes the older checkpoints in the output_dir. Default is unlimited checkpointssave_total_limitz:Random seed that will be set at the beginning of training.seedz*Random seed to be used with data samplers. data_seedzWhether to transform checkpoint according to parallel config. auto_trans_ckptzHWhether to load training context info, such as optimizer and epoch num. resume_trainingz.Whether to eval current model while Training. do_evalzDNum of step intervals between each eval, -1 means no step end eval. eval_step_intervalzKNum of epoch intervals between each eval, 1 means eval on every epoch end. eval_epoch_interval RANK_SIZE1 DEVICE_ID0RANK_IDcCsf|jdurtj|j|_|jdur2t|jj|_|jdurJt|jj|_|j durbt |j j|_ dS)N) rospath expanduserr'r valuer(r r*r selfr r r __post_init__s    zTrainingArguments.__post_init__cCs|jS)zget device num for training.) device_numr@r r rget_device_numsz TrainingArguments.get_device_numcCs|jS)zget device id for training.) device_idr@r r r get_device_idszTrainingArguments.get_device_idcCs|jS)zget rank id for training.)rank_idr@r r r get_rank_idszTrainingArguments.get_rank_id) task_configcCs|durt}||||||||||t|j|j|_t|j|j|_t|j |j |_ t|j |j |_ t|j |j |_ t|j |j |_ t|j |j|_ t|j|j|_t|j|j|_t|j|j|_t|j|j|_t|j|j|_dS)zOconvert training arguments to mindformer config type for adapting hugging-face.N)r_adapt_dataset_config_adapt_runner_config_adapt_lr_schedule_config_adapt_optimizer_config_adapt_save_checkpoint_configrrr/r0rrrload_checkpointr1r2r3r4r5r6rArIr r r"convert_args_to_mindformers_configsB     z4TrainingArguments.convert_args_to_mindformers_configcCs`t|jr t|jj|j|j_n tdt|jrRt|jj|j|j_|j_n tddS)zadapt dataset config.zThis task does not support training at the moment; it does not have a default training data configuration,so the per_device_train_batch_size setting will not take effectzThis task does not support evaluate at the moment;it does not have a default evaluate data configuration,so the per_device_eval_batch_size setting will not take effectN) r train_datasetrrrrwarning eval_datasetrrPr r rrJs    z'TrainingArguments._adapt_dataset_configcCsrt|jrnt|jj|j|j_t|jj|j|j_t|jj|j|j_t|jj|j|j_t|jj|j|j_dS)zadapt runner config.N) r runner_configrepochsr&rrZper_epoch_sizerrPr r rrKs       z&TrainingArguments._adapt_runner_configcCsnt|jrj|jjdur*t|jj|j|j_|jjdurJt|jj|j|j_|jjdurjt|jj|j|j_dS)zadapt lr schedule config.N)r lr_scheduletyperr'r r)rPr r rrLs       z+TrainingArguments._adapt_lr_schedule_configcCst|jr|jjdur*t|jj|j|j_|jjdurJt|jj|j|j_|jjdurjt|jj|j|j_|jjdurt|jj|j|j_|jj durt|jj |j |j_ |jj durt|jj |j |j_ dS)zadapt optimizer config.N) r optimizerrXrr(r r!beta1r"beta2r#epsr$rPr r rrMs2             z)TrainingArguments._adapt_optimizer_configcCs|jdurt|jtr|jdurt|jD]\}}t|tr*|ddkr*|jdkr`|j|q*|jdkrt|j|d|j|j|d<n*|jdkrt|j|d|j |j|d<|j r|j nd |j|d <t|j|d |j |j|d <q*dS) zadapt save checkpoint config.NrXCheckpointMointornostepssave_checkpoint_stepssecondssave_checkpoint_secondskeep_checkpoint_maxr+) callbacksr listr* enumeratedictpoprr,r-r.r+)rArIicallbackr r rrNs2       z/TrainingArguments._adapt_save_checkpoint_config)N)@__name__ __module__ __qualname____doc__rrstr__annotations__rboolrrrrrintrrr floatr!r"r#r$r%r&r'rr r(r r)r*r r+r,r-r.rr/r0r1r2r3r4r5r6r<getenvrCrErGrBpropertyrDrFrHrrQrJrKrLrMrNr r r rr*s        r)ror<typingrr dataclassesrrmindformers.tools.registerrmindformers.toolsrutilsr r r rrrr r r rs  pengcheng.mind-test/mindformers/trainer/__pycache__/utils.cpython-39.pyc000066400000000000000000000400171471400134000267050ustar00rootroot00000000000000a >cfa@sdZddlZddlZddlZddlmZddlmZddlZddl Z ddl m Z m Z ddl mZmZmZddl mZddlmZdd lmZdd lmZmZdd lmZerddlZd ZGd ddeeZ Gddde Z!Gddde Z"Gddde Z#Gddde Z$d>e%dddZd?ddZ&ddZ'dd Z(d!d"Z)d#d$Z*d%d&Z+d'd(Z,d@d)d*Z-d+d,Z.dAd.d/Z/dBd0d1Z0d2d3Z1dCd4d5Z2d6d7Z3dDd8d9Z4dEd:d;Z5d}d|vr|ddkr|jjd urd|vr|d|jj_qqd S) z=resume training, load training info from checkpoint to config-The load_checkpoint must be correct, but get loss_scale epoch_numFWhen distributed loads are sliced weights, sink_mode must be set True.rirrNCheckpointMointorN)rkrlrealpathrexistsrsrmrvr@rDrrGrC callbacksr] scale_senseloss_scale_value)rJZ resume_dictcallbackrrr#load_resume_context_from_checkpoints&   rFcCs~|js2tj|jr"tj|js2td|jtddvrRt |||||d|j rlt |}t ||dt |||ddS)z load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt rw parallel_modesemi_auto_parallel auto_parallelhybrid_parallel)do_eval do_predict)dst_ckpt_strategy)rWN)only_save_strategyrkrlr|rr}rsrget_auto_parallel_context build_modelauto_trans_ckptget_dst_strategytransform_ckpt load_ckpt)rJmodelnetworkrKrWrrrrrrtransform_and_load_checkpoints  rcCstddvr|jjstd|r6|jt|nh|rF||nX|jjdkr||jj dkr|td|jjd|jj dt d|j ||jj|jj d |j rtd n|j rtd d S) z#build model, generate strategy filerrrzzGWhen distributed loads are sliced weights, it does not supportepochs = z > 1 and sink_size = z" = 1,sink_size must be more than 1z .........Building model.........)rOepochrEzCOnly_save_strategy is True, model.compile() finished, system exit! zkonly_save_strategy is True, but stand_alone and data_parallel mode do not have strategy file, system exit! N)rrr@rDrinfer_predict_layoutnextcreate_tuple_iteratorrArEr rIbuildr SystemExit)rJrrKrrrrrr8s*     rc Cshttdd}ttdd}d}|dkr0|Strtj|jd}tj ||j j }tj|tj |}tj ||tdtjtd}|d d krt|||n t||td ttj|d }||D]}td |qtdt||jjdkr|d d kr|tdt|} tj | } tj|| } tj | | td| | td| }nd}n|}ntdtjtd}|d krPt||td ttj|d }||D]}td |qtdt||jjdkrF|d d kr@tdt|} td| }nd}n|d }ntdd|_d}|S)z get strategyrgrh RANK_SIZE1Nrstrategyz%.........Collecting strategy.........rz'.........All strtegy as follow......... *_rank_*.ckptz strategy: %sz(.........Collecting %d strategy.........z".........Merging strategy.........z Save %s to %sz!.........Merging succeed.........z.Can't collecting all strategy, device num > 8!F)rGrkrpr rlrnremote_save_urlmoxfile make_dirsparallelstrategy_ckpt_save_filebasenamecopyr rIr wait_collect_all_strategyrsortlenparallel_configpipeline_stagerrFr) rJrank_id world_sizeZdst_strategy_pathZ obs_save_dirlocal_strategy_pathobs_strategy_pathZlocal_strategy_dirlocal_strategy_pathsZmerged_strategy_pathZmerged_strategy_nameZobs_merged_strategy_pathrrrrTsp                rc Cs|jr tnd}|jrtnd}tjtd}tj|dd|ddkr t dt |j }t d|t d |j t d |t d |t |j |d ||t d tj|d|d}t|d}|tr tj|jd}tj|tj|d|d} tj|| t|||trtj|jd}tj|d|} tjtdd|} tj| | td|d| d| ||_ dS)zauto transform ckptrrtransformed_checkpointT)exist_okrz#.........Transforming ckpt.........zSrc ckpt strategy: %sz Src ckpt: %szDst ckpt strategy: %sz Dst ckpt: %sZ checkpoint_z$.........Transform succeed!.........transform_succeed_rank_.txtwrank_Rank z: Save z to N) use_parallelrrrkrlrnr makedirsr rIrsrc_strategy_path_or_dirrmstransform_checkpointsopencloser rrrrrwait_transform copy_parallelprint) rJrrrtransformed_ckpt_dirZsrc_ckpt_strategytransform_succeed_txtftransformed_ckpt_dir_obsZtransform_succeed_txt_obsZrank_transformed_ckpt_dir_obsZrank_transformed_ckpt_dirrrrrsD         rc s@td|dd}trtj|jdfddtd|dD}t|}d}t}|sd}|D]} t j | r~|d 7}qdqqd||krd }qXt|} | t krt d d }qtd qXn^tjtd} tj| d } t}tj | s"t|} | t krd }q"td qtd|d|rsz"wait_transform..rrrTz(Automatic weight transform is timed out!ztransform_succeed_rank_0.txtz: Transform succeed!)rr rkrlrnrrangertimerrr} ELAPSE_TIMEr rFsleepr ) rJrrtime_outZtransform_succeed_txts_obs total_numZall_node_transform_succeed start_timeZnode_transform_succeed_numr elapsed_timerrrrrsJ         rc Cst}d}|rltjtj|d}t||kr>tdq |D](}tj|tj |}tj ||qBttj|d}t||krqq t|} | t krt dd}qtdq |rdSdS)zwait all strategy collect overFz*.ckptrrz%Collection of all strategy timed out!T)rrrrrkrlrnrrrrrr rF) strategy_dirrobs_strategy_dirrrZobs_strategy_pathsrrrrrrrrs,      rcCshtdtj|jr"t|}n t|j}t||}tdt||rdt||}tdt|dS)zload checkpointzD.............Start load checkpoint from checkpoint..................z%Network parameters are not loaded: %sz'Optimizer parameters are not loaded: %sN) r rIrkrlrmrrvrstr)rJrrWruZnot_load_network_paramsZnot_load_optim_paramsrrrrs     rcs\tjstdddtD}|s6dSt|fddd}tj|dS) z-get last checkpoint for resuming or finetune.z is not a real directory,When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/checkpoint/**.ckptcSsg|]}|dr|qS)z.ckpt)endswith)r checkpointrrrr,s z'get_last_checkpoint..Ncstjtj|S)N)rkrlgetmtimern)xrtrr3z%get_last_checkpoint..)rfr>)rkrlrmNotADirectoryErrorlistdirsortedrn)rtZoutput_checkpoint_pathrrrrq$s  rq)r)r)N)NFF)FF)N)N)N)7rrkrr6renumrnumpyr7 mindsporer"mindspore.communication.managementrrrrrr r8mindformers.tools.loggerr mindformers.tools.registerr mindformers.tools.utilsr r Z mindformers.tools.transform_ckptrmoxingrrrrrr#r*r2rGr=rLrSrUr\r_rcrvrrrrrrrrrqrrrrsP              N )+  pengcheng.mind-test/mindformers/trainer/base_trainer.py000066400000000000000000001266711471400134000237270ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Base Trainer.""" import math import os import shutil import time from pprint import pprint from functools import partial from typing import Optional, Union, List import numpy as np from PIL.Image import Image import mindspore as ms from mindspore import Tensor, DatasetHelper from mindspore.train.model import Model from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn.wrap.cell_wrapper import _VirtualDatasetCell from mindspore.nn import TrainOneStepCell, Optimizer, Cell, \ PipelineCell, MicroBatchInterleaved from mindspore.common import initializer as init from mindformers.mindformer_book import MindFormerBook from mindformers.core import build_lr, build_optim, build_callback, build_metric from mindformers.core.callback.callback import EvalCallBack from mindformers.core.parallel_config import build_parallel_config from mindformers.dataset import build_dataset, check_dataset_config, BaseDataset from mindformers.models import build_model, build_processor, build_tokenizer, \ BaseModel, BaseTokenizer, BaseImageProcessor from mindformers.pipeline import pipeline from mindformers.wrapper import build_wrapper from mindformers.tools.register import MindFormerConfig from mindformers.tools.logger import logger from mindformers.tools.utils import count_params from mindformers.auto_class import AutoModel from .config_args import ConfigArguments from .training_args import TrainingArguments from .utils import check_runner_config, transform_and_load_checkpoint, load_resume_context_from_checkpoint, load_merged_checkpoint from .optimizer_grouped_parameters import get_optimizer_grouped_parameters, get_mup_optimizer_grouped_parameters from .utils import set_seed, check_train_data_loader_type, \ check_eval_data_loader_type, check_optimizer_and_lr_type, check_wrapper_config SUPPORT_TASKS = MindFormerBook().get_trainer_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_PIPELINE_INPUT_DATA = MindFormerBook().get_pipeline_support_input_data_list() CURRENT_PROJECT_PATH = MindFormerBook().get_project_path() DEFAULT_CONFIG_DIR = 'configs' def download_ckpt_from_obs(remote_url, local_url, rank): """ Download the dataset from the obs. src_data_url (Str): should be the dataset path in the obs tgt_data_path (Str): the local dataset path rank (Int): the current rank id """ import moxing as mox # print(f"begin download ckpt from {obs_ckpt_url}") # if not os.path.exists(os.path.join(cache_url, f"rank_{rank}")): # os.makedirs(os.path.join(cache_url, f"rank_{rank}"), exist_ok=True) # if not os.path.exists(cache_url): # os.makedirs(cache_url, exist_ok=True) time.sleep(rank % 4) try: mox.file.copy(src_url=remote_url, dst_url=local_url) except Exception as error: print(f"Error: {error}, download again...") time.sleep(10) mox.file.copy(src_url=remote_url, dst_url=local_url) print(f"Rank {rank} download ckpt {remote_url} succeed!", flush=True) class BaseTrainer: r"""Base Task Trainer. Args: task (str): The task name supported. model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, task: str = None, model_name: str = None): if model_name is None: model_name = "model name unspecified." if task is None: task = "task name unspecified." logger.info("Now Running Task is: %s, Model is: %s", task, model_name) self.model_name = model_name self.task = task self.config = None self.default_task_config = None self.train_dataset = None self.eval_dataset = None self.network = None self.optimizer = None self.image_processor = None self.audio_processor = None self.tokenizer = None self.callbacks = None self.eval_callbacks = None self.model_wrapper = None self.compute_metrics = None self.kwargs = None self.pipeline_task = None if not os.path.exists(os.path.join('.', DEFAULT_CONFIG_DIR)): configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) if os.path.exists(os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR)): mindformers_configs_directory = os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR) # python 3.7 版本不支持dirs_exist_ok入参, python 3.8及以上版本支持 try: # adapt to python 3.8+ # pylint: disable=E1123 shutil.copytree(mindformers_configs_directory, configs_directory, dirs_exist_ok=True) except TypeError: try: # adapt to python 3.7 to avoid dirs_exist_ok=False shutil.copytree(mindformers_configs_directory, configs_directory) except FileExistsError: pass if task not in SUPPORT_TASKS.keys(): logger.warning("Input task name is not in the supported list or unspecified.") if task in SUPPORT_TASKS.keys() and model_name not in SUPPORT_TASKS.get(task).keys(): logger.warning("Input model name is not in the supported list or unspecified.") logger.warning("See the list of supported task and model name: %s", SUPPORT_TASKS) logger.warning("The default model config: %s will now be used for the %s task ", SUPPORT_TASKS.get(self.task).get("common"), task) self.model_name = "common" def set_config(self, config: Optional[Union[dict, str, ConfigArguments, TrainingArguments]] = None, is_full_config: bool = False): """Set the task config for task trainer.""" if config is not None: if is_full_config: self.config = config else: self.setup_task_config() self.config = self._merge_config(config) else: self.setup_task_config() self.config = self.default_task_config build_parallel_config(self.config) self._check_global_batch_size_for_auto_parallel() return self.config def setup_task_config(self): """Setup the default task config.""" task_config = None if self.task in SUPPORT_TASKS.keys() and self.model_name in SUPPORT_TASKS.get(self.task).keys(): task_config = MindFormerConfig(SUPPORT_TASKS.get(self.task).get(self.model_name)) if isinstance(task_config, MindFormerConfig): self.default_task_config = task_config else: logger.warning("If the default config arguments is not specified," "you must define the required model or wrapper, optimizer, and so on" "in the train or evaluate or predict attribute function.") def _check_global_batch_size_for_auto_parallel(self): """Check global batch size in auto parallel mode.""" batch_size = self.config.runner_config.batch_size dp = self.config.parallel_config.data_parallel micro_batch_num = self.config.parallel_config.micro_batch_num micro_batch_interleave_num = self.config.micro_batch_interleave_num parallel_mode = ms.get_auto_parallel_context("parallel_mode") full_batch = ms.get_auto_parallel_context("full_batch") pp = self.get_pipeline_stages() if parallel_mode in ["semi_auto_parallel", "auto_parallel"]: if full_batch: if pp > 1: logger.info("Pipeline parallel was opened: pipeline_stages = %s, full batch is True, " "global batch size will be changed: " "global_batch_size = " "batch_size * data_parallel * micro_batch_num * micro_batch_interleave_num " "= %s * %s * %s * %s = %s).", pp, batch_size, dp, micro_batch_num, micro_batch_interleave_num, batch_size * dp * micro_batch_num * micro_batch_interleave_num) self.config.runner_config.batch_size = \ batch_size * dp * micro_batch_num * micro_batch_interleave_num self._reset_wrapper_for_pipeline_parallel() else: logger.info("The current parallel mode is %s, full batch is True," "so global batch size will be changed: " "global_batch_size = batch_size * data_parallel * micro_batch_interleave_num " "= %s * %s * %s = %s", parallel_mode, batch_size, dp, micro_batch_interleave_num, batch_size * dp * micro_batch_interleave_num) self.config.runner_config.batch_size = batch_size * dp * micro_batch_interleave_num else: if pp > 1: logger.info("Pipeline parallel was opened: pipeline_stages = %s, full batch is False, " "batch size per card will be changed: " "per_batch_size = batch_size * micro_batch_num * micro_batch_interleave_num " "= %s * %s * %s = %s).", pp, batch_size, micro_batch_num, micro_batch_interleave_num, batch_size * micro_batch_num * micro_batch_interleave_num) logger.info( "global_batch_size = per_batch_size * device_num " "= %s * %s = %s", batch_size * micro_batch_num * micro_batch_interleave_num, int(os.getenv('RANK_SIZE', '1')), batch_size * micro_batch_num * micro_batch_interleave_num * int(os.getenv('RANK_SIZE', '1'))) self.config.runner_config.batch_size = \ batch_size * micro_batch_num * micro_batch_interleave_num self._reset_wrapper_for_pipeline_parallel() else: logger.info("The current parallel mode is %s, batch size per card will not be changed: " "batch_size_per_card = %s", parallel_mode, batch_size) logger.info( "global_batch_size = batch_size_per_card * device_num " "= %s * %s = %s", batch_size, int(os.getenv('RANK_SIZE', '1')), batch_size * int(os.getenv('RANK_SIZE', '1'))) self.config.parallel_config.data_parallel = 1 self.config.parallel_config.model_parallel = 1 self.config.parallel_config.pipeline_stage = 1 self.config.parallel_config.optimizer_shard = False self.config.parallel_config.micro_batch_num = 1 logger.info("parallel_config will be change to default config: %s.", self.config.parallel_config) def _reset_wrapper_for_pipeline_parallel(self): """Reset wrapper when pipeline parallel.""" if self.config.runner_wrapper is not None: self.config.runner_wrapper.type = "MFPipelineWithLossScaleCell" \ if self.config.runner_wrapper.type != "MFPipelineWithLossScaleCell" else self.config.runner_wrapper.type self.config.runner_wrapper.micro_batch_num = self.config.parallel_config.micro_batch_num logger.warning( "When using the pipeline parallel mode, " "the MFPipelineWithLossScaleCell class is used by default.") else: logger.info( "When using the pipeline parallel mode, " "because the wrapper class is not specified, " "MindSpore's built-in PipelineCell is used by default") logger.info("PipelineWrapper under evaluate or predict mode will not take effect.") def _reset_dataset_batch_size(self): """Reset dataset batch size according to the global batch size of runner config.""" check_dataset_config(self.config) def _merge_config(self, args): """Merge config from default task config.""" if self.default_task_config is None: logger.warning("default task config is None, you will not be able to merge config parameters.") return args if isinstance(args, dict): self.default_task_config.merge_from_dict(args) elif isinstance(args, str): if not (os.path.realpath(args) and os.path.exists(args)): raise FileNotFoundError(f"config path must be exist, but get {args}.") if not args.endswith(('.yaml', '.yml')): raise ValueError(f"config file must be end with .yaml or .yml, but get {args}") self.default_task_config = MindFormerConfig(args) elif isinstance(args, ConfigArguments): if hasattr(args, 'train_dataset'): check_train_data_loader_type(args, self.default_task_config) if hasattr(args, 'eval_dataset'): check_eval_data_loader_type(args, self.default_task_config) if hasattr(args, 'optimizer'): check_optimizer_and_lr_type(args, self.default_task_config) if hasattr(args, 'runner_wrapper'): check_wrapper_config(args, self.default_task_config) self.default_task_config.merge_from_dict(args.__dict__) elif isinstance(args, TrainingArguments): args.convert_args_to_mindformers_config(self.default_task_config) else: logger.warning( "The type of config parameter to merge is not supported, " "currently supported types are [dict, config_path(str), ConfigArguments, TrainingArguments], " "but get %s", type(args)) return self.default_task_config def create_train_dataset(self, default_args: dict = None): """Create the train dataset for training.""" logger.info(".........Build Dataset From Config..........") self._reset_dataset_batch_size() train_dataset = build_dataset(self.config.train_dataset_task, default_args=default_args) return train_dataset def create_eval_dataset(self, default_args: dict = None): """Create the eval dataset for evaluate.""" logger.info(".........Build Dataset From Config..........") self._reset_dataset_batch_size() # reduce batch size on eval mode, for that micro_batch will not take effect on eval. parallel_mode = ms.get_auto_parallel_context("parallel_mode") if parallel_mode in ["semi_auto_parallel", "auto_parallel"]: self.config.eval_dataset.batch_size = \ self.config.eval_dataset.batch_size // self.config.parallel_config.micro_batch_num self.config.eval_dataset.batch_size = \ self.config.eval_dataset.batch_size // self.config.micro_batch_interleave_num eval_dataset = build_dataset(self.config.eval_dataset_task, default_args=default_args) return eval_dataset def create_network(self, default_args: dict = None): """Create the network for task trainer.""" logger.info(".........Build Network From Config..........") eval_network = build_model(self.config.model, default_args=default_args) network = eval_network micro_batch_interleave_num = self.config.micro_batch_interleave_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = MicroBatchInterleaved(network, micro_batch_interleave_num) return network, eval_network def create_pipeline_network(self, default_args: dict = None): """Create the network of pipeline parallel for task trainer.""" logger.info(".........Build Pipeline Network From Config..........") eval_network = build_model(self.config.model, default_args=default_args) network = eval_network micro_batch_interleave_num = self.config.micro_batch_interleave_num micro_batch_num = self.config.parallel_config.micro_batch_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = PipelineCell(MicroBatchInterleaved(network, micro_batch_interleave_num), micro_size=micro_batch_num) else: network = PipelineCell(network, micro_size=micro_batch_num) network = _VirtualDatasetCell(network) return network, eval_network def create_image_processor(self, default_args: dict = None): """Create the image processor for predict.""" logger.info(".........Build Image Processor From Config..........") self.image_processor = build_processor( self.config.processor.image_processor, default_args=default_args) return self.image_processor def create_tokenizer(self, default_args: dict = None): """Create the tokenizer for task trainer.""" logger.info(".........Build Text Tokenizer From Config..........") self.tokenizer = build_tokenizer( self.config.processor.tokenizer, default_args=default_args) return self.tokenizer def create_optimizer_scheduler(self, network, layer_scale=False): """Create the optimizer for training.""" logger.info(".........Build Optimizer From Config..........") # learning rate scale for multi-nodes training learning_scale = self.config.lr_scale scale_factor = self.config.lr_scale_factor # build learning rate schedule lr_schedule = self.create_lr_scheduler(learning_scale, scale_factor) weight_decay = self.config.optimizer.weight_decay if self.config.optimizer.weight_decay else 0. layer_decay = self.config.layer_decay if self.config.layer_decay else 1.0 if self.config.model.model_config.mup_width_base is not None: ratio = self.config.model.model_config.hidden_size / self.config.model.model_config.mup_width_base group_params = get_mup_optimizer_grouped_parameters(network, weight_decay, ratio, self.config.lr_schedule) else: group_params = get_optimizer_grouped_parameters(network, weight_decay, lr_schedule, layer_scale=layer_scale, layer_decay=layer_decay) if lr_schedule is not None: self.optimizer = build_optim( self.config.optimizer, default_args={"params": group_params, "learning_rate": lr_schedule}) else: if self.config.optimizer.learning_rate is None: raise ValueError("learning_rate must be input") self.config.optimizer.learning_rate = self.learning_rate_scale( self.config.optimizer.learning_rate, scale_factor) \ if learning_scale and scale_factor is not None else self.config.optimizer.learning_rate self.optimizer = build_optim( self.config.optimizer, default_args={"params": group_params}) return self.optimizer def create_lr_scheduler(self, learning_scale: bool = False, scale_factor: int = 256): """Create the learning rate scheduler.""" logger.info(".........Build LR Schedule From Config..........") train_data_size = self.get_train_data_size() if self.config.lr_schedule: warmup_epochs = self.config.lr_schedule.pop("warmup_epochs", None) warmup_ratio = self.config.lr_schedule.pop("warmup_ratio", None) # WSD decay_ratio = self.config.lr_schedule.pop("decay_ratio", None) if not self.config.runner_config.sink_mode: total_steps = int(self.config.runner_config.epochs * train_data_size) else: total_steps = int(self.config.runner_config.epochs * self.config.runner_config.sink_size) if warmup_epochs is not None and warmup_ratio is not None: logger.warning("warmup_epochs and warmup_ratio are set simultaneously," "warmup_ratio takes precedence.") warmup_epochs = None if warmup_epochs is not None: logger.warning("warmup_epochs was set in lr_schedule," "it will multiply the data size to represent the warmup steps") self.config.lr_schedule.warmup_steps = int(warmup_epochs * train_data_size) self.config.lr_schedule.total_steps = total_steps \ if self.config.lr_schedule.total_steps is None or self.config.lr_schedule.total_steps == -1 \ else int(self.config.lr_schedule.total_steps) if warmup_ratio is not None: self.config.lr_schedule.warmup_steps = int(self.config.lr_schedule.total_steps * warmup_ratio) if decay_ratio is not None: #WSD self.config.lr_schedule.decay_last_steps = int(self.config.lr_schedule.total_steps * decay_ratio) self.config.lr_schedule.learning_rate = self.learning_rate_scale( self.config.lr_schedule.learning_rate, scale_factor) \ if learning_scale and scale_factor is not None else self.config.lr_schedule.learning_rate lr_schedule = build_lr(self.config.lr_schedule) return lr_schedule def create_model_wrapper(self, network, optimizer): """Create the model wrapper for training.""" logger.info(".........Build Model Wrapper for Train From Config..........") model_wrapper = build_wrapper(self.config.runner_wrapper, default_args={"network": network, "optimizer": optimizer, "parallel_config": self.config.parallel_config}) return model_wrapper def create_callbacks(self, default_args: dict = None): """Create the callback list for training.""" logger.info(".........Build Callbacks for Train From Config..........") self.callbacks = [] if self.config.profile: self.callbacks.append(self.config.profile_cb) self.callbacks.extend(build_callback(self.config.callbacks, default_args=default_args)) return self.callbacks def create_eval_callbacks(self, default_args: dict = None): """Create the eval callback list for training.""" logger.info(".........Build Callbacks for Evaluate From Config..........") self.eval_callbacks = [] self.eval_callbacks.extend(build_callback(self.config.eval_callbacks, default_args=default_args)) return self.eval_callbacks def create_metrics(self, metric_name: str = None): """Create Metrics For Evaluate or Fit.""" if metric_name is None: metric_name = self.model_name + "_metric" self.compute_metrics = {metric_name: build_metric(self.config.metric)} return self.compute_metrics def count_parameters(self): """Count network parameters number.""" if self.network is not None: logger.info("Network Parameters: %s M.", str(count_params(self.network))) elif self.model_wrapper is not None: logger.info("Network Parameters: %s M.", str(count_params(self.model_wrapper.network))) else: logger.warning("Network is None, parameters incalculable.") def set_seed(self, seed: int = None): """Set seed for training.""" if seed is None: if self.config.seed is None: raise ValueError("seed is not set in config, it is None.") set_seed(self.config.seed) else: set_seed(seed) def set_train_dataset(self, dataset): """Set the attribute of train dataset.""" if dataset is None: raise ValueError("Train dataset is None") self.train_dataset = dataset def set_eval_dataset(self, dataset): """Set the attribute of eval dataset .""" if dataset is None: raise ValueError("Eval dataset is None") self.eval_dataset = dataset def set_network(self, network, is_train: bool = True): """Set the attribute of network.""" if network is None: raise ValueError("network is None") if isinstance(network, (Cell, BaseModel)): network.set_train(is_train) self.network = network def set_model_wrapper(self, model_wrapper): """Set the attribute of model_wrapper.""" if model_wrapper is None: raise ValueError("model wrapper is None") self.model_wrapper = model_wrapper def get_train_data_size(self): """Get train dataset size.""" if self.train_dataset is None: raise NotImplementedError("train dataset is None") return self.train_dataset.get_dataset_size() def get_eval_data_size(self): """Get eval dataset size.""" if self.eval_dataset is None: raise NotImplementedError("train dataset is None") return self.eval_dataset.get_dataset_size() def get_pipeline_stages(self): """Get pipeline stages for task trainer.""" pipeline_stages = ms.get_auto_parallel_context("pipeline_stages") return pipeline_stages def learning_rate_scale(self, base_learning_rate: float = 0., scale_factor: Optional[Union[float, int]] = 256.): """Scale learning rate for training.""" if not isinstance(base_learning_rate, float): raise ValueError(f"learning rate must be float type, but get {type(base_learning_rate)}") if not isinstance(scale_factor, (float, int)): raise ValueError(f"scale_factor must be float or int type, but get {type(scale_factor)}") device_num = int(os.getenv("RANK_SIZE", "1")) per_device_batch_size = self.config.train_dataset.batch_size learning_rate = (base_learning_rate * device_num * per_device_batch_size) / scale_factor return learning_rate def training_process( self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, optimizer: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): """Train or Fine-tune for BaseTrainer in MindFormers.""" self.kwargs = kwargs is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) if config.resume_training and config.load_checkpoint: logger.info(".............Start load resume context from checkpoint..................") rank = int(os.getenv("RANK_ID", "0")) if config.resume_merged_ckpt: logger.info(".............Start load merged checkpoint..................") obs_ckpt_url = os.path.join(config.load_checkpoint, config.restore_ckpt_name_prefix) local_url = f"/cache/checkpoint/{config.restore_ckpt_name_prefix}" config.load_checkpoint = local_url if rank%8==0: download_ckpt_from_obs(obs_ckpt_url, local_url, rank) with open('/cache/checkpoint/download_successed.txt', 'w')as fl: fl.write("") else: while not os.path.isfile('/cache/checkpoint/download_successed.txt'): time.sleep(3) else: download_rank = rank%config.parallel.optimizer_weight_shard_size if config.reuse_ckpt else rank #download_rank = rank % 256 obs_ckpt_url = os.path.join(config.load_checkpoint, f"rank_{download_rank}", f"{config.restore_ckpt_name_prefix.replace('*', str(download_rank))}") local_url = f"/cache/checkpoint/rank_{rank}.ckpt" download_ckpt_from_obs(obs_ckpt_url, local_url, rank) config.load_checkpoint = local_url load_resume_context_from_checkpoint(config) # build dataset logger.info(".........Build Dataset For Train..........") if dataset is None: dataset = self.create_train_dataset() self.set_train_dataset(dataset) check_runner_config(config, dataset) config.runner_config.epochs += config.runner_config.initial_epoch if config.runner_config.sink_mode: epoch_num = math.ceil((config.runner_config.epochs - config.runner_config.initial_epoch) * config.runner_config.sink_size / dataset.get_dataset_size()) # pylint: disable=W0212 dataset._dataset_helper = DatasetHelper(dataset, config.runner_config.sink_mode, config.runner_config.sink_size, epoch_num) # build network logger.info(".........Build Net For Train..........") eval_network = None if network is None and wrapper is None and \ self.model_wrapper is None and self.network is None: # If neither network nor wrapper exists, create a network if self.get_pipeline_stages() > 1: network, eval_network = self.create_pipeline_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) else: network, eval_network = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) elif network is None and wrapper is None and self.network is not None: logger.info(".........Using The Existing Network For Train:: %s", self.network.__class__.__name__) network = self.network if network is not None: self.set_network(network, is_train=True) if eval_network is None: eval_network = network if wrapper is not None: self.set_model_wrapper(wrapper) self.count_parameters() # build optimizer logger.info(".........Build Optimizer For Train..........") if optimizer is None and wrapper is None and self.model_wrapper is None: optimizer = self.create_optimizer_scheduler(network, layer_scale=config.layer_scale) # build model wrapper if wrapper is None and self.model_wrapper is None: logger.info(".........Build Running Wrapper From Config For Train..........") wrapper = self.create_model_wrapper(network, optimizer) elif wrapper is None and self.model_wrapper is not None: logger.info(".........Using The Existing Model Wrapper: %s", self.model_wrapper.__class__.__name__) wrapper = self.model_wrapper # build callback logger.info(".........Build Callbacks For Train..........") if callbacks is None: callbacks = self.create_callbacks(default_args={ "learning_rate": optimizer.learning_rate if optimizer else wrapper.optimizer.learning_rate, "origin_epochs": config.runner_config.origin_epochs, "dataset_size": config.data_size, "micro_batch_interleave_num": config.micro_batch_interleave_num, "micro_batch_num": config.parallel_config.micro_batch_num, "initial_epoch": config.runner_config.initial_epoch, "global_batch_size": self.config.runner_config.batch_size, "device_num": int(os.getenv('RANK_SIZE', '1'))}) # define compute metrics for evaluate in training compute_metrics = None if config.do_eval: compute_metrics = self.create_metrics() # define Model and begin training logger.info(".........Starting Init Train Model..........") if wrapper is not None: model = Model(wrapper, metrics=compute_metrics, eval_network=eval_network) else: model = Model(network, optimizer=optimizer, metrics=compute_metrics, eval_network=eval_network) # resume checkpoint if config.load_checkpoint or config.only_save_strategy: if config.resume_training: logger.info(".............Start resume training from checkpoint..................") if config.resume_merged_ckpt: load_merged_checkpoint(config, model, network, dataset, optimizer=optimizer) else: transform_and_load_checkpoint(config, model, network, dataset, optimizer=optimizer) else: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path transform_and_load_checkpoint(config, model, network, dataset) # build evaluate in training if config.do_eval: logger.info(".........Build Evaluate in Training Callback..........") eval_dataset = kwargs.get('eval_dataset', None) if eval_dataset is None: eval_dataset = self.create_eval_dataset() eval_callback = EvalCallBack( partial( self._evaluate_in_training, model=model, eval_dataset=eval_dataset, ), step_interval=config.eval_step_interval if config.eval_step_interval else 100, epoch_interval=config.eval_epoch_interval if config.eval_epoch_interval else -1, ) callbacks.append(eval_callback) logger.info(".........Starting Training Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) logger.info(".........Model Compiling, Please Wait a Moment...........") model.train(config.runner_config.epochs, dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode, sink_size=config.runner_config.sink_size, initial_epoch=config.runner_config.initial_epoch) logger.info(".........Training Over!.............") def evaluate_process( self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): """Evaluate for BaseTrainer in MindFormers.""" metric_name = kwargs.get("metric_name") is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) # build dataset logger.info(".........Build Dataset For Evaluate..........") if dataset is None: dataset = self.create_eval_dataset() self.set_eval_dataset(dataset) logger.info("Create evaluate dataset finish, dataset size:%d", dataset.get_dataset_size()) # build network eval_network = None if network is None and self.network is None: network, eval_network = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) elif network is None and self.network is not None: logger.info(".........Using The Existing Network For Evaluate: %s", self.network.__class__.__name__) network = self.network self.set_network(network, is_train=False) if eval_network is None: eval_network = network self.count_parameters() # build metric logger.info(".........Build Compute Metrics For Evaluate..........") if compute_metrics is None: compute_metrics = self.create_metrics(metric_name=metric_name) # build callback logger.info(".........Build Callbacks For Evaluate..........") if callbacks is None: callbacks = self.create_eval_callbacks() logger.info(".........Starting Init Evaluate Model..........") model = Model(network, metrics=compute_metrics, eval_network=eval_network) if config.load_checkpoint or config.only_save_strategy: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path logger.info(".............Start load checkpoint for eval..................") transform_and_load_checkpoint(config, model, network, dataset, do_eval=True) logger.info(".........Starting Evaluate Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) output = model.eval(dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode) logger.info('%s = %s', metric_name, str(output)) logger.info(".........Evaluate Over!.............") def predict_process(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, task: str = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseImageProcessor] = None, **kwargs): """Predict for BaseTrainer in MindFormers.""" if not self.pipeline_task: is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) if ms.context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: if task not in ["translation", "text_generation"]: raise SystemExit("Currently distributed predict only support translation and text_generation. " "Process exit!") if config.parallel_config.pipeline_stage > 1: raise SystemExit("Currently distributed predict dose not support pipeline parallel. " "Process exit!") # build network if network is None: network, _ = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) self.set_network(network, is_train=False) network.model_name = kwargs.get("model_name", None) self.count_parameters() if tokenizer is None and config.processor.tokenizer: tokenizer = build_tokenizer(config.processor.tokenizer) if image_processor is None and config.processor.image_processor: image_processor = build_processor(config.processor.image_processor) if audio_processor is None and config.processor.audio_processor: audio_processor = build_processor(config.processor.audio_processor) model = Model(network) if config.load_checkpoint or config.only_save_strategy: if ms.context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: seq_length = config.model.model_config.seq_length infer_data = Tensor(shape=(1, seq_length), dtype=ms.int32, init=init.One()) transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True) else: transform_and_load_checkpoint(config, model, network, None, do_predict=True) self.pipeline_task = pipeline( task=task, model=network, tokenizer=tokenizer, image_processor=image_processor, audio_processor=audio_processor, **kwargs ) top_k = kwargs.pop("top_k", None) if top_k is None: if config.top_k is not None: top_k = config.top_k elif config.model and config.model.model_config.top_k: top_k = config.model.model_config.top_k else: top_k = 1 save_file = kwargs.pop("save_file", None) if save_file is None: if config.save_file is not None: save_file = config.save_file else: save_file = f"{task}_result.txt" if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) output_results = self.pipeline_task(input_data, top_k=top_k) output_info = [] for one_output in output_results: if isinstance(one_output, dict) and "info" in one_output: output_info.append(one_output["info"]) else: output_info.append(one_output) with open(save_file, 'w') as file: file.write(str(output_info)) file.close() logger.info("output result is: %s", str(output_info)) logger.info("output result is saved at: %s", save_file) logger.info(".........Predict Over!.............") return output_results def _evaluate_in_training(self, model: Model, eval_dataset: BaseDataset): origin_phase = model.eval_network.phase model.eval_network.set_train(False) output = model.eval( eval_dataset, dataset_sink_mode=self.config.runner_config.sink_mode ) model.eval_network.set_train(origin_phase) return output pengcheng.mind-test/mindformers/trainer/build_trainer.py000066400000000000000000000105141471400134000241000ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Trainer API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from . import ImageClassificationTrainer, ZeroShotImageClassificationTrainer, \ MaskedImageModelingTrainer, MaskedLanguageModelingTrainer, ImageToTextRetrievalTrainer, \ TranslationTrainer, TokenClassificationTrainer, TextClassificationTrainer, \ ContrastiveLanguageImagePretrainTrainer, QuestionAnsweringTrainer, GeneralTaskTrainer def build_trainer( config: dict = None, default_args: dict = None, module_type: str = 'trainer', class_name: str = None, **kwargs): r"""Build trainer API. Instantiate the task trainer from MindFormerRegister's registry. Args: config (dict): The task trainer's config. Default: None. default_args (dict): The default argument of trainer API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'trainer'. class_name (str): The class name of task trainer API. Default: None. Return: The function instance of task trainer API. Examples: >>> from mindformers import build_trainer >>> trainer_config = {'type': 'image_classification', 'model_name': 'vit'} >>> # 1) use config dict to build trainer >>> cls_trainer_config = build_trainer(trainer_config) >>> # 2) use class name to build trainer >>> cls_trainer_class_name = build_trainer(class_name='image_classification', model_name='vit') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.TRAINER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_mf_trainer(): """ register MindFomrers builtin LR class. """ # adapt huggingface MindFormerRegister.register_cls( ImageClassificationTrainer, module_type=MindFormerModuleType.TRAINER, alias="image_classification") MindFormerRegister.register_cls( ZeroShotImageClassificationTrainer, module_type=MindFormerModuleType.TRAINER, alias="zero_shot_image_classification") MindFormerRegister.register_cls( MaskedImageModelingTrainer, module_type=MindFormerModuleType.TRAINER, alias="masked_image_modeling") MindFormerRegister.register_cls( MaskedLanguageModelingTrainer, module_type=MindFormerModuleType.TRAINER, alias="fill_mask") MindFormerRegister.register_cls( TokenClassificationTrainer, module_type=MindFormerModuleType.TRAINER, alias="token_classification") MindFormerRegister.register_cls( TextClassificationTrainer, module_type=MindFormerModuleType.TRAINER, alias="text_classification") MindFormerRegister.register_cls( ContrastiveLanguageImagePretrainTrainer, module_type=MindFormerModuleType.TRAINER, alias="contrastive_language_image_pretrain") MindFormerRegister.register_cls( TranslationTrainer, module_type=MindFormerModuleType.TRAINER, alias="translation") MindFormerRegister.register_cls( QuestionAnsweringTrainer, module_type=MindFormerModuleType.TRAINER, alias="question_answering") MindFormerRegister.register_cls( ImageToTextRetrievalTrainer, module_type=MindFormerModuleType.TRAINER, alias="image_to_text_retrieval") MindFormerRegister.register_cls( GeneralTaskTrainer, module_type=MindFormerModuleType.TRAINER, alias="general") register_mf_trainer() pengcheng.mind-test/mindformers/trainer/causal_language_modeling/000077500000000000000000000000001471400134000256735ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/causal_language_modeling/__init__.py000066400000000000000000000014651471400134000300120ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Language Modeling Trainer.""" from .causal_language_modeling import CausalLanguageModelingTrainer __all__ = ['CausalLanguageModelingTrainer'] pengcheng.mind-test/mindformers/trainer/causal_language_modeling/__pycache__/000077500000000000000000000000001471400134000301035ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/causal_language_modeling/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005211471400134000342710ustar00rootroot00000000000000a ?cf5@sdZddlmZdgZdS)z!Masked Language Modeling Trainer.)CausalLanguageModelingTrainerrN)__doc__causal_language_modelingr__all__rr`/home/ma-user/work/mindformers-baichuan/mindformers/trainer/causal_language_modeling/__init__.pys causal_language_modeling.cpython-39.pyc000066400000000000000000000264721471400134000374610ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/causal_language_modeling/__pycache__a ?cfl<@sBdZddlZddlZddlZddlmZmZmZddlmZddl Z ddl m Z ddl mZddlmZmZmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZmZm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddgZ/e&0Z1e2ej3Gddde,Z4dS)zCausal Image Modeling Trainer.N)OptionalListUnion)pprint)Model)Callback)TrainOneStepCell OptimizerCell)GeneratorDataset) BaseDataset) BaseModel BaseTokenizerbuild_tokenizer)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) build_metric) AutoModel)MindFormerBook)ConfigArguments)TrainingArguments) BaseTrainer)transform_and_load_checkpoint ADGENMetric EmF1Metricc s,eZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd d d Zdd d Zdeeee e e feeeeefeee e feedddZZS)CausalLanguageModelingTraineraCausalLanguageModelingTrainer Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers import CausalLanguageModelingTrainer >>> gen_trainer = CausalLanguageModelingTrainer(model_name="gpt2") >>> gen_trainer.train() >>> res = gen_trainer.predict(input_data = "hello world [MASK]") Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Ntext_generation)superr__init__)selfr __class__p/home/ma-user/work/mindformers-baichuan/mindformers/trainer/causal_language_modeling/causal_language_modeling.pyr";sz&CausalLanguageModelingTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for CausalLanguageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )r(r)r-r*r+r,N)training_process)r#r(r)r*r+r,r-kwargsr&r&r'train>s%z#CausalLanguageModelingTrainer.trainr(r)r*r-compute_metricscKsX|jj}|d||tvr8|j|f|||d|n|jf|||||d|dS)aEvaluate task for CausalLanguageModeling Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It supports CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It supports dict or set in MindSpore's Metric class. Default: None. metric_name)r)r*r2r1N)metrictype setdefaultGENERATE_METRIC_NAMESgenerate_evaluateevaluate_process)r#r(r)r*r-r2r/r3r&r&r'evaluatels* z&CausalLanguageModelingTrainer.evaluatec% Ks|d}|dd}|||}t|jjj} |dkr\| r\|jjdkr\t d|d|j_t d|durv| }| ||dur|j |j |jd d \} }|j|dd |t d |durt|j}|t d |dur|jjrt|jj}t dt||d} |js,|jrd|jtvrHt|jj|_t dt|| ||ddt dtt !ddddkrt"||jjj#} |jjj$} |jjj%}|jjj&}d}d}|j'}|(}t)|*D]\}}|d+}|d+}g}t,|j-dD](}|.t/0t/1|||kdqt/2|}| r\|d|j3jjj}t44}| j5j6|| || |d}g}t,|j-dD]$}|.||t||dqt44}|||j-d}d}t,t7|D]} ||| j-d7}q|dkr||7}|||7}||d}!||d|!}"t d|dd|d||dd d!|dd"d#|||dd$d%||dd&t8j9t|"d'|j:|dd(}#|j:|dd(}$|;|#|$q|<t d)dS)*zYEvaluate the text generate task. Return metrics with Rouge-1, Rouge-2, Rouge-l and BLEU. r3is_full_configFrzSFor metric %s, it only supports batch size equals 1, so reset batch size to 1 here.z-.........Build Dataset For Evaluate..........N)parallel_config moe_config) default_args)is_trainz5.........Build Compute Metrics For Evaluate..........z/.........Build tokenizer For Evaluate..........z/.........Starting Init Evaluate Model..........) eval_networkz=.............Start load checkpoint for eval..................T)do_evalz*.........Starting Evaluate Model..........RANK_ID0rg-C6? input_idslabels) do_sample max_lengthtop_ptop_kzStep[/z ], cost time z.4fzs, zevery example cost time is z, zgenerate speed: z tokens/s, z avg speed: z tokens/s, remaining time: )seconds)skip_special_tokensz(...........Evaluate Over!...............)=get set_configboolmodel model_configmax_new_tokens runner_config batch_sizerinfocreate_eval_datasetset_eval_datasetcreate_networkr=r> set_networkcount_parametersrr4clear processor tokenizerrrload_checkpointonly_save_strategySUPPORT_MODEL_NAMESrfrom_pretrained default_checkpoint_download_pathrintosgetenvrrHrJrKmax_decode_length pad_token_idget_dataset_size enumeratecreate_dict_iteratorasnumpyrangeshapeappendnpmaxargwherearrayr(timepredict_networkgeneratelendatetime timedeltadecodeupdateeval)%r#r(r)r*r2r_r/r3r;Zenable_max_new_tokens_rRrHrJrKrIZtotal_tokens_num total_timeriZ len_datasetiinputsrFrGvalid_length_each_examplej start_timeoutputs output_idsend_timeZ avg_cost_timeZ tokens_num batch_indexZavg_timeZ remain_timeZpres_str labels_strr&r&r'r8s                     &  "   $ z/CausalLanguageModelingTrainer.generate_evaluate)r( input_datar)r_c Ks|dur|j}t|tttfs0tdt|t|trtj |rt |d(}g}|D]}| |qZWdn1s~0Y|}|j f||d||d|S)a Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. It supports 1) a text string to be translated, 1) a file name where each line is a text to be translated and 3) a generator dataset. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. NzMInput data's type must be one of [str, list, GeneratorDataset], but got type rr )r(rtaskr)r_) r isinstancestrlistr ValueErrorr5rfpathisfileopenextendpredict_process) r#r(rr)r_r/fpinput_data_listliner&r&r'predicts( *z%CausalLanguageModelingTrainer.predict)N)NNNNNN)NNNNN)NNNN)NNNN)__name__ __module__ __qualname____doc__rr"rrdictrrrr r r r rr rrr0setr:r8rrr __classcell__r&r&r$r'r-sV  / 4 wr)5rrfrurytypingrrrrnumpyrq mindsporerZmindspore.trainr mindspore.nnrr r mindspore.datasetr mindformers.datasetr mindformers.modelsr rrmindformers.tools.loggerrmindformers.tools.registerrrrmindformers.corermindformers.auto_classrmindformers.mindformer_bookr config_argsr training_argsr base_trainerrutilsrr7get_model_name_support_listrbregisterTRAINERrr&r&r&r's2               pengcheng.mind-test/mindformers/trainer/causal_language_modeling/causal_language_modeling.py000066400000000000000000000361541471400134000332470ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Causal Image Modeling Trainer.""" import os import time import datetime from typing import Optional, List, Union from pprint import pprint import numpy as np from mindspore import Model from mindspore.train import Callback from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer, build_tokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from mindformers.core import build_metric from mindformers.auto_class import AutoModel from mindformers.mindformer_book import MindFormerBook from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer from ..utils import transform_and_load_checkpoint GENERATE_METRIC_NAMES = ['ADGENMetric', 'EmF1Metric'] SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() @MindFormerRegister.register(MindFormerModuleType.TRAINER) class CausalLanguageModelingTrainer(BaseTrainer): r"""CausalLanguageModelingTrainer Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers import CausalLanguageModelingTrainer >>> gen_trainer = CausalLanguageModelingTrainer(model_name="gpt2") >>> gen_trainer.train() >>> res = gen_trainer.predict(input_data = "hello world [MASK]") Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(CausalLanguageModelingTrainer, self).__init__("text_generation", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for CausalLanguageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for CausalLanguageModeling Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It supports CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It supports dict or set in MindSpore's Metric class. Default: None. """ metric_name = config.metric.type kwargs.setdefault("metric_name", metric_name) if metric_name in GENERATE_METRIC_NAMES: self.generate_evaluate( config, network=network, dataset=dataset, compute_metrics=compute_metrics, **kwargs) else: self.evaluate_process( config=config, network=network, dataset=dataset, callbacks=callbacks, compute_metrics=compute_metrics, **kwargs) def generate_evaluate(self, config, network=None, dataset=None, compute_metrics=None, tokenizer=None, **kwargs): r"""Evaluate the text generate task. Return metrics with Rouge-1, Rouge-2, Rouge-l and BLEU. """ metric_name = kwargs.get("metric_name") is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) enable_max_new_tokens = bool(config.model.model_config.max_new_tokens) # it does not support max_new_tokens as input parameter in text_generate, so reset batch_size to 1 when the # follow scenario happens if metric_name == "EmF1Metric" and enable_max_new_tokens and config.runner_config.batch_size != 1: logger.info("For metric %s, it only supports batch size equals 1, so reset batch size to 1 here.", metric_name) config.runner_config.batch_size = 1 # build dataset logger.info(".........Build Dataset For Evaluate..........") if dataset is None: dataset = self.create_eval_dataset() self.set_eval_dataset(dataset) # build network if network is None: _, network = self.create_network(default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) self.set_network(network, is_train=False) self.count_parameters() # build metric logger.info(".........Build Compute Metrics For Evaluate..........") if compute_metrics is None: compute_metrics = build_metric(config.metric) compute_metrics.clear() # build tokenizer logger.info(".........Build tokenizer For Evaluate..........") if tokenizer is None and config.processor.tokenizer: tokenizer = build_tokenizer(config.processor.tokenizer) logger.info(".........Starting Init Evaluate Model..........") model = Model(network, eval_network=network) if config.load_checkpoint or config.only_save_strategy: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path logger.info(".............Start load checkpoint for eval..................") transform_and_load_checkpoint(config, model, network, dataset, do_eval=True) logger.info('.........Starting Evaluate Model..........') if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) # generate config do_sample = config.model.model_config.do_sample top_p = config.model.model_config.top_p top_k = config.model.model_config.top_k max_length = config.model.model_config.max_decode_length total_tokens_num = 0 total_time = 0.0001 pad_token_id = tokenizer.pad_token_id len_dataset = dataset.get_dataset_size() for i, inputs in enumerate(dataset.create_dict_iterator()): input_ids = inputs['input_ids'].asnumpy() labels = inputs['labels'].asnumpy() valid_length_each_example = [] for j in range(input_ids.shape[0]): # As the nonzero returns the index and we need length valid_length_each_example.append(np.max(np.argwhere(input_ids[j] != pad_token_id)) + 1) valid_length_each_example = np.array(valid_length_each_example) if enable_max_new_tokens: # When we act as it, the batch_size is 1. it will be replaced when text_generator supports batch_size # inference quickly or text_generator supports max_new_tokens as the input parameter. max_length = valid_length_each_example[0] + self.config.model.model_config.max_new_tokens start_time = time.time() outputs = model.predict_network.generate(input_ids, do_sample=do_sample, max_length=max_length, top_p=top_p, top_k=top_k) output_ids = [] for j in range(input_ids.shape[0]): output_ids.append(outputs[j][int(valid_length_each_example[j]):]) end_time = time.time() avg_cost_time = (end_time - start_time) / input_ids.shape[0] tokens_num = 0 for batch_index in range(len(output_ids)): tokens_num += output_ids[batch_index].shape[0] if i != 0: total_tokens_num += tokens_num total_time += end_time - start_time # compute time remaining avg_time = total_time / (i + 1) remain_time = (len_dataset - i - 1) * avg_time logger.info(f"Step[{i+1}/{len_dataset}], cost time {end_time-start_time:.4f}s, "+ f"every example cost time is {avg_cost_time:.4f}, "+ f"generate speed: {tokens_num/(end_time-start_time):.4f} tokens/s, "+ f"avg speed: {total_tokens_num/total_time:.4f} tokens/s, " f"remaining time: {datetime.timedelta(seconds=int(remain_time))}") # decode input_id and label to string pres_str = tokenizer.decode(output_ids, skip_special_tokens=True) labels_str = tokenizer.decode(labels, skip_special_tokens=True) compute_metrics.update(pres_str, labels_str) compute_metrics.eval() logger.info('...........Evaluate Over!...............') def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list, GeneratorDataset]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. It supports 1) a text string to be translated, 1) a file name where each line is a text to be translated and 3) a generator dataset. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ if input_data is None: input_data = config.input_data if not isinstance(input_data, (str, list, GeneratorDataset)): raise ValueError("Input data's type must be one of " f"[str, list, GeneratorDataset], but got type {type(input_data)}") if isinstance(input_data, str) and os.path.isfile(input_data): with open(input_data, 'r') as fp: input_data_list = [] for line in fp: input_data_list.extend(line) input_data = input_data_list return self.predict_process(config=config, input_data=input_data, task='text_generation', network=network, tokenizer=tokenizer, **kwargs) pengcheng.mind-test/mindformers/trainer/config_args.py000066400000000000000000000325541471400134000235460ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers' Config API.""" import os from typing import Optional, Union from dataclasses import dataclass import inspect from mindformers.core.callback import CheckpointMointor from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType __all__ = ['BaseArgsConfig', 'RunnerConfig', 'DatasetConfig', 'DataLoaderConfig', 'ConfigArguments', 'ContextConfig', 'CloudConfig', 'CheckpointConfig', 'ParallelContextConfig', 'OptimizerConfig', 'LRConfig', 'WrapperConfig'] @dataclass class BaseArgsConfig: """Base Argument config.""" _support_kwargs = [] def __init__(self, **kwargs): if kwargs is not None: for key, value in kwargs.items(): assert key in self._support_kwargs, \ f"The Config Class support input argument is {self._support_kwargs}, but get {key}" if value is None: continue if isinstance(value, BaseArgsConfig): value = value.__dict__ self.__setattr__(key, value) @dataclass class ContextConfig(BaseArgsConfig): """Context Config.""" _support_kwargs = [ 'mode', 'precompile_only', 'device_target', 'device_id', 'save_graphs', 'save_graphs_path', 'enable_dump', 'auto_tune_mode', 'save_dump_path', 'enable_reduce_precision', 'variable_memory_max_size', 'enable_profiling', 'profiling_options', 'enable_auto_mixed_precision', 'enable_graph_kernel', 'reserve_class_name_in_scope', 'check_bprop', 'max_device_memory', 'print_file_path', 'enable_sparse', 'max_call_depth', 'env_config_path', 'graph_kernel_flags', 'save_compile_cache', 'runtime_num_threads', 'load_compile_cache', 'grad_for_scalar', 'pynative_synchronize', 'mempool_block_size' ] def __init__(self, mode: Optional[Union[int, str]] = 0, device_target: str = "Ascend", device_id: int = int(os.getenv('DEVICE_ID', '0')), save_graphs: bool = False, save_graphs_path: str = ".", **kwargs): super(ContextConfig, self).__init__(mode=mode, device_id=device_id, device_target=device_target, save_graphs=save_graphs, save_graphs_path=save_graphs_path, **kwargs) @dataclass class ParallelContextConfig(BaseArgsConfig): """Parallel Context Config.""" _support_kwargs = [ 'device_num', 'global_rank', 'gradients_mean', 'gradient_fp32_sync', 'parallel_mode', 'auto_parallel_search_mode', 'search_mode', 'parameter_broadcast', 'strategy_ckpt_load_file', 'strategy_ckpt_save_file', 'full_batch', 'enable_parallel_optimizer', 'enable_alltoall', 'all_reduce_fusion_config', 'pipeline_stages', 'grad_accumulation_step', 'parallel_optimizer_config', 'comm_fusion' ] def __init__(self, parallel_mode: str = 'STAND_ALONE', device_num: int = int(os.getenv('RANK_SIZE', '1')), gradients_mean: bool = False, **kwargs): super(ParallelContextConfig, self).__init__(parallel_mode=parallel_mode, device_num=device_num, gradients_mean=gradients_mean, **kwargs) @dataclass class CloudConfig(BaseArgsConfig): """Cloud Config For ModelArts.""" _support_kwargs = [ 'obs_path', 'root_path', 'rank_id', 'upload_frequence', 'keep_last', 'retry', 'retry_time' ] def __init__(self, obs_path: str = None, root_path: str = '/cache', rank_id: int = None, upload_frequence: int = 1, keep_last: bool = False, **kwargs): super(CloudConfig, self).__init__(obs_path=obs_path, root_path=root_path, rank_id=rank_id, upload_frequence=upload_frequence, keep_last=keep_last, **kwargs) @dataclass class RunnerConfig(BaseArgsConfig): """MindFormers' config when running model.""" _support_kwargs = [ 'epochs', 'batch_size', 'sink_mode', 'sink_size', 'initial_epoch', 'has_trained_epoches', 'has_trained_steps', 'image_size', 'num_classes', 'sink_size', ] def __init__(self, epochs: int = None, batch_size: int = None, sink_mode: bool = None, sink_size: int = None, initial_epoch: int = None, has_trained_epoches: int = None, has_trained_steps: int = None, **kwargs): super(RunnerConfig, self).__init__(epochs=epochs, batch_size=batch_size, sink_mode=sink_mode, sink_size=sink_size, initial_epoch=initial_epoch, has_trained_steps=has_trained_steps, has_trained_epoches=has_trained_epoches, **kwargs) @dataclass class CheckpointConfig(BaseArgsConfig): """MindFormers' save checkpoint config.""" _support_kwargs = inspect.getfullargspec(CheckpointMointor).args def __init__(self, prefix: str = 'mindformers', directory: str = None, save_checkpoint_steps: int = 1, keep_checkpoint_max: int = 1, integrated_save: bool = True, async_save: bool = False, saved_network: bool = None, **kwargs): super(CheckpointConfig, self).__init__(prefix=prefix, directory=directory, saved_network=saved_network, save_checkpoint_steps=save_checkpoint_steps, keep_checkpoint_max=keep_checkpoint_max, integrated_save=integrated_save, async_save=async_save, **kwargs) @dataclass class LRConfig(BaseArgsConfig): """MindFormers' learning rate schedule config.""" _support_kwargs = [ 'type', 'max_lr', 'min_lr', 'decay_steps', 'decay_rate', 'power', 'end_learning_rate', 'warmup_steps' ] def __init__(self, lr_type: str = None, **kwargs): if lr_type is not None: lr_schedule = MindFormerRegister.get_cls( module_type=MindFormerModuleType.LR, class_name=lr_type) self._support_kwargs.extend(inspect.getfullargspec(lr_schedule).args) super(LRConfig, self).__init__(type=lr_type, **kwargs) @dataclass class OptimizerConfig(BaseArgsConfig): """MindFormers' optimizer config.""" _support_kwargs = [ 'type', 'learning_rate', 'beta1', 'beta2', 'eps', 'epsilon', 'weight_decay', 'loss_scale', 'momentum' ] def __init__(self, optim_type: str = None, learning_rate: Optional[Union[BaseArgsConfig, float]] = None, **kwargs): if optim_type is not None: optimizer = MindFormerRegister.get_cls( module_type=MindFormerModuleType.OPTIMIZER, class_name=optim_type) self._support_kwargs.extend(inspect.getfullargspec(optimizer).args) super(OptimizerConfig, self).__init__(type=optim_type, learning_rate=learning_rate, **kwargs) @dataclass class WrapperConfig(BaseArgsConfig): """MindFormers' wrapper config.""" _support_kwargs = [ 'type', 'sens', 'scale_sense' ] def __init__(self, wrapper_type: str = None, **kwargs): if wrapper_type is not None: wrapper = MindFormerRegister.get_cls( module_type=MindFormerModuleType.WRAPPER, class_name=wrapper_type) self._support_kwargs.extend(inspect.getfullargspec(wrapper).args) super(WrapperConfig, self).__init__(type=wrapper_type, **kwargs) @dataclass class DataLoaderConfig(BaseArgsConfig): """MindFormers' data loader config.""" _support_kwargs = [ 'type', 'dataset_dir', 'num_samples', 'num_parallel_workers', 'shuffle', 'sampler', 'extensions', 'class_indexing', 'language_pair', 'decode', 'num_shards', 'shard_id', 'cache', 'decrypt', 'task', 'usage', 'test_set', 'valid_set', 'padded_sample', 'num_padded' ] def __init__(self, dataloader_type: str = None, dataset_dir: str = None, **kwargs): if dataloader_type is not None: dataloader = MindFormerRegister.get_cls( MindFormerModuleType.DATASET_LOADER, class_name=dataloader_type) self._support_kwargs.extend(inspect.getfullargspec(dataloader).args) super(DataLoaderConfig, self).__init__(type=dataloader_type, dataset_dir=dataset_dir, **kwargs) @dataclass class DatasetConfig(BaseArgsConfig): """MindFormers' dataset config.""" _support_kwargs = [ 'data_loader', 'input_columns', 'output_columns', 'column_order', 'drop_remainder', 'repeat', 'batch_size', 'image_size', 'num_parallel_workers', 'per_batch_map', 'python_multiprocessing', 'max_rowsize', 'cache', 'offload' ] def __init__(self, data_loader: Optional[Union[dict, BaseArgsConfig]] = None, input_columns: Optional[Union[str, list]] = None, output_columns: Optional[Union[str, list]] = None, column_order: Optional[Union[str, list]] = None, drop_remainder: bool = True, repeat: int = 1, batch_size: int = None, image_size: Optional[Union[int, list, tuple]] = None, **kwargs): super(DatasetConfig, self).__init__(data_loader=data_loader, batch_size=batch_size, image_size=image_size, repeat=repeat, input_columns=input_columns, output_columns=output_columns, column_order=column_order, drop_remainder=drop_remainder, **kwargs) @dataclass class ConfigArguments(BaseArgsConfig): """MindFormers' config arguments.""" _support_kwargs = [ 'output_dir', 'profile', 'auto_tune', 'filepath_prefix', 'autotune_per_step', 'train_dataset', 'eval_dataset', 'predict_dataset', 'runner_config', 'optimizer', 'lr_schedule', 'save_checkpoint', 'cloud_config', 'seed', 'runner_wrapper' ] def __init__(self, output_dir: str = './output', profile: bool = False, auto_tune: bool = False, filepath_prefix: str = './autotune', autotune_per_step: int = 10, seed: int = None, train_dataset: Optional[Union[dict, BaseArgsConfig]] = None, eval_dataset: Optional[Union[dict, BaseArgsConfig]] = None, runner_config: Optional[Union[dict, BaseArgsConfig]] = None, optimizer: Optional[Union[dict, BaseArgsConfig]] = None, runner_wrapper: Optional[Union[dict, BaseArgsConfig]] = None, lr_schedule: Optional[Union[dict, BaseArgsConfig]] = None, save_checkpoint: Optional[Union[dict, BaseArgsConfig]] = None, cloud_config: Optional[Union[dict, BaseArgsConfig]] = None): super(ConfigArguments, self).__init__(output_dir=output_dir, profile=profile, auto_tune=auto_tune, seed=seed, filepath_prefix=filepath_prefix, autotune_per_step=autotune_per_step, train_dataset=train_dataset, eval_dataset=eval_dataset, runner_config=runner_config, optimizer=optimizer, runner_wrapper=runner_wrapper, lr_schedule=lr_schedule, save_checkpoint=save_checkpoint, cloud_config=cloud_config) pengcheng.mind-test/mindformers/trainer/contrastive_language_image_pretrain/000077500000000000000000000000001471400134000301545ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/contrastive_language_image_pretrain/__init__.py000066400000000000000000000015421471400134000322670ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Contrastive Language Image Pretrain Trainer API.""" from .contrastive_language_image_pretrain import ContrastiveLanguageImagePretrainTrainer __all__ = ['ContrastiveLanguageImagePretrainTrainer'] pengcheng.mind-test/mindformers/trainer/contrastive_language_image_pretrain/__pycache__/000077500000000000000000000000001471400134000323645ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000006001471400134000364710ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/contrastive_language_image_pretrain/__pycache__a ?cfb@sdZddlmZdgZdS)z0Contrastive Language Image Pretrain Trainer API.)'ContrastiveLanguageImagePretrainTrainerrN)__doc__#contrastive_language_image_pretrainr__all__rrk/home/ma-user/work/mindformers-baichuan/mindformers/trainer/contrastive_language_image_pretrain/__init__.pys contrastive_language_image_pretrain.cpython-39.pyc000066400000000000000000000101621471400134000442100ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/contrastive_language_image_pretrain/__pycache__a ?cf@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZmZd d lmZd d lmZd d lmZeejGd ddeZdS)z,Contrastive Language Image Pretrain Trainer.)OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerc seZdZdZd edfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZd d Zd d ZZS)'ContrastiveLanguageImagePretrainTraineraContrastive Language Image Pretrain Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers import ContrastiveLanguageImagePretrainTrainer >>> trainer = ContrastiveLanguageImagePretrainTrainer(model_name="clip_vit_b_b32") >>> trainer.train() N) model_namecstt|d|dS)N#contrastive_language_image_pretrain)superr__init__)selfr __class__/home/ma-user/work/mindformers-baichuan/mindformers/trainer/contrastive_language_image_pretrain/contrastive_language_image_pretrain.pyr0s z0ContrastiveLanguageImagePretrainTrainer.__init__)confignetworkdataset optimizerwrapper callbacksc Ks"|jf||||||d|dS)aTrain For Trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The optimizer used for training .Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It supports CallBack or CallBack List of MindSpore. Default: None. )rrr"rr r!N)training_process)rrrrr r!r"kwargsrrrtrain4sz-ContrastiveLanguageImagePretrainTrainer.traincKs tddS)NzGThe contrastive language image pretrain task does not support evaluate.NotImplementedErrorrr$rrrevaluateZsz0ContrastiveLanguageImagePretrainTrainer.evaluatecKs tddS)NzFThe contrastive language image pretrain task does not support predict.r&r(rrrpredict^sz/ContrastiveLanguageImagePretrainTrainer.predict)N)NNNNNN)__name__ __module__ __qualname____doc__strrrrdictrrrr r r rrrrrr%r)r* __classcell__rrrrr s$ &rN)r.typingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr mindformers.tools.registerr r r config_argsr training_argsr base_trainerrregisterTRAINERrrrrrs        contrastive_language_image_pretrain.py000066400000000000000000000103201471400134000377150ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/contrastive_language_image_pretrain# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Contrastive Language Image Pretrain Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer @MindFormerRegister.register(MindFormerModuleType.TRAINER) class ContrastiveLanguageImagePretrainTrainer(BaseTrainer): r"""Contrastive Language Image Pretrain Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers import ContrastiveLanguageImagePretrainTrainer >>> trainer = ContrastiveLanguageImagePretrainTrainer(model_name="clip_vit_b_b32") >>> trainer.train() """ def __init__(self, model_name: str = None): super(ContrastiveLanguageImagePretrainTrainer, self).__init__( "contrastive_language_image_pretrain", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, optimizer: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train For Trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The optimizer used for training .Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It supports CallBack or CallBack List of MindSpore. Default: None. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, optimizer=optimizer, wrapper=wrapper, **kwargs) def evaluate(self, **kwargs): raise NotImplementedError( "The contrastive language image pretrain task does not support evaluate.") def predict(self, **kwargs): raise NotImplementedError( "The contrastive language image pretrain task does not support predict.") pengcheng.mind-test/mindformers/trainer/general_task_trainer/000077500000000000000000000000001471400134000250655ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/general_task_trainer/__init__.py000066400000000000000000000014331471400134000271770ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """General Task Example For Trainer.""" from .general_task_trainer import GeneralTaskTrainer __all__ = ['GeneralTaskTrainer'] pengcheng.mind-test/mindformers/trainer/general_task_trainer/__pycache__/000077500000000000000000000000001471400134000272755ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/general_task_trainer/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004761471400134000334740ustar00rootroot00000000000000a ?cf@sdZddlmZdgZdS)z!General Task Example For Trainer.)GeneralTaskTrainerrN)__doc__general_task_trainerr__all__rr\/home/ma-user/work/mindformers-baichuan/mindformers/trainer/general_task_trainer/__init__.pys general_task_trainer.cpython-39.pyc000066400000000000000000000204631471400134000360370ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/general_task_trainer/__pycache__a ?cfT$@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZmZd d lmZd d lmZd d lmZeejGd ddeZdS)z!General Task Example For Trainer.)OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerc seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd d d Zd d ZZS)GeneralTaskTraineraK General Task Example For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell, Accuracy >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.models import ViTForImageClassification, ViTConfig, ViTImageProcessor >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self._label = [np.ones(1000, np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index], self._label[index] ... ... def __len__(self): ... return len(self._data) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']) >>> train_dataset = train_dataset.batch(batch_size=2) >>> eval_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']) >>> eval_dataset = eval_dataset.batch(batch_size=2) >>> general_task = GeneralTaskTrainer(model_name='common') >>> vit_config = ViTConfig(batch_size=2) >>> network_with_loss = ViTForImageClassification(vit_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> general_task.train(wrapper=wrapper, dataset=train_dataset) >>> compute_metrics = {"Accuracy": Accuracy(eval_type='classification')} >>> general_task.evaluate(network=network_with_loss, dataset=eval_dataset, compute_metrics=compute_metrics) Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Ngeneral)superr__init__)selfr __class__h/home/ma-user/work/mindformers-baichuan/mindformers/trainer/general_task_trainer/general_task_trainer.pyrHszGeneralTaskTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc KsJ|durtd|dur(|dur(td|jf||||||d|dS)aTrain task for General-Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If network or dataset not implemented. Nz+train dataset must be define, but get None.z+train network must be define, but get None.)rrr"rr r!)NotImplementedErrortraining_process)rrrrr r!r"kwargsrrrtrainKs%zGeneralTaskTrainer.train)rrrr"compute_metricscKs`|durtd|dur td|dur0tdd}|d||jf|||||d|dS)aEvaluate task for General-Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callback, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. Raises: NotImplementedError: If network or dataset or compute_metrics not implemented. Nz*eval dataset must be define, but get None.z*eval network must be define, but get None.z*eval metrics must be define, but get None.zGeneral Task Metrics metric_name)rrrr'r")r# setdefaultevaluate_process)rrrrr"r'r%r(rrrevaluate~s"" zGeneralTaskTrainer.evaluatecKs tddS)NzOThe general task does not support predict, please customize pipeline inference.)r#)rr%rrrpredictszGeneralTaskTrainer.predict)N)NNNNNN)NNNNN)__name__ __module__ __qualname____doc__strrrrdictrrrr r r rrrrrr&setr+r, __classcell__rrrrrs:' 4 5rN)r0typingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr mindformers.tools.registerr r r config_argsr training_argsr base_trainerrregisterTRAINERrrrrrs        pengcheng.mind-test/mindformers/trainer/general_task_trainer/general_task_trainer.py000066400000000000000000000221241471400134000316230ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """General Task Example For Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer @MindFormerRegister.register(MindFormerModuleType.TRAINER) class GeneralTaskTrainer(BaseTrainer): r"""General Task Example For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell, Accuracy >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.models import ViTForImageClassification, ViTConfig, ViTImageProcessor >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self._label = [np.ones(1000, np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index], self._label[index] ... ... def __len__(self): ... return len(self._data) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']) >>> train_dataset = train_dataset.batch(batch_size=2) >>> eval_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']) >>> eval_dataset = eval_dataset.batch(batch_size=2) >>> general_task = GeneralTaskTrainer(model_name='common') >>> vit_config = ViTConfig(batch_size=2) >>> network_with_loss = ViTForImageClassification(vit_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> general_task.train(wrapper=wrapper, dataset=train_dataset) >>> compute_metrics = {"Accuracy": Accuracy(eval_type='classification')} >>> general_task.evaluate(network=network_with_loss, dataset=eval_dataset, compute_metrics=compute_metrics) Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(GeneralTaskTrainer, self).__init__("general", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for General-Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If network or dataset not implemented. """ if dataset is None: raise NotImplementedError("train dataset must be define, but get None.") if network is None and wrapper is None: raise NotImplementedError("train network must be define, but get None.") self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for General-Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callback, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. Raises: NotImplementedError: If network or dataset or compute_metrics not implemented. """ if dataset is None: raise NotImplementedError("eval dataset must be define, but get None.") if network is None: raise NotImplementedError("eval network must be define, but get None.") if compute_metrics is None: raise NotImplementedError("eval metrics must be define, but get None.") metric_name = "General Task Metrics" kwargs.setdefault("metric_name", metric_name) self.evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, **kwargs): raise NotImplementedError( "The general task does not support predict, please customize pipeline inference.") pengcheng.mind-test/mindformers/trainer/image_classification/000077500000000000000000000000001471400134000250375ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_classification/__init__.py000066400000000000000000000016401471400134000271510ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Trainer API.""" from .image_classification import ImageClassificationTrainer from .zero_shot_image_classification import ZeroShotImageClassificationTrainer __all__ = ['ImageClassificationTrainer', 'ZeroShotImageClassificationTrainer'] pengcheng.mind-test/mindformers/trainer/image_classification/__pycache__/000077500000000000000000000000001471400134000272475ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_classification/__pycache__/__init__.cpython-39.pyc000066400000000000000000000006461471400134000334450ustar00rootroot00000000000000a ?cf@s(dZddlmZddlmZddgZdS)z!Image Classification Trainer API.)ImageClassificationTrainer)"ZeroShotImageClassificationTrainerrrN)__doc__image_classificationrzero_shot_image_classificationr__all__rr\/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_classification/__init__.pys  image_classification.cpython-39.pyc000066400000000000000000000244571471400134000357720ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_classification/__pycache__a ?cf-@sdZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZmZddlmZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZmZddl m!Z!ddl"m#Z#ddl$m%Z%dgZ&ej'ej(ddGddde%Z)dS)zImage Classification Trainer.)OptionalListUnionN)Image)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell)Tensor) BaseDataset) BaseModelBaseImageProcessor)logger) load_image)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerImageClassificationTrainerimage_classification)aliasc s(eZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd d d Zdeeee e e feeeejeeefeee e feed d dZZS)ra ImageClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell, Accuracy >>> from mindformers.trainer import ImageClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import ViTForImageClassification, ViTConfig, ViTImageProcessor >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self._label = [np.eye(1000)[i] for i in range(64)] ... ... def __getitem__(self, index): ... return self._data[index], self._label[index] ... ... def __len__(self): ... return len(self._data) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']).batch(batch_size=2) >>> eval_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']).batch(batch_size=2) >>> input_data = np.uint8(np.random.random((5, 3, 255, 255))) >>> #1) use config to train >>> cls_task = ImageClassificationTrainer(model_name='vit_base_p16') >>> cls_task.train(dataset=train_dataset) >>> cls_task.evaluate(dataset=eval_dataset) >>> cls_task.predict(input_data=input_data, top_k=5) >>> #2) use instance function to train >>> vit_config = ViTConfig(batch_size=2) >>> network_with_loss = ViTForImageClassification(vit_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> cls_task.train(wrapper=wrapper, dataset=train_dataset) >>> compute_metrics = {"Accuracy": Accuracy(eval_type='classification')} >>> cls_task.evaluate(network=network_with_loss, dataset=eval_dataset, compute_metrics=compute_metrics) >>> image_processor = ViTImageProcessor(size=224) >>> cls_task.predict(input_data=input_data, image_processor=image_processor, top_k=5) N) model_namecstd|dS)Nr)super__init__)selfr __class__h/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_classification/image_classification.pyr]sz#ImageClassificationTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for ImageClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )r#r$r(r%r&r'N)Ztraining_process)rr#r$r%r&r'r(kwargsr!r!r"train`s%z ImageClassificationTrainer.train)r#r$r%r(compute_metricscKs0d}|d||jf|||||d|dS)aEvaluate task for ImageClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. z Top1 Accuracy metric_name)r#r$r%r+r(N) setdefaultZevaluate_process)rr#r$r%r(r+r)r,r!r!r"evaluates z#ImageClassificationTrainer.evaluate)r# input_datar$image_processorcKstd|dur|j}t|ttjttt fs6t dg}t|trT| t |n(t|t rx|D]}| t |qbn|}|j f||d||d|S)apPredict task for ImageClassification Trainer. This function is used to predict the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, tokenizer, image_processor, audio_processor. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. z/.........Build Input Data For Predict..........NzTInput data's type must be one of [str, ms.Tensor, np.ndarray, PIL.Image.Image, list]r)r#r/taskr$r0)rinfor/ isinstancer npndarrayrstrlist ValueErrorappendrZpredict_process)rr#r/r$r0r)Zbatch_input_data data_pathr!r!r"predicts(   z"ImageClassificationTrainer.predict)N)NNNNNN)NNNNN)NNNN)__name__ __module__ __qualname____doc__r6rrrdictrrrr r r rrr rrr*setr.r r4r5rr7rr; __classcell__r!r!rr"r(sL3 / +)*r?typingrrrnumpyr4Z PIL.ImagerZmindspore.trainrmindspore.datasetr mindspore.nnrr r mindsporer mindformers.datasetr mindformers.modelsr rmindformers.tools.loggerrZmindformers.tools.image_toolsrmindformers.tools.registerrrr config_argsr training_argsr base_trainerr__all__registerTRAINERrr!r!r!r"s$          zero_shot_image_classification.cpython-39.pyc000066400000000000000000000143521471400134000400570ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_classification/__pycache__a ?cf@sdZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZmZmZdd lmZdd lmZmZmZd dlmZddlm Z ddl!m"Z"ddl#m$Z$dgZ%e&ej'Gddde$Z(dS)zImage Classification Trainer.)OptionalListUnionN)Image)Callback)Tensor)Cell)GeneratorDataset) BaseDataset) BaseModel BaseTokenizerBaseImageProcessor)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_dataset_loader)ConfigArguments)TrainingArguments) BaseTrainer"ZeroShotImageClassificationTrainerc seZdZdZdedfdd ZddZdeee e e e feee efeeeefeeeeefeee efdd d Zdeee e e e feeeeejeeefeee efeeeed d d ZZS)raZeroShotImageClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers import ZeroShotImageClassificationTrainer >>> trainer = ZeroShotImageClassificationTrainer(model_name="clip_vit_b_32") >>> trainer.evaluate() >>> trainer.predict() N) model_namecstt|d|dS)Nzero_shot_image_classification)superr__init__)selfr __class__r/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_classification/zero_shot_image_classification.pyr8sz+ZeroShotImageClassificationTrainer.__init__cOs tddS)Nzs$            pengcheng.mind-test/mindformers/trainer/image_classification/group_ic_params.py000066400000000000000000000110161471400134000305620ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/microsoft/SimMIM # ============================================================================ """group parameters for masked image modeling.""" import json from functools import partial from mindformers.core.lr import LearningRateWiseLayer from mindformers.tools import logger def get_group_parameters(config, model, base_lr): """get finetune param groups""" if config.model.arch.type == 'SwinForImageClassification': depths = config.model.model_config.depths num_layers = sum(depths) get_layer_func = partial(get_swin_layer, num_layers=num_layers + 2, depths=depths) elif config.model.arch.type == 'ViTForImageClassification': num_layers = config.model.model_config.num_hidden_layers get_layer_func = partial(get_vit_layer, num_layers=num_layers + 2) else: raise NotImplementedError layer_decay = config.runner_config.layer_decay scales = list(layer_decay ** i for i in reversed(range(num_layers + 2))) skip = {} skip_keywords = {} if hasattr(model, 'no_weight_decay'): skip = model.no_weight_decay() logger.info('No weight decay: %s', skip) if hasattr(model, 'no_weight_decay_keywords'): skip_keywords = model.no_weight_decay_keywords() logger.info('No weight decay keywords: %s', skip_keywords) parameter_group_names = {} parameter_group_vars = {} for param in model.trainable_params(): if len(param.shape) == 1 or param.name.endswith(".bias") or (param.name in skip) or \ check_keywords_in_name(param.name, skip_keywords): group_name = "no_decay" this_weight_decay = 0. else: group_name = "decay" this_weight_decay = config.optimizer.weight_decay if get_layer_func is not None: layer_id = get_layer_func(param.name) group_name = "layer_%d_%s" % (layer_id, group_name) else: layer_id = None if group_name not in parameter_group_names: if scales is not None: scale = scales[layer_id] else: scale = 1. parameter_group_names[group_name] = { "weight_decay": this_weight_decay, "params": [], } parameter_group_vars[group_name] = { "weight_decay": this_weight_decay, "params": [], "lr": LearningRateWiseLayer(base_lr, scale), } parameter_group_vars[group_name]["params"].append(param) parameter_group_names[group_name]["params"].append(param.name) param_groups = json.dumps(parameter_group_names, indent=2) logger.info("Param groups = %s", param_groups) return list(parameter_group_vars.values()) def get_vit_layer(name, num_layers): """get vit layer""" if name.endswith(("cls_tokens", "mask_tokens", "pos_embed")): layer_num = 0 elif name.startswith("vit.patch_embed"): layer_num = 0 elif name.startswith("vit.blocks"): layer_id = int(name.split('.')[1]) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num def get_swin_layer(name, num_layers, depths): """get swin layer""" if name in ("encoder.mask_token",): layer_num = 0 elif name.startswith("encoder.patch_embed"): layer_num = 0 elif name.startswith("encoder.layers"): layer_id = int(name.split('.')[2]) block_id = name.split('.')[4] if block_id in ('reduction', 'norm'): layer_num = sum(depths[:layer_id + 1]) else: layer_id = sum(depths[:layer_id]) + int(block_id) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num def check_keywords_in_name(name, keywords=()): isin = False for keyword in keywords: if keyword in name: isin = True break return isin pengcheng.mind-test/mindformers/trainer/image_classification/image_classification.py000066400000000000000000000267411471400134000315600ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Trainer.""" from typing import Optional, List, Union import numpy as np from PIL.Image import Image from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore import Tensor from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.logger import logger from mindformers.tools.image_tools import load_image from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer __all__ = ['ImageClassificationTrainer'] @MindFormerRegister.register(MindFormerModuleType.TRAINER, alias="image_classification") class ImageClassificationTrainer(BaseTrainer): r""" ImageClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell, Accuracy >>> from mindformers.trainer import ImageClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import ViTForImageClassification, ViTConfig, ViTImageProcessor >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self._label = [np.eye(1000)[i] for i in range(64)] ... ... def __getitem__(self, index): ... return self._data[index], self._label[index] ... ... def __len__(self): ... return len(self._data) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']).batch(batch_size=2) >>> eval_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']).batch(batch_size=2) >>> input_data = np.uint8(np.random.random((5, 3, 255, 255))) >>> #1) use config to train >>> cls_task = ImageClassificationTrainer(model_name='vit_base_p16') >>> cls_task.train(dataset=train_dataset) >>> cls_task.evaluate(dataset=eval_dataset) >>> cls_task.predict(input_data=input_data, top_k=5) >>> #2) use instance function to train >>> vit_config = ViTConfig(batch_size=2) >>> network_with_loss = ViTForImageClassification(vit_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> cls_task.train(wrapper=wrapper, dataset=train_dataset) >>> compute_metrics = {"Accuracy": Accuracy(eval_type='classification')} >>> cls_task.evaluate(network=network_with_loss, dataset=eval_dataset, compute_metrics=compute_metrics) >>> image_processor = ViTImageProcessor(size=224) >>> cls_task.predict(input_data=input_data, image_processor=image_processor, top_k=5) """ def __init__(self, model_name: str = None): super().__init__('image_classification', model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for ImageClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for ImageClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. """ metric_name = "Top1 Accuracy" kwargs.setdefault("metric_name", metric_name) self.evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[Tensor, np.ndarray, Image, str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): r"""Predict task for ImageClassification Trainer. This function is used to predict the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, tokenizer, image_processor, audio_processor. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. """ logger.info(".........Build Input Data For Predict..........") if input_data is None: input_data = config.input_data if not isinstance(input_data, (Tensor, np.ndarray, Image, str, list)): raise ValueError("Input data's type must be one of " "[str, ms.Tensor, np.ndarray, PIL.Image.Image, list]") batch_input_data = [] if isinstance(input_data, str): batch_input_data.append(load_image(input_data)) elif isinstance(input_data, list): for data_path in input_data: batch_input_data.append(load_image(data_path)) else: batch_input_data = input_data return self.predict_process(config=config, input_data=batch_input_data, task='image_classification', network=network, image_processor=image_processor, **kwargs) pengcheng.mind-test/mindformers/trainer/image_classification/zero_shot_image_classification.py000066400000000000000000000170251471400134000336470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Trainer.""" from typing import Optional, List, Union import numpy as np from PIL.Image import Image from mindspore.train import Callback from mindspore import Tensor from mindspore.nn import Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer, BaseImageProcessor from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ...dataset.dataloader import build_dataset_loader from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer __all__ = ['ZeroShotImageClassificationTrainer'] @MindFormerRegister.register(MindFormerModuleType.TRAINER) class ZeroShotImageClassificationTrainer(BaseTrainer): r"""ZeroShotImageClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers import ZeroShotImageClassificationTrainer >>> trainer = ZeroShotImageClassificationTrainer(model_name="clip_vit_b_32") >>> trainer.evaluate() >>> trainer.predict() """ def __init__(self, model_name: str = None): super(ZeroShotImageClassificationTrainer, self).__init__("zero_shot_image_classification", model_name) def train(self, *args, **kwargs): raise NotImplementedError( "The ZeroShotImageClassification task does not support train.") def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for ZeroShotImageClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It supports CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It supports dict or set in MindSpore's Metric class. Default: None. """ metric_name = "Top1 Accuracy" kwargs.setdefault("metric_name", metric_name) self.evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): r"""Predict task for ZeroShotImageClassification Trainer. This function is used to predict the network. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. input_data (Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]]): The dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): Used for text process. image_processor (Optional[BaseImageProcessor]): Used for image process. """ config = self.set_config(config) logger.info(".........Build Input Data For Predict..........") if input_data is None and config.input_data is not None: input_data = config.input_data if input_data is None: input_data = build_dataset_loader(config.eval_dataset.data_loader) candidate_labels = kwargs.pop("candidate_labels", None) if candidate_labels is None: if hasattr(input_data, "label_names"): candidate_labels = input_data.label_names else: candidate_labels = ["sunflower", "tree", "dog", "cat", "toy"] hypothesis_template = kwargs.pop("hypothesis_template", None) if hypothesis_template is None: if config.eval_dataset.data_loader.hypothesis_template is not None: hypothesis_template = config.eval_dataset.data_loader.hypothesis_template else: hypothesis_template = "{}" return self.predict_process(config=config, input_data=input_data, task='zero_shot_image_classification', network=network, tokenizer=tokenizer, image_processor=image_processor, candidate_labels=candidate_labels, hypothesis_template=hypothesis_template, **kwargs) pengcheng.mind-test/mindformers/trainer/image_to_text_generation/000077500000000000000000000000001471400134000257455ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_to_text_generation/__init__.py000066400000000000000000000014671471400134000300660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image to text generation Trainer API.""" from .image_to_text_generation import ImageToTextGenerationTrainer __all__ = ['ImageToTextGenerationTrainer'] pengcheng.mind-test/mindformers/trainer/image_to_text_generation/__pycache__/000077500000000000000000000000001471400134000301555ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_to_text_generation/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005241471400134000343460ustar00rootroot00000000000000a ?cf7@sdZddlmZdgZdS)z%Image to text generation Trainer API.)ImageToTextGenerationTrainerrN)__doc__image_to_text_generationr__all__rr`/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_generation/__init__.pys image_to_text_generation.cpython-39.pyc000066400000000000000000000106601471400134000375750ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_to_text_generation/__pycache__a ?cf{@sdZddlmZmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZmZd d lmZd dlmZd dlmZd dlmZdgZ e!ej"GdddeZ#dS)zImageToTextGeneration Trainer.)OptionalUnionN)Image)Tensor)Cell)GeneratorDataset) BaseModel BaseTokenizerBaseImageProcessor)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_dataset_loader)ConfigArguments)TrainingArguments) BaseTrainerImageToTextGenerationTrainerc seZdZdZd edfdd ZddZdd Zdee e e e e fee eeejeeefee eefeeeed d d ZZS)raImageToTextGenerationTrainer Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers.trainer import ImageToTextGenerationTrainer >>> trainer = ImageToTextGenerationTrainer(model_name="blip2_stage2_vit_g_llama_7b") >>> trainer.predict() N) model_namecstt|d|dS)Nimage_to_text_generation)superr__init__)selfr __class__p/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_generation/image_to_text_generation.pyr5sz%ImageToTextGenerationTrainer.__init__cOs tddS)Nz=The ImageToTextGenerationTrainer task does not support train.NotImplementedErrorrargskwargsrrrtrain8sz"ImageToTextGenerationTrainer.traincOs tddS)Nz@The ImageToTextGenerationTrainer task does not support evaluate.rr!rrrevaluate<sz%ImageToTextGenerationTrainer.evaluate)config input_datanetwork tokenizerimage_processorc Ks||}td|dur,|jdur,|j}|dur@t|jj}|dd}|durr|jj j durn|jj j }nd}|dd}|dur|jj j dur|jj j }nd}|dd} | dur|jjj dur|jjj } nd} |j f||d|||||| d |S) a;Predict task for ZeroShotImageToTextGenerationTrainer Trainer. This function is used to predict the network. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. input_data (Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]]): The dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): Used for text process. image_processor (Optional[BaseImageProcessor]): Used for image process. z/.........Build Input Data For Predict..........N max_length paddinghypothesis_templater) r&r'taskr(r)r*r+r-r.) set_configr infor'r eval_dataset data_loaderpop processorr)r+r-r.predict_process) rr&r'r(r)r*r#r+r-r.rrrpredict@sB          z$ImageToTextGenerationTrainer.predict)N)NNNNN)__name__ __module__ __qualname____doc__strrr$r%rrdictrrrrrnpndarrayrlistrrr r r8 __classcell__rrrrr%s$ )$r<typingrrnumpyr?Z PIL.Imager mindsporer mindspore.nnrmindspore.datasetrmindformers.modelsrr r mindformers.tools.loggerr mindformers.tools.registerr r rZdataset.dataloaderr config_argsr training_argsr base_trainerr__all__registerTRAINERrrrrrs           pengcheng.mind-test/mindformers/trainer/image_to_text_generation/image_to_text_generation.py000066400000000000000000000131731471400134000333670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ImageToTextGeneration Trainer.""" from typing import Optional, Union import numpy as np from PIL.Image import Image from mindspore import Tensor from mindspore.nn import Cell from mindspore.dataset import GeneratorDataset from mindformers.models import BaseModel, BaseTokenizer, BaseImageProcessor from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ...dataset.dataloader import build_dataset_loader from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer __all__ = ['ImageToTextGenerationTrainer'] @MindFormerRegister.register(MindFormerModuleType.TRAINER) class ImageToTextGenerationTrainer(BaseTrainer): r"""ImageToTextGenerationTrainer Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers.trainer import ImageToTextGenerationTrainer >>> trainer = ImageToTextGenerationTrainer(model_name="blip2_stage2_vit_g_llama_7b") >>> trainer.predict() """ def __init__(self, model_name: str = None): super(ImageToTextGenerationTrainer, self).__init__("image_to_text_generation", model_name) def train(self, *args, **kwargs): raise NotImplementedError( "The ImageToTextGenerationTrainer task does not support train.") def evaluate(self, *args, **kwargs): raise NotImplementedError( "The ImageToTextGenerationTrainer task does not support evaluate.") def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): r"""Predict task for ZeroShotImageToTextGenerationTrainer Trainer. This function is used to predict the network. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. input_data (Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]]): The dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): Used for text process. image_processor (Optional[BaseImageProcessor]): Used for image process. """ config = self.set_config(config) logger.info(".........Build Input Data For Predict..........") if input_data is None and config.input_data is not None: input_data = config.input_data if input_data is None: input_data = build_dataset_loader(config.eval_dataset.data_loader) max_length = kwargs.pop("max_length", None) if max_length is None: if config.processor.tokenizer.max_length is not None: max_length = config.processor.tokenizer.max_length else: max_length = 32 padding = kwargs.pop("padding", None) if padding is None: if config.processor.tokenizer.padding is not None: padding = config.processor.tokenizer.padding else: padding = "max_length" hypothesis_template = kwargs.pop("hypothesis_template", None) if hypothesis_template is None: if config.eval_dataset.data_loader.hypothesis_template is not None: hypothesis_template = config.eval_dataset.data_loader.hypothesis_template else: hypothesis_template = "" return self.predict_process(config=config, input_data=input_data, task='image_to_text_generation', network=network, tokenizer=tokenizer, image_processor=image_processor, max_length=max_length, padding=padding, hypothesis_template=hypothesis_template, **kwargs) pengcheng.mind-test/mindformers/trainer/image_to_text_retrieval/000077500000000000000000000000001471400134000256075ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_to_text_retrieval/__init__.py000066400000000000000000000014621471400134000277230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image-to-text Retrieval Trainer API.""" from .image_to_text_retrieval import ImageToTextRetrievalTrainer __all__ = ['ImageToTextRetrievalTrainer'] pengcheng.mind-test/mindformers/trainer/image_to_text_retrieval/__pycache__/000077500000000000000000000000001471400134000300175ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_to_text_retrieval/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005201471400134000342040ustar00rootroot00000000000000a ?cf2@sdZddlmZdgZdS)z$Image-to-text Retrieval Trainer API.)ImageToTextRetrievalTrainerrN)__doc__image_to_text_retrievalr__all__rr_/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_retrieval/__init__.pys eval_utils.cpython-39.pyc000066400000000000000000000073211471400134000345430ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_to_text_retrieval/__pycache__a ?cf@sRdZddlZddlmZddlmZddlmZddl m Z ddZ d d d Z dS)z&Image-to-text Retrieval Trainer Utils.N)Tensor)logger)topk)MultiImgCapDataLoaderc sT|}t|dr$|jdur$|jd}qt|tr@|j}|j|jfStdt ||j d|j df|j d|j dfks|J|j d}|j d}||krddt |D}||fS||kr|n||kr|n|}|fddt |D} g} t |dD]} | | g7} q|d} | |dg| 7} |krL| | fS| | fS) zextract_image_text_mapping from eval_dataloader. Args: eval_dataloader: evaluation dataloader score_i2t, score_t2i: two score matrix (I2T, T2I) Returns: img2txt, txt2img: ground truth image-text mapping childrenNrzexpect the eval dataset to be generate from MultiImgCapDataLoader, but is %s, will generate image-text mapping with accumulate indexes by default.cSsg|]}|qSr.0irra/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_retrieval/eval_utils.py .z.extract_image_text_mapping..cs&g|]fddtDqS)csg|]}t|qSr)minr )biggerfactorkrr r 3rz9extract_image_text_mapping...)range)r rr)rr r 3r) hasattrr isinstancersourceimg2txttxt2imgrwarningtypeshaper) Zeval_dataloader score_i2t score_t2i dataloaderdatasetZ image_numZtext_numindsZsmallerZ smaller_indsZ bigger_indsr last_numrrr extract_image_text_mappings6    ,   r#FcCs"td|g}|d|d|dd}}}|}|}|D]>}t|j|j|jj|jt||j} | d} | | qDtj |dd}t |jd|jdfd} t |D]\} } | ddkrtd | d |jdd | jt| j|\}}|j}|j}|| | |f<|rt|}|j|| ||d d }| | |f|7<q|j}t |jd|jdfd}t |D]\} } | ddkrtd| d |jdd | jt| j|\}}|j}|j}||| |f<|rzt|}|j|| ||dd }|| |f|7<qz| |fS)a compute image-text matching scores, in matrix format. Args: network (BaseModel): network for evaluate ITM eval_inputs (tuple): inputs for evaluate itm scores. k_test (int, optional): k_test num, Defaults to 128. add_extra_itm_score (bool, optional): whether to add extra scores (model decides), Defaults to False. Returns: score_matrix_i2t, score_matrix_t2i: two score matrix (I2T, T2I) z"========= k_text num: %d =========rrN)axisgY2zI2T: /z - sims.shape: T)Zi2tzT2I: F)rinfoasnumpyprintrdtypeTnpmatmulmaxappendstackfull enumeraterr from_numpyZcompute_extra_itm)network eval_inputsk_testadd_extra_itm_scoreZ sims_matrixZ image_featsZ text_feats extra_argsZ image_featZsim_q2tZsim_i2tZscore_matrix_i2tr ZsimsZtopk_simZtopk_idxZ topk_idx_msscoreZscore_matrix_t2irrr compute_itm_scores=sR     "  "  r<)r$F) __doc__numpyr. mindsporermindformers.tools.loggerrZmindformers.generation.utilsrZ9mindformers.dataset.dataloader.multi_image_cap_dataloaderrr#r<rrrr s    &image_to_text_retrieval.cpython-39.pyc000066400000000000000000000204721471400134000373030ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/image_to_text_retrieval/__pycache__a ?cf2@shdZddlZddlmZmZmZddlmZddlZddl Z ddl m Z ddl mZddlmZddlmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m&Z&ddl6m(Z(e.j7e/j8ddGddde&Z9dS) Image-to-text Retrieval Trainer.N)ListOptionalUnion)pprint)dtype)Model)DynamicLossScaleUpdateCell)TrainOneStepCell Optimizer)Callback) build_datasetcheck_dataset_config BaseDataset) build_model BaseModel)build_callback)build_lr) build_optim) build_wrapper) BaseTrainer)check_runner_config)logger) count_params)MindFormerRegisterMindFormerModuleType)compute_itm_scoresextract_image_text_mapping)ConfigArgumentsimage_to_text_retrieval)aliasc seZdZdZdedfdd Zdeeee feeee feeee fee ee eeeeefdddZdeeee feeee feeee feeeeefd d d Zd d ZZS)ImageToTextRetrievalTrainerrN) model_namecs"tt|d|||_d|_dS)Nr!)superr#__init__r$kwargs)selfr$ __class__n/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_retrieval/image_to_text_retrieval.pyr&1sz$ImageToTextRetrievalTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc sHtjdstdtj||_fddtdt|durRtj }t |td|durt j j jdd }|tj}td tt|td |durZtd tj}|} tt| } ttfd d| } | jjdd| id| ig} |dur6tj| |dd }n$jjsHJdtjd| id }|durg}jr||j| t!j"d|jid tdt#dddd} |durt$j%||| dd }n t&|t'r|}ntd|tdt(|}t)t*+dddd kr$t,|j-j.j/||j.j0j.j1d!dS)"zH Training task for ImageToTextRetrievalTrainer Trainer. z2.0zNCurrently, filip only support training under mindspore2.0, but with mindspore cs$j}|D]}||jvr dSq dS)NFT)no_decay_paramsname) param_namer3keyword)r-r+r, decay_filterJs  z7ImageToTextRetrievalTrainer.train..decay_filter .........Build Dataset..........N.........Build Net..........)parallel_config moe_config default_argsu网络参数量：%s M.z".........Build Optimizer..........z$.........Build LR Schedule..........cs | S)Nr+)x)r7r+r,lz3ImageToTextRetrievalTrainer.train..)params weight_decayrA order_params)rA learning_ratezlearning_rate must be inputrDz(.........Build Running Wrapper..........g0Ari)loss_scale_value scale_factor scale_window)r.r1Zscale_update_cellzINow not support this wrapper,it should be TrainOneStepCell type, but get z,.........Starting Init Train Model..........RANK_ID0r)r2dataset_sink_mode sink_size)2ms __version__ startswithNotImplementedErrorr'rinforr train_dataset_taskrrmodelr:r;to_floatmstypefloat16strrr lr_scheduletrainable_paramslistfilterr1rBrrDprofileappend profile_cbextendrr2r rrunner_wrapper isinstancer rintosgetenvrtrain runner_configepochs sink_moderL)r(r-r.r/r0r1r2r'rXrAZ decay_params other_params group_params update_cellrSr+)r-r7r,re6s                      z!ImageToTextRetrievalTrainer.train)r-r.r/r2cKs||_|dd}|||}tdt||durBt|j}td|td|dur~t |j |j |j ddd}| tj}td |j jjdur||j jtd tt|td |durg}|jr||j|t|jtd ttd dddkr(t|||}|j j!durH|j j!nd}|"d|}|j j#} |"d| } t$|||| d\} } t%|j | | \} } |&| | | | }t|td|S)zJ Evaluation task for ImageToTextRetrievalTrainer Trainer. is_full_configFr8Nz+Create eval dataset finish, dataset size:%dr9)r:r; is_trainingr<z%.........Loading Checkpoint..........zNetwork Parameters: %s M.z/.........Build Callbacks for Evaluate..........z).........Starting Evaling Model..........rHrIrJrk_testadd_extra_itm_score)rorpz$.........Evaluate Over!.............)'r'get set_configrrQrr eval_dataset_taskget_dataset_sizerrSr:r;rTrUrV model_configcheckpoint_name_or_pathload_checkpointrWrr\r]r^r_reval_callbacksrbrcrdrZprepare_inputs_for_itm_eval eval_datasetropoprprr_report_metrics)r(r-r.r/r2r'rlZ eval_inputsrorpZ score_i2tZ score_t2iimg2txttxt2img eval_resultr+r+r,evaluatesh                   z$ImageToTextRetrievalTrainer.evaluatec Csdd}t|jd}t|D]L\}}td|||t|ddd} td| || ||||<q dtt|d kdt|} dtt|d kdt|} dtt|d kdt|} t|jd}t|D]L\}}td |||t|ddd} td| || ||||<qdtt|d kdt|} dtt|d kdt|}dtt|d kdt|}| | | d }| ||d }||d}| | | d }| | | || |||||d }|S)a report metrics for image-text matching Args: scores_i2t: image-to-text similarity score matrix scores_t2i: text-to-image similarity score matrix img2txt: image-to-text ground truth mapping txt2img: text-to-image ground truth mapping Returns: eval_result: A dictionary containing r1, r5, r10 scores cSsft|tr t||kddSt|ts2Jdd}|D]&}t||kdd}||kr:|}q:|S)Nrz:img2txt or txt2img should be list[int] or list[list[int]]!g@xD)rarbnpwhererZ)ranks ground_truthrankitmpr+r+r,get_lowest_from_rankss zJImageToTextRetrievalTrainer._report_metrics..get_lowest_from_ranksrzimg2txt[{}]: {}Nzmost similarity:gY@r ztxt2img[{}]: {}r) Ztxt_r1Ztxt_r5Ztxt_r10Z txt_r_meanZimg_r1Zimg_r5Zimg_r10Z img_r_meanr_mean agg_metrics) rzerosshape enumerateprintformatargsortlenr)r(Z scores_i2tZ scores_t2ir|r}rrindexscoreindstr1Ztr5Ztr10Zir1Zir5Zir10Ztr_meanZir_meanrrr~r+r+r,r{sF   """ """  z+ImageToTextRetrievalTrainer._report_metrics)N)NNNNNN)NNNN)__name__ __module__ __qualname____doc__rWr&rrdictr rrr r r rrerr{ __classcell__r+r+r)r,r#.s6 j Rr#):rrctypingrrrrnumpyr mindsporerMrrUmindspore.train.modelrmindspore.nn.wrap.loss_scaler mindspore.nnr r Zmindspore.trainr mindformers.datasetr rrmindformers.modelsrrmindformers.core.callbackrZmindformers.core.lrrZmindformers.core.optimrmindformers.wrapperrZ mindformers.trainer.base_trainerrZmindformers.trainer.utilsrmindformers.tools.loggerrmindformers.tools.utilsrmindformers.tools.registerrrZ eval_utilsrr config_argsr base_trainerutilsregisterTRAINERr#r+r+r+r,s6                pengcheng.mind-test/mindformers/trainer/image_to_text_retrieval/eval_utils.py000066400000000000000000000117021471400134000303310ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image-to-text Retrieval Trainer Utils.""" import numpy as np from mindspore import Tensor from mindformers.tools.logger import logger from mindformers.generation.utils import topk from mindformers.dataset.dataloader.multi_image_cap_dataloader import MultiImgCapDataLoader def extract_image_text_mapping(eval_dataloader, score_i2t, score_t2i): """extract_image_text_mapping from eval_dataloader. Args: eval_dataloader: evaluation dataloader score_i2t, score_t2i: two score matrix (I2T, T2I) Returns: img2txt, txt2img: ground truth image-text mapping """ dataloader = eval_dataloader while hasattr(dataloader, "children") and dataloader.children is not None: dataloader = dataloader.children[0] if isinstance(dataloader, MultiImgCapDataLoader): dataset = dataloader.source return dataset.img2txt, dataset.txt2img logger.warning("expect the eval dataset to be generate \ from MultiImgCapDataLoader, but is %s, will generate \ image-text mapping with accumulate indexes by default.", type(dataloader)) assert (score_i2t.shape[0], score_i2t.shape[1]) == (score_t2i.shape[1], score_t2i.shape[0]) image_num = score_i2t.shape[0] text_num = score_t2i.shape[0] if image_num == text_num: inds = [i for i in range(image_num)] return inds, inds bigger = image_num if image_num > text_num else text_num smaller = text_num if bigger == image_num else image_num factor = bigger // smaller smaller_inds = [[min(i + k * factor, bigger) for i in range(factor)] for k in range(smaller)] bigger_inds = [] for i in range(smaller - 1): bigger_inds += [i] * factor last_num = bigger - (smaller - 1) * factor bigger_inds += [smaller - 1] * last_num if bigger == image_num: return bigger_inds, smaller_inds return smaller_inds, bigger_inds def compute_itm_scores(network, eval_inputs, k_test=128, add_extra_itm_score=False): """ compute image-text matching scores, in matrix format. Args: network (BaseModel): network for evaluate ITM eval_inputs (tuple): inputs for evaluate itm scores. k_test (int, optional): k_test num, Defaults to 128. add_extra_itm_score (bool, optional): whether to add extra scores (model decides), Defaults to False. Returns: score_matrix_i2t, score_matrix_t2i: two score matrix (I2T, T2I) """ logger.info("========= k_text num: %d =========", k_test) sims_matrix = [] image_feats, text_feats, extra_args = eval_inputs[0], eval_inputs[1], eval_inputs[2:] image_feats = image_feats.asnumpy() text_feats = text_feats.asnumpy() for image_feat in image_feats: print(image_feat.shape, image_feat.dtype, text_feats.T.shape, text_feats.dtype) sim_q2t = np.matmul(image_feat, text_feats.T) sim_i2t = sim_q2t.max(0) sims_matrix.append(sim_i2t) sims_matrix = np.stack(sims_matrix, axis=0) score_matrix_i2t = np.full( (image_feats.shape[0], text_feats.shape[0]), -100.0 ) for i, sims in enumerate(sims_matrix): if i % 50 == 0: print(f"I2T: {i}/{sims_matrix.shape[0]} - sims.shape: {sims.shape}") topk_sim, topk_idx = topk(sims.T, k_test) topk_sim = topk_sim.T topk_idx = topk_idx.T score_matrix_i2t[i, topk_idx] = topk_sim # .astype(mstype.float32) if add_extra_itm_score: topk_idx_ms = Tensor.from_numpy(topk_idx) score = network.compute_extra_itm(extra_args, i, k_test, topk_idx_ms, i2t=True) score_matrix_i2t[i, topk_idx] += score.asnumpy() sims_matrix = sims_matrix.T score_matrix_t2i = np.full( (text_feats.shape[0], image_feats.shape[0]), -100.0 ) for i, sims in enumerate(sims_matrix): if i % 50 == 0: print(f"T2I: {i}/{sims_matrix.shape[0]} - sims.shape: {sims.shape}") topk_sim, topk_idx = topk(sims.T, k_test) topk_sim = topk_sim.T topk_idx = topk_idx.T score_matrix_t2i[i, topk_idx] = topk_sim if add_extra_itm_score: topk_idx_ms = Tensor.from_numpy(topk_idx) score = network.compute_extra_itm(extra_args, i, k_test, topk_idx_ms, i2t=False) score_matrix_t2i[i, topk_idx] += score.asnumpy() return score_matrix_i2t, score_matrix_t2i pengcheng.mind-test/mindformers/trainer/image_to_text_retrieval/image_to_text_retrieval.py000066400000000000000000000313421471400134000330710ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image-to-text Retrieval Trainer.""" import os from typing import List, Optional, Union from pprint import pprint import numpy as np import mindspore as ms from mindspore import dtype as mstype from mindspore.train.model import Model from mindspore.nn.wrap.loss_scale import DynamicLossScaleUpdateCell from mindspore.nn import TrainOneStepCell, Optimizer from mindspore.train import Callback from mindformers.dataset import build_dataset, check_dataset_config, BaseDataset from mindformers.models import build_model, BaseModel from mindformers.core.callback import build_callback from mindformers.core.lr import build_lr from mindformers.core.optim import build_optim from mindformers.wrapper import build_wrapper from mindformers.trainer.base_trainer import BaseTrainer from mindformers.trainer.utils import check_runner_config from mindformers.tools.logger import logger from mindformers.tools.utils import count_params from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from .eval_utils import compute_itm_scores, extract_image_text_mapping from ..config_args import ConfigArguments from ..base_trainer import BaseTrainer from ..utils import check_runner_config @MindFormerRegister.register(MindFormerModuleType.TRAINER, alias="image_to_text_retrieval") class ImageToTextRetrievalTrainer(BaseTrainer): """Image-to-text Retrieval Trainer.""" def __init__(self, model_name: str = None): super(ImageToTextRetrievalTrainer, self).__init__("image_to_text_retrieval", model_name) self.model_name = model_name self.kwargs = None def train(self, config: Optional[Union[dict, ConfigArguments]] = None, network: Optional[Union[str, BaseModel]] = None, dataset: Optional[Union[str, BaseDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): """ Training task for ImageToTextRetrievalTrainer Trainer. """ # check mindspore version # currently, filip only support training under mindspore2.0 if not ms.__version__.startswith('2.0'): raise NotImplementedError(f"Currently, filip only support training under mindspore2.0, " f"but with mindspore {ms.__version__}") self.kwargs = kwargs # no_weight_decay_params filter def decay_filter(param_name): no_decay_params = config.no_decay_params for keyword in no_decay_params: if keyword in param_name.name: return False return True # build dataset logger.info(".........Build Dataset..........") check_dataset_config(config) if dataset is None: dataset = build_dataset(config.train_dataset_task) check_runner_config(config, dataset) # build network logger.info(".........Build Net..........") if network is None: network = build_model(config.model, default_args={ "parallel_config": config.parallel_config, "moe_config": config.moe_config}) network = network.to_float(mstype.float16) logger.info("网络参数量：%s M.", str(count_params(network))) # build optimizer logger.info(".........Build Optimizer..........") if optimizer is None: # build learning rate schedule logger.info(".........Build LR Schedule..........") lr_schedule = build_lr(config.lr_schedule) params = network.trainable_params() decay_params = list(filter(decay_filter, params)) other_params = list(filter(lambda x: not decay_filter(x), params)) group_params = [{'params': decay_params, 'weight_decay': config.optimizer.weight_decay}, {'params': other_params}, {'order_params': params}] if lr_schedule is not None: optimizer = build_optim( config.optimizer, default_args={"params": group_params, "learning_rate": lr_schedule}) else: assert config.optimizer.learning_rate, "learning_rate must be input" optimizer = build_optim( config.optimizer, default_args={"params": group_params}) # build callback if callbacks is None: callbacks = [] if config.profile: callbacks.append(config.profile_cb) callbacks.extend(build_callback( config.callbacks, default_args={"learning_rate": optimizer.learning_rate})) # build runner wrapper logger.info(".........Build Running Wrapper..........") update_cell = DynamicLossScaleUpdateCell(loss_scale_value=2. ** 20, scale_factor=2, scale_window=1000) if wrapper is None: model = build_wrapper(config.runner_wrapper, default_args={"network": network, "optimizer": optimizer, "scale_update_cell": update_cell}) elif isinstance(wrapper, TrainOneStepCell): model = wrapper else: raise NotImplementedError(f"Now not support this wrapper," f"it should be TrainOneStepCell type, but get {wrapper}") # define Model and begin training logger.info(".........Starting Init Train Model..........") model = Model(model) if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) model.train( config.runner_config.epochs, dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode, sink_size=config.runner_config.sink_size) def evaluate(self, config: Optional[Union[dict, ConfigArguments]] = None, network: Optional[Union[str, BaseModel]] = None, dataset: Optional[Union[str, BaseDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): """ Evaluation task for ImageToTextRetrievalTrainer Trainer. """ self.kwargs = kwargs is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) # build dataset logger.info(".........Build Dataset..........") check_dataset_config(config) if dataset is None: dataset = build_dataset(config.eval_dataset_task) logger.info("Create eval dataset finish, dataset size:%d", dataset.get_dataset_size()) # build network logger.info(".........Build Net..........") if network is None: network = build_model(config.model, default_args={ "parallel_config": config.parallel_config, "moe_config": config.moe_config, "is_training": False}) network = network.to_float(mstype.float16) # checkpoint logger.info(".........Loading Checkpoint..........") if config.model.model_config.checkpoint_name_or_path is not None: network.load_checkpoint(config.model.model_config) logger.info("Network Parameters: %s M.", str(count_params(network))) # build callback logger.info(".........Build Callbacks for Evaluate..........") if callbacks is None: callbacks = [] if config.profile: callbacks.append(config.profile_cb) callbacks.extend(build_callback(config.eval_callbacks)) logger.info(".........Starting Evaling Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) # prepare inputs for computing simliarity matrix eval_inputs = network.prepare_inputs_for_itm_eval(dataset) # k_test value, for topk k_test = config.eval_dataset.k_test if \ config.eval_dataset.k_test is not None else 128 # trainer arguments overrides top_k k_test = kwargs.pop('k_test', k_test) # whether adding additional itm score add_extra_itm_score = config.eval_dataset.add_extra_itm_score # trainer arguments overrides add_extra_itm_score add_extra_itm_score = kwargs.pop('add_extra_itm_score', add_extra_itm_score) # compute image-to-text/text-to-image similarity scores score_i2t, score_t2i = compute_itm_scores(network, eval_inputs, k_test=k_test, add_extra_itm_score=add_extra_itm_score) # ground-truth image-text mapping img2txt, txt2img = extract_image_text_mapping(config.eval_dataset, score_i2t, score_t2i) # report evaluation results eval_result = self._report_metrics( score_i2t, score_t2i, img2txt, txt2img, ) logger.info(eval_result) logger.info(".........Evaluate Over!.............") return eval_result def _report_metrics(self, scores_i2t, scores_t2i, img2txt, txt2img): """ report metrics for image-text matching Args: scores_i2t: image-to-text similarity score matrix scores_t2i: text-to-image similarity score matrix img2txt: image-to-text ground truth mapping txt2img: text-to-image ground truth mapping Returns: eval_result: A dictionary containing r1, r5, r10 scores """ def get_lowest_from_ranks(ranks, ground_truth): if isinstance(ground_truth, int): return np.where(ranks == ground_truth)[0][0] assert isinstance(ground_truth, list), "img2txt or txt2img should be list[int] or list[list[int]]!" rank = 1e20 for i in ground_truth: tmp = np.where(ranks == i)[0][0] if tmp < rank: rank = tmp return rank # Images->Text ranks = np.zeros(scores_i2t.shape[0]) for index, score in enumerate(scores_i2t): print("img2txt[{}]: {}".format(index, img2txt[index])) inds = np.argsort(score)[::-1] print("most similarity:", inds) # Score ranks[index] = get_lowest_from_ranks(inds, img2txt[index]) # 计算度量, 100 是百分比基数, # ranks为每张图ground-truth对应的 # text在similarity score中排位的最小值。 tr1 = 100.0 * len(np.where(ranks < 1)[0]) / len(ranks) tr5 = 100.0 * len(np.where(ranks < 5)[0]) / len(ranks) tr10 = 100.0 * len(np.where(ranks < 10)[0]) / len(ranks) # Text->Images ranks = np.zeros(scores_t2i.shape[0]) for index, score in enumerate(scores_t2i): print("txt2img[{}]: {}".format(index, txt2img[index])) inds = np.argsort(score)[::-1] print("most similarity:", inds) ranks[index] = get_lowest_from_ranks(inds, txt2img[index]) # 此段逻辑同上注释，以text为基准。 ir1 = 100.0 * len(np.where(ranks < 1)[0]) / len(ranks) ir5 = 100.0 * len(np.where(ranks < 5)[0]) / len(ranks) ir10 = 100.0 * len(np.where(ranks < 10)[0]) / len(ranks) tr_mean = (tr1 + tr5 + tr10) / 3 ir_mean = (ir1 + ir5 + ir10) / 3 r_mean = (tr_mean + ir_mean) / 2 agg_metrics = (tr1 + tr5 + tr10) / 3 eval_result = { "txt_r1": tr1, "txt_r5": tr5, "txt_r10": tr10, "txt_r_mean": tr_mean, "img_r1": ir1, "img_r5": ir5, "img_r10": ir10, "img_r_mean": ir_mean, "r_mean": r_mean, "agg_metrics": agg_metrics, } return eval_result pengcheng.mind-test/mindformers/trainer/masked_image_modeling/000077500000000000000000000000001471400134000251665ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/masked_image_modeling/__init__.py000066400000000000000000000014621471400134000273020ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Trainer.""" from .masked_image_modeling_pretrain import MaskedImageModelingTrainer __all__ = ['MaskedImageModelingTrainer'] pengcheng.mind-test/mindformers/trainer/masked_image_modeling/__pycache__/000077500000000000000000000000001471400134000273765ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/masked_image_modeling/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005161471400134000335700ustar00rootroot00000000000000a ?cf2@sdZddlmZdgZdS)zMasked Image Modeling Trainer.)MaskedImageModelingTrainerrN)__doc__Zmasked_image_modeling_pretrainr__all__rr]/home/ma-user/work/mindformers-baichuan/mindformers/trainer/masked_image_modeling/__init__.pys masked_image_modeling_pretrain.cpython-39.pyc000066400000000000000000000207631471400134000401500ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/masked_image_modeling/__pycache__a ?cf$@sdZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&e'ej(Gddde Z)dS)zMasked Image Modeling Trainer.)OptionalListUnionN)Image)Tensor)Callback)TrainOneStepCell OptimizerCell)GeneratorDataset) BaseDataset) BaseModel)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainer)BaseImageProcessor)logger) load_imagec seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZd d Zdeeee e e feeeejeeefeee e feed d d ZZS)MaskedImageModelingTrainera MaskedImageModeling Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import ViTMAEForPreTraining, ViTMAEConfig >>> class MyDataLoader: ... def __init__(self): ... self.image = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self.mask = [np.zeros((196,), np.int32) for _ in range(64)] ... self.ids_restore = [np.zeros((196,), np.int32) for _ in range(64)] ... self.unmask_index = [np.zeros((49,), np.int32) for _ in range(64)] ... def __getitem__(self, index): ... return self.image[index], self.mask[index], self.ids_restore[index], self.unmask_index[index] ... def __len__(self): ... return len(self.image) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), ... column_names=["image", "mask", "ids_restore", "unmask_index"]).batch(2) >>> #1) use config to train >>> mae_trainer = MaskedImageModelingTrainer(model_name='mae_vit_base_p16') >>> mae_trainer.train(dataset=train_dataset) >>> #2) use instance function to train >>> mae_config = ViTMAEConfig(batch_size=2) >>> network_with_loss = ViTMAEForPreTraining(mae_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> mae_trainer.train(wrapper=wrapper, optimizer=optimizer, dataset=train_dataset) Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Nmasked_image_modeling)superr__init__)selfr __class__s/home/ma-user/work/mindformers-baichuan/mindformers/trainer/masked_image_modeling/masked_image_modeling_pretrain.pyrNsz#MaskedImageModelingTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for MaskedImageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )r!r"r&r#r$r%N)training_process)rr!r"r#r$r%r&kwargsrrr trainQs%z MaskedImageModelingTrainer.traincOs tddS)Nz7The MaskedImageModeling task does not support evaluate.)NotImplementedError)rargsr(rrr evaluatesz#MaskedImageModelingTrainer.evaluate)r! input_datar"image_processorcKstd|dur|j}t|ttjttt fs6t dg}t|trT| t |n(t|t rx|D]}| t |qbn|}|j f||d||d|S)aPredict task for MaskedImageModeling Trainer. This function is used to predict the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, tokenizer, image_processor, audio_processor. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. z/.........Build Input Data For Predict..........NzTInput data's type must be one of [str, ms.Tensor, np.ndarray, PIL.Image.Image, list]r)r!r-taskr"r.)rinfor- isinstancernpndarrayrstrlist ValueErrorappendrpredict_process)rr!r-r"r.r(batch_input_data data_pathrrr predicts(   z"MaskedImageModelingTrainer.predict)N)NNNNNN)NNNN)__name__ __module__ __qualname____doc__r4rrrdictrrrr r r r rr rrr)r,rr2r3rr5rr; __classcell__rrrr r%s6' .r)*r?typingrrrnumpyr2Z PIL.Imager mindsporerZmindspore.trainr mindspore.nnrr r mindspore.datasetr mindformers.datasetr mindformers.modelsr mindformers.tools.registerrrrmindformers.trainer.config_argsrZ!mindformers.trainer.training_argsrZ mindformers.trainer.base_trainerrZ!mindformers.models.base_processorrmindformers.tools.loggerrZmindformers.tools.image_toolsrregisterTRAINERrrrrr s$             pengcheng.mind-test/mindformers/trainer/masked_image_modeling/group_mim_parameters.py000066400000000000000000000042251471400134000317640ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/microsoft/SimMIM # ============================================================================ """group parameters for masked image modeling.""" from mindformers.tools import logger def get_group_parameters(config, model): """get pretrain param groups""" skip = {} skip_keywords = {} if hasattr(model, 'no_weight_decay'): skip = model.no_weight_decay() logger.info('No weight decay: %s', skip) if hasattr(model, 'no_weight_decay_keywords'): skip_keywords = model.no_weight_decay_keywords() logger.info('No weight decay keywords: %s', skip_keywords) has_decay = [] no_decay = [] has_decay_name = [] no_decay_name = [] for param in model.trainable_params(): if len(param.shape) == 1 or param.name.endswith(".bias") or (param.name in skip) or \ check_keywords_in_name(param.name, skip_keywords): no_decay.append(param) no_decay_name.append(param.name) else: has_decay.append(param) has_decay_name.append(param.name) logger.info('No decay params: %s', no_decay_name) logger.info('Has decay params: %s', has_decay_name) return [{'params': has_decay, 'weight_decay': config.optimizer.weight_decay}, {'params': no_decay, 'weight_decay': 0.0}, {'order_params': model.trainable_params()}] def check_keywords_in_name(name, keywords=()): isin = False for keyword in keywords: if keyword in name: isin = True break return isin pengcheng.mind-test/mindformers/trainer/masked_image_modeling/masked_image_modeling_pretrain.py000066400000000000000000000223731471400134000337370ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Trainer.""" from typing import Optional, List, Union import numpy as np from PIL.Image import Image from mindspore import Tensor from mindspore.train import Callback from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from mindformers.trainer.config_args import ConfigArguments from mindformers.trainer.training_args import TrainingArguments from mindformers.trainer.base_trainer import BaseTrainer from mindformers.models.base_processor import BaseImageProcessor from mindformers.tools.logger import logger from mindformers.tools.image_tools import load_image @MindFormerRegister.register(MindFormerModuleType.TRAINER) class MaskedImageModelingTrainer(BaseTrainer): r"""MaskedImageModeling Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import ViTMAEForPreTraining, ViTMAEConfig >>> class MyDataLoader: ... def __init__(self): ... self.image = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self.mask = [np.zeros((196,), np.int32) for _ in range(64)] ... self.ids_restore = [np.zeros((196,), np.int32) for _ in range(64)] ... self.unmask_index = [np.zeros((49,), np.int32) for _ in range(64)] ... def __getitem__(self, index): ... return self.image[index], self.mask[index], self.ids_restore[index], self.unmask_index[index] ... def __len__(self): ... return len(self.image) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), ... column_names=["image", "mask", "ids_restore", "unmask_index"]).batch(2) >>> #1) use config to train >>> mae_trainer = MaskedImageModelingTrainer(model_name='mae_vit_base_p16') >>> mae_trainer.train(dataset=train_dataset) >>> #2) use instance function to train >>> mae_config = ViTMAEConfig(batch_size=2) >>> network_with_loss = ViTMAEForPreTraining(mae_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> mae_trainer.train(wrapper=wrapper, optimizer=optimizer, dataset=train_dataset) Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(MaskedImageModelingTrainer, self).__init__("masked_image_modeling", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for MaskedImageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, *args, **kwargs): raise NotImplementedError( "The MaskedImageModeling task does not support evaluate.") def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[Tensor, np.ndarray, Image, str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): r"""Predict task for MaskedImageModeling Trainer. This function is used to predict the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, tokenizer, image_processor, audio_processor. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. """ logger.info(".........Build Input Data For Predict..........") if input_data is None: input_data = config.input_data if not isinstance(input_data, (Tensor, np.ndarray, Image, str, list)): raise ValueError("Input data's type must be one of " "[str, ms.Tensor, np.ndarray, PIL.Image.Image, list]") batch_input_data = [] if isinstance(input_data, str): batch_input_data.append(load_image(input_data)) elif isinstance(input_data, list): for data_path in input_data: batch_input_data.append(load_image(data_path)) else: batch_input_data = input_data return self.predict_process(config=config, input_data=batch_input_data, task='masked_image_modeling', network=network, image_processor=image_processor, **kwargs) pengcheng.mind-test/mindformers/trainer/masked_language_modeling/000077500000000000000000000000001471400134000256675ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/masked_language_modeling/__init__.py000066400000000000000000000014761471400134000300100ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Language Modeling Trainer.""" from .masked_language_modeling_pretrain import MaskedLanguageModelingTrainer __all__ = ['MaskedLanguageModelingTrainer'] pengcheng.mind-test/mindformers/trainer/masked_language_modeling/__pycache__/000077500000000000000000000000001471400134000300775ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/masked_language_modeling/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005321471400134000342670ustar00rootroot00000000000000a ?cf>@sdZddlmZdgZdS)z!Masked Language Modeling Trainer.)MaskedLanguageModelingTrainerrN)__doc__Z!masked_language_modeling_pretrainr__all__rr`/home/ma-user/work/mindformers-baichuan/mindformers/trainer/masked_language_modeling/__init__.pys masked_language_modeling_pretrain.cpython-39.pyc000066400000000000000000000157311471400134000413510ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/masked_language_modeling/__pycache__a ?cf@sdZddlmZmZmZddlmZddlmZm Z m Z ddl m Z ddl mZddlmZmZddlmZmZmZd d lmZd d lmZd d lmZeejGd ddeZdS)zMasked Image Modeling Trainer.)OptionalListUnion)Callback)TrainOneStepCell OptimizerCell)GeneratorDataset) BaseDataset) BaseModel BaseTokenizer)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerc seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZd d Zdeeee e e feeeefeeee feed d d ZZS)MaskedLanguageModelingTrainera]MaskedLanguageModeling Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers import MaskedLanguageModelingTrainer >>> def generator(): >>> data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32) >>> input_mask = np.ones_like(data) >>> token_type_id = np.zeros_like(data) >>> next_sentence_lables = np.array([1]).astype(np.int32) >>> masked_lm_positions = np.array([1, 2]).astype(np.int32) >>> masked_lm_ids = np.array([1, 2]).astype(np.int32) >>> masked_lm_weights = np.ones_like(masked_lm_ids) >>> train_data = (data, input_mask, token_type_id, next_sentence_lables, ... masked_lm_positions, masked_lm_ids, masked_lm_weights) >>> for _ in range(512): ... yield train_data >>> dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids", ... "next_sentence_labels", "masked_lm_positions", ... "masked_lm_ids", "masked_lm_weights"]) >>> dataset = dataset.batch(batch_size=16) >>> mlm_trainer = MaskedLanguageModelingTrainer(model_name="bert_tiny_uncased") >>> mlm_trainer.train(dataset=dataset) >>> res = mlm_trainer.predict(input_data = "hello world [MASK]") Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)N fill_mask)superr__init__)selfr __class__y/home/ma-user/work/mindformers-baichuan/mindformers/trainer/masked_language_modeling/masked_language_modeling_pretrain.pyr>sz&MaskedLanguageModelingTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for MaskedLanguageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )rrr#r r!r"N)training_process)rrrr r!r"r#kwargsrrrtrainAs%z#MaskedLanguageModelingTrainer.traincOs tddS)Nz:The MaskedLanguageModeling task does not support evaluate.)NotImplementedError)rargsr%rrrevaluateosz&MaskedLanguageModelingTrainer.evaluate)r input_datar tokenizerc Ks||}d|jj_|dur"|j}t|ttfsBtdt |d|jj_ |rX|j j n|jjj }|j f||d|||dd|S)a Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[str, BaseModel]]): The network for trainer. It support model name supported or BaseModel class. Supported model name can refer to model support list. For . Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. FNz;Input data's type must be one of [str, list], but got type r max_length)rr*taskrr+r-padding) set_configmodel model_config is_trainingr* isinstancestrlist ValueErrortype batch_sizer seq_lengthpredict_process)rrr*rr+r%r-rrrpredictss(   z%MaskedLanguageModelingTrainer.predict)N)NNNNNN)NNNN)__name__ __module__ __qualname____doc__r5rrrdictrrrrr r r rrrrr&r)r6r r< __classcell__rrrrr s6 .rN)r@typingrrrZmindspore.trainr mindspore.nnrrrmindspore.datasetr mindformers.datasetr mindformers.modelsr r mindformers.tools.registerr rr config_argsr training_argsr base_trainerrregisterTRAINERrrrrrs       masked_language_modeling_pretrain.py000066400000000000000000000173401471400134000350600ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/masked_language_modeling# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer @MindFormerRegister.register(MindFormerModuleType.TRAINER) class MaskedLanguageModelingTrainer(BaseTrainer): r"""MaskedLanguageModeling Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers import MaskedLanguageModelingTrainer >>> def generator(): >>> data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32) >>> input_mask = np.ones_like(data) >>> token_type_id = np.zeros_like(data) >>> next_sentence_lables = np.array([1]).astype(np.int32) >>> masked_lm_positions = np.array([1, 2]).astype(np.int32) >>> masked_lm_ids = np.array([1, 2]).astype(np.int32) >>> masked_lm_weights = np.ones_like(masked_lm_ids) >>> train_data = (data, input_mask, token_type_id, next_sentence_lables, ... masked_lm_positions, masked_lm_ids, masked_lm_weights) >>> for _ in range(512): ... yield train_data >>> dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids", ... "next_sentence_labels", "masked_lm_positions", ... "masked_lm_ids", "masked_lm_weights"]) >>> dataset = dataset.batch(batch_size=16) >>> mlm_trainer = MaskedLanguageModelingTrainer(model_name="bert_tiny_uncased") >>> mlm_trainer.train(dataset=dataset) >>> res = mlm_trainer.predict(input_data = "hello world [MASK]") Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(MaskedLanguageModelingTrainer, self).__init__("fill_mask", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for MaskedLanguageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, *args, **kwargs): raise NotImplementedError( "The MaskedLanguageModeling task does not support evaluate.") def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[str, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[str, BaseModel]]): The network for trainer. It support model name supported or BaseModel class. Supported model name can refer to model support list. For . Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ config = self.set_config(config) config.model.model_config.is_training = False if input_data is None: input_data = config.input_data if not isinstance(input_data, (str, list)): raise ValueError("Input data's type must be one of " f"[str, list], but got type {type(input_data)}") config.model.model_config.batch_size = 1 max_length = network.config.seq_length if network else config.model.model_config.seq_length return self.predict_process(config=config, input_data=input_data, task='fill_mask', network=network, tokenizer=tokenizer, max_length=max_length, padding="max_length", **kwargs) pengcheng.mind-test/mindformers/trainer/optimizer_grouped_parameters.py000066400000000000000000000173541471400134000272600ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Grouped parameters of optimizer.""" import json from functools import partial from typing import Optional import copy from mindspore.nn import Cell from mindspore.nn.learning_rate_schedule import LearningRateSchedule from mindformers.models import BaseModel from mindformers.core.lr import LearningRateWiseLayer from mindformers.core import build_lr from mindformers.tools import logger from .utils import check_keywords_in_name def get_optimizer_grouped_parameters(model: Optional[BaseModel] = None, weight_decay: float = 0.0, dynamic_lr_schedule: Optional[LearningRateSchedule] = None, layer_scale: bool = False, layer_decay: float = 1.0): """Get grouped parameters of the network for training.""" if not isinstance(model, (Cell, BaseModel)): raise TypeError(f"model type should be BaseModel, but get {type(model)}") skip_params = {} skip_keywords = {} if hasattr(model, 'no_weight_decay'): skip_params = model.no_weight_decay() logger.info('No weight decay: %s', skip_params) if hasattr(model, 'no_weight_decay_keywords'): skip_keywords = model.no_weight_decay_keywords() logger.info('No weight decay keywords: %s', skip_keywords) decay_parameters_names = [ param.name for param in model.trainable_params() if not (len(param.shape) == 1 or param.name.endswith(".bias") or (param.name in skip_params) or check_keywords_in_name(param.name, skip_keywords)) ] get_layer_id_func = None scales_list = [] if dynamic_lr_schedule is not None: if layer_scale: if model.__class__.__name__ == 'SwinForImageClassification': depths = model.config.depths num_layers = sum(depths) get_layer_id_func = partial(get_swin_layer, num_layers=num_layers + 2, depths=depths) scales_list = list(layer_decay ** i for i in reversed(range(num_layers + 2))) elif model.__class__.__name__ == 'ViTForImageClassification': num_layers = model.config.depth get_layer_id_func = partial(get_vit_layer, num_layers=num_layers + 2) scales_list = list(layer_decay ** i for i in reversed(range(num_layers + 2))) else: logger.warning("Now only support VitModel and SwinModel," "if use dynamic_lr_schedule and layer_scale, they will be reset and invalid.") layer_scale = False dynamic_lr_schedule = None else: logger.warning("dynamic_lr_schedule will be reset and invalid when layer_scale is False.") dynamic_lr_schedule = None parameter_group_names = {} parameter_group_vars = {} scale = 1. for param in model.trainable_params(): if param.name in decay_parameters_names: group_name = 'decay' else: group_name = 'no_decay' weight_decay = 0. if get_layer_id_func is not None: layer_id = get_layer_id_func(param.name) group_name = "layer_%d_%s" % (layer_id, group_name) if scales_list is not None: scale = scales_list[layer_id] if group_name not in parameter_group_names: parameter_group_names[group_name] = { "weight_decay": weight_decay, "params": [], } parameter_group_vars[group_name] = { "weight_decay": weight_decay, "params": [], } if isinstance(dynamic_lr_schedule, LearningRateSchedule): if layer_scale: parameter_group_vars[group_name]["lr"] = LearningRateWiseLayer(dynamic_lr_schedule, scale) else: parameter_group_vars[group_name]["lr"] = dynamic_lr_schedule parameter_group_vars[group_name]["params"].append(param) parameter_group_names[group_name]["params"].append(param.name) param_groups = json.dumps(parameter_group_names, indent=2) logger.info("Param groups = %s", param_groups) return list(parameter_group_vars.values()) def get_mup_optimizer_grouped_parameters(model: Optional[BaseModel] = None, weight_decay: float = 0.0, mup_ratio: float = 0.0, lr_schedule_config = None): """Get grouped parameters of the network for training.""" if not isinstance(model, (Cell, BaseModel)): raise TypeError(f"model type should be BaseModel, but get {type(model)}") def find_in_name(name, key_list): for k in key_list: if name.find(k) >= 0: return True return False middle_weight_key_list = [ "attention.wo", "attention.wq", "attention.wk", "attention.wv", "feed_forward", "lm_head" ] parameter_group_vars = {} middle_weight_group = {"params": []} # mlp and attention for param in model.trainable_params(): if find_in_name(param.name, middle_weight_key_list): middle_weight_group["params"].append(param) middle_lr_config = copy.deepcopy(lr_schedule_config) middle_lr_config.learning_rate = middle_lr_config.learning_rate / mup_ratio middle_lr_config.lr_middle = middle_lr_config.lr_middle / mup_ratio middle_lr_config.lr_end = middle_lr_config.lr_end / mup_ratio middle_lr = build_lr(middle_lr_config) middle_weight_group["lr"] = middle_lr middle_weight_group["weight_decay"] = weight_decay parameter_group_vars["middle_output"] = middle_weight_group norm_key_list = [ "norm", "embedding" ] norm_weight_group = {"params": []} for param in model.trainable_params(): if find_in_name(param.name, norm_key_list): norm_weight_group["params"].append(param) norm_weight_group["weight_decay"] = 0.0 parameter_group_vars["norm_key"] = norm_weight_group return list(parameter_group_vars.values()) def get_vit_layer(name, num_layers): """get vit layer""" if name.endswith(("cls_tokens", "mask_tokens", "pos_embed")): layer_num = 0 elif name.startswith("vit.patch_embed"): layer_num = 0 elif name.startswith("vit.blocks"): layer_id = int(name.split('.')[2]) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num def get_swin_layer(name, num_layers, depths): """get swin layer""" if name in ("encoder.mask_token",): layer_num = 0 elif name.startswith("encoder.patch_embed"): layer_num = 0 elif name.startswith("encoder.layers"): layer_id = int(name.split('.')[2]) block_id = name.split('.')[4] if block_id in ('reduction', 'norm'): layer_num = sum(depths[:layer_id + 1]) else: layer_id = sum(depths[:layer_id]) + int(block_id) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num pengcheng.mind-test/mindformers/trainer/question_answering/000077500000000000000000000000001471400134000246265ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/question_answering/__init__.py000066400000000000000000000014371471400134000267440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Question Answering Trainer.""" from .question_answering import QuestionAnsweringTrainer __all__ = ['QuestionAnsweringTrainer'] pengcheng.mind-test/mindformers/trainer/question_answering/__pycache__/000077500000000000000000000000001471400134000270365ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/question_answering/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004721471400134000332310ustar00rootroot00000000000000a ?cf@sdZddlmZdgZdS)zQuestion Answering Trainer.)QuestionAnsweringTrainerrN)__doc__question_answeringr__all__rrZ/home/ma-user/work/mindformers-baichuan/mindformers/trainer/question_answering/__init__.pys question_answering.cpython-39.pyc000066400000000000000000000177411471400134000353460ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/question_answering/__pycache__a ?cf$@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZdd lmZmZmZd d lmZd d lmZd d lmZdgZeej GdddeZ!dS)z Question Answer Trainer. )OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel BaseTokenizer)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) BaseTrainer)ConfigArguments)TrainingArgumentsQuestionAnsweringTrainerc s$eZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd fd d Zdeeee e e feeeefeee e feed d dZZS)raQuestionAnswering Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForQuestionAnswering, BertConfig >>> config = MindFormerConfig("configs/qa/run_qa_bert_base_uncased.yaml") >>> #1) use config to train >>> cls_task = QuestionAnsweringTrainer(model_name='qa_bert_base_uncased') >>> cls_task.train(config=config) >>> #2) use instance function to train Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Nquestion_answering)superr__init__)selfr __class__d/home/ma-user/work/mindformers-baichuan/mindformers/trainer/question_answering/question_answering.pyr7sz!QuestionAnsweringTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for TokenClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )rr r$r!r"r#N)training_process)rrr r!r"r#r$kwargsrrrtrain:s%zQuestionAnsweringTrainer.train)rr r!r$compute_metricsc s2d}|d|tjf|||||d|dS)aEvaluate task for TokenClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. z QA Metric metric_name)rr r!r(r$N) setdefaultrevaluate_process)rrr r!r$r(r&r)rrrevaluatehs z!QuestionAnsweringTrainer.evaluate)r input_datar tokenizercKstd|dur$|jr |j}nd}t|ttfs:tdt|tr`|D]}t|tsHtdqHd|jj_ |j f||d||d|S) ai Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. z/.........Build Input Data For Predict..........Nz;My name is Wolfgang and I live in Berlin - Where do I live?z,Input data's type must be one of [str, list]z*The element of input data list must be strr)rr-taskr r.) r infor- isinstancestrlist ValueErrormodel model_config batch_sizepredict_process)rrr-r r.r&itemrrrpredicts(     z QuestionAnsweringTrainer.predict)N)NNNNNN)NNNNN)NNNN)__name__ __module__ __qualname____doc__r3rrrdictrrrr r r rrrrrr'setr,r4r r; __classcell__rrrrr"sL /+N)"r?typingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr r mindformers.tools.loggerr mindformers.tools.registerrrr base_trainerr config_argsr training_argsr__all__registerTRAINERrrrrrs        pengcheng.mind-test/mindformers/trainer/question_answering/question_answering.py000066400000000000000000000223461471400134000311330ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Question Answer Trainer. """ from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..base_trainer import BaseTrainer from ..config_args import ConfigArguments from ..training_args import TrainingArguments __all__ = ['QuestionAnsweringTrainer'] @MindFormerRegister.register(MindFormerModuleType.TRAINER) class QuestionAnsweringTrainer(BaseTrainer): r"""QuestionAnswering Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForQuestionAnswering, BertConfig >>> config = MindFormerConfig("configs/qa/run_qa_bert_base_uncased.yaml") >>> #1) use config to train >>> cls_task = QuestionAnsweringTrainer(model_name='qa_bert_base_uncased') >>> cls_task.train(config=config) >>> #2) use instance function to train Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(QuestionAnsweringTrainer, self).__init__("question_answering", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for TokenClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for TokenClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. """ metric_name = "QA Metric" kwargs.setdefault("metric_name", metric_name) super().evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ logger.info(".........Build Input Data For Predict..........") if input_data is None: if config.input_data: input_data = config.input_data else: input_data = "My name is Wolfgang and I live in Berlin - Where do I live?" if not isinstance(input_data, (str, list)): raise ValueError("Input data's type must be one of [str, list]") if isinstance(input_data, list): for item in input_data: if not isinstance(item, str): raise ValueError("The element of input data list must be str") # This is a known issue, you need to specify batch size equal to 1 when creating bert model. config.model.model_config.batch_size = 1 return self.predict_process(config=config, input_data=input_data, task='question_answering', network=network, tokenizer=tokenizer, **kwargs) pengcheng.mind-test/mindformers/trainer/text_classfication/000077500000000000000000000000001471400134000245705ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/text_classfication/__init__.py000066400000000000000000000014431471400134000267030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Classification Trainer.""" from .text_classification import TextClassificationTrainer __all__ = ['TextClassificationTrainer'] pengcheng.mind-test/mindformers/trainer/text_classfication/__pycache__/000077500000000000000000000000001471400134000270005ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/text_classfication/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004751471400134000331760ustar00rootroot00000000000000a ?cf#@sdZddlmZdgZdS)zText Classification Trainer.)TextClassificationTrainerrN)__doc__text_classificationr__all__rrZ/home/ma-user/work/mindformers-baichuan/mindformers/trainer/text_classfication/__init__.pys text_classification.cpython-39.pyc000066400000000000000000000204331471400134000354130ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/text_classfication/__pycache__a ?cf&@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZdd lmZmZmZd d lmZd d lmZd d lmZeejGdddeZ dS)zText Classification Trainer.)OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel BaseTokenizer)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerc seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee eeeefeeeeefeeeefd d d Zdeeee e e feeeefeee eed d dZZS)TextClassificationTraineraTextClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import TextClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForMultipleChoice, BertConfig >>> config = MindFormerConfig("configs/txtcls/run_txtcls_bert_base_uncased.yaml") >>> #1) use default config to train >>> txtcls_task = TextClassificationTrainer(model_name='bert_for_multiple_choice') >>> txtcls_task.train(config=config) >>> txtcls_task.evaluate(config=config) >>> input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", ... "i don't know um do you do a lot of camping-I know exactly."] >>> res = txtcls_task.predict(input_data=input_data) >>> #2) use instance function to train Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Ntext_classification)superr__init__)selfr __class__e/home/ma-user/work/mindformers-baichuan/mindformers/trainer/text_classfication/text_classification.pyr9sz"TextClassificationTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for TextClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )rr r$r!r"r#N)training_process)rrr r!r"r#r$kwargsrrrtrain<s%zTextClassificationTrainer.train)rr r!r$compute_metricscKs0d}|d||jf|||||d|dS)aEvaluate task for TextClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. z Top1 Accuracy metric_name)rr r!r(r$N) setdefaultevaluate_process)rrr r!r$r(r&r)rrrevaluatejs z"TextClassificationTrainer.evaluate)r input_datar tokenizerc Ks||}td|dur"|j}t|ttfs8tdt|tr^|D]}t|tsFtdqFd|jj _ |rt|j j n|jj j }|j f||d|||dd|S) ak Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. z/.........Build Input Data For Predict..........Nz,Input data's type must be one of [str, list]z*The element of input data list must be strr max_length)rr-taskr r.r0padding) set_configr infor- isinstancestrlist ValueErrormodel model_config batch_sizer seq_lengthpredict_process)rrr-r r.r&itemr0rrrpredicts,      z!TextClassificationTrainer.predict)N)NNNNNN)NNNNN)NNNN)__name__ __module__ __qualname____doc__r6rrrdictrrrr r r rrrrrr'setr,r7r r? __classcell__rrrrr sL /  + rN)!rCtypingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr r mindformers.tools.loggerr mindformers.tools.registerrrr config_argsr training_argsr base_trainerrregisterTRAINERrrrrrs        pengcheng.mind-test/mindformers/trainer/text_classfication/text_classification.py000066400000000000000000000232621471400134000312060ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Classification Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer @MindFormerRegister.register(MindFormerModuleType.TRAINER) class TextClassificationTrainer(BaseTrainer): r"""TextClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import TextClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForMultipleChoice, BertConfig >>> config = MindFormerConfig("configs/txtcls/run_txtcls_bert_base_uncased.yaml") >>> #1) use default config to train >>> txtcls_task = TextClassificationTrainer(model_name='bert_for_multiple_choice') >>> txtcls_task.train(config=config) >>> txtcls_task.evaluate(config=config) >>> input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", ... "i don't know um do you do a lot of camping-I know exactly."] >>> res = txtcls_task.predict(input_data=input_data) >>> #2) use instance function to train Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(TextClassificationTrainer, self).__init__("text_classification", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for TextClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for TextClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. """ metric_name = "Top1 Accuracy" kwargs.setdefault("metric_name", metric_name) self.evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ config = self.set_config(config) logger.info(".........Build Input Data For Predict..........") if input_data is None: input_data = config.input_data if not isinstance(input_data, (str, list)): raise ValueError("Input data's type must be one of [str, list]") if isinstance(input_data, list): for item in input_data: if not isinstance(item, str): raise ValueError("The element of input data list must be str") # bert模型已知issue，由于bert模型在创建的时候需要batch_size参数， # 同时pipeline是一个样本一个样本进行处理，所以这里设定为1 config.model.model_config.batch_size = 1 max_length = network.config.seq_length if network else config.model.model_config.seq_length return self.predict_process(config=config, input_data=input_data, task='text_classification', network=network, tokenizer=tokenizer, max_length=max_length, padding="max_length", **kwargs) pengcheng.mind-test/mindformers/trainer/token_classification/000077500000000000000000000000001471400134000250755ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/token_classification/__init__.py000066400000000000000000000014471471400134000272140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Token Classification Trainer.""" from .token_classification import TokenClassificationTrainer __all__ = ['TokenClassificationTrainer'] pengcheng.mind-test/mindformers/trainer/token_classification/__pycache__/000077500000000000000000000000001471400134000273055ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/token_classification/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005021471400134000334720ustar00rootroot00000000000000a ?cf'@sdZddlmZdgZdS)zToken Classification Trainer.)TokenClassificationTrainerrN)__doc__token_classificationr__all__rr\/home/ma-user/work/mindformers-baichuan/mindformers/trainer/token_classification/__init__.pys token_classification.cpython-39.pyc000066400000000000000000000214661471400134000360630ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/token_classification/__pycache__a ?cf (@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZdd lmZmZmZd d lmZd d lmZd d lmZddlmZe ej!GdddeZ"dS)zToken Classification Trainer.)OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel BaseTokenizer)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) BaseTrainer)ConfigArguments)TrainingArguments)cluener_labelsc s$eZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd fd d Zdeeee e e feeeefeee e feed d dZZS)TokenClassificationTraineruTokenClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import TokenClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForTokenClassification, BertConfig >>> config = MindFormerConfig("configs/tokcls/run_tokcls_bert_base_chinese.yaml") >>> #1) use config to train or evaluate or predict >>> tokcls_task = TokenClassificationTrainer(model_name='tokcls_bert_base_chinese') >>> tokcls_task.train(config=config) >>> tokcls_task.evaluate(config=config) >>> input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。", "的时间会去玩玩星际2。"] >>> res = tokcls_task.predict(input_data=input_data) >>> print(res) [[{'entity_group': 'address', 'word': '日内瓦', 'start': 6, 'end': 9}], [{'entity_group': 'game', 'word': '星际2', 'start': 7, 'end': 10}]] >>> #2) use instance function to train or evaluate or predict Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Ntoken_classification)superr__init__)selfr __class__h/home/ma-user/work/mindformers-baichuan/mindformers/trainer/token_classification/token_classification.pyr<sz#TokenClassificationTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for TokenClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )r!r"r&r#r$r%N)training_process)rr!r"r#r$r%r&kwargsrrr train?s%z TokenClassificationTrainer.train)r!r"r#r&compute_metricsc s2d}|d|tjf|||||d|dS)aEvaluate task for TokenClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. z Entity Metric metric_name)r!r"r#r*r&N) setdefaultrevaluate_process)rr!r"r#r&r*r(r+rrr evaluatems z#TokenClassificationTrainer.evaluate)r! input_datar" tokenizerc Ks||}td|dur"|j}t|ttfs8tdt|tr^|D]}t|tsFtdqFd|jj _ |rt|j j n|jj j }ddt tD}|jf||d|||d |d |S) ai Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. z/.........Build Input Data For Predict..........Nz,Input data's type must be one of [str, list]z*The element of input data list must be strcSsi|]\}}||qSrr).0label_idlabelrrr z6TokenClassificationTrainer.predict..r max_length)r!r/taskr"r0r7paddingid2label) set_configr infor/ isinstancestrlist ValueErrormodel model_config batch_sizer! seq_length enumeraterpredict_process) rr!r/r"r0r(itemr7r:rrr predicts0      z"TokenClassificationTrainer.predict)N)NNNNNN)NNNNN)NNNN)__name__ __module__ __qualname____doc__r>rrrdictrrrr r r rrrrrr)setr.r?r rH __classcell__rrrr r sL /+rN)#rLtypingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr r mindformers.tools.loggerr mindformers.tools.registerrrr base_trainerr config_argsr training_argsrdataset.labelsrregisterTRAINERrrrrr s         pengcheng.mind-test/mindformers/trainer/token_classification/token_classification.py000066400000000000000000000240131471400134000316420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Token Classification Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..base_trainer import BaseTrainer from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ...dataset.labels import cluener_labels @MindFormerRegister.register(MindFormerModuleType.TRAINER) class TokenClassificationTrainer(BaseTrainer): r"""TokenClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import TokenClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForTokenClassification, BertConfig >>> config = MindFormerConfig("configs/tokcls/run_tokcls_bert_base_chinese.yaml") >>> #1) use config to train or evaluate or predict >>> tokcls_task = TokenClassificationTrainer(model_name='tokcls_bert_base_chinese') >>> tokcls_task.train(config=config) >>> tokcls_task.evaluate(config=config) >>> input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。", "的时间会去玩玩星际2。"] >>> res = tokcls_task.predict(input_data=input_data) >>> print(res) [[{'entity_group': 'address', 'word': '日内瓦', 'start': 6, 'end': 9}], [{'entity_group': 'game', 'word': '星际2', 'start': 7, 'end': 10}]] >>> #2) use instance function to train or evaluate or predict Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(TokenClassificationTrainer, self).__init__("token_classification", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for TokenClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for TokenClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. """ metric_name = "Entity Metric" kwargs.setdefault("metric_name", metric_name) super().evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ config = self.set_config(config) logger.info(".........Build Input Data For Predict..........") if input_data is None: input_data = config.input_data if not isinstance(input_data, (str, list)): raise ValueError("Input data's type must be one of [str, list]") if isinstance(input_data, list): for item in input_data: if not isinstance(item, str): raise ValueError("The element of input data list must be str") # This is a known issue, you need to specify batch size equal to 1 when creating bert model. config.model.model_config.batch_size = 1 max_length = network.config.seq_length if network else config.model.model_config.seq_length id2label = {label_id: label for label_id, label in enumerate(cluener_labels)} return self.predict_process(config=config, input_data=input_data, task='token_classification', network=network, tokenizer=tokenizer, max_length=max_length, padding="max_length", id2label=id2label, **kwargs) pengcheng.mind-test/mindformers/trainer/trainer.py000066400000000000000000001504651471400134000227330ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Trainer API For Import.""" import os import shutil from collections import OrderedDict from typing import List, Optional, Union import numpy as np from PIL.Image import Image import mindspore as ms from mindspore import Tensor from mindspore.common import set_seed from mindspore import load_checkpoint, load_param_into_net from mindspore.nn import TrainOneStepCell, Optimizer, Cell, \ PipelineCell, MicroBatchInterleaved from mindspore.nn.wrap.cell_wrapper import _VirtualDatasetCell from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.dataset.engine.datasets import BatchDataset, RepeatDataset from mindformers.core.parallel_config import build_parallel_config from mindformers.dataset import build_dataset, build_dataset_loader, \ check_dataset_config, BaseDataset from mindformers.mindformer_book import MindFormerBook from mindformers.models import build_model, BaseModel, BaseImageProcessor, \ BaseTokenizer, BaseAudioProcessor from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerConfig from mindformers.tools.register.config import ordered_yaml_dump from .build_trainer import build_trainer from .config_args import ConfigArguments from .training_args import TrainingArguments from .utils import check_train_data_loader_type, check_eval_data_loader_type, \ check_optimizer_and_lr_type, check_wrapper_config, config2dict __all__ = ['Trainer'] SUPPORT_TASKS = MindFormerBook().get_trainer_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_PIPELINE_INPUT_DATA = MindFormerBook().get_pipeline_support_input_data_list() CURRENT_PROJECT_PATH = MindFormerBook().get_project_path() DEFAULT_CHECKPOINT_DIR = 'checkpoint' DEFAULT_CONFIG_DIR = 'configs' class Trainer: r""" Trainer package to train\evaluate\predict class. The trainer interface is used to quickly start training, evaluation and predict for integrated tasks. It also allows users to customize the model, optimizer, dataset, tokenizer, processor, train_one_step, callback, and metric. Args: args (Optional[Union[str, dict, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It support yaml path or config dict or ConfigArguments class. Default: None. task (str): The task name supported. Please refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: 'general'. model (Optional[Union[str, Cell, BaseModel]]): The network for trainer. It support model name supported or BaseModel or MindSpore Cell class. Supported model name can refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: None. model_name (Optional[Union[str]]): The model name supported. Default: None. train_dataset (Optional[Union[str, BaseDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. eval_dataset (Optional[Union[str, BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for text preprocessing. It support BaseTokenizer class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. audio_processor (Optional[BaseAudioProcessor]): The processor for audio preprocessing. It support BaseAudioProcessor class. Default: None. optimizers (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. eval_callbacks (Optional[Union[Callback, List[Callback]]]): The evaluate callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. save_config (bool): Save current the config of task. Default: False. Raises: KeyError: If 'task' or 'model' not in supported trainer. Examples: >>> from mindformers import Trainer >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index] ... ... def __len__(self): ... return len(self._data) >>> #1) input task name and model name to init trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train') >>> #2) input config to init trainer >>> from mindformers.trainer.config_args import ConfigArguments, OptimizerConfig, \ ... RunnerConfig, LRConfig, WrapperConfig >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindspore.train.callback import LossMonitor >>> runner_config = RunnerConfig(epochs=10, batch_size=2, image_size=224) >>> lr_schedule_config = LRConfig(lr_type='WarmUpLR', learning_rate=0.001, warmup_steps=10) >>> optim_config = OptimizerConfig(optim_type='Adam', beta1=0.009, learning_rate=lr_schedule_config) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper_config = WrapperConfig(wrapper_type='TrainOneStepWithLossScaleCell', scale_sense=loss_scale) >>> dataset = GeneratorDataset(source=MyDataLoader(), column_names='image') >>> dataset = dataset.batch(batch_size=2) >>> config = ConfigArguments(seed=2022, runner_config=runner_config, ... optimizer=optim_config, runner_wrapper=wrapper_config) >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... args=config, train_dataset=dataset) >>> #3) input instance to init trainer >>> from mindformers.models import ViTForImageClassification >>> vit_model_with_loss = ViTForImageClassification() >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=vit_model_with_loss.trainable_params()) >>> loss_cb = LossMonitor(per_print_times=2) >>> callbacks = [loss_cb] >>> task_trainer = Trainer(task='image_classification', ... model=vit_model_with_loss, ... args=config, ... optimizers=optimizer, ... train_dataset=dataset, ... callbacks=callbacks) """ def __init__(self, args: Optional[Union[str, dict, ConfigArguments, TrainingArguments]] = None, task: Optional[str] = 'general', model: Optional[Union[str, Cell, BaseModel]] = None, model_name: Optional[Union[str]] = None, train_dataset: Optional[Union[str, BaseDataset]] = None, eval_dataset: Optional[Union[str, BaseDataset]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, optimizers: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, pet_method: Optional[str] = '', callbacks: Optional[Union[Callback, List[Callback]]] = None, eval_callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, save_config: bool = False, **kwargs): self.task = task self.model = model self.model_name = model_name self.pet_method = pet_method self.train_dataset = train_dataset self.eval_dataset = eval_dataset self.optimizers = optimizers self.wrapper = wrapper self.tokenizer = tokenizer self.image_processor = image_processor self.audio_processor = audio_processor self.callbacks = callbacks self.eval_callbacks = eval_callbacks self.compute_metrics = compute_metrics self.default_checkpoint_name_or_path = None self.configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) self.kwargs = kwargs if not os.path.exists(os.path.join('.', DEFAULT_CONFIG_DIR)): configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) if os.path.exists(os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR)): mindformers_configs_directory = os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR) # python 3.7 版本不支持dirs_exist_ok入参, python 3.8及以上版本支持 try: # adapt to python 3.8+ # pylint: disable=E1123 shutil.copytree(mindformers_configs_directory, configs_directory, dirs_exist_ok=True) except TypeError: try: # adapt to python 3.7 shutil.copytree(mindformers_configs_directory, configs_directory) except FileExistsError: pass if wrapper is not None: if model is not None: logger.warning( 'wrapper has existed, input model invalid, it should be include in wrapper.') if optimizers is not None: logger.warning( 'wrapper has existed, input optimizers invalid, it should be include in wrapper.') assert task in SUPPORT_TASKS.keys(), \ f"task name must be in {SUPPORT_TASKS.keys()}, but get {task}." if isinstance(model, str): model = model + '_{}'.format(pet_method) if pet_method else model assert model in SUPPORT_MODEL_NAMES, \ f"model must be in {SUPPORT_MODEL_NAMES} when model's type is string, but get {model}." self.model_name = model self.model = None if isinstance(self.model, (Cell, BaseModel)): logger.info("The model instance has been entered, " "and the model will not be created from model_config") if pet_method: logger.warning("pet_method is not valid when a model instance is passed in." "Currently, only part of the model in MindFormers is supported." "Please pass in the model keyword") self.is_model_instance = True else: self.is_model_instance = False if self.is_model_instance and self.model_name is None: logger.warning( "Recognizing that a model instance is sent and model_name is None,") logger.warning( "it is recommended to select a model configuration that corresponds " "to the support of MindFormers based on the instance model and set model_name.") logger.warning( "Otherwise, they will default to a general configuration." "You are advised to pass instances such as optimizers, metric, tokenizer, and processor") self.model_name = 'common' task_config = MindFormerConfig(SUPPORT_TASKS.get(self.task).get(self.model_name)) if self.model_name == "common": if self.model is not None: task_config.trainer.model_name = self.model.__class__.__name__ if self.wrapper is not None: task_config.trainer.model_name = self.wrapper.network.__class__.__name__ if args is None: self.config = task_config else: if isinstance(args, dict): task_config.merge_from_dict(args) elif isinstance(args, str): assert os.path.realpath(args) and os.path.exists(args), \ f"config path must be exist, but get {args}." assert args.endswith(('.yaml', '.yml')), \ f"config file must be end with .yaml or .yml, but get {args}" task_config = MindFormerConfig(args) elif isinstance(args, ConfigArguments): if hasattr(args, 'train_dataset'): check_train_data_loader_type(args, task_config) if hasattr(args, 'eval_dataset'): check_eval_data_loader_type(args, task_config) if hasattr(args, 'optimizer'): check_optimizer_and_lr_type(args, task_config) if hasattr(args, 'runner_wrapper'): check_wrapper_config(args, task_config) task_config.merge_from_dict(args.__dict__) elif isinstance(args, TrainingArguments): logger.warning( "When using the TrainingArguments class, " "its arguments will override the default config configuration.") args.convert_args_to_mindformers_config(task_config) self.config = task_config if save_config: self.save_config_to_yaml(self.config) logger.info("save running config success of %s_new.", task_config.trainer.model_name.lower()) # check dataset config if isinstance(train_dataset, str): assert os.path.exists(train_dataset), \ f"train dataset path must be exist, but get {train_dataset}." self.config.train_dataset.data_loader.dataset_dir = train_dataset self.train_dataset = None if isinstance(eval_dataset, str): assert os.path.exists(eval_dataset), \ f"eval dataset path must be exist, but get {eval_dataset}." self.config.eval_dataset.data_loader.dataset_dir = eval_dataset self.eval_dataset = None if tokenizer is not None: if self.config.train_dataset is not None: self.config.train_dataset.tokenizer = tokenizer if self.config.eval_dataset is not None: self.config.eval_dataset.tokenizer = tokenizer check_dataset_config(self.config) # build parallel config self.rank_id = int(os.getenv("RANK_ID", "0")) self.device_num = int(os.getenv("RANK_SIZE", "1")) self.config.rank_id = self.rank_id self.config.device_num = self.device_num self.is_set_parallel_config = False self.is_set_moe_config = False self.is_set_recompute_config = False # set seed if self.config.seed and \ ms.context.get_auto_parallel_context("parallel_mode") \ not in ["semi_auto_parallel", "auto_parallel"]: set_seed(self.config.seed) np.random.seed(self.config.seed) # set output directory os.environ.setdefault("LOCAL_DEFAULT_PATH", self.config.output_dir) # build task trainer self.trainer = build_trainer(self.config.trainer) if self.trainer is None: raise ModuleNotFoundError("config must be contain 'trainer' key, but get None.") def train(self, train_checkpoint: Optional[Union[str, bool]] = False, resume_training: Optional[bool] = None, auto_trans_ckpt: Optional[bool] = None, do_eval: bool = False, **kwargs): r"""Train task for Trainer. This function is used to train or fine-tune the network. Args: train_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It supports real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. resume_training (bool): Whether to perform resume training. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if resume_or_finetune_from_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/eval') >>> # 1) default train task to reproduce model. >>> task_trainer.train() >>> # 2) eval network when train task to reproduce model. >>> task_trainer.train(do_eval=True) >>> # 3) resume train task to auto load the last checkpoint, if training break after 10 epochs. >>> task_trainer.train(train_checkpoint=True, resume_training=True) >>> # 4) resume train task according to checkpoint path, if training break after 10 epochs. >>> task_trainer.train( ... resume_or_finetune_from_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... resume_training=True) """ if train_checkpoint is not None and \ not isinstance(train_checkpoint, (bool, str)): raise TypeError(f"resume_or_finetune_from_checkpoint must be one of [None, string, bool], " f"but get {train_checkpoint}") if train_checkpoint is False: train_checkpoint = None do_eval = do_eval or self.config.do_eval if do_eval: if self.eval_dataset is None: logger.info("do_eval is enabled, building eval_dataset from config.") self.eval_dataset = build_dataset(self.config.eval_dataset_task) if self.eval_dataset is None: raise ValueError(f"if do_eval is true, eval_dataset must be input, " f"the task {self.task} is not support eval now.") # open do_eval for trainer config self.config.do_eval = True if train_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(train_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = train_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None if resume_training is not None: self.config.resume_training = resume_training if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=True) self.trainer.train( config=self.config, network=self.model, dataset=self.train_dataset, optimizer=self.optimizers, eval_dataset=self.eval_dataset if do_eval else None, wrapper=self.wrapper, callbacks=self.callbacks, is_full_config=True, **kwargs) def finetune(self, finetune_checkpoint: Optional[Union[str, bool]] = False, resume_training: Optional[bool] = None, auto_trans_ckpt: Optional[bool] = None, do_eval: bool = False, **kwargs): r"""Finetune task for Trainer. This function is used to fine-tune the network. Args: finetune_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. if resume_training is true, this checkpoint will be used to restore training of the network. Default: False. resume_training (bool): Whether to perform resume training. Default: False auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if load_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='text_generation', ... model='gpt2', ... pet_method='lora', ... train_dataset='./train', ... eval_dataset='./eval') >>> # 1) default finetune task. >>> task_trainer.finetune() >>> # 2) eval network when finetune model. >>> task_trainer.finetune(do_eval=True) >>> # 3) The last weight in the output directory is automatically loaded to resume training. >>> task_trainer.finetune(finetune_checkpoint=True, resume_training=True) >>> # 4) The last weight in the output directory is automatically loaded for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint=True) >>> # 5) Specify weights to fine-tune. >>> task_trainer.finetune(finetune_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') >>> # 6) Automatically load preset weights for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint='gpt2') """ if finetune_checkpoint is not None and \ not isinstance(finetune_checkpoint, (bool, str)): raise TypeError(f"load_checkpoint must be one of [None, string, bool], " f"but get {finetune_checkpoint}") if finetune_checkpoint is False: finetune_checkpoint = None do_eval = do_eval or self.config.do_eval if do_eval: if self.eval_dataset is None: logger.info("do_eval is enabled, building eval_dataset from config.") self.eval_dataset = build_dataset(self.config.eval_dataset_task) if self.eval_dataset is None: raise ValueError(f"if do_eval is true, eval_dataset must be input, " f"the task {self.task} is not support eval now.") # open do_eval for trainer config self.config.do_eval = True if finetune_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(finetune_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = finetune_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be fine-tune are specified" "by the finetune_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None self.config.model.model_config.checkpoint_name_or_path = None if resume_training is not None: self.config.resume_training = resume_training if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=True) self.trainer.train( config=self.config, network=self.model, dataset=self.train_dataset, optimizer=self.optimizers, eval_dataset=self.eval_dataset if do_eval else None, wrapper=self.wrapper, callbacks=self.callbacks, is_full_config=True, **kwargs) def evaluate(self, eval_checkpoint: Optional[Union[str, bool]] = False, auto_trans_ckpt: Optional[bool] = None, **kwargs): r"""Evaluate task for Trainer. This function is used to evaluate the network. Args: eval_checkpoint (Optional[Union[str, bool]]): Used to evaluate the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model Raises: TypeError: if eval_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... eval_dataset='data/imagenet/train') >>> # 1) default evaluate task to test model. >>> task_trainer.evaluate() >>> # 2) evaluate task to auto load the last checkpoint. >>> task_trainer.evaluate(eval_checkpoint=True) >>> # 3) evaluate task according to checkpoint path. >>> task_trainer.evaluate(eval_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') """ if eval_checkpoint is not None and not isinstance(eval_checkpoint, (bool, str)): raise TypeError(f"eval_checkpoint must be one of [None, string, bool], " f"but get {eval_checkpoint}") if eval_checkpoint is False: eval_checkpoint = None if eval_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(eval_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = eval_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be evaluated are specified" "by the eval_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=False) self.trainer.evaluate( config=self.config, network=self.model, dataset=self.eval_dataset, callbacks=self.eval_callbacks, is_full_config=True, **kwargs) def predict(self, predict_checkpoint: Optional[Union[str, bool]] = None, auto_trans_ckpt: Optional[bool] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, **kwargs): r"""Predict task for Trainer. This function is used to predict the network. Args: predict_checkpoint (Optional[Union[str, bool]]): Used to predict the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. Return: predict result (dict). Raises: TypeError: if predict_checkpoint is not bool or str type. TypeError: if input_data is not Tensor or np.ndarray or Image or str or list. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16') >>> input_data = "./sunflower.png" >>> # 1) predict task to auto load the last checkpoint. >>> task_trainer.predict(predict_checkpoint=True, input_data=input_data) >>> # 2) predict task according to checkpoint path. >>> task_trainer.predict(predict_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... input_data=input_data) >>> # 3) download and auto load the checkpoint on obs and predict. >>> task_trainer.predict(input_data=input_data) """ if predict_checkpoint is not None and not isinstance(predict_checkpoint, (bool, str)): raise TypeError(f"predict_checkpoint must be one of [None, string, bool], " f"but get {predict_checkpoint}") if self.task not in SUPPORT_PIPELINES.keys(): raise NotImplementedError(f"The {self.task} not support predict, " f"now this tasks {SUPPORT_PIPELINES.keys()} is support predict.") if predict_checkpoint is False: predict_checkpoint = None if predict_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(predict_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = predict_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be evaluated are specified" "by the eval_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if input_data is None: input_data = build_dataset_loader(self.config.eval_dataset.data_loader) logger.info("dataset by config is used as input_data.") assert isinstance(input_data, (GeneratorDataset, BaseDataset, RepeatDataset, BatchDataset, Tensor, np.ndarray, Image, str, list)), \ "Input data's type must be one of [GeneratorDataset," \ " str, ms.Tensor, np.ndarray, PIL.Image.Image]" if self.is_model_instance: self.reset_model_instance(is_train=False) kwargs['model_name'] = self.model_name output_result = self.trainer.predict( config=self.config, input_data=input_data, network=self.model, image_processor=self.image_processor, audio_processor=self.audio_processor, tokenizer=self.tokenizer, is_full_config=True, **kwargs) return output_result def build_network(self, input_checkpoint: Optional[Union[str, bool]] = None, is_train: bool = True): """build network for trainer.""" if self.model is None and self.task != 'general': logger.info("...........Start Init Network..........") self.model = build_model(self.config.model) # set running mode if self.model is not None: self.model.set_train(is_train) else: logger.warning("network will be create in %s task trainer class.", self.task) if self.is_model_instance and input_checkpoint: self._load_model_checkpoint() def set_parallel_config( self, data_parallel=1, model_parallel=1, expert_parallel=1, pipeline_stage=1, micro_batch_interleave_num=1, micro_batch_num=1, use_seq_parallel=False, optimizer_shard=False, gradient_aggregation_group=4, vocab_emb_dp=True): r""" set_parallel_config for the setting global data parallel, model parallel and fusion group. The parallel configure setting for Trainer. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. use_seq_parallel (bool): Whether to enable sequence parallel. Default False. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. micro_batch_interleave_num (int): split num of batch size. Default: 1. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_parallel_config(data_parallel=2, model_parallel=2) """ self.config.parallel_config.data_parallel = data_parallel self.config.parallel_config.model_parallel = model_parallel self.config.parallel_config.expert_parallel = expert_parallel self.config.parallel_config.pipeline_stage = pipeline_stage self.config.parallel_config.use_seq_parallel = use_seq_parallel self.config.parallel_config.optimizer_shard = optimizer_shard self.config.parallel_config.micro_batch_num = micro_batch_num self.config.parallel_config.vocab_emb_dp = vocab_emb_dp self.config.parallel_config.gradient_aggregation_group = gradient_aggregation_group self.config.micro_batch_interleave_num = micro_batch_interleave_num self.is_set_parallel_config = True def set_recompute_config(self, recompute=False, parallel_optimizer_comm_recompute=False, select_recompute=False, mp_comm_recompute=True, recompute_slice_activation=False): r"""Set recompute config. TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. select_recompute (bool): Only Enable recomputation of the attention layer or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_recompute_config(recompute=True) """ self.config.recompute_config.recompute = recompute self.config.recompute_config.select_recompute = select_recompute self.config.recompute_config.parallel_optimizer_comm_recompute = parallel_optimizer_comm_recompute self.config.recompute_config.mp_comm_recompute = mp_comm_recompute self.config.recompute_config.recompute_slice_activation = recompute_slice_activation self.is_set_recompute_config = True def set_moe_config(self, expert_num=1, capacity_factor=1.1, aux_loss_factor=0.05, num_experts_chosen=1, expert_group_size=None, group_wise_a2a=False, comp_comm_parallel=False, comp_comm_parallel_degree=2): r"""The configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_moe_config(expert_num=2, capacity_factor=1.2, aux_loss_factor=0.001) """ self.config.moe_config.expert_num = expert_num self.config.moe_config.capacity_factor = capacity_factor self.config.moe_config.aux_loss_factor = aux_loss_factor self.config.moe_config.num_experts_chosen = num_experts_chosen self.config.moe_config.expert_group_size = expert_group_size self.config.moe_config.group_wise_a2a = group_wise_a2a self.config.moe_config.comp_comm_parallel = comp_comm_parallel self.config.moe_config.comp_comm_parallel_degree = comp_comm_parallel_degree self.is_set_moe_config = True def reset_model_instance(self, is_train=True): """Reset model instance for new model config.""" if True not in [self.is_set_parallel_config, self.is_set_moe_config, self.is_set_recompute_config]: return if self.is_set_parallel_config: logger.info("The incoming model will be configured in parallel.") if self.is_set_recompute_config: logger.info("The incoming model will be configured in recompute.") if self.is_set_moe_config: logger.info("The incoming model will be configured in moe.") if not isinstance(self.model, BaseModel): raise NotImplementedError("Currently only the integrated model structure in MindFormers is supported.") build_parallel_config(self.config) model_config = self.model.config model_config.parallel_config = self.config.parallel_config model_config.moe_config = self.config.moe_config self.model.__init__(model_config) self.model.set_train(is_train) if not is_train: return network = self.model micro_batch_interleave_num = self.config.micro_batch_interleave_num if ms.get_auto_parallel_context("pipeline_stages") > 1: micro_batch_num = self.config.parallel_config.micro_batch_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = PipelineCell(MicroBatchInterleaved(self.model, micro_batch_interleave_num), micro_size=micro_batch_num) else: network = PipelineCell(self.model, micro_size=micro_batch_num) network = _VirtualDatasetCell(network) else: if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = MicroBatchInterleaved(self.model, micro_batch_interleave_num) del self.model self.model = network def get_train_dataloader(self): """get train dataloader of mindspore.""" return build_dataset_loader(self.config.train_dataset.data_loader) def get_eval_dataloader(self): """get eval dataloader of mindspore.""" return build_dataset_loader(self.config.eval_dataset.data_loader) def get_last_checkpoint(self): """get last checkpoint for resuming or finetune.""" output_folder = self.config.output_dir checkpoint_dir = os.path.join( output_folder, DEFAULT_CHECKPOINT_DIR, 'rank_{}'.format(self.rank_id)) output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] if not output_checkpoint_path: return None output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) return os.path.join(checkpoint_dir, output_checkpoint_path[-1]) def save_config_to_yaml(self, config: dict = None): """save now config file to yaml file.""" if config is None: config = self.config model_name = self.config.trainer.model_name config_dict = _reset_config_for_save(config) config_dir = os.path.join( self.configs_directory, model_name.lower() + '_new') if not os.path.exists(config_dir): os.makedirs(config_dir, exist_ok=True) run_yaml_path = os.path.join(config_dir, 'run_{}.yaml'.format(model_name.lower())) _save_config_to_yaml(run_yaml_path, config_dict) def _load_model_checkpoint(self): """Load model checkpoint to network.""" checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if checkpoint_name_or_path is None: logger.warning("checkpoint_name_or_path is None, not load input checkpoint.") elif isinstance(checkpoint_name_or_path, str): is_exist_path = os.path.exists(checkpoint_name_or_path) is_checkpoint_name = checkpoint_name_or_path in SUPPORT_MODEL_NAMES if is_exist_path: logger.info("now input valid checkpoint path, it will load to network.") checkpoint_dict = load_checkpoint(checkpoint_name_or_path) not_load_params = load_param_into_net(self.model, checkpoint_dict) logger.info("not load parameters is: %s", str(not_load_params)) elif is_checkpoint_name: logger.info("now input valid checkpoint name, it will load to network.") if isinstance(self.model, (Cell, BaseModel)): self.model.load_checkpoint(self.config.model.model_config) else: logger.warning("model must be BaseModel or Cell type, but get %s", type(self.model)) else: logger.warning("input checkpoint args is invalid, " "it must be valid and real checkpoint path or a valid checkpoint name," "but get %s", checkpoint_name_or_path) else: raise TypeError(f"checkpoint_name_or_path type error, " f"it should be one of [None, str], " f"but get {type(checkpoint_name_or_path)}") def _check_checkpoint_config(self, checkpoint: Optional[Union[str, bool]] = None): """check checkpoint config.""" if checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = self.get_last_checkpoint() elif isinstance(checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = checkpoint else: if self.default_checkpoint_name_or_path is not None: self.config.model.model_config.checkpoint_name_or_path = self.default_checkpoint_name_or_path def _save_config_to_yaml(save_file_path: str = None, save_config: dict = None): r"""Save Config to Yaml File. Args: save_file_path (str): The real path to save yaml file. Default: None. save_config (dict): The task config. Default: None. """ if save_config is None: save_config = {} with open(save_file_path, 'w', encoding='utf-8') as file_pointer: file_pointer.write( ordered_yaml_dump( save_config, default_flow_style=False, allow_unicode=True, sort_keys=False)) def _reset_config_for_save(config: dict = None): r"""Reset Config According to Yaml File Number. Args: config (dict): The task config. Default: None. model_name (str): The model name to save. Default: 'common'. """ if config is None: config = {} config = config.copy() config_dict = OrderedDict() if config.get('model') is not None: model_config = config2dict(config.pop('model')) config_dict.setdefault('model', model_config) if config.get('processor') is not None: processor_config = config2dict(config.pop('processor')) config_dict.setdefault('processor', processor_config) if config.get('train_dataset_task') is not None and config.get('train_dataset') is not None: train_dataset_config = config2dict(config.pop('train_dataset')) train_dataset_task_config = config2dict(config.pop('train_dataset_task')) config_dict.setdefault('train_dataset', train_dataset_config) config_dict.setdefault('train_dataset_task', train_dataset_task_config) if config.get('eval_dataset_task') is not None and config.get('eval_dataset') is not None: eval_dataset_config = config2dict(config.pop('eval_dataset')) eval_dataset_task_config = config2dict(config.pop('eval_dataset_task')) config_dict.setdefault('train_dataset', eval_dataset_config) config_dict.setdefault('train_dataset_task', eval_dataset_task_config) if config.get('context') is not None: context_config = config2dict(config.pop('context')) parallel_context_config = config2dict(config.pop('parallel')) moe_conifg = config2dict(config.pop('moe_config')) recompute_config = config2dict(config.pop('recompute_config')) parallel_config = config2dict(config.pop('parallel_config')) config_dict.setdefault('context', context_config) config_dict.setdefault('parallel', parallel_context_config) config_dict.setdefault('moe_conifg', moe_conifg) config_dict.setdefault('recompute_config', recompute_config) config_dict.setdefault('parallel_config', parallel_config) if config.get('runner_config') is not None: runner_config = config2dict(config.pop('runner_config')) config_dict.setdefault('runner_config', runner_config) if config.get('runner_wrapper') is not None: wrapper_config = config2dict(config.pop('runner_wrapper')) config_dict.setdefault('runner_wrapper', wrapper_config) if config.get('optimizer') is not None: optim_config = config2dict(config.pop('optimizer')) config_dict.setdefault('optimizer', optim_config) if config.get('lr_schedule') is not None: lr_config = config2dict(config.pop('lr_schedule')) config_dict.setdefault('lr_schedule', lr_config) if config.get('callbacks') is not None: cb_config = config2dict(config.pop('callbacks')) config_dict.setdefault('callbacks', cb_config) run_config = config2dict(config) for key, value in run_config.items(): config_dict.setdefault(key, value) return config_dict pengcheng.mind-test/mindformers/trainer/training_args.py000066400000000000000000000344241471400134000241120ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py # ============================================================================ """Default Training Arguments for Trainer.""" import os from typing import Optional, Union from dataclasses import dataclass, field from mindformers.tools.register import MindFormerConfig from mindformers.tools import logger from .utils import LRType, OptimizerType, SaveIntervalStrategy def _check_task_config(check_config): """check task config for adapting hugging-face.""" if check_config is not None and isinstance(check_config, MindFormerConfig): return True return False def _check_training_args(ori_value, new_value): """check training arguments for adapt MindFormers.""" if new_value is not None: return new_value return ori_value @dataclass class TrainingArguments: """ TrainingArguments is the subset of the arguments we use in our default config **which is relate to the training in MindSpore**. """ output_dir: str = field( default=None, metadata={"help": "The output directory where the model predictions and checkpoints will be written."}, ) use_parallel: bool = field( default=None, metadata={"help": "The use_parallel is used to enable distribute parallel of the network."} ) profile: bool = field( default=None, metadata={"help": "The profile is used to enable profiling of the network."} ) only_save_strategy: bool = field( default=None, metadata={"help": "The only_save_strategy is used to enable save strategy of the network."} ) sink_mode: bool = field( default=None, metadata={"help": "The sink_mode is used to enable data sink of the network."} ) sink_size: bool = field( default=None, metadata={"help": "The sink_size is used to enable data sink number per step for training or evaluation."} ) batch_size: int = field( default=None, metadata={"help": "Global batch size per GPU/TPU core/CPU for training and evaluation."} ) per_device_train_batch_size: int = field( default=None, metadata={"help": "Batch size per GPU/TPU core/CPU for training."} ) per_device_eval_batch_size: int = field( default=None, metadata={"help": "Batch size m per GPU/TPU core/CPU for evaluation."} ) learning_rate: float = field(default=None, metadata={"help": "The initial learning rate for AdamW."}) weight_decay: float = field(default=None, metadata={"help": "Weight decay for AdamW if we apply some."}) adam_beta1: float = field(default=None, metadata={"help": "Beta1 for AdamW optimizer"}) adam_beta2: float = field(default=None, metadata={"help": "Beta2 for AdamW optimizer"}) adam_epsilon: float = field(default=None, metadata={"help": "Epsilon for AdamW optimizer."}) max_grad_norm: float = field(default=None, metadata={"help": "Max gradient norm."}) num_train_epochs: float = field(default=None, metadata={"help": "Total number of training epochs to perform."}) lr_scheduler_type: Union[LRType, str] = field( default=None, metadata={"help": "The lr scheduler type to use."}, ) optim: Union[OptimizerType, str] = field( default=None, metadata={"help": "The optimizer type to use."}, ) warmup_steps: int = field(default=0, metadata={"help": "Linear warmup over warmup_steps."}) save_strategy: Union[SaveIntervalStrategy, str] = field( default=None, metadata={"help": "The checkpoint save strategy to use."}, ) integrated_save: bool = field( default=None, metadata={ "help": ( " Whether to merge and save the split Tensor in the automatic parallel scenario. " "Integrated save function is only supported in automatic parallel scene, not supported" "in manual parallel." ) } ) save_steps: int = field(default=None, metadata={"help": "Save checkpoint every X updates steps."}) save_seconds: int = field(default=None, metadata={"help": "Save checkpoint every X updates seconds."}) save_total_limit: Optional[int] = field( default=None, metadata={ "help": ( "Limit the total amount of checkpoints. " "Deletes the older checkpoints in the output_dir. Default is unlimited checkpoints" ) }, ) seed: int = field(default=None, metadata={"help": "Random seed that will be set at the beginning of training."}) data_seed: Optional[int] = field(default=None, metadata={"help": "Random seed to be used with data samplers."}) resume_from_checkpoint: Optional[str] = field( default=None, metadata={"help": "The path to a folder with a valid checkpoint for your model."}, ) auto_trans_ckpt: bool = field( default=None, metadata={"help": "Whether to transform checkpoint according to parallel config. "} ) resume_training: bool = field( default=None, metadata={"help": "Whether to load training context info, such as optimizer and epoch num. "} ) do_eval: bool = field( default=None, metadata={"help": "Whether to eval current model while Training. "} ) eval_step_interval: int = field( default=None, metadata={"help": "Num of step intervals between each eval, -1 means no step end eval. "} ) eval_epoch_interval: int = field( default=None, metadata={"help": "Num of epoch intervals between each eval, 1 means eval on every epoch end. "} ) device_num = int(os.getenv("RANK_SIZE", "1")) device_id = int(os.getenv("DEVICE_ID", "0")) rank_id = int(os.getenv("RANK_ID", "0")) def __post_init__(self): if self.output_dir is not None: self.output_dir = os.path.expanduser(self.output_dir) if self.lr_scheduler_type is not None: self.lr_scheduler_type = LRType(self.lr_scheduler_type).value if self.optim is not None: self.optim = OptimizerType(self.optim).value if self.save_strategy is not None: self.save_strategy = SaveIntervalStrategy(self.save_strategy).value @property def get_device_num(self): """get device num for training.""" return self.device_num @property def get_device_id(self): """get device id for training.""" return self.device_id @property def get_rank_id(self): """get rank id for training.""" return self.rank_id def convert_args_to_mindformers_config(self, task_config: MindFormerConfig = None): """convert training arguments to mindformer config type for adapting hugging-face.""" if task_config is None: task_config = MindFormerConfig() self._adapt_dataset_config(task_config) self._adapt_runner_config(task_config) self._adapt_lr_schedule_config(task_config) self._adapt_optimizer_config(task_config) self._adapt_save_checkpoint_config(task_config) task_config.output_dir = _check_training_args(task_config.output_dir, self.output_dir) task_config.seed = _check_training_args(task_config.seed, self.seed) task_config.data_seed = _check_training_args(task_config.data_seed, self.data_seed) task_config.profile = _check_training_args(task_config.profile, self.profile) task_config.only_save_strategy = _check_training_args(task_config.only_save_strategy, self.only_save_strategy) task_config.use_parallel = _check_training_args(task_config.use_parallel, self.use_parallel) task_config.load_checkpoint = _check_training_args( task_config.load_checkpoint, self.resume_from_checkpoint) task_config.auto_trans_ckpt = _check_training_args( task_config.auto_trans_ckpt, self.auto_trans_ckpt) task_config.resume_training = _check_training_args( task_config.resume_training, self.resume_training) task_config.do_eval = _check_training_args( task_config.do_eval, self.do_eval) task_config.eval_step_interval = _check_training_args( task_config.eval_step_interval, self.eval_step_interval) task_config.eval_epoch_interval = _check_training_args( task_config.eval_epoch_interval, self.eval_epoch_interval) def _adapt_dataset_config(self, task_config): """adapt dataset config.""" if _check_task_config(task_config.train_dataset): task_config.train_dataset.batch_size = _check_training_args( task_config.train_dataset.batch_size, self.per_device_train_batch_size) else: logger.warning( "This task does not support training at the moment; " "it does not have a default training data configuration," "so the per_device_train_batch_size setting will not take effect") if _check_task_config(task_config.eval_dataset): task_config.eval_dataset.batch_size = task_config.train_dataset.batch_size = _check_training_args( task_config.eval_dataset.batch_size, self.per_device_eval_batch_size) else: logger.warning( "This task does not support evaluate at the moment;" "it does not have a default evaluate data configuration," "so the per_device_eval_batch_size setting will not take effect") def _adapt_runner_config(self, task_config): """adapt runner config.""" if _check_task_config(task_config.runner_config): task_config.runner_config.epochs = _check_training_args( task_config.runner_config.epochs, self.num_train_epochs) task_config.runner_config.batch_size = _check_training_args( task_config.runner_config.batch_size, self.batch_size) task_config.runner_config.sink_size = _check_training_args( task_config.runner_config.sink_size, self.sink_size) task_config.runner_config.per_epoch_size = _check_training_args( task_config.runner_config.per_epoch_size, self.sink_size) task_config.runner_config.sink_mode = _check_training_args( task_config.runner_config.sink_mode, self.sink_mode) def _adapt_lr_schedule_config(self, task_config): """adapt lr schedule config.""" if _check_task_config(task_config.lr_schedule): if task_config.lr_schedule.type is not None: task_config.lr_schedule.type = _check_training_args( task_config.lr_schedule.type, self.lr_scheduler_type) if task_config.lr_schedule.learning_rate is not None: task_config.lr_schedule.learning_rate = _check_training_args( task_config.lr_schedule.learning_rate, self.learning_rate) if task_config.lr_schedule.warmup_steps is not None: task_config.lr_schedule.warmup_steps = _check_training_args( task_config.lr_schedule.warmup_steps, self.warmup_steps) def _adapt_optimizer_config(self, task_config): """adapt optimizer config.""" if _check_task_config(task_config.optimizer): if task_config.optimizer.type is not None: task_config.optimizer.type = _check_training_args( task_config.optimizer.type, self.optim) if task_config.optimizer.learning_rate is not None: task_config.optimizer.learning_rate = _check_training_args( task_config.optimizer.learning_rate, self.learning_rate) if task_config.optimizer.weight_decay is not None: task_config.optimizer.weight_decay = _check_training_args( task_config.optimizer.weight_decay, self.weight_decay) if task_config.optimizer.beta1 is not None: task_config.optimizer.beta1 = _check_training_args( task_config.optimizer.beta1, self.adam_beta1) if task_config.optimizer.beta2 is not None: task_config.optimizer.beta2 = _check_training_args( task_config.optimizer.beta2, self.adam_beta2) if task_config.optimizer.eps is not None: task_config.optimizer.eps = _check_training_args( task_config.optimizer.eps, self.adam_epsilon) def _adapt_save_checkpoint_config(self, task_config): """adapt save checkpoint config.""" if task_config.callbacks is not None and \ isinstance(task_config.callbacks, list) and \ self.save_strategy is not None: for i, callback in enumerate(task_config.callbacks): if isinstance(callback, dict) and callback['type'] == "CheckpointMointor": if self.save_strategy == 'no': task_config.callbacks.pop(i) continue if self.save_strategy == 'steps': task_config.callbacks[i]['save_checkpoint_steps'] = _check_training_args( task_config.callbacks[i]['save_checkpoint_steps'], self.save_steps) elif self.save_strategy == 'seconds': task_config.callbacks[i]['save_checkpoint_seconds'] = _check_training_args( task_config.callbacks[i]['save_checkpoint_seconds'], self.save_seconds) task_config.callbacks[i]['keep_checkpoint_max'] = self.save_total_limit \ if self.save_total_limit else 5 task_config.callbacks[i]['integrated_save'] = _check_training_args( task_config.callbacks[i]['integrated_save'], self.integrated_save) pengcheng.mind-test/mindformers/trainer/translation/000077500000000000000000000000001471400134000232405ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/translation/__init__.py000066400000000000000000000014331471400134000253520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Language Modeling Trainer.""" from .translation_finetune import TranslationTrainer __all__ = ['TranslationTrainer'] pengcheng.mind-test/mindformers/trainer/translation/__pycache__/000077500000000000000000000000001471400134000254505ustar00rootroot00000000000000pengcheng.mind-test/mindformers/trainer/translation/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004651471400134000316450ustar00rootroot00000000000000a ?cf@sdZddlmZdgZdS)z!Masked Language Modeling Trainer.)TranslationTrainerrN)__doc__Ztranslation_finetuner__all__rrS/home/ma-user/work/mindformers-baichuan/mindformers/trainer/translation/__init__.pys pengcheng.mind-test/mindformers/trainer/translation/__pycache__/translation_finetune.cpython-39.pyc000066400000000000000000000143511471400134000343400ustar00rootroot00000000000000a ?cf@sdZddlZddlmZmZmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZdd lmZmZmZd d lmZd d lmZd d lmZeejGdddeZ dS)zTranslation Modeling Trainer.N)OptionalListUnion)Callback)TrainOneStepCell OptimizerCell)GeneratorDataset) BaseDataset) BaseModel BaseTokenizer)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) BaseTrainer)ConfigArguments)TrainingArgumentsc seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZd d Zdeeee e e feeeeefeee e feed d d ZZS)TranslationTraineraTranslation Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers.trainer import TranslationTrainer >>> from mindformers import T5ForConditionalGeneration, TranslationTrainer >>> # follow the instruction in t5 section in the README.md and download wmt16 dataset. >>> # change the dataset_files path of configs/t5/wmt16_dataset.yaml >>> trans_trainer = TranslationTrainer(model_name='t5_small') >>> trans_trainer.train() >>> model = T5ForConditionalGeneration.from_pretrained('t5_small') >>> trans_trainer = TranslationTrainer(model_name="t5_small") >>> res = trans_trainer.predict(input_data="translate the English to Romanian: a good boy!", network=model) [{'translation_text': ['hello world']}] Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)N translation)superr__init__)selfr __class___/home/ma-user/work/mindformers-baichuan/mindformers/trainer/translation/translation_finetune.pyr3szTranslationTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for Translation Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )rrr#r r!r"N)training_process)rrrr r!r"r#kwargsrrrtrain6s$zTranslationTrainer.traincOs tddS)Nz/The Translation task does not support evaluate.)NotImplementedError)rargsr%rrrevaluatecszTranslationTrainer.evaluate)r input_datar tokenizerc Ks|dur|j}t|trjtj|rjt|d,}g}|D]}||q8|}Wdn1s`0Y|jf||d||d|S)a Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. It supports 1) a text string to be translated, 1) a file name where each line is a text to be translated and 3) a generator dataset. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. Nrr)rr*taskrr+) r* isinstancestrospathisfileopenappendpredict_process) rrr*rr+r%fpZinput_data_listlinerrrpredictgs   "zTranslationTrainer.predict)N)NNNNNN)NNNN)__name__ __module__ __qualname____doc__r/rrrdictrrrrr r r rrrrr&r)listr r8 __classcell__rrrrr s6 -r)!r<os.pathr0typingrrrZmindspore.trainr mindspore.nnrrrmindspore.datasetr mindformers.datasetr mindformers.modelsr r mindformers.tools.registerr rr base_trainerr config_argsr training_argsrregisterTRAINERrrrrrs       pengcheng.mind-test/mindformers/trainer/translation/translation_finetune.py000066400000000000000000000156451471400134000300600ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Translation Modeling Trainer.""" import os.path from typing import Optional, List, Union from mindspore.train import Callback from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.register import MindFormerRegister,\ MindFormerModuleType, MindFormerConfig from ..base_trainer import BaseTrainer from ..config_args import ConfigArguments from ..training_args import TrainingArguments @MindFormerRegister.register(MindFormerModuleType.TRAINER) class TranslationTrainer(BaseTrainer): r"""Translation Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers.trainer import TranslationTrainer >>> from mindformers import T5ForConditionalGeneration, TranslationTrainer >>> # follow the instruction in t5 section in the README.md and download wmt16 dataset. >>> # change the dataset_files path of configs/t5/wmt16_dataset.yaml >>> trans_trainer = TranslationTrainer(model_name='t5_small') >>> trans_trainer.train() >>> model = T5ForConditionalGeneration.from_pretrained('t5_small') >>> trans_trainer = TranslationTrainer(model_name="t5_small") >>> res = trans_trainer.predict(input_data="translate the English to Romanian: a good boy!", network=model) [{'translation_text': ['hello world']}] Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(TranslationTrainer, self).__init__("translation", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for Translation Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, *args, **kwargs): raise NotImplementedError( "The Translation task does not support evaluate.") def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list, GeneratorDataset]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. It supports 1) a text string to be translated, 1) a file name where each line is a text to be translated and 3) a generator dataset. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ if input_data is None: input_data = config.input_data if isinstance(input_data, str) and os.path.isfile(input_data): with open(input_data, 'r') as fp: input_data_list = [] for line in fp: input_data_list.append(line) input_data = input_data_list return self.predict_process(config=config, input_data=input_data, task='translation', network=network, tokenizer=tokenizer, **kwargs) pengcheng.mind-test/mindformers/trainer/utils.py000066400000000000000000000651111471400134000224200ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Trainer Utils.""" import os import time import random from glob import glob from enum import Enum import numpy as np import mindspore as ms from mindspore.communication.management import get_rank, get_group_size from mindspore import context, load_checkpoint, load_param_into_net from mindspore import set_seed as ms_set_seed from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerConfig from mindformers.tools.utils import check_in_modelarts, get_output_root_path from mindformers.tools.transform_ckpt import get_strategy if check_in_modelarts(): import moxing as mox ELAPSE_TIME = 7200 class BaseEnum(str, Enum): """ Base Enum for MindFormers. """ @classmethod def _missing_(cls, value): """Enum with more explicit error message for missing values.""" raise ValueError( f"{value} is not a valid {cls.__name__}, please select one of {list(cls._value2member_map_.keys())}" ) class SaveIntervalStrategy(BaseEnum): """ Stores the acceptable string identifiers for save checkpoint monitor. """ NO = "no" STEPS = "steps" SECONDS = "seconds" class LRType(BaseEnum): """ Stores the acceptable string identifiers for learning rate schedule. """ # will be support item for future. LINEAR = "linear" COSINE = "cosine" COSINE_WITH_RESTARTS = "cosine_with_restarts" POLYNOMIAL = "polynomial" CONSTANT_WITH_WARMUP = "constant_with_warmup" class OptimizerType(BaseEnum): """ Stores the acceptable string identifiers for optimizers. """ # supported item for test, will be delete in the future. ADAMWEIGHTDECAY = 'AdamWeightDecay' # will be support item for future. ADAMW = "adamw" ADAM = "adam" SGD = "sgd" ADAGRAD = "adagrad" ADAFACTOR = "adafactor" class WrapperType(BaseEnum): """ Stores the acceptable string identifiers for training wrapper. """ # will be support item for future. MFWRAPPER = 'mf_wrapper' TRAINONESTEP = 'wrapper' TRAINONESTEPWITHLOSSSCALE = 'loss_scale_wrapper' def set_seed(seed: int = 0): """ Helper function for reproducible behavior to set the seed in `random`, `numpy`, `MindSpore`. Args: seed (`int`): The seed to set. """ random.seed(seed) np.random.seed(seed) ms_set_seed(seed) def check_keywords_in_name(name, keywords=()): """ Check keywords in name. """ isin = False for keyword in keywords: if keyword in name: isin = True return isin def check_runner_config(config, dataset): """ Check runner config. """ data_size = dataset.get_dataset_size() new_epochs = config.runner_config.epochs config.runner_config.origin_epochs = new_epochs if config.runner_config.initial_epoch is None: config.runner_config.initial_epoch = 0 if config.runner_config.sink_mode: if config.runner_config.sink_size != -1: if config.runner_config.sink_size <= 0: raise ValueError("per epoch size must be more than 0 or equal to -1, " f"but get {config.runner_config.sink_size}") if data_size < config.runner_config.sink_size: logger.warning("The data size %s (get from dataset.get_dataset_size()) is smaller " "than the sink_size %s (get from config.runner_config.sink_size), " "you should set the config.runner_config.sink_size to %s", data_size, config.runner_config.sink_size, data_size) config.runner_config.epochs = int((data_size / config.runner_config.sink_size) * new_epochs) else: config.runner_config.sink_size = data_size else: logger.warning("Sink mode is False, per epoch size is invalid, it will reset -1.") config.runner_config.sink_size = -1 config.data_size = data_size logger.info("Will be Training epochs:%d, sink_size:%d", config.runner_config.origin_epochs, config.runner_config.sink_size) logger.info("Create training dataset finish, dataset size:%d", data_size) def check_train_data_loader_type(new_config, old_config): """Check train data loader config type.""" if new_config.train_dataset is None: return None if new_config.train_dataset.get('data_loader') is None: return None train_data_loader_type = new_config.train_dataset.get('data_loader').get('type') if old_config.train_dataset is not None and train_data_loader_type is not None: default_train_data_loader_type = old_config.train_dataset.data_loader.type if train_data_loader_type != default_train_data_loader_type: logger.warning("train dataset's data_loader type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", train_data_loader_type) old_config.train_dataset.data_loader = {} return None def check_eval_data_loader_type(new_config, old_config): """Check eval data loader config type.""" if new_config.eval_dataset is None: return None if new_config.eval_dataset.get('data_loader') is None: return None eval_data_loader_type = new_config.eval_dataset.get('data_loader').get('type') if old_config.eval_dataset is not None and eval_data_loader_type is not None: default_eval_data_loader_type = old_config.eval_dataset.data_loader.type if eval_data_loader_type != default_eval_data_loader_type: logger.warning("eval dataset's data_loader type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", eval_data_loader_type) old_config.eval_dataset.data_loader = {} return None def check_optimizer_and_lr_type(new_config, old_config): """Check optimizer and lr schedule config type.""" optimizer_type = new_config.optimizer.get('type') if old_config.optimizer is not None and optimizer_type is not None: default_optimizer_type = old_config.optimizer.type if optimizer_type != default_optimizer_type: logger.warning( "optimizer type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually except (params).", optimizer_type) old_config.optimizer = {} if hasattr(new_config.optimizer, 'learning_rate'): lr_type = new_config.optimizer.learning_rate.get('type') if old_config.lr_schedule is not None and lr_type is not None: default_lr_type = old_config.lr_schedule.type if lr_type != default_lr_type: logger.warning( "lr schedule type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", lr_type) old_config.lr_schedule = None def check_wrapper_config(new_config, old_config): """Check wrapper config.""" wrapper_type = new_config.runner_wrapper.get('type') if old_config.runner_wrapper is not None and wrapper_type is not None: default_wrapper_type = old_config.runner_wrapper.type if wrapper_type != default_wrapper_type: logger.warning( "wrapper type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", wrapper_type) old_config.runner_wrapper = {} def config2dict(config): """MindFormerConfig Type Convert to Dict.""" if not isinstance(config, (dict, MindFormerConfig)): return config new_dict = {} for key, value in config.items(): if isinstance(value, MindFormerConfig): value = config2dict(value) new_dict.setdefault(key, value) return new_dict def load_distributed_checkpoint(config, specify_prefix=None): """Load Checkpoint in Parallel Mode.""" checkpoint_dir = config.load_checkpoint if os.path.isdir(checkpoint_dir): logger.info( "When distributed loads are sliced weights," "load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*}," "The directory structure is as follows: **checkpoint_root_dir/checkpoint/rank_{0-*}/**.ckpt") distribute_checkpoint_dir = os.path.join( checkpoint_dir, "rank_{}".format(int(os.getenv("RANK_ID", "0")))) distribute_checkpoint_path = get_last_checkpoint(distribute_checkpoint_dir) elif os.path.isfile(checkpoint_dir): logger.info("Your load_checkpoint is file, it will be load in network.") distribute_checkpoint_path = checkpoint_dir else: raise FileNotFoundError(f"{checkpoint_dir} is not found.") checkpoint_dict = load_checkpoint(distribute_checkpoint_path, specify_prefix=specify_prefix) logger.info("Distribute load is success.") return checkpoint_dict def load_resume_context_from_checkpoint(config): """resume training, load training info from checkpoint to config""" if not os.path.realpath(config.load_checkpoint) or \ not os.path.exists(config.load_checkpoint): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") if os.path.isdir(config.load_checkpoint): resume_dict = load_distributed_checkpoint(config, ["loss_scale", "epoch_num"]) if not config.runner_config.sink_mode: raise ValueError("When distributed loads are sliced weights, sink_mode must be set True.") else: resume_dict = load_checkpoint(config.load_checkpoint, specify_prefix=["loss_scale", "epoch_num"]) if "epoch_num" in resume_dict: config.runner_config.initial_epoch = int(resume_dict["epoch_num"]) else: config.runner_config.initial_epoch = 0 for callback in config.callbacks: if "type" in callback and callback["type"] == "CheckpointMointor": if config.runner_wrapper.scale_sense is not None and "loss_scale" in resume_dict: config.runner_wrapper.scale_sense.loss_scale_value = resume_dict["loss_scale"] break def transform_and_load_checkpoint(config, model, network, dataset, optimizer=None, do_eval=False, do_predict=False): """ load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt """ if not config.only_save_strategy and (not os.path.realpath(config.load_checkpoint) or not os.path.exists(config.load_checkpoint)): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: # 1. build net if parallel mode is auto_parallel build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) if config.auto_trans_ckpt: # 2. get strategy dst_ckpt_strategy = get_dst_strategy(config) # 3. transform checkpoint if needed transform_ckpt(config, dst_ckpt_strategy=dst_ckpt_strategy) # 4. load ckpt load_ckpt(config, network, optimizer=optimizer) def load_merged_checkpoint(config, model, network, dataset, optimizer=None, do_eval=False, do_predict=False): """ load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt """ if not os.path.realpath(config.load_checkpoint or not os.path.exists(config.load_checkpoint)): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") # 1. load merged ckpt load_ckpt(config, network, optimizer=optimizer) if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: # 2. build net if parallel mode is auto_parallel build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) def build_model(config, model, dataset, do_eval=False, do_predict=False): """build model, generate strategy file""" if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: if not config.runner_config.sink_mode: raise ValueError("When distributed loads are sliced weights, sink_mode must be set True.") if do_eval: model.infer_predict_layout(*next(dataset.create_tuple_iterator())) elif do_predict: model.infer_predict_layout(dataset) else: if config.runner_config.epochs > 1 and config.runner_config.sink_size == 1: raise ValueError(f"When distributed loads are sliced weights, it does not support" f"epochs = {config.runner_config.epochs} > 1 and " f"sink_size = {config.runner_config.sink_size} = 1," f"sink_size must be more than 1") logger.info(".........Building model.........") model.build(train_dataset=dataset, epoch=config.runner_config.epochs, sink_size=config.runner_config.sink_size) if config.only_save_strategy: raise SystemExit("Only_save_strategy is True, model.compile() finished, system exit! ") elif config.only_save_strategy: raise SystemExit("only_save_strategy is True, " "but stand_alone and data_parallel mode do not have strategy file, system exit! ") def get_dst_strategy(config): """get strategy""" rank_id = int(os.getenv('RANK_ID', '0')) world_size = int(os.getenv('RANK_SIZE', '1')) dst_strategy_path = None if world_size == 1: return dst_strategy_path if check_in_modelarts(): # local send all strategy file to obs obs_save_dir = os.path.join(config.remote_save_url, "strategy") mox.file.make_dirs(obs_save_dir) local_strategy_path = config.parallel.strategy_ckpt_save_file obs_strategy_path = os.path.join(obs_save_dir, os.path.basename(local_strategy_path)) mox.file.copy(local_strategy_path, obs_strategy_path) # obs send all strategy to each node logger.info(".........Collecting strategy.........") local_strategy_dir = os.path.join(get_output_root_path(), "strategy") if rank_id % 8 == 0: wait_collect_all_strategy(local_strategy_dir, world_size, obs_save_dir) else: wait_collect_all_strategy(local_strategy_dir, world_size) logger.info(".........All strtegy as follow.........") local_strategy_paths = glob(os.path.join(local_strategy_dir, "*_rank_*.ckpt")) local_strategy_paths.sort() for local_strategy_path in local_strategy_paths: logger.info("strategy: %s", local_strategy_path) logger.info(".........Collecting %d strategy.........", len(local_strategy_paths)) # merge strategy if pipeline_stage > 1 if config.parallel_config.pipeline_stage > 1: if rank_id % 8 == 0: logger.info(".........Merging strategy.........") merged_strategy_path = get_strategy(local_strategy_dir) merged_strategy_name = os.path.basename(merged_strategy_path) obs_merged_strategy_path = os.path.join(obs_save_dir, merged_strategy_name) mox.file.copy(merged_strategy_path, obs_merged_strategy_path) logger.info("Save %s to %s", merged_strategy_path, obs_merged_strategy_path) logger.info(".........Merging succeed.........") dst_strategy_path = merged_strategy_path else: dst_strategy_path = None else: dst_strategy_path = local_strategy_path else: logger.info(".........Collecting strategy.........") local_strategy_dir = os.path.join(get_output_root_path(), "strategy") if world_size <= 8: wait_collect_all_strategy(local_strategy_dir, world_size) logger.info(".........All strtegy as follow.........") local_strategy_paths = glob(os.path.join(local_strategy_dir, "*_rank_*.ckpt")) local_strategy_paths.sort() for local_strategy_path in local_strategy_paths: logger.info("strategy: %s", local_strategy_path) logger.info(".........Collecting %d strategy.........", len(local_strategy_paths)) # merge strategy if pipeline_stage > 1 if config.parallel_config.pipeline_stage > 1: if rank_id % 8 == 0: logger.info(".........Merging strategy.........") merged_strategy_path = get_strategy(local_strategy_dir) logger.info(".........Merging succeed.........") dst_strategy_path = merged_strategy_path else: dst_strategy_path = None else: dst_strategy_path = local_strategy_paths[0] else: logger.warning("Can't collecting all strategy, device num > 8!") config.auto_trans_ckpt = False dst_strategy_path = None return dst_strategy_path def transform_ckpt(config, dst_ckpt_strategy=None): """auto transform ckpt""" rank_id = get_rank() if config.use_parallel else 0 world_size = get_group_size() if config.use_parallel else 1 transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") os.makedirs(transformed_ckpt_dir, exist_ok=True) if rank_id % 8 == 0: logger.info(".........Transforming ckpt.........") src_ckpt_strategy = get_strategy(config.src_strategy_path_or_dir) logger.info("Src ckpt strategy: %s", src_ckpt_strategy) logger.info("Src ckpt: %s", config.load_checkpoint) logger.info("Dst ckpt strategy: %s", dst_ckpt_strategy) logger.info("Dst ckpt: %s", transformed_ckpt_dir) ms.transform_checkpoints(config.load_checkpoint, transformed_ckpt_dir, 'checkpoint_', src_ckpt_strategy, dst_ckpt_strategy) logger.info(".........Transform succeed!.........") transform_succeed_txt = os.path.join(transformed_ckpt_dir, f'transform_succeed_rank_{rank_id}.txt') f = open(transform_succeed_txt, 'w') f.close() if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") mox.file.make_dirs(transformed_ckpt_dir_obs) transform_succeed_txt_obs = os.path.join(transformed_ckpt_dir_obs, f'transform_succeed_rank_{rank_id}.txt') mox.file.copy(transform_succeed_txt, transform_succeed_txt_obs) wait_transform(config, rank_id, world_size) if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") rank_transformed_ckpt_dir_obs = os.path.join(transformed_ckpt_dir_obs, f'rank_{rank_id}') rank_transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint", f'rank_{rank_id}') mox.file.copy_parallel(rank_transformed_ckpt_dir, rank_transformed_ckpt_dir_obs) print(f"Rank {rank_id}: Save {rank_transformed_ckpt_dir} to {rank_transformed_ckpt_dir_obs}") config.load_checkpoint = transformed_ckpt_dir def wait_transform(config, rank_id, world_size): """wait all node transform over""" print(f'Rank {rank_id}: Waiting transforming ckpt......') time_out = False if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") transform_succeed_txts_obs = [os.path.join(transformed_ckpt_dir_obs, f'transform_succeed_rank_{int(i/8)*8}.txt') for i in range(0, world_size, 8)] total_num = len(transform_succeed_txts_obs) all_node_transform_succeed = False start_time = time.time() while not all_node_transform_succeed: node_transform_succeed_num = 0 for transform_succeed_txt in transform_succeed_txts_obs: if mox.file.exists(transform_succeed_txt): node_transform_succeed_num += 1 else: break if node_transform_succeed_num == total_num: all_node_transform_succeed = True else: elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: logger.warning("Automatic weight transform is timed out!") time_out = True break time.sleep(5) else: transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") transform_succeed_txt = os.path.join(transformed_ckpt_dir, f'transform_succeed_rank_0.txt') start_time = time.time() while not os.path.exists(transform_succeed_txt): elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: time_out = True break time.sleep(5) print(f'Rank {rank_id}: Transform succeed!') if time_out: return False return True def wait_collect_all_strategy(strategy_dir, total_num, obs_strategy_dir=None): """wait all strategy collect over""" start_time = time.time() time_out = False while True: if obs_strategy_dir: obs_strategy_paths = mox.file.glob(os.path.join(obs_strategy_dir, "*.ckpt")) if len(obs_strategy_paths) < total_num: time.sleep(5) continue for obs_strategy_path in obs_strategy_paths: local_strategy_path = os.path.join(strategy_dir, os.path.basename(obs_strategy_path)) mox.file.copy(obs_strategy_path, local_strategy_path) local_strategy_paths = glob(os.path.join(strategy_dir, "*_rank_*.ckpt")) if len(local_strategy_paths) == total_num: break else: elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: logger.warning("Collection of all strategy timed out!") time_out = True break time.sleep(5) if time_out: return False return True def load_ckpt(config, network, optimizer=None): """load checkpoint""" # load checkpoint params into dict logger.info(".............Start load checkpoint from checkpoint..................") if os.path.isdir(config.load_checkpoint): checkpoint_dict = load_distributed_checkpoint(config) else: checkpoint_dict = load_checkpoint(config.load_checkpoint) # taoht if config.reset_steps: logger.info(".............Reset checkpoint's global_step=0..................") from mindspore.common import Parameter from mindspore.common.tensor import Tensor import mindspore.common.dtype as mstype checkpoint_dict["global_step"] = Parameter(Tensor([0], dtype=mstype.int32), name="global_step") checkpoint_dict["current_iterator_step"] = Parameter(Tensor(0, dtype=mstype.int32), name="current_iterator_step") checkpoint_dict["step_num"] = Parameter(Tensor(0,dtype=mstype.int32), name="step_num") checkpoint_dict["epoch_num"] = Parameter(Tensor(0, dtype=mstype.int32), name="epoch_num") checkpoint_dict["last_overflow_iterator_step"] = Parameter(Tensor(0, dtype=mstype.int32), name="last_overflow_iterator_step") if config.unload_adam: logger.info(".............unload checkpoint's adam and accu_grads parameters..................") parame_list = {} for k, v in checkpoint_dict.items(): if 'adam' in k or 'accu_grads' in k: # del checkpoint_dict[k] continue else: parame_list[k]=v checkpoint_dict = parame_list # load params into net not_load_network_params = load_param_into_net(network, checkpoint_dict) logger.info("Network parameters are not loaded: %s", str(not_load_network_params)) if optimizer: not_load_optim_params = load_param_into_net(optimizer, checkpoint_dict) logger.info("Optimizer parameters are not loaded: %s", str(not_load_optim_params)) def get_last_checkpoint(checkpoint_dir): """get last checkpoint for resuming or finetune.""" if not os.path.isdir(checkpoint_dir): raise NotADirectoryError( f"{checkpoint_dir} is not a real directory," "When distributed loads are sliced weights," "load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*}," "The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/checkpoint/**.ckpt") output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] if not output_checkpoint_path: return None output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) return os.path.join(checkpoint_dir, output_checkpoint_path[-1]) pengcheng.mind-test/mindformers/version_control.py000066400000000000000000000072651471400134000230470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindSpore Version Control""" import os import mindspore as ms from mindspore import nn import mindspore.ops.operations as P import mindspore.ops.functional as F from .tools.utils import is_version_ge from .tools.logger import logger def get_cell_reuse(): """Cell reuse decorator.""" def decorator(func): if os.getenv("MS_DEV_CELL_REUSE", "0") == "0" or not is_version_ge(ms.__version__, "2.1.0"): return func logger.info("Enable cell use mode at %s.", func.__class__.__name__) from mindspore._extends import cell_attr_register return cell_attr_register()(func) return decorator def get_dropout(dropout_prob): if is_version_ge(ms.__version__, '1.11.0'): dropout = nn.Dropout(p=dropout_prob) else: dropout = nn.Dropout(keep_prob=1 - dropout_prob) return dropout def get_tril(): if is_version_ge(ms.__version__, '1.11.0'): tril = P.Tril() else: tril = nn.Tril() return tril def get_norm(): """return ops.norm""" # pylint: disable=C0103 # pylint: disable=W0622 def tensor_norm1(A, ord=None, dim=None, keepdim=False, dtype=None): return F.norm(A, ord=ord, dim=dim, keepdim=keepdim, dtype=dtype) # pylint: disable=C0103 # pylint: disable=W0622 def tensor_norm2(A, ord=2, dim=None, keepdim=False, dtype=None): if dtype is not None: logger.warning("The 'dtype' is not available when mindspore version < '1.11.0'") if not isinstance(ord, int): raise TypeError("The type of 'ord' should be int when mindspore version < '1.11.0'") return F.norm(A, dim, p=ord, keep_dims=keepdim) if is_version_ge(ms.__version__, '1.11.0'): return tensor_norm1 return tensor_norm2 def get_dataset_map(dataset, operations, input_columns=None, output_columns=None, num_parallel_workers=None, **kwargs): if is_version_ge(ms.__version__, "1.11.0"): return dataset.map(operations, input_columns=input_columns, output_columns=output_columns, num_parallel_workers=num_parallel_workers, **kwargs) return dataset.map(operations, input_columns=input_columns, output_columns=output_columns, column_order=output_columns, num_parallel_workers=num_parallel_workers, python_multiprocessing=kwargs.get('python_multiprocessing', False), max_rowsize=kwargs.get('max_rowsize', 16), cache=kwargs.get('cache', None), callbacks=kwargs.get('callbacks', None), offload=kwargs.get('offload', None)) def get_identity(): if is_version_ge(ms.__version__, "1.11.0"): return nn.Identity() return F.identity def fix_optim_global_step_sig(): # when the version of mindspore bigger than 2.2.0, it should update global step explicitly. return is_version_ge(ms.__version__, "2.2.0") pengcheng.mind-test/mindformers/wrapper/000077500000000000000000000000001471400134000207165ustar00rootroot00000000000000pengcheng.mind-test/mindformers/wrapper/.ipynb_checkpoints/000077500000000000000000000000001471400134000245075ustar00rootroot00000000000000pengcheng.mind-test/mindformers/wrapper/.ipynb_checkpoints/build_wrapper-checkpoint.py000066400000000000000000000100251471400134000320430ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Wrapper.""" import inspect import mindspore as ms from mindspore import nn, Tensor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .wrapper import MFTrainOneStepCell, MFPipelineWithLossScaleCell WRAPPERS_MINDFORMERS_DEFINED = ["MFTrainOneStepCell", "MFPipelineWithLossScaleCell"] def build_wrapper(config: dict = None, default_args: dict = None, module_type: str = 'wrapper', class_name: str = None, **kwargs): r"""Build Wrapper For MindFormer. Instantiate the wrapper from MindFormerRegister's registry. Args: config (dict): The task wrapper's config. Default: None. default_args (dict): The default argument of wrapper API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'wrapper'. class_name (str): The class name of wrapper API. Default: None. Return: The function instance of wrapper API. Examples: >>> from mindformers import build_wrapper >>> wrapper_config = {'type': 'TrainOneStepCell', 'sens': 1024} >>> # 1) use config dict to build wrapper >>> wrapper_from_config = build_wrapper(wrapper_config) >>> # 2) use class name to build wrapper >>> wrapper_class_name = build_wrapper(class_name='TrainOneStepCell', sens=1024) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.type not in WRAPPERS_MINDFORMERS_DEFINED: if default_args and "parallel_config" in default_args: del default_args["parallel_config"] if config.scale_sense is not None: if isinstance(config.scale_sense, dict) and config.scale_sense.type is not None: config.scale_sense = build_wrapper(config.scale_sense) elif isinstance(config.scale_sense, int): config.scale_sense = Tensor(config.scale_sense, ms.float32) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.WRAPPER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_wrap(): """ register MindSpore builtin wrapper class. """ for module_name in dir(nn.wrap): if module_name.startswith('__'): continue wrap = getattr(nn.wrap, module_name) if inspect.isclass(wrap): MindFormerRegister.register_cls( wrap, MindFormerModuleType.WRAPPER) def register_mf_wrapper(): """ register MindFormers builtin wrapper class. """ # Support built-in model wrapper of MindFormers. MindFormerRegister.register_cls( nn.wrap.TrainOneStepCell, module_type=MindFormerModuleType.WRAPPER, alias="wrapper") MindFormerRegister.register_cls( nn.wrap.TrainOneStepWithLossScaleCell, module_type=MindFormerModuleType.WRAPPER, alias="loss_scale_wrapper") MindFormerRegister.register_cls( MFTrainOneStepCell, module_type=MindFormerModuleType.WRAPPER, alias="mf_wrapper") MindFormerRegister.register_cls( MFPipelineWithLossScaleCell, module_type=MindFormerModuleType.WRAPPER, alias="pipeline_wrapper") register_ms_wrap() register_mf_wrapper() pengcheng.mind-test/mindformers/wrapper/.ipynb_checkpoints/wrapper-checkpoint.py000066400000000000000000000353051471400134000306740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Self-Define Wrapper.""" from copy import deepcopy from mindspore.common.tensor import Tensor from mindspore.common import RowTensor from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore import nn, Parameter, ParallelMode from mindspore.parallel._utils import _get_enable_parallel_optimizer import mindspore.common.dtype as mstype from mindformers.core.clip_grad import ClipGradNorm from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.version_control import get_identity __all__ = ['MFTrainOneStepCell', 'MFPipelineWithLossScaleCell'] _grad_scale = C.MultitypeFuncGraph("grad_scale") reciprocal = P.Reciprocal() @_grad_scale.register("Tensor", "Tensor") def tensor_grad_scale(scale, grad): return F.cast(grad, mstype.float32) * reciprocal(scale) @_grad_scale.register("Tensor", "RowTensor") def tensor_grad_scale_row_tensor(scale, grad): return RowTensor(grad.indices, grad.values * F.cast(reciprocal(scale), F.dtype(grad.values)), grad.dense_shape) @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class DynamicLossScaleUpdateWithMinimalCell(nn.Cell): def __init__(self, loss_scale_value, scale_factor, scale_window, minimum_loss_scale): super(DynamicLossScaleUpdateWithMinimalCell, self).__init__() self.scale_window = Tensor(scale_window, dtype=mstype.int32) self.scale_factor = Tensor(scale_factor, dtype=mstype.float32) self.loss_scale_value = loss_scale_value self.cur_iter = Parameter(Tensor(1, dtype=mstype.int32), name="current_iterator_step") self.last_overflow_iter = Parameter(Tensor(0, dtype=mstype.int32), name="last_overflow_iterator_step") self.select = P.Select() self.max = P.Maximum() self.minimum_loss_scale = Tensor(minimum_loss_scale, dtype=mstype.float32) self.reciprocal = P.Reciprocal() self.less_equal = P.LessEqual() self.logic_and = P.LogicalAnd() self.logic_not = P.LogicalNot() self.logic_or = P.LogicalOr() self.const_true = Tensor(True, dtype=mstype.bool_) print("build dynamic loss scale with minimal value") def get_loss_scale(self): """ Get Loss Scale value. Returns: float, the loss scale value. Examples: >>> from mindspore import nn >>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=212, scale_factor=2, scale_window=1000) >>> output = manager.get_loss_scale() >>> print(output) 212 """ return self.loss_scale_value def construct(self, loss_scale, overflow): overflow_cond = overflow loss_scale_on_overflow = self.select(overflow_cond, self.max(loss_scale * self.reciprocal(self.scale_factor), self.minimum_loss_scale), loss_scale) should_inc = self.less_equal(self.scale_window, self.cur_iter - self.last_overflow_iter) last_iter_cond = self.logic_or(overflow_cond, should_inc) last_overflow_iter = self.select(last_iter_cond, self.cur_iter, self.last_overflow_iter) last_iter = F.assign(self.last_overflow_iter, last_overflow_iter) update_scale_cond = self.logic_and(should_inc, self.logic_not(overflow_cond)) scale_mul_res = loss_scale_on_overflow * self.scale_factor scaled_loss_scale = self.select(update_scale_cond, scale_mul_res, loss_scale_on_overflow) F.assign(loss_scale, scaled_loss_scale) inc_cur_iter = self.cur_iter + 1 inc_cur_iter = F.depend(inc_cur_iter, last_iter) F.assign(self.cur_iter, inc_cur_iter) return overflow @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class MFTrainOneStepCell(nn.TrainOneStepWithLossScaleCell): r"""TrainOneStep For MindFormer. Network training with loss scaling, grad clip, gradient accumulation, exponential moving average and so on. This is a training step with loss scaling. It takes a network, an optimizer and a scale update Cell(or a Tensor) as args. The loss scale value can be updated in both host side or device side. If you want to update it on host side, using a value of Tensor type as `scale_sense`, otherwise, using a Cell instance for updating loss scale as `scale_sense`. Args: network (Cell): The training network. The network only supports single output. optimizer (Cell): Optimizer for updating the network parameters. use_clip_grad (bool): Whether to use the gradient clipping function. Default: False. max_grad_norm (float): Maximum gradient value. Default: 1.0. scale_sense (Union[Tensor, Cell]): If this value is a Cell, it will be called by `MFTrainOneStepCell` to update loss scale. If this value is a Tensor, the loss scale can be modified by `set_sense_scale`, the shape should be :math:`()` or :math:`(1,)`. Inputs: - **(*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). """ def __init__(self, network, optimizer, use_clip_grad=False, max_grad_norm=0.5, scale_sense=1.0, **kwargs): super(MFTrainOneStepCell, self).__init__(network, optimizer, scale_sense) self.use_clip_grad = use_clip_grad self.clip_grad_norm = ClipGradNorm(max_norm=max_grad_norm) self.parallel_config = kwargs.pop("parallel_config", None) self.learning_rate = deepcopy(self.optimizer.learning_rate) def construct(self, *inputs): """forward and backward.""" weights = self.weights loss = self.network(*inputs) scaling_sens = self.scale_sense status, scaling_sens = self.start_overflow_check(loss, scaling_sens) scaling_sens_filled = C.ones_like(loss) * F.cast(scaling_sens, F.dtype(loss)) grads = self.grad(self.network, weights)(*inputs, scaling_sens_filled) grads = self.hyper_map(F.partial(_grad_scale, scaling_sens), grads) # apply grad reducer on grads grads = self.grad_reducer(grads) learning_rate = self.learning_rate if self.optimizer.dynamic_lr: if self.optimizer.is_group_lr: learning_rate = self.learning_rate[-1](self.optimizer.global_step).reshape(()) else: learning_rate = self.learning_rate(self.optimizer.global_step).reshape(()) # get the overflow buffer cond = self.get_overflow_status(status, grads) overflow = self.process_loss_scale(cond) # if there is no overflow, do optimize if not overflow: if self.use_clip_grad: grads, _ = self.clip_grad_norm(grads) loss = F.depend(loss, self.optimizer(grads)) return loss, overflow, scaling_sens, learning_rate grad_scale = C.MultitypeFuncGraph("grad_scale") shard_grad_scale = C.MultitypeFuncGraph("shard_grad_scale") @grad_scale.register("Tensor", "Tensor", "Tensor") def tensor_grad_scale_pipeline(scale, grad, accu_grad): accu_grad = F.depend(accu_grad, grad) new_grad = accu_grad * reciprocal(scale) accu_grad = F.depend(accu_grad, new_grad) zeros = F.tensor_mul(accu_grad, 0.0) new_grad = F.depend(new_grad, F.assign(accu_grad, zeros)) return new_grad @shard_grad_scale.register("Tensor", "Tensor", "Tensor") def tensor_shard_grad_scale_pipeline(scale, grad, accu_grad): new_grad = grad * reciprocal(scale) accu_grad = F.depend(accu_grad, new_grad) new_grad = F.depend(new_grad, F.assign(accu_grad, F.zeros_like(accu_grad))) return new_grad @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class MFPipelineWithLossScaleCell(nn.TrainOneStepCell): r""" Append an train one step cell with loss scale of pipeline parallel for MindFormers. Args: network (Cell): The training network. Note that loss function should have been added. optimizer (Optimizer): Optimizer for updating the weights. use_clip_grad (bool): Whether to use gradient clipping. Default: True. max_grad_norm (float): Maximum gradient constraint value. Default: 1.0. scale_sense (Cell): Cell to do the loss scale. Default: 1.0. micro_batch_num (int): Micro batch number of pipeline parallel. Default: 1. Inputs: - **(\*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). """ def __init__(self, network, optimizer, use_clip_grad=True, max_grad_norm=0.5, scale_sense=1.0, micro_batch_num=1, **kwargs): super(MFPipelineWithLossScaleCell, self).__init__(network, optimizer, sens=None) self.network = network self.network.add_flags(defer_inline=True) self.weights = optimizer.parameters self.accu_grads = self.weights.clone(prefix="accu_grads", init="zeros") self.optimizer = optimizer self.grad = C.GradOperation(get_by_list=True, sens_param=True) self.grad_reducer = get_identity() self.degree = 1 self.cast = P.Cast() self.alloc_status = P.NPUAllocFloatStatus() self.get_status = P.NPUGetFloatStatus() self.clear_before_grad = P.NPUClearFloatStatus() self.reduce_sum = P.ReduceSum(keep_dims=False) if self.parallel_mode not in [ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL]: raise ValueError(f"ParallelMode must be one of " f"[ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL], but found " f"{self.parallel_mode}.") self.allreduce = P.AllReduce() self.base = Tensor(1, mstype.float32) self.less_equal = P.LessEqual() self.hyper_map = C.HyperMap() self.reshape = P.Reshape() self.loss_scaling_manager = None if isinstance(scale_sense, nn.Cell): self.loss_scaling_manager = scale_sense self.scale_sense = Parameter(Tensor(scale_sense.get_loss_scale(), dtype=mstype.float32), name="scale_sense") elif isinstance(scale_sense, Tensor): if scale_sense.shape == (1,) or scale_sense.shape == (): self.scale_sense = Parameter(scale_sense, name='scale_sense') else: raise ValueError("The shape of 'scale_sense' must be (1,) or (), but got {}" .format(scale_sense.shape)) else: raise TypeError("The 'scale_sense' must be Cell or Tensor, but got {}".format(type(scale_sense))) self.opt_shard = _get_enable_parallel_optimizer() self.use_clip_grad = use_clip_grad self.clip_grad_norm = ClipGradNorm(max_norm=max_grad_norm) self.micro_size = micro_batch_num self.parallel_config = kwargs.pop("parallel_config", None) self.learning_rate = deepcopy(self.optimizer.learning_rate) @C.add_flags(has_effect=True) def construct(self, *inputs): """The construct processes of pipeline wrapper cell.""" loss = self.network(*inputs) scaling_sens = self.scale_sense scaling_sens_filled = C.ones_like(loss) * F.cast(scaling_sens, F.dtype(loss)) init = self.alloc_status() status_clear = self.clear_before_grad(init) scaling_sens_filled = F.depend(scaling_sens_filled, status_clear) grads = self.grad(self.network, self.weights)(*inputs, self.cast(scaling_sens_filled / self.micro_size, mstype.float32)) init = F.depend(init, grads) get_status = self.get_status(init) init = F.depend(init, get_status) flag_sum = self.reduce_sum(init, (0,)) loss = F.depend(loss, status_clear) if self.opt_shard: grads = self.grad_reducer(grads) grads = self.hyper_map(F.partial(shard_grad_scale, scaling_sens * self.degree), grads, self.accu_grads) else: accu_grads = self.grad_reducer(self.accu_grads) grads = self.hyper_map(F.partial(grad_scale, scaling_sens * self.degree), grads, accu_grads) if self.use_clip_grad: grads, _ = self.clip_grad_norm(grads) learning_rate = self.learning_rate if self.optimizer.dynamic_lr: if self.optimizer.is_group_lr: learning_rate = self.learning_rate[-1](self.optimizer.global_step).reshape(()) else: learning_rate = self.learning_rate(self.optimizer.global_step).reshape(()) # sum overflow flag over devices flag_reduce = self.allreduce(flag_sum) cond = self.less_equal(self.base, flag_reduce) overflow = cond if self.loss_scaling_manager is not None: overflow = self.loss_scaling_manager(self.scale_sense, cond) if not overflow: loss = F.depend(loss, self.optimizer(grads)) return loss, overflow, scaling_sens.value(), learning_rate pengcheng.mind-test/mindformers/wrapper/__init__.py000066400000000000000000000014501471400134000230270ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Wrapper API.""" from .build_wrapper import build_wrapper from .wrapper import * __all__ = [] __all__.extend(wrapper.__all__) pengcheng.mind-test/mindformers/wrapper/__pycache__/000077500000000000000000000000001471400134000231265ustar00rootroot00000000000000pengcheng.mind-test/mindformers/wrapper/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004541471400134000273210ustar00rootroot00000000000000a >cf(@s,dZddlmZddlTgZeejdS)zMindFormers Wrapper API.) build_wrapper)*N)__doc__rwrapper__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/wrapper/__init__.pys pengcheng.mind-test/mindformers/wrapper/__pycache__/build_wrapper.cpython-39.pyc000066400000000000000000000053701471400134000304230ustar00rootroot00000000000000a >cf@sdZddlZddlZddlmZmZddlmZmZm Z ddl m Z m Z ddgZ deeeed d d Zd dZddZeedS)zBuild Wrapper.N)nnTensor)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)MFTrainOneStepCellMFPipelineWithLossScaleCellrr wrapper)config default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|jtvrZ|rZd|vrZ|d=|jdurt|jtr|jjdurt|j|_nt|jtrt|jt j |_t j |t j|dSt j||fi|S)a_Build Wrapper For MindFormer. Instantiate the wrapper from MindFormerRegister's registry. Args: config (dict): The task wrapper's config. Default: None. default_args (dict): The default argument of wrapper API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'wrapper'. class_name (str): The class name of wrapper API. Default: None. Return: The function instance of wrapper API. Examples: >>> from mindformers import build_wrapper >>> wrapper_config = {'type': 'TrainOneStepCell', 'sens': 1024} >>> # 1) use config dict to build wrapper >>> wrapper_from_config = build_wrapper(wrapper_config) >>> # 2) use class name to build wrapper >>> wrapper_class_name = build_wrapper(class_name='TrainOneStepCell', sens=1024) Nparallel_config)r ) isinstancedictrtypeWRAPPERS_MINDFORMERS_DEFINED scale_sense build_wrapperintrmsfloat32rget_instance_from_cfgrWRAPPER get_instance)r r r rkwargsrL/home/ma-user/work/mindformers-baichuan/mindformers/wrapper/build_wrapper.pyrs"    rcCsDttjD]4}|drq ttj|}t|r t|t j q dS)z+ register MindSpore builtin wrapper class. __N) dirrwrap startswithgetattrinspectisclassr register_clsrr) module_namer!rrrregister_ms_wrapEs   r(cCsTtjtjjtjddtjtjjtjddtjttjddtjt tjdddS)z- register MindFormers builtin wrapper class. r )r aliasloss_scale_wrapper mf_wrapperZpipeline_wrapperN) rr&rr!TrainOneStepCellrrTrainOneStepWithLossScaleCellrr rrrrregister_mf_wrapperPs r.)NNr N)__doc__r$ mindsporerrrmindformers.tools.registerrrrr rr rrstrrr(r.rrrrs ( pengcheng.mind-test/mindformers/wrapper/__pycache__/wrapper.cpython-39.pyc000066400000000000000000000267471471400134000272570ustar00rootroot00000000000000a `of<@sdZddlmZddlmZddlmZddlmZ ddlm Z ddlm Z ddlmZmZmZdd lmZdd lmmZdd lmZdd lmZmZdd lmZddgZe dZ!e "Z#e!$ddddZ%e!$ddddZ&e$ej'Gdddej(Z)e$ej'Gdddej*Z+e dZ,e dZ-e,$dddddZ.e-$dddddZ/e$ej'Gdddej0Z1d S) zSelf-Define Wrapper.)deepcopy)Tensor) RowTensor) composite) functional) operations)nn Parameter ParallelMode)_get_enable_parallel_optimizerN) ClipGradNorm)MindFormerRegisterMindFormerModuleType) get_identityMFTrainOneStepCellMFPipelineWithLossScaleCell grad_scalercCst|tjt|SN)Fcastmstypefloat32 reciprocalscalegradrF/home/ma-user/work/mindformers-baichuan/mindformers/wrapper/wrapper.pytensor_grad_scale%srrc Cs*t|j|jtt|t|j|jSr)rindicesvaluesrrrdtype dense_shaperrrrtensor_grad_scale_row_tensor*sr#cs,eZdZfddZddZddZZS)%DynamicLossScaleUpdateWithMinimalCellcstt|t|tjd|_t|tjd|_||_ t tdtjddd|_ t tdtjddd|_ t |_t |_t|tjd|_t |_t |_t |_t |_t |_tdtjd|_tddS) Nr!current_iterator_stepnamerlast_overflow_iterator_stepTz+build dynamic loss scale with minimal value) superr$__init__rrint32 scale_windowr scale_factorloss_scale_valuer cur_iterlast_overflow_iterPSelectselectMaximummaxminimum_loss_scale Reciprocalr LessEqual less_equal LogicalAnd logic_and LogicalNot logic_not LogicalOrlogic_orbool_ const_trueprint)selfr0r/r.r8 __class__rrr,2s        z.DynamicLossScaleUpdateWithMinimalCell.__init__cCs|jS)ao Get Loss Scale value. Returns: float, the loss scale value. Examples: >>> from mindspore import nn >>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=212, scale_factor=2, scale_window=1000) >>> output = manager.get_loss_scale() >>> print(output) 212 )r0)rErrrget_loss_scaleJsz4DynamicLossScaleUpdateWithMinimalCell.get_loss_scalec Cs|}||||||j|j|}||j|j|j}| ||}|||j|j}t |j|}| || |} ||j} || | |} t || |jd} t | |} t |j| |S)Nr&)r5r7rr/r8r;r.r1r2rArassignr=r?depend) rE loss_scaleoverflow overflow_condloss_scale_on_overflow should_inclast_iter_condr2 last_iterupdate_scale_cond scale_mul_resscaled_loss_scale inc_cur_iterrrr constructZs$     z/DynamicLossScaleUpdateWithMinimalCell.construct)__name__ __module__ __qualname__r,rHrV __classcell__rrrFrr$0s r$cs*eZdZdZdfdd ZddZZS) raTrainOneStep For MindFormer. Network training with loss scaling, grad clip, gradient accumulation, exponential moving average and so on. This is a training step with loss scaling. It takes a network, an optimizer and a scale update Cell(or a Tensor) as args. The loss scale value can be updated in both host side or device side. If you want to update it on host side, using a value of Tensor type as `scale_sense`, otherwise, using a Cell instance for updating loss scale as `scale_sense`. Args: network (Cell): The training network. The network only supports single output. optimizer (Cell): Optimizer for updating the network parameters. use_clip_grad (bool): Whether to use the gradient clipping function. Default: False. max_grad_norm (float): Maximum gradient value. Default: 1.0. scale_sense (Union[Tensor, Cell]): If this value is a Cell, it will be called by `MFTrainOneStepCell` to update loss scale. If this value is a Tensor, the loss scale can be modified by `set_sense_scale`, the shape should be :math:`()` or :math:`(1,)`. Inputs: - **(*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). F?c sVtt||||||_t|d|_|dd|_t|j j |_ t dgt j |_dS)Nmax_normparallel_config)r+rr, use_clip_gradr clip_grad_normpopr^r optimizer learning_raterrrgrad_norm_base)rEnetworkrcr` max_grad_norm scale_sensekwargsrFrrr,s  zMFTrainOneStepCell.__init__c Gs.|j}|j|}|j}|||\}}t|t|t|}| |j|g||R}| t t ||}| |}|j}|jjr|jjr|jd|jjd}n||jjd}|||} || } |j} | s |jr||\}} t|||}n|j|jj|jj|| ||| fS)zforward and backward.r)weightsrfrhstart_overflow_checkC ones_likerrr!r hyper_mappartial _grad_scale grad_reducerrdrc dynamic_lr is_group_lr global_stepreshapeget_overflow_statusprocess_loss_scalerer`rarJ assignaddglobal_step_increase_tensor) rEinputsrkloss scaling_sensstatusscaling_sens_filledgradsrdcondrL grad_normrrrrVs,    zMFTrainOneStepCell.construct)Fr[r[)rWrXrY__doc__r,rVrZrrrFrrks #shard_grad_scalecCsHt||}|t|}t||}t|d}t|t||}|S)Nr_)rrJr tensor_mulrI)rr accu_gradnew_gradzerosrrrtensor_grad_scale_pipelines     rc Cs6|t|}t||}t|t|t|}|Sr)rrrJrI zeros_like)rrrrrrr tensor_shard_grad_scale_pipelines  rcs6eZdZdZd fdd Zejdddd ZZS) ra Append an train one step cell with loss scale of pipeline parallel for MindFormers. Args: network (Cell): The training network. Note that loss function should have been added. optimizer (Optimizer): Optimizer for updating the weights. use_clip_grad (bool): Whether to use gradient clipping. Default: True. max_grad_norm (float): Maximum gradient constraint value. Default: 1.0. scale_sense (Cell): Cell to do the loss scale. Default: 1.0. micro_batch_num (int): Micro batch number of pipeline parallel. Default: 1. Inputs: - **(\*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). Tr[r&c stt|j||dd||_|jjdd|j|_|jjddd|_||_ t j ddd|_ t |_d|_t|_t|_t|_t|_tjd d |_|jtjtjfvrtd |jd t |_!t"dt#j$|_%t&|_'t (|_)t*|_+d|_,t-|t.j/r0||_,t0t"|1t#j$d dd|_2nXt-|t"rv|j3dksT|j3dkrdt0|dd|_2ntd4|j3nt5d4t6|t7|_8||_9t:|d|_;||_<|=dd|_>t?|j j@|_@t"dgt#j$|_AdS)N)sensT) defer_inline accu_gradsr)prefixinit) get_by_list sens_paramr&F) keep_dimszeParallelMode must be one of [ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL], but found .r%rhr()r&rz9The shape of 'scale_sense' must be (1,) or (), but got {}z4The 'scale_sense' must be Cell or Tensor, but got {}r\r^r_)Br+rr,rf add_flags parametersrkclonerrcrm GradOperationrrrrdegreer3CastrNPUAllocFloatStatus alloc_statusNPUGetFloatStatus get_statusNPUClearFloatStatusclear_before_grad ReduceSum reduce_sum parallel_moder SEMI_AUTO_PARALLEL AUTO_PARALLEL ValueError AllReduce allreducerrrbaser:r;HyperMaproReshapervloss_scaling_manager isinstancerCellr rHrhshapeformat TypeErrortyper opt_shardr`r ra micro_sizerbr^rrdre)rErfrcr`rgrhmicro_batch_numrirFrrr,sV            z$MFPipelineWithLossScaleCell.__init__)Z has_effectcGs|j|}|j}t|t|t|}|}||}t ||}| |j|j g||||j t jR}t ||}||}t ||}||d} t ||}|jr||}|tt||j||j}n(||j} |tt||j|| }|j} |jr&||\}} |j} |jjrn|jjrZ|jd|jj !d} n||jj !d} |"| } |#|j$| }|}|j%dur|%|j|}|st |||}n|j&|jj |jj'|||(| | fS)z1The construct processes of pipeline wrapper cell.)rrjrN))rfrhrmrnrrr!rrrJrrkrrrrrrrrrorprrrrrer`rardrcrsrtrurvrr;rrryrzvalue)rEr{r|r}rrZ status_clearrrflag_sumrrrd flag_reducerrLrrrrVsL                z%MFPipelineWithLossScaleCell.construct)Tr[r[r&) rWrXrYrr,rmrrVrZrrrFrrs . )2rcopyrmindspore.common.tensorrmindspore.commonr mindspore.opsrrmrrrr3 mindsporerr r mindspore.parallel._utilsr Zmindspore.common.dtypecommonr!rZmindformers.core.clip_gradr mindformers.tools.registerr rmindformers.version_controlr__all__MultitypeFuncGraphrqr9rregisterrr#WRAPPERrr$TrainOneStepWithLossScaleCellrrrrrTrainOneStepCellrrrrrs>               : S      pengcheng.mind-test/mindformers/wrapper/build_wrapper.py000066400000000000000000000100251471400134000241250ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Wrapper.""" import inspect import mindspore as ms from mindspore import nn, Tensor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .wrapper import MFTrainOneStepCell, MFPipelineWithLossScaleCell WRAPPERS_MINDFORMERS_DEFINED = ["MFTrainOneStepCell", "MFPipelineWithLossScaleCell"] def build_wrapper(config: dict = None, default_args: dict = None, module_type: str = 'wrapper', class_name: str = None, **kwargs): r"""Build Wrapper For MindFormer. Instantiate the wrapper from MindFormerRegister's registry. Args: config (dict): The task wrapper's config. Default: None. default_args (dict): The default argument of wrapper API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'wrapper'. class_name (str): The class name of wrapper API. Default: None. Return: The function instance of wrapper API. Examples: >>> from mindformers import build_wrapper >>> wrapper_config = {'type': 'TrainOneStepCell', 'sens': 1024} >>> # 1) use config dict to build wrapper >>> wrapper_from_config = build_wrapper(wrapper_config) >>> # 2) use class name to build wrapper >>> wrapper_class_name = build_wrapper(class_name='TrainOneStepCell', sens=1024) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.type not in WRAPPERS_MINDFORMERS_DEFINED: if default_args and "parallel_config" in default_args: del default_args["parallel_config"] if config.scale_sense is not None: if isinstance(config.scale_sense, dict) and config.scale_sense.type is not None: config.scale_sense = build_wrapper(config.scale_sense) elif isinstance(config.scale_sense, int): config.scale_sense = Tensor(config.scale_sense, ms.float32) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.WRAPPER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_wrap(): """ register MindSpore builtin wrapper class. """ for module_name in dir(nn.wrap): if module_name.startswith('__'): continue wrap = getattr(nn.wrap, module_name) if inspect.isclass(wrap): MindFormerRegister.register_cls( wrap, MindFormerModuleType.WRAPPER) def register_mf_wrapper(): """ register MindFormers builtin wrapper class. """ # Support built-in model wrapper of MindFormers. MindFormerRegister.register_cls( nn.wrap.TrainOneStepCell, module_type=MindFormerModuleType.WRAPPER, alias="wrapper") MindFormerRegister.register_cls( nn.wrap.TrainOneStepWithLossScaleCell, module_type=MindFormerModuleType.WRAPPER, alias="loss_scale_wrapper") MindFormerRegister.register_cls( MFTrainOneStepCell, module_type=MindFormerModuleType.WRAPPER, alias="mf_wrapper") MindFormerRegister.register_cls( MFPipelineWithLossScaleCell, module_type=MindFormerModuleType.WRAPPER, alias="pipeline_wrapper") register_ms_wrap() register_mf_wrapper() pengcheng.mind-test/mindformers/wrapper/wrapper.py000066400000000000000000000362521471400134000227600ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Self-Define Wrapper.""" from copy import deepcopy from mindspore.common.tensor import Tensor from mindspore.common import RowTensor from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore import nn, Parameter, ParallelMode from mindspore.parallel._utils import _get_enable_parallel_optimizer import mindspore.common.dtype as mstype from mindformers.core.clip_grad import ClipGradNorm from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.version_control import get_identity __all__ = ['MFTrainOneStepCell', 'MFPipelineWithLossScaleCell'] _grad_scale = C.MultitypeFuncGraph("grad_scale") reciprocal = P.Reciprocal() @_grad_scale.register("Tensor", "Tensor") def tensor_grad_scale(scale, grad): return F.cast(grad, mstype.float32) * reciprocal(scale) @_grad_scale.register("Tensor", "RowTensor") def tensor_grad_scale_row_tensor(scale, grad): return RowTensor(grad.indices, grad.values * F.cast(reciprocal(scale), F.dtype(grad.values)), grad.dense_shape) @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class DynamicLossScaleUpdateWithMinimalCell(nn.Cell): def __init__(self, loss_scale_value, scale_factor, scale_window, minimum_loss_scale): super(DynamicLossScaleUpdateWithMinimalCell, self).__init__() self.scale_window = Tensor(scale_window, dtype=mstype.int32) self.scale_factor = Tensor(scale_factor, dtype=mstype.float32) self.loss_scale_value = loss_scale_value self.cur_iter = Parameter(Tensor(1, dtype=mstype.int32), name="current_iterator_step") self.last_overflow_iter = Parameter(Tensor(0, dtype=mstype.int32), name="last_overflow_iterator_step") self.select = P.Select() self.max = P.Maximum() self.minimum_loss_scale = Tensor(minimum_loss_scale, dtype=mstype.float32) self.reciprocal = P.Reciprocal() self.less_equal = P.LessEqual() self.logic_and = P.LogicalAnd() self.logic_not = P.LogicalNot() self.logic_or = P.LogicalOr() self.const_true = Tensor(True, dtype=mstype.bool_) print("build dynamic loss scale with minimal value") def get_loss_scale(self): """ Get Loss Scale value. Returns: float, the loss scale value. Examples: >>> from mindspore import nn >>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=212, scale_factor=2, scale_window=1000) >>> output = manager.get_loss_scale() >>> print(output) 212 """ return self.loss_scale_value def construct(self, loss_scale, overflow): overflow_cond = overflow loss_scale_on_overflow = self.select(overflow_cond, self.max(loss_scale * self.reciprocal(self.scale_factor), self.minimum_loss_scale), loss_scale) should_inc = self.less_equal(self.scale_window, self.cur_iter - self.last_overflow_iter) last_iter_cond = self.logic_or(overflow_cond, should_inc) last_overflow_iter = self.select(last_iter_cond, self.cur_iter, self.last_overflow_iter) last_iter = F.assign(self.last_overflow_iter, last_overflow_iter) update_scale_cond = self.logic_and(should_inc, self.logic_not(overflow_cond)) scale_mul_res = loss_scale_on_overflow * self.scale_factor scaled_loss_scale = self.select(update_scale_cond, scale_mul_res, loss_scale_on_overflow) F.assign(loss_scale, scaled_loss_scale) inc_cur_iter = self.cur_iter + 1 inc_cur_iter = F.depend(inc_cur_iter, last_iter) F.assign(self.cur_iter, inc_cur_iter) return overflow @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class MFTrainOneStepCell(nn.TrainOneStepWithLossScaleCell): r"""TrainOneStep For MindFormer. Network training with loss scaling, grad clip, gradient accumulation, exponential moving average and so on. This is a training step with loss scaling. It takes a network, an optimizer and a scale update Cell(or a Tensor) as args. The loss scale value can be updated in both host side or device side. If you want to update it on host side, using a value of Tensor type as `scale_sense`, otherwise, using a Cell instance for updating loss scale as `scale_sense`. Args: network (Cell): The training network. The network only supports single output. optimizer (Cell): Optimizer for updating the network parameters. use_clip_grad (bool): Whether to use the gradient clipping function. Default: False. max_grad_norm (float): Maximum gradient value. Default: 1.0. scale_sense (Union[Tensor, Cell]): If this value is a Cell, it will be called by `MFTrainOneStepCell` to update loss scale. If this value is a Tensor, the loss scale can be modified by `set_sense_scale`, the shape should be :math:`()` or :math:`(1,)`. Inputs: - **(*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). """ def __init__(self, network, optimizer, use_clip_grad=False, max_grad_norm=1.0, scale_sense=1.0, **kwargs): super(MFTrainOneStepCell, self).__init__(network, optimizer, scale_sense) self.use_clip_grad = use_clip_grad self.clip_grad_norm = ClipGradNorm(max_norm=max_grad_norm) self.parallel_config = kwargs.pop("parallel_config", None) self.learning_rate = deepcopy(self.optimizer.learning_rate) self.grad_norm_base = Tensor([0.0], mstype.float32) def construct(self, *inputs): """forward and backward.""" weights = self.weights loss = self.network(*inputs) scaling_sens = self.scale_sense status, scaling_sens = self.start_overflow_check(loss, scaling_sens) scaling_sens_filled = C.ones_like(loss) * F.cast(scaling_sens, F.dtype(loss)) grads = self.grad(self.network, weights)(*inputs, scaling_sens_filled) grads = self.hyper_map(F.partial(_grad_scale, scaling_sens), grads) # apply grad reducer on grads grads = self.grad_reducer(grads) learning_rate = self.learning_rate if self.optimizer.dynamic_lr: if self.optimizer.is_group_lr: learning_rate = self.learning_rate[-1](self.optimizer.global_step).reshape(()) else: learning_rate = self.learning_rate(self.optimizer.global_step).reshape(()) # get the overflow buffer cond = self.get_overflow_status(status, grads) overflow = self.process_loss_scale(cond) # if there is no overflow, do optimize grad_norm = self.grad_norm_base if not overflow: if self.use_clip_grad: grads, grad_norm = self.clip_grad_norm(grads) loss = F.depend(loss, self.optimizer(grads)) else: self.optimizer.assignadd(self.optimizer.global_step, self.optimizer.global_step_increase_tensor) return loss, overflow, scaling_sens, learning_rate, grad_norm grad_scale = C.MultitypeFuncGraph("grad_scale") shard_grad_scale = C.MultitypeFuncGraph("shard_grad_scale") @grad_scale.register("Tensor", "Tensor", "Tensor") def tensor_grad_scale_pipeline(scale, grad, accu_grad): accu_grad = F.depend(accu_grad, grad) new_grad = accu_grad * reciprocal(scale) accu_grad = F.depend(accu_grad, new_grad) zeros = F.tensor_mul(accu_grad, 0.0) new_grad = F.depend(new_grad, F.assign(accu_grad, zeros)) return new_grad @shard_grad_scale.register("Tensor", "Tensor", "Tensor") def tensor_shard_grad_scale_pipeline(scale, grad, accu_grad): new_grad = grad * reciprocal(scale) accu_grad = F.depend(accu_grad, new_grad) new_grad = F.depend(new_grad, F.assign(accu_grad, F.zeros_like(accu_grad))) return new_grad @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class MFPipelineWithLossScaleCell(nn.TrainOneStepCell): r""" Append an train one step cell with loss scale of pipeline parallel for MindFormers. Args: network (Cell): The training network. Note that loss function should have been added. optimizer (Optimizer): Optimizer for updating the weights. use_clip_grad (bool): Whether to use gradient clipping. Default: True. max_grad_norm (float): Maximum gradient constraint value. Default: 1.0. scale_sense (Cell): Cell to do the loss scale. Default: 1.0. micro_batch_num (int): Micro batch number of pipeline parallel. Default: 1. Inputs: - **(\*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). """ def __init__(self, network, optimizer, use_clip_grad=True, max_grad_norm=1.0, scale_sense=1.0, micro_batch_num=1, **kwargs): super(MFPipelineWithLossScaleCell, self).__init__(network, optimizer, sens=None) self.network = network self.network.add_flags(defer_inline=True) self.weights = optimizer.parameters self.accu_grads = self.weights.clone(prefix="accu_grads", init="zeros") self.optimizer = optimizer self.grad = C.GradOperation(get_by_list=True, sens_param=True) self.grad_reducer = get_identity() self.degree = 1 self.cast = P.Cast() self.alloc_status = P.NPUAllocFloatStatus() self.get_status = P.NPUGetFloatStatus() self.clear_before_grad = P.NPUClearFloatStatus() self.reduce_sum = P.ReduceSum(keep_dims=False) if self.parallel_mode not in [ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL]: raise ValueError(f"ParallelMode must be one of " f"[ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL], but found " f"{self.parallel_mode}.") self.allreduce = P.AllReduce() self.base = Tensor(1, mstype.float32) self.less_equal = P.LessEqual() self.hyper_map = C.HyperMap() self.reshape = P.Reshape() self.loss_scaling_manager = None if isinstance(scale_sense, nn.Cell): self.loss_scaling_manager = scale_sense self.scale_sense = Parameter(Tensor(scale_sense.get_loss_scale(), dtype=mstype.float32), name="scale_sense") elif isinstance(scale_sense, Tensor): if scale_sense.shape == (1,) or scale_sense.shape == (): self.scale_sense = Parameter(scale_sense, name='scale_sense') else: raise ValueError("The shape of 'scale_sense' must be (1,) or (), but got {}" .format(scale_sense.shape)) else: raise TypeError("The 'scale_sense' must be Cell or Tensor, but got {}".format(type(scale_sense))) self.opt_shard = _get_enable_parallel_optimizer() self.use_clip_grad = use_clip_grad self.clip_grad_norm = ClipGradNorm(max_norm=max_grad_norm) self.micro_size = micro_batch_num self.parallel_config = kwargs.pop("parallel_config", None) self.learning_rate = deepcopy(self.optimizer.learning_rate) self.grad_norm_base = Tensor([0.0], mstype.float32) @C.add_flags(has_effect=True) def construct(self, *inputs): """The construct processes of pipeline wrapper cell.""" loss = self.network(*inputs) scaling_sens = self.scale_sense scaling_sens_filled = C.ones_like(loss) * F.cast(scaling_sens, F.dtype(loss)) init = self.alloc_status() status_clear = self.clear_before_grad(init) scaling_sens_filled = F.depend(scaling_sens_filled, status_clear) grads = self.grad(self.network, self.weights)(*inputs, self.cast(scaling_sens_filled / self.micro_size, mstype.float32)) init = F.depend(init, grads) get_status = self.get_status(init) init = F.depend(init, get_status) flag_sum = self.reduce_sum(init, (0,)) loss = F.depend(loss, status_clear) if self.opt_shard: grads = self.grad_reducer(grads) grads = self.hyper_map(F.partial(shard_grad_scale, scaling_sens * self.degree), grads, self.accu_grads) else: accu_grads = self.grad_reducer(self.accu_grads) grads = self.hyper_map(F.partial(grad_scale, scaling_sens * self.degree), grads, accu_grads) grad_norm = self.grad_norm_base if self.use_clip_grad: grads, grad_norm = self.clip_grad_norm(grads) learning_rate = self.learning_rate if self.optimizer.dynamic_lr: if self.optimizer.is_group_lr: learning_rate = self.learning_rate[-1](self.optimizer.global_step).reshape(()) else: learning_rate = self.learning_rate(self.optimizer.global_step).reshape(()) # sum overflow flag over devices flag_reduce = self.allreduce(flag_sum) cond = self.less_equal(self.base, flag_reduce) overflow = cond if self.loss_scaling_manager is not None: overflow = self.loss_scaling_manager(self.scale_sense, cond) if not overflow: loss = F.depend(loss, self.optimizer(grads)) else: self.optimizer.assignadd(self.optimizer.global_step, self.optimizer.global_step_increase_tensor) return loss, overflow, scaling_sens.value(), learning_rate, grad_norm pengcheng.mind-test/requirements.back000077500000000000000000000003131471400134000202560ustar00rootroot00000000000000numpy setuptools sentencepiece>=0.1.97 ftfy>=6.1.1 regex>=2022.10.31 tqdm>=4.65.0 pyyaml>=6.0 jieba>=0.42.1 rouge_chinese>=1.0.3 nltk>=2.0 mindpet==1.0.1 pydantic==1.10.11 mdtex2html gradio opencv-pythonpengcheng.mind-test/run_chat_web.py000077500000000000000000000320171471400134000177310ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Run chat web demo. """ import argparse import time from threading import Thread import mdtex2html import mindspore as ms import gradio as gr from mindformers import AutoModel, AutoTokenizer, TextIteratorStreamer, AutoConfig, logger from mindformers.tools.utils import str2bool def get_model_and_tokenizer(model_config, tokenizer_name): """Get model and tokenizer instance""" return AutoModel.from_config(model_config), AutoTokenizer.from_pretrained(tokenizer_name) parser = argparse.ArgumentParser() parser.add_argument('--device_target', default="Ascend", type=str, choices=['Ascend', 'CPU'], help='The target device to run, support "Ascend" and "CPU". Default: Ascend.') parser.add_argument('--device_id', default=0, type=int, help='Which device to run service. Default: 0.') parser.add_argument('--model', type=str, help='Which model to generate text.') parser.add_argument('--tokenizer', type=str, help='Which tokenizer to tokenize text.') parser.add_argument('--checkpoint_path', type=str, default=None, help='The path of model checkpoint.') parser.add_argument('--seq_length', default="512", type=int, help="Sequence length of the model. Default: 512.") parser.add_argument('--use_past', default=False, type=str2bool, help='Whether to enable incremental inference. Default: False.') parser.add_argument('--host', default="127.0.0.1", type=str, help="Which host ip to run the service. Default: 127.0.0.1.") parser.add_argument('--port', default=None, type=int, help='Which port to run the service. Default: None.') args = parser.parse_args() ms.set_context(mode=ms.GRAPH_MODE, device_target=args.device_target, device_id=args.device_id) config = AutoConfig.from_pretrained(args.model) config.seq_length = args.seq_length config.use_past = args.use_past if args.checkpoint_path: config.checkpoint_name_or_path = args.checkpoint_path logger.info("Config: %s", config) model, tokenizer = get_model_and_tokenizer(config, args.tokenizer) streamer = TextIteratorStreamer(tokenizer=tokenizer, skip_prompt=True, skip_special_tokens=True) # pre-build the network sample_input = tokenizer("hello") sample_output = model.generate(sample_input["input_ids"], max_length=10) prompt_examples = [["Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{}\n\n### Response:"], ["A chat between a curious user and an artificial intelligence assistant. The assistant gives " "helpful," "detailed, and polite answers to user\'s questions. USER: {} ASSISTANT: "], ["Assume you are a dog, you must response \"Woofs\" at first whatever any instruction\n\n" "### Instruction:\n{}\n\n### Response:"], [": {} \n: "], ["问：{}\n\n答："]] is_multi_round = args.model.startswith("glm2") default_prompt = "Below is an instruction that describes a task. Write a response that appropriately completes the " \ "request.\n\n### Instruction:\n{}\n\n### Response:" if args.model.startswith("llama_7b_lora") else "" default_prompt = "问：{}\n\n答：" if args.model.startswith("glm2") else "" class Chatbot(gr.Chatbot): """Chatbot with overrode postprocess method""" def postprocess(self, y): """postprocess""" if y is None: return [] for i, (message, response) in enumerate(y): y[i] = ( None if message is None else mdtex2html.convert(message), None if response is None else mdtex2html.convert(response), ) return y def parse_text(text): """copy from https://github.com/GaiZhenbiao/ChuanhuChatGPT/""" lines = text.split("\n") lines = [line for line in lines if line != ""] count = 0 for i, line in enumerate(lines): if "```" in line: count += 1 items = line.split('`') if count % 2 == 1: lines[i] = f'

'
            else:
                lines[i] = f'
' else: if i > 0: if count % 2 == 1: line = line.replace("`", r"\`") line = line.replace("<", "<") line = line.replace(">", ">") line = line.replace(" ", " ") line = line.replace("*", "*") line = line.replace("_", "_") line = line.replace("-", "-") line = line.replace(".", ".") line = line.replace("!", "!") line = line.replace("(", "(") line = line.replace(")", ")") line = line.replace("$", "$") lines[i] = "
" + line text = "".join(lines) return text def build_prompt(inputs, prompt): """Build prompt for inputs""" if prompt == "": return inputs if prompt.find("{}") != -1: return prompt.format(inputs) raise gr.Error("The prompt is invalid! Please make sure your prompt contains placeholder '{}' to replace user " "input.") def build_multi_round(inputs, history): """Build multi-round prompt for inputs""" prev_rounds = "" if args.model.startswith("glm2"): for i, (query, response) in enumerate(history): prev_rounds += "[Round {}]\n\n{}{}\n\n".format(i, query, response) prev_rounds += "[Round {}]\n\n".format(len(history)) return prev_rounds + inputs def generate(**kwargs): """generate function with timer and exception catcher""" gen_model = kwargs.pop("model") try: start_time = time.time() gen_model.generate(**kwargs) end_time = time.time() logger.info("Total time: %.2f s", (end_time-start_time)) # pylint: disable=W0703 except Exception as e: logger.error(repr(e)) streamer.text_queue.put("") def predict(inputs, bot, history, do_sample, top_k, top_p, temperature, repetition_penalty, max_length, prompt): """predict""" if inputs == "": raise gr.Error("Input cannot be empty!") logger.info("Received user input: %s", inputs) bot.append((parse_text(inputs), "")) prompted_input = build_prompt(inputs, prompt) logger.info("Prompt: %s", prompt) logger.info("User input after prompted: %s", prompted_input) round_prompted_input = build_multi_round(prompted_input, history) logger.info("User input after multi-round prompted: %s", round_prompted_input) input_tokens = tokenizer(round_prompted_input) if max_length > args.seq_length: raise gr.Error("Max length must be set smaller than the sequence length of model! The model sequence length " "is {}.".format(args.seq_length)) if max_length <= len(input_tokens["input_ids"]): raise gr.Error("Max length must be larger than the length of input tokens! The current length of input " "tokens is {}.".format(len(input_tokens["input_ids"]))) generation_kwargs = dict(model=model, input_ids=input_tokens["input_ids"], streamer=streamer, do_sample=do_sample, top_k=top_k, top_p=top_p, temperature=temperature, repetition_penalty=repetition_penalty, max_length=max_length) thread = Thread(target=generate, kwargs=generation_kwargs) logger.info("Start to generate text. Generate args: {input_ids: %s, do_sample: %s, top_k: %s, top_p: %s, " "temperature: %s, repetition_penalty: %s, max_length: %s}", input_tokens["input_ids"], do_sample, top_k, top_p, temperature, repetition_penalty, max_length) thread.start() interval_time = 0. output = "" for response in streamer: if response == "": raise gr.Error("An error occurred! Please make sure all the settings are correct!") output += response new_history = history + [(prompted_input, output)] bot[-1] = (parse_text(inputs), parse_text(output)) interval_start = time.time() yield bot, new_history interval_end = time.time() interval_time = interval_time + interval_end - interval_start logger.info("Generate output: %s", output) logger.info("Web communication time: %.2f s", interval_time) def reset_user_input(): """reset user input""" return gr.update(value='') def reset_state(): """reset state""" return [], [] def set_do_sample_args(do_sample): return {top_k_slider: gr.update(visible=do_sample), top_p_slider: gr.update(visible=do_sample), temp_number: gr.update(visible=do_sample)} with gr.Blocks() as demo: gr.HTML(f"""

Chat Web Demo powered by MindFormers

""") with gr.Row(): with gr.Column(scale=10): with gr.Group(): chatbot = gr.Chatbot(label=fr"Chatbot {args.model}") user_input = gr.Textbox(show_label=False, placeholder="Ask something...", lines=5) with gr.Row(): with gr.Column(scale=6): submit_btn = gr.Button("Submit", variant="primary") with gr.Column(scale=6): empty_btn = gr.Button("Clear") with gr.Column(scale=2): with gr.Group(): do_sample_checkbox = gr.Checkbox(value=config.do_sample if config.do_sample else False, label="sampling", info="Whether to sample on the candidate ids") top_k_slider = gr.Slider(value=config.top_k if config.top_k else 1, label="top k", maximum=10, minimum=0, step=1, visible=config.do_sample if config.do_sample else False, info="Determine the topK numbers token id as candidate") top_p_slider = gr.Slider(value=config.top_p if config.top_p else 1, label="top p", maximum=1, minimum=0.01, step=0.01, visible=config.do_sample if config.do_sample else False, info="The accumulation probability of the candidate token ids, " "below the top_p will be select as the candidate ids") temp_number = gr.Number(value=config.temperature if config.temperature else 1, label="temperature", visible=config.do_sample if config.do_sample else False, info="The value used to modulate the next token probabilities") rp_number = gr.Number(value=config.repetition_penalty if config.repetition_penalty else 1, label="repetition penalty", minimum=0, info="The penalty factor of the frequency that generated words") max_len_number = gr.Number(value=config.max_length if config.max_length else 512, minimum=0, label="max length", info="The maximum length of the generated words") prompt_input = gr.Textbox(value=default_prompt, label="prompt", placeholder="No prompt...", info="Add prompt to input") gr.Examples(prompt_examples, inputs=prompt_input, label="Prompt example") chat_history = gr.State([]) submit_btn.click(predict, [user_input, chatbot, chat_history, do_sample_checkbox, top_k_slider, top_p_slider, temp_number, rp_number, max_len_number, prompt_input], [chatbot, chat_history], show_progress=True) submit_btn.click(reset_user_input, [], [user_input]) empty_btn.click(reset_state, outputs=[chatbot, chat_history], show_progress=True) do_sample_checkbox.change(set_do_sample_args, [do_sample_checkbox], [top_k_slider, top_p_slider, temp_number]) if __name__ == "__main__": demo.queue().launch(server_name=args.host, server_port=args.port) pengcheng.mind-test/run_infer_main.py000077500000000000000000000166161471400134000202730ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """lite infer main.""" import sys import argparse from threading import Thread # Avoid bugs when mslite and mindspore are not built from same commit, which may cause running error. # pylint: disable=W0611 import mindspore_lite as mslite from mindformers.models.base_tokenizer import Tokenizer from mindformers.models import BloomTokenizer, LlamaTokenizer from mindformers.models import ChatGLMTokenizer, ChatGLM2Tokenizer from mindformers.pipeline import pipeline from mindformers.generation import TextIteratorStreamer from mindformers.tools.utils import str2bool from mindformers.inference import InferConfig, InferTask def pipeline_from_model_paths(args_, tokenizer): """build infer pipeline for model paths.""" lite_pipeline = pipeline( task="text_generation", model=(args_.prefill_model_path, args_.increment_model_path), tokenizer=tokenizer, backend="mslite", model_name=args_.model_name, ge_config_path=args_.config_path, device_id=args_.device_id, infer_seq_length=args_.seq_length, ) return lite_pipeline def pipeline_from_model_name(args_, tokenizer): """build infer pipeline for model name.""" lite_pipeline = pipeline( task="text_generation", model=args_.model_name, tokenizer=tokenizer, backend="mslite", ge_config_path=args_.config_path, device_id=args_.device_id, infer_seq_length=args_.seq_length, ) return lite_pipeline def pipeline_from_model_dir(args_, tokenizer): """build infer pipeline for model dir.""" lite_pipeline = pipeline( task="text_generation", model=args_.model_dir, tokenizer=tokenizer, backend="mslite", model_name=args_.model_name, ge_config_path=args_.config_path, device_id=args_.device_id, infer_seq_length=args_.seq_length, ) return lite_pipeline def pipeline_from_infer_config(args_, tokenizer): """build infer pipeline for infer config.""" lite_config = InferConfig( prefill_model_path=args_.prefill_model_path, increment_model_path=args_.increment_model_path, model_type="mindir", model_name=args_.model_name, ge_config_path=args_.config_path, device_id=args_.device_id, infer_seq_length=args_.seq_length, ) lite_pipeline = InferTask.get_infer_task("text_generation", lite_config, tokenizer=tokenizer) return lite_pipeline # the model name list that mslite inference has supported. LITE_SUPPORT_MODELS = ('bloom', 'glm', 'glm2', 'llama') def get_tokenizer(model_name: str) -> Tokenizer: """get tokenizer with model name.""" tokenizer = None if model_name == 'bloom': tokenizer = BloomTokenizer.from_pretrained("bloom_560m") elif model_name == 'glm': tokenizer = ChatGLMTokenizer.from_pretrained("glm_6b") elif model_name == 'glm2': tokenizer = ChatGLM2Tokenizer.from_pretrained("glm2_6b") elif model_name == 'llama': tokenizer = LlamaTokenizer.from_pretrained("llama_7b") else: raise ValueError( f"model must be in {LITE_SUPPORT_MODELS} when getting tokenizer, but got input {model_name}.") return tokenizer def infer_main(args_): """lite infer main.""" tokenizer = get_tokenizer(args_.model_name.lower()) lite_pipeline = pipeline_from_model_paths( args_, tokenizer ) while True: user_input = input("Please enter your predict data: \n") if user_input == "exit": print("Task is over.") sys.exit() output = lite_pipeline(user_input, is_sample_acceleration=args_.is_sample_acceleration, add_special_tokens=args_.add_special_tokens) print(output) def infer_stream_main(args_): """main entry for infer stream.""" tokenizer = get_tokenizer(args_.model_name.lower()) streamer = TextIteratorStreamer(tokenizer, skip_prompt=True) lite_pipeline = pipeline_from_model_paths( args_, tokenizer ) while True: user_input = input("Please enter your predict data(input 'exit' to quit): \n") if user_input == "exit": print("Quit now, this may take a while.") sys.exit() generation_kwargs = dict(inputs=user_input, streamer=streamer, is_sample_acceleration=args_.is_sample_acceleration, add_special_tokens=args_.add_special_tokens) thread = Thread(target=lite_pipeline, kwargs=generation_kwargs) thread.start() output = "" for new_text in streamer: print(new_text) output += new_text print(output) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument( '--device_id', default=0, type=int, help='ID of the target device, the value must be in [0, device_num_per_host-1], ' 'while device_num_per_host should be no more than 4096. Default: None') parser.add_argument( '--rank_id', default=0, type=int, help='ID of the target device, the value must be in [0, device_num_per_host-1], ' 'while device_num_per_host should be no more than 4096. Default: None') parser.add_argument( '--model_dir', default=None, type=str, help="This model dir path. " "Default: None") parser.add_argument( '--model_name', default="common", type=str, help=f"The model name, only supports name in {LITE_SUPPORT_MODELS}. " "Default: None") parser.add_argument( '--seq_length', default=2048, type=int, help="This model dir path. " "Default: None") parser.add_argument( '--prefill_model_path', default=None, type=str, help="This full model path. " "Default: None") parser.add_argument( '--increment_model_path', default=None, type=str, help="When use kv-cache, this is cache mode path. " "Default: None") parser.add_argument( '--config_path', default=None, type=str, help="ge config file path. " "Default: None") parser.add_argument( '--is_sample_acceleration', default=False, type=str2bool, help="Whether postprocess in graph or not." "Default: False") parser.add_argument( '--add_special_tokens', default=False, type=str2bool, help="Whether preprocess add special tokens or not." "Default: False") parser.add_argument( '--stream', default=False, type=str2bool, help="Whether decode in stream or not." "Default: False") args = parser.parse_args() if args.stream: infer_stream_main(args) else: infer_main(args) pengcheng.mind-test/run_mindformer.py000077500000000000000000000455021471400134000203220ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Run MindFormer.""" import argparse import os import shutil import numpy as np import mindspore as ms from mindspore.common import set_seed from mindformers.tools.register import MindFormerConfig, ActionDict from mindformers.core.parallel_config import build_parallel_config from mindformers.tools.utils import str2bool, set_remote_save_url, check_in_modelarts, parse_value from mindformers.core.context import build_context, build_profile_cb from mindformers.trainer import build_trainer from mindformers.tools.cloud_adapter import cloud_monitor from mindformers.tools.logger import logger from mindformers.tools import get_output_root_path from mindformers.mindformer_book import MindFormerBook if check_in_modelarts(): import moxing as mox SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() def update_checkpoint_config(config, is_train=True): """update checkpoint config depending on is_train""" if (is_train and config.resume_training) or config.auto_trans_ckpt or os.path.isdir(config.load_checkpoint): logger.info("Leave load_checkpoint may because: ") logger.info("1. resume training need resume training info. ") logger.info("2. need load distributed shard checkpoint. ") if not config.load_checkpoint: config.load_checkpoint = config.model.model_config.checkpoint_name_or_path config.model.model_config.checkpoint_name_or_path = None else: if config.run_mode in ('train', 'finetune'): config.model.model_config.checkpoint_name_or_path = config.load_checkpoint elif config.run_mode in ['eval', 'predict'] and config.load_checkpoint: config.model.model_config.checkpoint_name_or_path = config.load_checkpoint config.load_checkpoint = None def clear_auto_trans_output(config): """clear transformed_checkpoint and strategy""" if check_in_modelarts(): obs_strategy_dir = os.path.join(config.remote_save_url, "strategy") if mox.file.exists(obs_strategy_dir) and config.local_rank == 0: mox.file.remove(obs_strategy_dir, recursive=True) obs_transformed_ckpt_dir = os.path.join(config.remote_save_url, "transformed_checkpoint") if mox.file.exists(obs_transformed_ckpt_dir) and config.local_rank == 0: mox.file.remove(obs_transformed_ckpt_dir, recursive=True) mox.file.make_dirs(obs_strategy_dir) mox.file.make_dirs(obs_transformed_ckpt_dir) else: strategy_dir = os.path.join(get_output_root_path(), "strategy") if os.path.exists(strategy_dir) and config.local_rank % 8 == 0: shutil.rmtree(strategy_dir) transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") if os.path.exists(transformed_ckpt_dir) and config.local_rank % 8 == 0: shutil.rmtree(transformed_ckpt_dir) os.makedirs(strategy_dir, exist_ok=True) os.makedirs(transformed_ckpt_dir, exist_ok=True) @cloud_monitor() def main(config): """main.""" # init context build_context(config) if config.seed and \ ms.context.get_auto_parallel_context("parallel_mode") \ not in ["semi_auto_parallel", "auto_parallel"]: set_seed(config.seed) np.random.seed(config.seed) # build context config logger.info(".........Build context config..........") if config.run_mode == 'predict': if config.use_parallel and \ config.parallel.parallel_mode in ['semi_auto_parallel', 1] and \ config.parallel_config.data_parallel != 1: raise ValueError("The value of data parallel can only be set to 1, since the batch size of input is 1. ") build_parallel_config(config) logger.info("context config is: %s", config.parallel_config) logger.info("moe config is: %s", config.moe_config) if config.run_mode == 'train': update_checkpoint_config(config) if config.run_mode == 'finetune': if not config.load_checkpoint: raise ValueError("if run status is finetune, " "load_checkpoint must be input") update_checkpoint_config(config) if config.run_mode in ['eval', 'predict']: update_checkpoint_config(config, is_train=False) # remote save url if check_in_modelarts() and config.remote_save_url: logger.info("remote_save_url is %s, the output file will be uploaded to here.", config.remote_save_url) set_remote_save_url(config.remote_save_url) # define callback and add profile callback if config.profile: config.profile_cb = build_profile_cb(config) if config.auto_trans_ckpt: clear_auto_trans_output(config) trainer = build_trainer(config.trainer) if config.run_mode == 'train' or config.run_mode == 'finetune': trainer.train(config, is_full_config=True) elif config.run_mode == 'eval': trainer.evaluate(config, is_full_config=True) elif config.run_mode == 'predict': trainer.predict(config, is_full_config=True) if __name__ == "__main__": work_path = os.path.dirname(os.path.abspath(__file__)) parser = argparse.ArgumentParser() parser.add_argument( '--config', default="configs/mae/run_mae_vit_base_p16_224_800ep.yaml", required=True, help='YAML config files') parser.add_argument( '--mode', default=None, type=int, help='Running in GRAPH_MODE(0) or PYNATIVE_MODE(1). Default: GRAPH_MODE(0).' 'GRAPH_MODE or PYNATIVE_MODE can be set by `mode` attribute and both modes support all backends,' 'Default: None') parser.add_argument( '--device_id', default=None, type=int, help='ID of the target device, the value must be in [0, device_num_per_host-1], ' 'while device_num_per_host should be no more than 4096. Default: None') parser.add_argument( '--device_target', default=None, type=str, help='The target device to run, support "Ascend", "GPU", and "CPU".' 'If device target is not set, the version of MindSpore package is used.' 'Default: None') parser.add_argument( '--run_mode', default=None, type=str, help='task running status, it support [train, finetune, eval, predict].' 'Default: None') parser.add_argument( '--do_eval', default=None, type=str2bool, help='whether do evaluate in training process.' 'Default: None') parser.add_argument( '--train_dataset_dir', default=None, type=str, help='dataset directory of data loader to train/finetune. ' 'Default: None') parser.add_argument( '--eval_dataset_dir', default=None, type=str, help='dataset directory of data loader to eval. ' 'Default: None') parser.add_argument( '--predict_data', default=None, type=str, help='input data for predict, it support real data path or data directory.' 'Default: None') parser.add_argument( '--load_checkpoint', default=None, type=str, help="load model checkpoint to train/finetune/eval/predict, " "it is also support input model name, such as 'mae_vit_base_p16', " "please refer to https://gitee.com/mindspore/mindformers#%E4%BB%8B%E7%BB%8D." "Default: None") parser.add_argument( '--src_strategy_path_or_dir', default=None, type=str, help="The strategy of load_checkpoint, " "if dir, it will be merged before transform checkpoint, " "if file, it will be used in transform checkpoint directly, " "Default: None, means load_checkpoint is a single whole ckpt, not distributed") parser.add_argument( '--auto_trans_ckpt', default=None, type=str2bool, help="if true, auto transform load_checkpoint to load in distributed model. ") parser.add_argument( '--only_save_strategy', default=None, type=str2bool, help="if true, when strategy files are saved, system exit. ") parser.add_argument( '--resume_training', default=None, type=str2bool, help="whether to load training context info, such as optimizer and epoch num") parser.add_argument( '--strategy_load_checkpoint', default=None, type=str, help='path to parallel strategy checkpoint to load, it support real data path or data directory.' 'Default: None') parser.add_argument( '--remote_save_url', default=None, type=str, help='remote save url, where all the output files will tansferred and stroed in here. ' 'Default: None') parser.add_argument( '--seed', default=None, type=int, help='global random seed to train/finetune.' 'Default: None') parser.add_argument( '--use_parallel', default=None, type=str2bool, help='whether use parallel mode. Default: None') parser.add_argument( '--profile', default=None, type=str2bool, help='whether use profile analysis. Default: None') parser.add_argument( '--options', nargs='+', action=ActionDict, help='override some settings in the used config, the key-value pair' 'in xxx=yyy format will be merged into config file') parser.add_argument( '--epochs', default=None, type=int, help='train epochs.' 'Default: None') parser.add_argument( '--batch_size', default=None, type=int, help='batch_size of datasets.' 'Default: None') parser.add_argument( '--sink_mode', default=None, type=str2bool, help='whether use sink mode. ' 'Default: None') parser.add_argument( '--num_samples', default=None, type=int, help='number of datasets samples used.' 'Default: None') parser.add_argument( '--learning_rate', default=None, type=float, help='start learning rate' 'Default: None') parser.add_argument( '--lr_middle', default=None, type=float, help='middle learning rate' 'Default: None') parser.add_argument( '--lr_end', default=None, type=float, help='learning rate end' 'Default: None') parser.add_argument( '--decay_ratio', default=None, type=float, help='learning rate decay ratio' 'Default: None') parser.add_argument( '--embed_scale', default=None, type=float, help='embedding scale' 'Default: None') parser.add_argument( '--depth_scale', default=None, type=float, help='depth scale' 'Default: None') parser.add_argument( '--init_std', default=None, type=float, help='init std' 'Default: None') parser.add_argument( '--consumed_samples', default=None, type=int, help='consumed samples' 'Default: None') parser.add_argument( '--restore_ckpt_name_prefix', default=None, type=str, help="substitute checkpoint path") parser.add_argument( '--data_parallel', default=None, type=int, help='data_parallel' 'Default: None') parser.add_argument( '--model_parallel', default=None, type=int, help='model_parallel' 'Default: None') parser.add_argument( '--pipeline_stage', default=None, type=int, help='pipeline_stage' 'Default: None') parser.add_argument( '--micro_batch_num', default=None, type=int, help='micro_batch_num' 'Default: None') parser.add_argument( '--optimizer_weight_shard_size', default=None, type=int, help='optimizer_weight_shard_size' 'Default: None') parser.add_argument( '--total_steps', default=None, type=int, help='total_steps' 'Default: None') parser.add_argument( '--eval_epoch_interval', default=None, type=int, help='eval epoch interval' 'Default: None') parser.add_argument( '--use_flash_attention', default=None, type=str2bool, help='whether use flash attention. ' 'Default: None') args_, rest_args_ = parser.parse_known_args() rest_args_ = [i for item in rest_args_ for i in item.split("=")] if len(rest_args_) % 2 != 0: raise ValueError(f"input arg key-values are not in pair, please check input args. ") if args_.config is not None: args_.config = os.path.join(work_path, args_.config) config_ = MindFormerConfig(args_.config) if args_.device_id is not None: config_.context.device_id = args_.device_id if args_.device_target is not None: config_.context.device_target = args_.device_target if args_.mode is not None: config_.context.mode = args_.mode if args_.run_mode is not None: config_.run_mode = args_.run_mode if args_.do_eval is not None: config_.do_eval = args_.do_eval if args_.seed is not None: config_.seed = args_.seed if args_.use_parallel is not None: config_.use_parallel = args_.use_parallel if args_.load_checkpoint is not None: config_.load_checkpoint = args_.load_checkpoint if args_.src_strategy_path_or_dir is not None: config_.src_strategy_path_or_dir = args_.src_strategy_path_or_dir if args_.auto_trans_ckpt is not None: config_.auto_trans_ckpt = args_.auto_trans_ckpt if args_.only_save_strategy is not None: config_.only_save_strategy = args_.only_save_strategy if args_.resume_training is not None: config_.resume_training = args_.resume_training if args_.strategy_load_checkpoint is not None: if os.path.isdir(args_.strategy_load_checkpoint): ckpt_list = [os.path.join(args_.strategy_load_checkpoint, file) for file in os.listdir(args_.strategy_load_checkpoint) if file.endwith(".ckpt")] args_.strategy_load_checkpoint = ckpt_list[0] config_.parallel.strategy_ckpt_load_file = args_.strategy_load_checkpoint if args_.remote_save_url is not None: config_.remote_save_url = args_.remote_save_url if args_.profile is not None: config_.profile = args_.profile if args_.options is not None: config_.merge_from_dict(args_.options) assert config_.run_mode in ['train', 'eval', 'predict', 'finetune'], \ f"run status must be in {['train', 'eval', 'predict', 'finetune']}, but get {config_.run_mode}" if args_.train_dataset_dir: config_.train_dataset.data_loader.dataset_dir = args_.train_dataset_dir if args_.eval_dataset_dir: config_.eval_dataset.data_loader.dataset_dir = args_.eval_dataset_dir if config_.run_mode == 'predict': if args_.predict_data is None: logger.info("dataset by config is used as input_data.") elif os.path.isdir(args_.predict_data) and os.path.exists(args_.predict_data): predict_data = [os.path.join(root, file) for root, _, file_list in os.walk(os.path.join(args_.predict_data)) for file in file_list if file.endswith(".jpg") or file.endswith(".png") or file.endswith(".jpeg") or file.endswith(".JPEG") or file.endswith("bmp")] args_.predict_data = predict_data config_.input_data = args_.predict_data if args_.epochs is not None: config_.runner_config.epochs = args_.epochs if args_.batch_size is not None: config_.runner_config.batch_size = args_.batch_size if args_.sink_mode is not None: config_.runner_config.sink_mode = args_.sink_mode if args_.num_samples is not None: if config_.train_dataset and config_.train_dataset.data_loader: config_.train_dataset.data_loader.num_samples = args_.num_samples if config_.eval_dataset and config_.eval_dataset.data_loader: config_.eval_dataset.data_loader.num_samples = args_.num_samples if args_.learning_rate is not None: config_.lr_schedule.learning_rate = args_.learning_rate if args_.lr_middle is not None: config_.lr_schedule.lr_middle = args_.lr_middle if args_.lr_end is not None: config_.lr_schedule.lr_end = args_.lr_end if args_.decay_ratio is not None: config_.lr_schedule.decay_ratio = args_.decay_ratio if args_.embed_scale is not None: config_.model.model_config.embed_scale = args_.embed_scale if args_.depth_scale is not None: config_.model.model_config.depth_scale = args_.depth_scale if args_.init_std is not None: config_.model.model_config.init_std = args_.init_std if args_.consumed_samples is not None: config_.train_dataset.consumed_samples = args_.consumed_samples if args_.restore_ckpt_name_prefix is not None: config_.restore_ckpt_name_prefix = args_.restore_ckpt_name_prefix if args_.data_parallel is not None: config_.parallel_config.data_parallel = args_.data_parallel if args_.model_parallel is not None: config_.parallel_config.model_parallel = args_.model_parallel if args_.pipeline_stage is not None: config_.parallel_config.pipeline_stage = args_.pipeline_stage if args_.micro_batch_num is not None: config_.parallel_config.micro_batch_num = args_.micro_batch_num if args_.optimizer_weight_shard_size is not None: config_.parallel.optimizer_weight_shard_size = args_.optimizer_weight_shard_size if args_.total_steps is not None: config_.lr_schedule.total_steps = args_.total_steps if args_.eval_epoch_interval is not None: config_.eval_epoch_interval = args_.eval_epoch_interval if args_.use_flash_attention is not None: config_.model.model_config.use_flash_attention = args_.use_flash_attention while rest_args_: key = rest_args_.pop(0) value = rest_args_.pop(0) if not key.startswith("--"): raise ValueError("Custom config key need to start with --.") dists = key[2:].split(".") dist_config = config_ while len(dists) > 1: if dists[0] not in dist_config: raise ValueError(f"{dists[0]} is not a key of {dist_config}, please check input arg keys. ") dist_config = dist_config[dists.pop(0)] dist_config[dists.pop()] = parse_value(value) main(config_) pengcheng.mind-test/scripts/000077500000000000000000000000001471400134000164005ustar00rootroot00000000000000pengcheng.mind-test/scripts/.ipynb_checkpoints/000077500000000000000000000000001471400134000221715ustar00rootroot00000000000000pengcheng.mind-test/scripts/.ipynb_checkpoints/create_train-checkpoint.py000066400000000000000000000016211471400134000273300ustar00rootroot00000000000000import moxing as mox import mindspore def get_per_slice_token(filename): d = mindspore.dataset.MindDataset([filename]) print(d.get_dataset_size()) target_tokens = 2 remote_dst_directory = f"obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/scaling_law_dataset/WSD_train_{target_tokens}B" remote_src_directory = "obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/ct_exp1_all" t = target_tokens * 1e9 / 4096 / 132498 for part in range(100): for s in range(10): local_file = f"ct_660_part{part}of100_0of1_mindrecord{s}" remote_src_file = f"{remote_src_directory}/{local_file}" remote_dst_file = f"{remote_dst_directory}/{local_file}" mox.file.copy_parallel(remote_src_file, remote_dst_file) mox.file.copy_parallel(f"{remote_src_file}.db", f"{remote_dst_file}.db") t = t - 1 if t < 0: exit(0) pengcheng.mind-test/scripts/create_eval.py000077500000000000000000000023731471400134000212340ustar00rootroot00000000000000import moxing as mox import mindspore import copy from mindspore.mindrecord import FileWriter def get_per_slice_token(filename): d = mindspore.dataset.MindDataset([filename]) print(d.get_dataset_size()) target_tokens = 1 filename = "ct_660_part99of100_0of1_mindrecord9" src_file = f"obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/ct_exp1_all/{filename}" dst_file = f"/cache/{filename}" #mox.file.copy_parallel(src_file, dst_file) #mox.file.copy_parallel(f"{src_file}.db", f"{dst_file}.db") sample_size = 64 file_list = [dst_file] d = mindspore.dataset.MindDataset(file_list, num_samples=sample_size) d_iter = d.create_dict_iterator() writer = FileWriter(file_name=f"eval_{sample_size}.mindrecord") schema_json = {"input_ids": {"type": "int32", "shape": [-1]}} writer.add_schema(schema_json, "test_schema") for item in d_iter: new_item = copy.deepcopy(item) data = [{"input_ids": new_item['input_ids'][:-1].asnumpy()}] writer.write_raw_data(data) writer.commit() dataset_name = f"eval_{sample_size}.mindrecord" mox.file.copy_parallel(dataset_name, f"obs://sunzhb/scaling_law/dataset/660_dataset/{dataset_name}") mox.file.copy_parallel(f"{dataset_name}.db", f"obs://sunzhb/scaling_law/dataset/660_dataset/{dataset_name}.db") pengcheng.mind-test/scripts/create_train.py000077500000000000000000000016211471400134000214150ustar00rootroot00000000000000import moxing as mox import mindspore def get_per_slice_token(filename): d = mindspore.dataset.MindDataset([filename]) print(d.get_dataset_size()) target_tokens = 2 remote_dst_directory = f"obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/scaling_law_dataset/WSD_train_{target_tokens}B" remote_src_directory = "obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/ct_exp1_all" t = target_tokens * 1e9 / 4096 / 132498 for part in range(100): for s in range(10): local_file = f"ct_660_part{part}of100_0of1_mindrecord{s}" remote_src_file = f"{remote_src_directory}/{local_file}" remote_dst_file = f"{remote_dst_directory}/{local_file}" mox.file.copy_parallel(remote_src_file, remote_dst_file) mox.file.copy_parallel(f"{remote_src_file}.db", f"{remote_dst_file}.db") t = t - 1 if t < 0: exit(0) pengcheng.mind-test/scripts/examples/000077500000000000000000000000001471400134000202165ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/contrastive_language_image_pretrain/000077500000000000000000000000001471400134000274705ustar00rootroot00000000000000clip_vit_b_32_pretrain_on_flickr8k.sh000066400000000000000000000024001471400134000365540ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/contrastive_language_image_pretrain#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/contrastive_language_image_pretrain/clip_vit_b_32_pretrain_on_flick8k.sh" echo "The data setting could refer to ./docs/task_cards/contrastive_language_image_pretrain.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yamlpengcheng.mind-test/scripts/examples/image_classification/000077500000000000000000000000001471400134000243535ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/image_classification/swin_base_p4w7_eval_on_imagenet1k.sh000066400000000000000000000023531471400134000333550ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/swin_base_p4w7_eval_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode evalpengcheng.mind-test/scripts/examples/image_classification/swin_base_p4w7_finetune_on_imagenet1k.sh000066400000000000000000000023631471400134000342440ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/swin_base_p4w7_finetune_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode finetunepengcheng.mind-test/scripts/examples/image_classification/swin_base_p4w7_predict_on_imagenet1k.sh000066400000000000000000000026551471400134000340650ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/swin_base_p4w7_predict_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode predict \ --predict_data "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/ XFormer_for_mindspore/clip/sunflower.png"pengcheng.mind-test/scripts/examples/image_classification/swin_base_p4w7_train_on_imagenet1k.sh000066400000000000000000000023341471400134000335420ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/swin_base_p4w7_train_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yamlpengcheng.mind-test/scripts/examples/image_classification/vit_base_p16_eval_on_imagenet1k.sh000066400000000000000000000023461471400134000330060ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/vit_base_p16_eval_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode evalpengcheng.mind-test/scripts/examples/image_classification/vit_base_p16_finetune_on_imagenet1k.sh000066400000000000000000000023561471400134000336750ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/vit_base_p16_finetune_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode finetunepengcheng.mind-test/scripts/examples/image_classification/vit_base_p16_predict_on_imagenet1k.sh000066400000000000000000000026501471400134000335070ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/vit_base_p16_predict_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode predict \ --predict_data "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/ XFormer_for_mindspore/clip/sunflower.png"pengcheng.mind-test/scripts/examples/image_classification/vit_base_p16_train_on_imagenet1k.sh000066400000000000000000000023271471400134000331730ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/vit_base_p16_train_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yamlpengcheng.mind-test/scripts/examples/masked_image_modeling/000077500000000000000000000000001471400134000245025ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/masked_image_modeling/mae_vit_base_p16_predict_on_imagenet1k.sh000066400000000000000000000025401471400134000344560ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/masked_image_modeling/mae_vit_base_p16_predict_on_imagenet1k.sh" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/mae/run_mae_vit_base_p16_224_800ep.yaml --run_mode predict \ --predict_data "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/ XFormer_for_mindspore/clip/sunflower.png"pengcheng.mind-test/scripts/examples/masked_image_modeling/mae_vit_base_p16_train_on_imagenet1k.sh000066400000000000000000000022171471400134000341420ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/masked_image_modeling/mae_vit_base_p16_train_on_imagenet1k.sh" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/mae/run_mae_vit_base_p16_224_800ep.yamlpengcheng.mind-test/scripts/examples/masked_language_modeling/000077500000000000000000000000001471400134000252035ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/masked_language_modeling/bert_pretrain_distributed_gpu.sh000066400000000000000000000032401471400134000336530ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/masked_language_modeling/bert_pretrain_distributed.sh DEVICE_NUM HOST_FILE" echo "for example: scripts/examples/masked_language_modeling/bert_pretrain_distributed.sh 8 hostfile" echo "It is better to use absolute path." echo "==============================================================================================================" RANK_SIZE=$1 HOSTFILE=$2 mpirun --allow-run-as-root -n $RANK_SIZE --hostfile $HOSTFILE \ --output-filename run_distributed_train_bert \ -x NCCL_IB_HCA -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x NCCL_SOCKET_IFNAME -n $RANK_SIZE \ python run_mindformer.py --config ./configs/bert/run_bert_base_uncased.yaml \ --use_parallel True \ --run_mode train \ --device_target GPU > distribute_train_gpu_log.txt 2>&1 & pengcheng.mind-test/scripts/examples/question_answering/000077500000000000000000000000001471400134000241425ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/question_answering/qa_bert_base_uncased_eval_on_squad.sh000066400000000000000000000024201471400134000335050ustar00rootroot00000000000000#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/question_answering/qa_bert_base_uncased_eval_on_squad.sh" echo "The data setting could refer to ./docs/task_cards/question_answering.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode eval --load_checkpoint qa_bert_base_uncased_squad pengcheng.mind-test/scripts/examples/question_answering/qa_bert_base_uncased_finetune_on_squad.sh000066400000000000000000000023531471400134000344000ustar00rootroot00000000000000#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/question_answering/qa_bert_base_uncased_finetune_on_squad.sh" echo "The data setting could refer to ./docs/task_cards/question_answering.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode finetune pengcheng.mind-test/scripts/examples/question_answering/qa_bert_base_uncased_predict_on_squad.sh000066400000000000000000000026401471400134000342140ustar00rootroot00000000000000#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/question_answering/qa_bert_base_uncased_predict_on_squad.sh" echo "The data setting could refer to ./docs/task_cards/question_answering.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml \ --run_mode eval --load_checkpoint qa_bert_base_uncased_squad \ --predict_data "My name is Wolfgang and I live in Berlin - Where do I live?" pengcheng.mind-test/scripts/examples/text_classification/000077500000000000000000000000001471400134000242555ustar00rootroot00000000000000txtcls_bert_base_uncased_finetune_on_mnli.sh000066400000000000000000000023701471400134000351750ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/text_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/text_classification/txtcls_bert_base_uncased_finetune_on_mnli.sh" echo "The data setting could refer to ./docs/task_cards/text_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode finetune txtcls_bert_base_uncased_mnli_eval_on_mnli.sh000066400000000000000000000024451471400134000353310ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/text_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/text_classification/txtcls_bert_base_uncased_mnli_eval_on_mnli.sh" echo "The data setting could refer to ./docs/task_cards/text_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode eval --load_checkpoint txtcls_bert_base_uncased_mnli txtcls_bert_base_uncased_mnli_predict_on_mnli.sh000066400000000000000000000026741471400134000360400ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/text_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/text_classification/txtcls_bert_base_uncased_mnli_predict_on_mnli.sh" echo "The data setting could refer to ./docs/task_cards/text_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml \ --run_mode predict --load_checkpoint txtcls_bert_base_uncased_mnli \ --predict_data "The new rights are nice enough-Everyone really likes the newest benefits " pengcheng.mind-test/scripts/examples/token_classification/000077500000000000000000000000001471400134000244115ustar00rootroot00000000000000tokcls_bert_base_chinese_eval_on_cluener.sh000066400000000000000000000024501471400134000351120ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/token_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/token_classification/tokcls_bert_base_chinese_eval_on_cluener.sh" echo "The data setting could refer to ./docs/task_cards/token_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode eval --load_checkpoint tokcls_bert_base_chinese_cluener tokcls_bert_base_chinese_finetune_on_cluener.sh000066400000000000000000000023751471400134000360060ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/token_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/token_classification/tokcls_bert_base_chinese_finetune_on_cluener.sh" echo "The data setting could refer to ./docs/task_cards/token_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode finetune tokcls_bert_base_chinese_predict_on_cluener.sh000066400000000000000000000030331471400134000356130ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/token_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/token_classification/tokcls_bert_base_chinese_predict_on_cluener.sh" echo "The data setting could refer to ./docs/task_cards/token_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml \ --run_mode predict --load_checkpoint tokcls_bert_base_chinese_cluener \ --predict_data "结果上周六他们主场0：3惨败给了中游球队瓦拉多利德，近7个多月以来西甲首次输球。"pengcheng.mind-test/scripts/examples/translation/000077500000000000000000000000001471400134000225545ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/translation/t5_small_translation_finetune_on_wmt16.sh000066400000000000000000000031261471400134000326770ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/translation/t5_small_translation_finetune_on_wmt16.sh /your_wmt_path" echo "The data setting could refer to ./docs/model_cards/t5.md" echo "It is better to use absolute path." echo "Please make the src_max_length and the tgt_max_length in config/t5/run_t5_small_on_wmt16.yaml" echo "to be consistent with the seq_length and max_decoder_length in config/t5/run_t5_small_on_wmt16.yaml" echo "Or it will raise error like " echo "TransformerEncoderLayer x shape must be one of [[1, 1024, 512], [1024, 512]],but got [1, 32, 512]" echo "==============================================================================================================" data_path=$1 python run_mindformer.py --config ./configs/t5/run_t5_small_on_wmt16.yaml --train_dataset_dir $data_pathpengcheng.mind-test/scripts/examples/zero_shot_image_classification/000077500000000000000000000000001471400134000264475ustar00rootroot00000000000000clip_vit_b_32_eval_on_cifar100.sh000066400000000000000000000024121471400134000344310ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/zero_shot_image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/zero_shot_image_classification/clip_vit_b_32_eval_on_cifar100.sh" echo "The data setting could refer to ./docs/task_cards/zero_shot_image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml clip_vit_b_32_predict_on_cifar100.sh000066400000000000000000000024371471400134000351430ustar00rootroot00000000000000pengcheng.mind-test/scripts/examples/zero_shot_image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/zero_shot_image_classification/clip_vit_b_32_predict_on_cifar100.sh" echo "The data setting could refer to ./docs/task_cards/zero_shot_image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml --run_mode predictpengcheng.mind-test/scripts/ms_to_torch.py000066400000000000000000000055461471400134000213040ustar00rootroot00000000000000import torch import mindspore as ms import argparse import moxing as mox if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( '--obs_merged_ckpt', type=str, help='') parser.add_argument( '--obs_hf_ckpt_save_dir', type=str, help='') args_opt = parser.parse_args() mox.file.copy(args_opt.obs_merged_ckpt, "/cache/dst_ckpt/rank_0/checkpoint_0.ckpt") print("loading mindspore ckpt.................") ms_ckpt = ms.load_checkpoint('/cache/dst_ckpt/rank_0/checkpoint_0.ckpt') torch_dict = {} torch_dict['model.embed_tokens.weight'] = torch.Tensor(ms_ckpt['model.tok_embeddings.embedding_weight'].numpy()).to(dtype=torch.float16) for i in range(40): print(f"trans layer"+str(i)+".................") torch_dict['model.layers.'+str(i)+'.input_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.post_attention_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.ffn_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.gate_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w1.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.down_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w2.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.up_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w3.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.self_attn.o_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wo.weight'].numpy()).to(dtype=torch.float16) q=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wq.weight'].numpy()).to(dtype=torch.float16) k=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wk.weight'].numpy()).to(dtype=torch.float16) v=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wv.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.self_attn.W_pack.weight']=torch.cat([q,k,v],dim=0) torch_dict['model.norm.weight'] = torch.Tensor(ms_ckpt['model.norm_out.weight'].numpy()).to(dtype=torch.float16) torch_dict['lm_head.weight'] = torch.Tensor(ms_ckpt['lm_head.weight'].numpy()).to(dtype=torch.float16) print("saving pytorch ckpt.................") torch.save(torch_dict,'/cache/pytorch_model.bin') if not mox.file.exists(args_opt.obs_hf_ckpt_save_dir): mox.file.make_dirs(args_opt.obs_hf_ckpt_save_dir) mox.file.copy("/cache/pytorch_model.bin", args_opt.obs_hf_ckpt_save_dir + "/pytorch_model.bin")pengcheng.mind-test/scripts/run_distribute.sh000077500000000000000000000136061471400134000220070ustar00rootroot00000000000000#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 4 ] && [ $# != 5 ] && [ $# != 6 ] then echo "Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] For Multiple Devices In Single Machine" echo "Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] [RANK_SIZE] For Multiple Devices In Multiple Machines" echo "Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] predict [PREDICT_DATA] For Multiple Devices Predict In Single Machine" echo "Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] predict [RANK_SIZE] [PREDICT_DATA] For Multiple Devices Predict In Multiple Machines" exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } PATH1=$(check_real_path $1) CONFIG_FILE=$(check_real_path $2) DEVICE_RANGE=$3 RUN_STATUS=$4 DEVICE_RANGE_LEN=${#DEVICE_RANGE} DEVICE_RANGE=${DEVICE_RANGE:1:DEVICE_RANGE_LEN-2} PREFIX=${DEVICE_RANGE%%","*} INDEX=${#PREFIX} START_DEVICE=${DEVICE_RANGE:0:INDEX} END_DEVICE=${DEVICE_RANGE:INDEX+1:DEVICE_RANGE_LEN-INDEX} if [ ! -f $PATH1 ] then echo "error: RANK_TABLE_FILE=$PATH1 is not a file" exit 1 fi if [ ! -f $CONFIG_FILE ] then echo "error: config_path=$CONFIG_FILE is not a file" exit 1 fi if [[ ! $START_DEVICE =~ ^[0-9]+$ ]]; then echo "error: start_device=$START_DEVICE is not a number" exit 1 fi if [[ ! $END_DEVICE =~ ^[0-9]+$ ]]; then echo "error: end_device=$END_DEVICE is not a number" exit 1 fi ulimit -u unlimited if [ $RUN_STATUS != "predict" ] then if [ $# == 4 ] then export RANK_SIZE=$(($END_DEVICE - $START_DEVICE)) else export RANK_SIZE=$5 fi else if [ $# == 5 ] then export RANK_SIZE=$(($END_DEVICE - $START_DEVICE)) PREDICT_DATA=$5 else export RANK_SIZE=$5 PREDICT_DATA=$6 fi fi export RANK_TABLE_FILE=$PATH1 export LOCAL_DEFAULT_PATH="../../output" export CHECKPOINT_DOWNLOAD_FOLDER="../../checkpoint_download" export CHECKPOINT_SAVE_FOLDER="../../checkpoint_save" shopt -s extglob if [ $RUN_STATUS != "predict" ] then if [ $# == 4 ] then for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export DEVICE_ID=${i} export RANK_ID=$((i-START_DEVICE)) rm -rf ./mf_parallel$i/!(rank_*) test -d ./mf_parallel$i || mkdir ./mf_parallel$i cp ../*.py ./mf_parallel$i cp -r ../configs ./mf_parallel$i cp -r ../mindformers ./mf_parallel$i cd ./mf_parallel$i || exit echo "start training for rank $RANK_ID, device $DEVICE_ID" env > env.log mkdir -p ../../output/log/rank_$RANK_ID python run_mindformer.py --config=$CONFIG_FILE --use_parallel=True --run_mode=$RUN_STATUS \ &> ../../output/log/rank_$RANK_ID/mindformer.log & cd .. done else for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export RANK_ID=${i} export DEVICE_ID=$((i-START_DEVICE)) rm -rf ./mf_parallel$i/!(rank_*) test -d ./mf_parallel$i || mkdir ./mf_parallel$i cp ../*.py ./mf_parallel$i cp -r ../configs ./mf_parallel$i cp -r ../mindformers ./mf_parallel$i cd ./mf_parallel$i || exit echo "start training for rank $RANK_ID, device $DEVICE_ID" env > env.log mkdir -p ../../output/log/rank_$RANK_ID python run_mindformer.py --config=$CONFIG_FILE --use_parallel=True --run_mode=$RUN_STATUS \ &> ../../output/log/rank_$RANK_ID/mindformer.log & cd .. done fi else if [ $# == 5 ] then for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export DEVICE_ID=${i} export RANK_ID=$((i-START_DEVICE)) rm -rf ./mf_parallel$i/!(rank_*) test -d ./mf_parallel$i || mkdir ./mf_parallel$i cp ../*.py ./mf_parallel$i cp -r ../configs ./mf_parallel$i cp -r ../mindformers ./mf_parallel$i cd ./mf_parallel$i || exit echo "start training for rank $RANK_ID, device $DEVICE_ID" env > env.log mkdir -p ../../output/log/rank_$RANK_ID python run_mindformer.py --config=$CONFIG_FILE --use_parallel=True --run_mode=$RUN_STATUS \ --predict_data "$PREDICT_DATA" &> ../../output/log/rank_$RANK_ID/mindformer.log & cd .. done else for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export RANK_ID=${i} export DEVICE_ID=$((i-START_DEVICE)) rm -rf ./mf_parallel$i/!(rank_*) test -d ./mf_parallel$i || mkdir ./mf_parallel$i cp ../*.py ./mf_parallel$i cp -r ../configs ./mf_parallel$i cp -r ../mindformers ./mf_parallel$i cd ./mf_parallel$i || exit echo "start training for rank $RANK_ID, device $DEVICE_ID" env > env.log mkdir -p ../../output/log/rank_$RANK_ID python run_mindformer.py --config=$CONFIG_FILE --use_parallel=True --run_mode=$RUN_STATUS \ --predict_data "$PREDICT_DATA" &> ../../output/log/rank_$RANK_ID/mindformer.log & cd .. done fi fi shopt -u extglob #cd ./pretrain_parallel${START_DEVICE} || exit #tail -f mindformer.log # if you want kill current job, you can use as follow: # kill -9 $(ps aux | grep "python run_mindformer.py" | grep -v grep | awk '{print $2}') pengcheng.mind-test/scripts/run_standalone.sh000077500000000000000000000031121471400134000217500ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 3 ] then echo "Usage Help: bash run_standalone.sh [CONFIG_PATH] [DEVICE_ID] [RUN_STATUS] " exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } CONFIG_FILE=$(check_real_path $1) DEVICE_ID=$2 RUN_STATUS=$3 if [ ! -f $CONFIG_FILE ] then echo "error: config_path=$CONFIG_FILE is not a file" exit 1 fi ulimit -u unlimited export DEVICE_ID=${DEVICE_ID} rm -rf ./mf_standalone mkdir ./mf_standalone cp ../*.py ./mf_standalone cp -r ../configs ./mf_standalone cp -r ../mindformers ./mf_standalone cd ./mf_standalone || exit echo "start training for device $DEVICE_ID" env > env.log python run_mindformer.py --config=$CONFIG_FILE --use_parallel=False --run_mode=$RUN_STATUS &> mindformer.log & cd .. # if you want kill current job, you can use as follow: # kill -9 $(ps aux | grep "python run_mindformer.py" | grep -v grep | awk '{print $2}')pengcheng.mind-test/setup.py000077500000000000000000000132421471400134000164300ustar00rootroot00000000000000#!/usr/bin/env python3 # encoding: utf-8 # Copyright 2022 Huawei Technologies Co., Ltd.All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== """Setup.""" import sys import os import shutil import stat import platform from importlib import import_module from setuptools import setup, find_packages from setuptools.command.egg_info import egg_info from setuptools.command.build_py import build_py from setuptools.command.install import install def get_configs_content(): pwd = os.path.dirname(os.path.realpath(__file__)) configs = [os.path.join(root, file) for root, _, file_list in os.walk(os.path.join(pwd, "configs")) for file in file_list if file.endswith('.yaml') or file.endswith(".yml") or file.endswith(".md")] return configs def get_readme_content(): pwd = os.path.dirname(os.path.realpath(__file__)) with open(os.path.join(pwd, 'README.md'), encoding='UTF-8') as f: return f.read() def get_platform(): """ Get platform name. Returns: str, platform name in lowercase. """ return platform.system().strip().lower() def get_description(): """ Get description. Returns: str, wheel package description. """ os_info = get_platform() cpu_info = platform.machine().strip() return 'mindformers platform: %s, cpu: %s' % (os_info, cpu_info) def get_install_requires(): """ Get install requirements. Returns: list, list of dependent packages. """ with open('requirements.txt') as file: return file.read().strip().splitlines() def update_permissions(path): """ Update permissions. Args: path (str): Target directory path. """ for dirpath, dirnames, filenames in os.walk(path): for dirname in dirnames: dir_fullpath = os.path.join(dirpath, dirname) os.chmod(dir_fullpath, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC | stat.S_IRGRP | stat.S_IXGRP) for filename in filenames: file_fullpath = os.path.join(dirpath, filename) os.chmod(file_fullpath, stat.S_IREAD) class EggInfo(egg_info): """Egg info.""" def run(self): egg_info_dir = os.path.join(os.path.dirname(__file__), 'mindformers.egg-info') shutil.rmtree(egg_info_dir, ignore_errors=True) super().run() update_permissions(egg_info_dir) class BuildPy(build_py): """Build py files.""" def run(self): mindspore_transformer_lib_dir = os.path.join(os.path.dirname(__file__), 'build', 'lib', 'mindformers') shutil.rmtree(mindspore_transformer_lib_dir, ignore_errors=True) super().run() update_permissions(mindspore_transformer_lib_dir) class Install(install): """Install.""" def run(self): super().run() if sys.argv[-1] == 'install': pip = import_module('pip') mindspore_transformer_dir = os.path.join(os.path.dirname(pip.__path__[0]), 'mindformers') update_permissions(mindspore_transformer_dir) if __name__ == '__main__': version_info = sys.version_info if (version_info.major, version_info.minor) < (3, 7): sys.stderr.write('Python version should be at least 3.7\r\n') sys.exit(1) setup( name='mindformers', version='2023.09.dev', author='The MindSpore Authors', author_email='contact@mindspore.cn', url='https://www.mindspore.cn', download_url='https://gitee.com/mindspore/mindformers/tags', project_urls={ 'Sources': 'https://gitee.com/mindspore/mindformers', 'Issue Tracker': 'https://gitee.com/mindspore/mindformers/issues', }, description=get_description(), long_description=get_readme_content(), long_description_content_type="text/markdown", test_suite="tests", packages=find_packages(exclude=["*tests*"]), platforms=[get_platform()], include_package_data=True, package_data={'mindformers': get_configs_content()}, cmdclass={ 'egg_info': EggInfo, 'build_py': BuildPy, 'install': Install, }, python_requires='>=3.7', install_requires=get_install_requires(), classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Console', 'Environment :: Web Environment', 'Intended Audience :: Science/Research', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python :: 3 :: Only', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Topic :: Scientific/Engineering', 'Topic :: Scientific/Engineering :: Artificial Intelligence', 'Topic :: Software Development', 'Topic :: Software Development :: Libraries', 'Topic :: Software Development :: Libraries :: Python Modules', ], license='Apache 2.0', keywords='mindformers', ) pengcheng.mind-test/start.sh000077500000000000000000000004521471400134000164060ustar00rootroot00000000000000pkill python export RANK_TABLE_FILE=/home/ma-user/work/rank_table_128.json export RANK_ID=0 export DEVICE_ID=0 export RANK_SIZE=128 export GLOG_v=2 python run_mindformer.py --config configs/official_training/run_minicpm_1e3_128.yaml --train_dataset=/home/ma-user/work/dataset/eval_2048.mindrecord pengcheng.mind-test/tests/000077500000000000000000000000001471400134000160535ustar00rootroot00000000000000pengcheng.mind-test/tests/__init__.py000066400000000000000000000000001471400134000201520ustar00rootroot00000000000000pengcheng.mind-test/tests/st/000077500000000000000000000000001471400134000165015ustar00rootroot00000000000000pengcheng.mind-test/tests/st/__init__.py000066400000000000000000000012451471400134000206140ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ST.""" pengcheng.mind-test/tests/st/test_build.py000066400000000000000000000265741471400134000212270ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the interface used for mindformers. How to run this: python tests/ut/test_build.py """ import os from dataclasses import dataclass from typing import Callable import pytest from mindspore.nn import AdamWeightDecay, CosineDecayLR, Accuracy,\ TrainOneStepWithLossScaleCell, L1Loss from mindspore.train.callback import LossMonitor, TimeMonitor from mindspore import Parameter, Tensor from mindformers.tools import logger from mindformers.tools import MindFormerConfig, MindFormerRegister, MindFormerModuleType from mindformers.core import build_lr, build_optim, build_loss, build_metric from mindformers.trainer import build_trainer, BaseTrainer from mindformers.models import build_model, BaseModel, build_processor from mindformers.dataset import build_dataset, build_sampler, check_dataset_config, \ build_dataset_loader, build_mask, build_transforms, BaseDataset from mindformers.pipeline import build_pipeline from mindformers.wrapper import build_wrapper path = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) yaml_path = os.path.join(path, 'tests', 'st', 'test_build.yaml') all_config = MindFormerConfig(yaml_path) @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class TestDataLoader: """Test DataLoader API For Register.""" def __init__(self, dataset_dir=None): self.dataset_dir = dataset_dir @MindFormerRegister.register(MindFormerModuleType.DATASET_SAMPLER) class TestSampler: """Test Sampler API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class TestTransforms1: """Test Transforms API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class TestTransforms2: """Test Transforms API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.MASK_POLICY) class TestModelMask: """Test Model Mask API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.MODULES) class TestAttentionModule: """Test Module API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.DATASET) class TestDataset(BaseDataset): """Test Dataset API For Register.""" def __init__(self, dataset_config: dict = None): super(TestDataset, self).__init__(dataset_config) self.config = dataset_config def __new__(cls, dataset_config: dict = None): if dataset_config is not None: build_dataset_loader(dataset_config.data_loader) logger.info("Test Build DataLoader Success") build_sampler(dataset_config.sampler) logger.info("Test Build Sampler Success") build_transforms(dataset_config.transforms) logger.info("Test Build Transforms Success") build_mask(dataset_config.mask_policy) logger.info("Test Build Mask Policy Success") else: build_dataset_loader(class_name='TestDataLoader') logger.info("Test Build DataLoader Success") build_sampler(class_name='TestSampler') logger.info("Test Build Sampler Success") build_transforms(class_name='TestTransforms1') logger.info("Test Build Transforms Success") build_mask(class_name='TestModelMask') logger.info("Test Build Mask Policy Success") @MindFormerRegister.register(MindFormerModuleType.MODELS) class TestModel(BaseModel): """Test Model API For Register.""" def __init__(self, config: dict = None): super(TestModel, self).__init__(config) self.model_config = config self.params = Parameter(Tensor([0.1])) @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class TestTokenizer: """Test Tokenizer API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.CONFIG) @dataclass class TestTextConfig: """Test TextConfig API For Register.""" seq_length: int = 12 @MindFormerRegister.register(MindFormerModuleType.CONFIG) @dataclass class TestVisionConfig: """Test VisionConfig API For Register.""" seq_length: int = 12 @MindFormerRegister.register(MindFormerModuleType.CONFIG) @dataclass class TestModelConfig: """Test ModelConfig API For Register.""" batch_size: int = 2 embed_dim: int = 768 text_config: Callable = TestTextConfig vision_config: Callable = TestVisionConfig @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class TestAdamWeightDecay(AdamWeightDecay): """Test AdamWeightDecay API For Register.""" def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0): super(TestAdamWeightDecay, self).__init__(params, learning_rate=learning_rate, beta1=beta1, beta2=beta2, eps=eps, weight_decay=weight_decay) self.param = params @MindFormerRegister.register(MindFormerModuleType.LR) class TestCosineDecayLR(CosineDecayLR): """Test CosineDecayLR API For Register.""" def __init__(self, min_lr, max_lr, decay_steps): super(TestCosineDecayLR, self).__init__(min_lr, max_lr, decay_steps) self.lr = max_lr @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class TestTrainOneStepWithLossScaleCell(TrainOneStepWithLossScaleCell): """Test TrainOneStepWithLossScaleCell API For Register.""" def __init__(self, network, optimizer, scale_sense): super(TestTrainOneStepWithLossScaleCell, self).__init__(network, optimizer, scale_sense) self.scale_sense = scale_sense @MindFormerRegister.register(MindFormerModuleType.METRIC) class TestAccuracy(Accuracy): """Test Accuracy API For Register.""" def __init__(self, eval_type='classification'): super(TestAccuracy, self).__init__(eval_type) self.eval = eval_type @MindFormerRegister.register(MindFormerModuleType.LOSS) class TestL1Loss(L1Loss): """Test L1Loss API For Register.""" def __init__(self, reduction='mean'): super(TestL1Loss, self).__init__(reduction) self.reduction = reduction @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class TestLLossMonitor(LossMonitor): """Test LossMonitor API For Register.""" def __init__(self, per_print_times=1): super(TestLLossMonitor, self).__init__(per_print_times) self.print = per_print_times @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class TestTimeMonitor(TimeMonitor): """Test TimeMonitor API For Register.""" def __init__(self, data_size=1): super(TestTimeMonitor, self).__init__(data_size) self.data_size = data_size @MindFormerRegister.register(MindFormerModuleType.PIPELINE) class TestPipeline: """Test Pipeline API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class TestProcessor: """Test Processor API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.TRAINER) class TestTaskTrainer(BaseTrainer): """Test TimeMonitor API For Register.""" def __init__(self, model_name='vit'): super(TestTaskTrainer, self).__init__(model_name=model_name) @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_build_from_config(): """ Feature: Build API from config Description: Test build function to instance API from config Expectation: TypeError """ # build dataset check_dataset_config(all_config) build_dataset(all_config.train_dataset_task) logger.info("Test Build Dataset Success") # build model model = build_model(all_config.model) logger.info("Test Build Network Success") # build lr lr = build_lr(all_config.lr_schedule) logger.info("Test Build LR Success") # build optimizer if lr is not None: optimizer = build_optim(all_config.optimizer, default_args={"params": model.trainable_params(), "learning_rate": lr}) else: optimizer = build_optim(all_config.optimizer, default_args={"params": model.trainable_params()}) logger.info("Test Build Optimizer Success") # build wrapper build_wrapper(all_config.runner_wrapper, default_args={"network": model, "optimizer": optimizer}) logger.info("Test Build Wrapper Success") build_loss(all_config.loss) logger.info("Test Build Loss Success") build_metric(all_config.metric) logger.info("Test Build Metric Success") build_processor(all_config.processor) logger.info("Test Build Processor Success") # build trainer build_trainer(all_config.trainer) logger.info("Test Build Trainer Success") # build pipeline build_pipeline(all_config.pipeline) logger.info("Test Build Pipeline Success") @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_build_from_class_name(): """ Feature: Build API from class name Description: Test build function to instance API from class name Expectation: TypeError """ # build dataset build_dataset(class_name='TestDataset') logger.info("Test Build Dataset Success") # build model model = build_model(class_name='TestModel') logger.info("Test Build Network Success") # build lr lr = build_lr(class_name='TestCosineDecayLR', min_lr=0., max_lr=0.001, decay_steps=1000) logger.info("Test Build LR Success") # build optimizer if lr is not None: optimizer = build_optim(class_name='TestAdamWeightDecay', params=model.trainable_params(), learning_rate=lr) else: optimizer = build_optim(class_name='TestAdamWeightDecay', params=model.trainable_params()) logger.info("Test Build Optimizer Success") # build loss build_loss(class_name='TestL1Loss') logger.info("Test Build Loss Success") # build metric build_metric(class_name='TestAccuracy') logger.info("Test Build Metric Success") # build wrapper scale_sense = Tensor(1.0) build_wrapper(class_name='TestTrainOneStepWithLossScaleCell', network=model, optimizer=optimizer, scale_sense=scale_sense) logger.info("Test Build Wrapper Success") # build processor build_processor(class_name='TestProcessor') logger.info("Test Build Processor Success") # build trainer build_trainer(class_name='TestTaskTrainer') logger.info("Test Build Trainer Success") # build pipeline build_pipeline(class_name='TestPipeline') logger.info("Test Build Pipeline Success") pengcheng.mind-test/tests/st/test_build.yaml000066400000000000000000000031351471400134000215250ustar00rootroot00000000000000model: model_config: type: TestModelConfig text_config: type: TestTextConfig vision_config: type: TestVisionConfig arch: type: TestModel loss: type: TestL1Loss reduction: 'mean' metric: type: TestAccuracy eval_type: 'classification' train_dataset: &train_dataset data_loader: type: TestDataLoader dataset_dir: "/dataset/imageNet-1k/train" transforms: - type: TestTransforms1 - type: TestTransforms1 sampler: type: TestSampler mask_policy: type: TestModelMask input_columns: ["image"] output_columns: [ "image", "mask"] column_order: [ "image", "mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 128 repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: TestDataset dataset_config: *train_dataset runner_wrapper: type: TestTrainOneStepWithLossScaleCell scale_sense: 1024 # optimizer optimizer: type: TestAdamWeightDecay beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 # lr sechdule lr_schedule: type: TestCosineDecayLR max_lr: 0.00015 min_lr: 0. decay_steps: 100 runner_config: epoch: 800 batch_size: 64 image_size: 224 sink_mode: True sink_size: 1 initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 load_checkpoint: callbacks: - type: TestLLossMonitor per_print_times: 10 - type: TestTimeMonitor data_size: 100 do_train: True do_eval: False do_predict: False trainer: type: TestTaskTrainer model_name: 'mae' pipeline: type: TestPipeline processor: type: TestProcessorpengcheng.mind-test/tests/st/test_do_eval.py000066400000000000000000000102061471400134000215220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing evaluating on training featrue used for mindformers. Uses gpt2 model as example How to run this: pytest tests/st/test_do_eval.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers import GPT2LMHeadModel, GPT2Config from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 65 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 64 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestDoEvalMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments( batch_size=4, num_train_epochs=1, do_eval=True, eval_step_interval=-1, eval_epoch_interval=2, ) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "input_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "input_mask"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = GPT2Config(num_layers=2, hidden_size=32, num_heads=2, seq_length=64) model = GPT2LMHeadModel(model_config) self.task_trainer = Trainer(task='text_generation', model=model, model_name='gpt2', args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) print(self.task_trainer.config) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() @pytest.mark.run(order=2) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.finetune(finetune_checkpoint=True) @pytest.mark.run(order=3) def test_config_change_train(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.do_eval = False self.task_trainer.config.eval_step_interval = 4 self.task_trainer.config.eval_epoch_interval = -1 self.task_trainer.train(do_eval=True) @pytest.mark.run(order=4) def test_config_change_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.finetune(finetune_checkpoint=True, do_eval=True) pengcheng.mind-test/tests/st/test_mindformer_book.py000066400000000000000000000061021471400134000232650ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of MindFormerBook class How to run this: windows: pytest .\\tests\\st\\test_mindformer_book.py linux: pytest ./tests/st/test_mindformer_book.py """ import pytest from mindformers import MindFormerBook from mindformers.tools import logger @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_mindformer_book_show(): """ Feature: mindformers book class, show content Description: Test to show content in MindFormerBook Expectation: success """ MindFormerBook.show_trainer_support_task_list() MindFormerBook.show_pipeline_support_task_list() MindFormerBook.show_model_support_list() MindFormerBook.show_model_ckpt_url_list() MindFormerBook.show_model_config_url_list() MindFormerBook.show_project_path() MindFormerBook.show_default_checkpoint_download_folder() MindFormerBook.show_default_checkpoint_save_folder() MindFormerBook.show_model_config_to_name() @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_mindformer_book_get(): """ Feature: mindformers book class, get content Description: Test to get content of MindFormerBook Expectation: success """ logger.info(MindFormerBook.get_trainer_support_task_list()) logger.info(MindFormerBook.get_pipeline_support_task_list()) logger.info(MindFormerBook.get_model_support_list()) logger.info(MindFormerBook.get_model_ckpt_url_list()) logger.info(MindFormerBook.get_model_config_url_list()) logger.info(MindFormerBook.get_project_path()) logger.info(MindFormerBook.get_default_checkpoint_download_folder()) logger.info(MindFormerBook.get_default_checkpoint_save_folder()) logger.info(MindFormerBook.get_model_config_to_name()) # default path is settable @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_mindformer_book_set(): """ Feature: mindformers book class, get content Description: Test to set content for MindFormerBook Expectation: success or TypeError """ path = MindFormerBook.get_project_path() MindFormerBook.set_default_checkpoint_download_folder(path) MindFormerBook.set_default_checkpoint_save_folder(path) logger.info(MindFormerBook.get_default_checkpoint_download_folder()) logger.info(MindFormerBook.get_default_checkpoint_save_folder()) MindFormerBook.set_model_config_to_name("ModelConfig", "ModelName") pengcheng.mind-test/tests/st/test_model/000077500000000000000000000000001471400134000206405ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/__init__.py000066400000000000000000000012551471400134000227540ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test model.""" pengcheng.mind-test/tests/st/test_model/test_bert_model/000077500000000000000000000000001471400134000240135ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_bert_model/__init__.py000066400000000000000000000012541471400134000261260ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test bert.""" pengcheng.mind-test/tests/st/test_model/test_bert_model/test_autoclass.py000066400000000000000000000072541471400134000274320ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bert interface used for mindformers. How to run this: pytest tests/st/test_model/test_bert_model/test_auto_class.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor ms.set_context(mode=0) class TestBertAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['bert_base_uncased'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_auto_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-test/tests/st/test_model/test_bert_model/test_pipeline.py000066400000000000000000000027151471400134000272360ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bert interface used for mindformers. How to run this: pytest tests/st/test_model/test_bert_model/test_pipeline.py """ import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestBertPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_list = ['bert_base_uncased'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_list: task_pipeline = pipeline(task='fill_mask', model=model_type, max_length=128, padding='max_length') task_pipeline([" Hello I am a [MASK] model.",]) pengcheng.mind-test/tests/st/test_model/test_bert_model/test_trainer.py000066400000000000000000000072601471400134000270750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bert interface used for mindformers. How to run this: pytest tests/st/test_model/test_bert_model/test_gpt_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models import BertConfig, BertForPreTraining from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator(): """dataset generator""" data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32) input_mask = np.ones_like(data) token_type_id = np.zeros_like(data) next_sentence_lables = np.array([1]).astype(np.int32) masked_lm_positions = np.array([1, 2]).astype(np.int32) masked_lm_ids = np.array([1, 2]).astype(np.int32) masked_lm_weights = np.ones_like(masked_lm_ids) train_data = (data, input_mask, token_type_id, next_sentence_lables, masked_lm_positions, masked_lm_ids, masked_lm_weights) for _ in range(4): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestBertTrainerMethod: """A test class for testing trainer.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=1, num_train_epochs=1) train_dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids", "next_sentence_labels", "masked_lm_positions", "masked_lm_ids", "masked_lm_weights"]) train_dataset = train_dataset.batch(batch_size=1) model_config = BertConfig(batch_size=1, num_hidden_layers=2) model = BertForPreTraining(model_config) self.task_trainer = Trainer(task='fill_mask', model=model, args=args, train_dataset=train_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() @pytest.mark.run(order=2) def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ input_data = [" Hello I am a [MASK] model.",] self.task_trainer.predict(input_data=input_data) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ pengcheng.mind-test/tests/st/test_model/test_blip2_model/000077500000000000000000000000001471400134000240675ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_blip2_model/__init__.py000066400000000000000000000012551471400134000262030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test blip2.""" pengcheng.mind-test/tests/st/test_model/test_blip2_model/test_autoclass.py000066400000000000000000000121731471400134000275020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the blip2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_blip2_model/test_autoclass.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor ms.set_context(mode=0, device_id=7) class TestBlip2AutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['blip2_stage1_vit_g'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') class TestBlip2SecondStageAutoClassMethod: """A test class for testing Model classes of Blip2 second stage""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['blip2_stage2_vit_g_llama_7b', 'itt_blip2_stage2_vit_g_llama_7b'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-test/tests/st/test_model/test_blip2_model/test_pipeline.py000066400000000000000000000047351471400134000273160ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the blip2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_blip2_model/test_pipeline.py """ from PIL import Image import mindspore as ms from mindformers import pipeline, BasePipeline ms.set_context(mode=0, device_id=7) class TestBlip2PipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['blip2_stage1_classification'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: classifier = pipeline("zero_shot_image_classification", model=model_type, candidate_labels=["sunflower", "tree", "dog", "cat", "toy"], hypothesis_template="A picture of {}.") image = Image.new('RGB', (512, 512)) classifier(image) class TestBlip2SecondStagePipelineMethod: """A test class for testing pipeline of ImageToTextGeneration.""" def setup_method(self): """setup method.""" self.test_llm_list = ['itt_blip2_stage2_vit_g_llama_7b'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: generator = pipeline("image_to_text_generation", model=model_type, max_length=32) assert isinstance(generator, BasePipeline) image = Image.new('RGB', (448, 448)) generator(image, hypothesis_template='a picture of ') pengcheng.mind-test/tests/st/test_model/test_blip2_model/test_trainer.py000066400000000000000000000243711471400134000271530ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the blip2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_blip2_model/test_trainer.py """ import numpy as np # import pytest from PIL import Image import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers import Trainer, TrainingArguments, Blip2Llama, LlamaTokenizer from mindformers.models.bert.bert_tokenizer import BertTokenizer from mindformers.models.blip2 import Blip2Qformer, Blip2Classifier, Blip2Config, Blip2ImageProcessor from mindformers.models.blip2.blip2_llama import Blip2ImageToTextGeneration from mindformers.models.blip2.qformer import QFormerConfig from mindformers.models.llama.llama_config import LlamaConfig from mindformers.models.vit.vit import ViTConfig ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 8 images = np.random.randint(low=0, high=255, size=(3, 32, 32)).astype(np.float32) input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(4): yield images, input_ids def generator_eval(): """eval dataset generator""" seq_len = 8 images = np.random.randint(low=0, high=255, size=(3, 32, 32)).astype(np.float32) input_ids = np.random.randint(low=0, high=15, size=(seq_len)).astype(np.int32) for _ in range(4): yield images, input_ids class TestBlip2TrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" train_dataset = GeneratorDataset(generator_train, column_names=["image", "input_ids"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["image", "input_ids"]) self.train_dataset = train_dataset.batch(batch_size=2) self.eval_dataset = eval_dataset.batch(batch_size=2) qformer_config = QFormerConfig(num_hidden_layers=1, num_attention_heads=2, hidden_size=2, head_embed_dim=2, encoder_width=2, intermediate_size=2, vocab_size=32, max_position_embeddings=80) vision_config = ViTConfig(image_size=32, hidden_size=2, num_hidden_layers=1, num_attention_heads=2, intermediate_size=2, num_labels=10) model_config = Blip2Config(vision_config=vision_config, qformer_config=qformer_config) # for train and evaluation self.blip2_qformer = Blip2Qformer(model_config) # for prediction self.blip2_classifier = Blip2Classifier(model_config) self.image_processor = Blip2ImageProcessor(image_size=32) self.tokenizer = BertTokenizer.from_pretrained('bert_base_uncased') self.training_args = TrainingArguments(batch_size=2, num_train_epochs=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ task_trainer = Trainer(task='contrastive_language_image_pretrain', model=self.blip2_qformer, model_name='blip2_stage1_vit_g', args=self.training_args, tokenizer=self.tokenizer, train_dataset=self.train_dataset, eval_dataset=self.eval_dataset) task_trainer.train() def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ task_evaluater = Trainer(task='image_to_text_retrieval', model=self.blip2_qformer, model_name='blip2_stage1_vit_g', tokenizer=self.tokenizer, train_dataset=self.train_dataset, eval_dataset=self.eval_dataset) task_evaluater.evaluate(k_test=1, add_extra_itm_score=False) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ task_predictor = Trainer(task='zero_shot_image_classification', model=self.blip2_classifier, model_name='blip2_stage1_classification', tokenizer=self.tokenizer, image_processor=self.image_processor, candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) # 加载输入，一张图片 input_data = Image.new('RGB', (32, 32)) # 加载指定的权重以完成推理 task_predictor.predict(input_data=input_data) class TestBlip2SecondStageTrainerMethod: """A test class for testing blip2 second stage trainers.""" def setup_method(self): """init task trainer.""" train_batch_size = 2 train_dataset = GeneratorDataset(generator_train, column_names=["image", "input_ids"]) self.train_dataset = train_dataset.batch(batch_size=train_batch_size) qformer_config = QFormerConfig(num_hidden_layers=1, num_attention_heads=2, hidden_size=2, head_embed_dim=2, encoder_width=2, intermediate_size=2, vocab_size=50, max_position_embeddings=512) vision_config = ViTConfig(image_size=32, hidden_size=2, num_hidden_layers=1, num_attention_heads=2, intermediate_size=2, num_labels=10, checkpoint_name_or_path="") text_config_for_trainer = LlamaConfig(seq_length=39, hidden_size=8, num_layers=2, num_heads=2, checkpoint_name_or_path="") model_config_for_trainer = Blip2Config(batch_size=train_batch_size, vision_config=vision_config, qformer_config=qformer_config, text_config=text_config_for_trainer, checkpoint_name_or_path="", max_txt_len=7) text_config_for_generator = LlamaConfig(seq_length=128, hidden_size=8, num_layers=2, num_heads=2, checkpoint_name_or_path="", use_past=True) model_config_for_generator = Blip2Config(batch_size=1, vision_config=vision_config, qformer_config=qformer_config, text_config=text_config_for_generator, checkpoint_name_or_path="", max_txt_len=8) self.blip2_llm = Blip2Llama(model_config_for_trainer) # for prediction self.blip2_generator = Blip2ImageToTextGeneration(model_config_for_generator) self.image_processor = Blip2ImageProcessor(image_size=32) self.tokenizer = LlamaTokenizer.from_pretrained('llama_7b') self.training_args = TrainingArguments(batch_size=4, num_train_epochs=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ task_trainer = Trainer(task='contrastive_language_image_pretrain', model=self.blip2_llm, model_name='blip2_stage2_vit_g_llama_7b', args=self.training_args, tokenizer=self.tokenizer, train_dataset=self.train_dataset) task_trainer.train(train_checkpoint='') def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ task_predictor = Trainer(task='image_to_text_generation', model=self.blip2_generator, model_name='itt_blip2_stage2_vit_g_llama_7b', tokenizer=self.tokenizer, image_processor=self.image_processor) # 加载输入，一张图片 input_data = Image.new('RGB', (32, 32)) # 加载指定的权重以完成推理 task_predictor.predict(input_data=input_data, max_length=8) pengcheng.mind-test/tests/st/test_model/test_bloom_model/000077500000000000000000000000001471400134000241675ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_bloom_model/__init__.py000066400000000000000000000012551471400134000263030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test bloom.""" pengcheng.mind-test/tests/st/test_model/test_bloom_model/test_autoclass.py000066400000000000000000000072301471400134000276000ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bloom interface used for mindformers. How to run this: pytest tests/st/test_model/test_bloom_model/test_autoclass.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor ms.set_context(mode=0) class TestBloomAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['bloom_560m'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_llm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_llm_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-test/tests/st/test_model/test_bloom_model/test_pipeline.py000066400000000000000000000026611471400134000274120ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bloom interface used for mindformers. How to run this: pytest tests/st/test_model/test_bloom_model/test_pipeline.py """ import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestBloomPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['bloom_560m'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("hello!", top_k=3) pengcheng.mind-test/tests/st/test_model/test_bloom_model/test_trainer.py000066400000000000000000000073541471400134000272550ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bloom interface used for mindformers. How to run this: pytest tests/st/test_model/test_bloom_model/test_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers import AutoTokenizer from mindformers import BloomLMHeadModel, BloomConfig from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 513 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(16): yield input_ids def generator_eval(): """eval dataset generator""" seq_len = 512 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(16): yield input_ids @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestBloomTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = BloomConfig(num_layers=2, hidden_size=32, num_heads=2, seq_length=512, max_decode_length=10) model = BloomLMHeadModel(model_config) tokenizer = AutoTokenizer.from_pretrained("bloom_560m") self.task_trainer = Trainer(task='text_generation', model=model, args=args, tokenizer=tokenizer, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() # @pytest.mark.run(order=2) # def test_eval(self): # """ # Feature: Trainer.evaluate() # Description: Test trainer for evaluate. # Expectation: TypeError, ValueError, RuntimeError # """ # self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", add_special_tokens=False, max_length=20) # @pytest.mark.run(order=4) # def test_finetune(self): # """ # Feature: Trainer.finetune() # Description: Test trainer for finetune. # Expectation: TypeError, ValueError, RuntimeError # """ # self.task_trainer.finetune() pengcheng.mind-test/tests/st/test_model/test_clip_model/000077500000000000000000000000001471400134000240065ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_clip_model/__init__.py000066400000000000000000000012541471400134000261210ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test clip.""" pengcheng.mind-test/tests/st/test_model/test_clip_model/test_clip_model.py000066400000000000000000000107101471400134000275250ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and CLIPModel class How to run this: windows: pytest .\\tests\\st\\test_model\\test_clip_model\\test_clip_model.py linux: pytest ./tests/st/test_model/test_clip_model/test_clip_model.py """ import os import time import mindspore as ms from mindformers import MindFormerBook, AutoConfig, AutoModel from mindformers.models import CLIPModel, BaseModel from mindformers.tools import logger ms.set_context(mode=0) class TestCLIPModelMethod: """A test class for testing Model classes""" def setup_method(self): """get_input""" self.model_type = "clip_vit_b_32" self.checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip') self.config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') self.config = AutoConfig.from_pretrained(self.model_type) self.checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip', self.model_type + '.ckpt') self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'clip') def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ time.sleep(5) AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # input model name, load model and weights model_a = AutoModel.from_pretrained(self.model_type) # input model directory, load model and weights model_b = AutoModel.from_pretrained(self.checkpoint_dir) # input yaml path, load model without weights model_c = AutoModel.from_config(self.config_path) # input config, load model without weights model_d = AutoModel.from_config(self.config) model_a.save_pretrained(self.save_directory, save_name=self.model_type) CLIPModel.show_support_list() support_list = CLIPModel.get_support_list() logger.info(support_list) # input model name, load model and weights model_i = CLIPModel.from_pretrained(self.model_type) # input model directory, load model and weights model_j = CLIPModel.from_pretrained(self.checkpoint_dir) # input config, load model weights model_k = CLIPModel(self.config) # input config, load model without weights self.config.checkpoint_name_or_path = None model_l = CLIPModel(self.config) model_i.save_pretrained(self.save_directory, save_name=self.model_type) # all models are ClipModel class， and inherited from BaseModel assert isinstance(model_i, CLIPModel) assert isinstance(model_j, CLIPModel) assert isinstance(model_k, CLIPModel) assert isinstance(model_l, CLIPModel) assert isinstance(model_i, BaseModel) assert isinstance(model_j, BaseModel) assert isinstance(model_k, BaseModel) assert isinstance(model_l, BaseModel) # all models are CLIPModel class， and inherited from BaseModel assert isinstance(model_a, CLIPModel) assert isinstance(model_b, CLIPModel) assert isinstance(model_c, CLIPModel) assert isinstance(model_d, CLIPModel) assert isinstance(model_a, BaseModel) assert isinstance(model_b, BaseModel) assert isinstance(model_c, BaseModel) assert isinstance(model_d, BaseModel) pengcheng.mind-test/tests/st/test_model/test_clip_model/test_clip_tokenizer.py000066400000000000000000000132311471400134000304400ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing Tokenizer class How to run this: linux: pytest ./tests/st/test_model/test_clip_model/test_clip_tokenizer.py """ import os import shutil import time import pytest from mindspore import Tensor from mindformers import Tokenizer, AutoTokenizer from mindformers import CLIPTokenizer def generate_fake_vocab(output_path): vocabs = ["[PAD]", "[unused1]", "[UNK]", "[CLS]", "[SEP]", "[MASK]", "hello", "world", "!"] with open(os.path.join(output_path, 'vocab_file.txt'), 'w') as fp: for item in vocabs: fp.write(item + '\n') class TestAutoTokenizerMethod: """A test class for testing the AutoTokenizer""" def setup_method(self): self.output_path = os.path.join(os.path.dirname(__file__), 'test_tokenizer_output') os.makedirs(self.output_path, exist_ok=True) generate_fake_vocab(self.output_path) def teardown_method(self): shutil.rmtree(self.output_path, ignore_errors=True) @pytest.mark.run(order=1) def test_save_and_load_using_bert_tokenizer(self): """ Feature: The test load and save function for the clip tokenizer Description: Load the tokenizer and then saved it Expectation: The restored kwargs is not expected version. """ clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") res = clip_tokenizer.tokenize("hello world?") assert isinstance(clip_tokenizer, CLIPTokenizer) assert res == ['hello', 'world', '?'] @pytest.mark.run(order=2) def test_load_from_yaml(self): """ Feature: The test load from yaml and save as the yaml for the tokenizer Description: Load the tokenizer and then saved it Expectation: The restored kwargs is not expected version. """ tokenizer = AutoTokenizer.from_pretrained("clip_vit_b_32") res = tokenizer.tokenize("hello world?") assert isinstance(tokenizer, CLIPTokenizer) assert res == ['hello', 'world', '?'] class TestPretrainedTokenizerMethod: """A test class for testing the PretrainedTokenizer""" def generate_fake_vocab(self): vocabs = ["[PAD]", "[unused1]", "[UNK]", "[CLS]", "[SEP]", "[MASK]", "hello", "world", "!"] with open(os.path.join(self.output_path, 'vocab.txt'), 'w') as fp: for item in vocabs: fp.write(item + '\n') def setup_method(self): self.output_path = os.path.join(os.path.dirname(__file__), 'test_tokenizer_output' + str(self)) os.makedirs(self.output_path, exist_ok=True) self.generate_fake_vocab() def teardown_method(self): shutil.rmtree(self.output_path, ignore_errors=True) def test_from_pretrained_tokenizer(self): """ Feature: The Tokenizer test using from python class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ time.sleep(10) tokenizer = Tokenizer.from_pretrained(self.output_path) with pytest.raises(NotImplementedError): tokenizer("hello world") class TestClipTokenizerMethod: """Test the basic usage of the CLIPTokenizer""" def test_padding(self): """ Feature: The CLIPTokenizer test using padding Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to the target. """ clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") res = clip_tokenizer("hello world?", max_length=8, padding='max_length') pad_id = clip_tokenizer.pad_token_id assert res == {'attention_mask': [1, 1, 1, 1, 1, 0, 0, 0], 'input_ids': [49406, 3306, 1002, 286, 49407, pad_id, pad_id, pad_id]}, f"The res is {res}." clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") res = clip_tokenizer("hello world?", max_length=8, padding='max_length', return_tensors='ms') assert len(res) == 2 for k in res.keys(): assert isinstance(res[k], Tensor) clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") batch_inputs = ["hello world?", "Who are you?", "I am find, thank you."] res = clip_tokenizer(batch_inputs, max_length=12, padding='max_length') assert len(res) == 2 assert res == {'input_ids': [[49406, 3306, 1002, 286, 49407, pad_id, pad_id, pad_id, pad_id, pad_id, pad_id, pad_id], [49406, 822, 631, 592, 286, 49407, pad_id, pad_id, pad_id, pad_id, pad_id, pad_id], [49406, 328, 687, 1416, 267, 1144, 592, 269, 49407, pad_id, pad_id, pad_id]], 'attention_mask': [[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0]]} pengcheng.mind-test/tests/st/test_model/test_glm2_model/000077500000000000000000000000001471400134000237205ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_glm2_model/__init__.py000066400000000000000000000012541471400134000260330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test glm2.""" pengcheng.mind-test/tests/st/test_model/test_glm2_model/test_auto_class.py000066400000000000000000000061341471400134000274720ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm2_model/test_auto_class.py """ import os from mindspore import context from mindformers import MindFormerBook, AutoModel, BaseModel, AutoConfig, BaseConfig, AutoTokenizer, BaseTokenizer class TestGLMAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" context.set_context(mode=0) self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ["glm2_6b"] def test_glm2_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights # Too time-cost, not used for now. for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + "_model", ) def test_glm2_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + "_config", ) def test_glm2_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name tokenizer_type = "glm2_6b" tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + "_tokenizer", ) pengcheng.mind-test/tests/st/test_model/test_glm2_model/test_trainer.py000066400000000000000000000115201471400134000267740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm2_model/test_trainer.py """ import numpy as np import pytest import mindspore from mindspore import context from mindspore.dataset import GeneratorDataset from mindformers import AutoTokenizer, ChatGLM2Config, ChatGLM2ForConditionalGeneration from mindformers import Trainer, TrainingArguments from mindformers.tools.utils import is_version_ge def generator_train(): """train dataset generator""" seq_len = 128 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) labels = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) train_data = (input_ids, labels) for _ in range(32): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 127 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) labels = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) eval_data = (input_ids, labels) for _ in range(8): yield eval_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGLM2TrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" context.set_context(mode=0, device_target="Ascend") args = TrainingArguments(num_train_epochs=1, batch_size=2) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "labels"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "labels"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = ChatGLM2Config(num_layers=2, seq_length=128, hidden_size=32, inner_hidden_size=None, num_heads=2, position_encoding_2d=True) model = ChatGLM2ForConditionalGeneration(model_config) self.tokenizer = AutoTokenizer.from_pretrained("glm2_6b") self.task_trainer = Trainer(task='text_generation', model=model, model_name='glm2_6b', tokenizer=self.tokenizer, args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ if is_version_ge(mindspore.__version__, "1.11.0"): self.task_trainer.train() @pytest.mark.run(order=2) def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ if is_version_ge(mindspore.__version__, "1.11.0"): self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ model_config = ChatGLM2Config(num_layers=1, hidden_size=32, inner_hidden_size=None, num_heads=2, position_encoding_2d=True) model = ChatGLM2ForConditionalGeneration(model_config) task_trainer = Trainer(task='text_generation', model=model, tokenizer=self.tokenizer) if is_version_ge(mindspore.__version__, "1.11.0"): task_trainer.predict(input_data="你好", max_length=20) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ if is_version_ge(mindspore.__version__, "1.11.0"): self.task_trainer.finetune() pengcheng.mind-test/tests/st/test_model/test_glm_model/000077500000000000000000000000001471400134000236365ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_glm_model/__init__.py000066400000000000000000000012531471400134000257500ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test glm.""" pengcheng.mind-test/tests/st/test_model/test_glm_model/test_auto_class.py000066400000000000000000000075331471400134000274140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm_model/test_auto_class.py """ import os from mindspore import context from mindformers import MindFormerBook from mindformers import AutoModel from mindformers import AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel from mindformers.models import BaseConfig, BaseTokenizer, BaseProcessor class TestGLMAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" context.set_context(mode=0) self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ["glm_6b", "glm_6b_chat"] def test_glm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights # Too time-cost, not used for now. for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + "_model", ) def test_glm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + "_config", ) def test_glm_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + "_processor", ) def test_glm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name tokenizer_type = "glm_6b" tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + "_tokenizer", ) pengcheng.mind-test/tests/st/test_model/test_glm_model/test_pipeline.py000066400000000000000000000027641471400134000270650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm_model/test_pipeline.py """ from mindspore import context from mindformers import pipeline class TestGLMPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" context.set_context(mode=0) self.test_llm_list = ['glm_6b', 'glm_6b_chat'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ # Too time-cost, not used for now. for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("你好", top_k=3) pengcheng.mind-test/tests/st/test_model/test_glm_model/test_trainer.py000066400000000000000000000112121471400134000267100ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm_model/test_trainer.py """ import numpy as np import pytest from mindspore import context from mindspore.dataset import GeneratorDataset from mindformers import AutoTokenizer from mindformers import GLMForPreTraining, GLMChatModel, GLMConfig from mindformers import Trainer, TrainingArguments def generator_train(): """train dataset generator""" seq_len = 128 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) label = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) position_ids = np.ones((2, seq_len)).astype(np.int64) attention_mask = np.ones(shape=(seq_len, seq_len)).astype(np.int32) train_data = (input_ids, label, position_ids, attention_mask) for _ in range(512): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 512 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) label = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) eval_data = (input_ids, label) for _ in range(8): yield eval_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGLMTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" context.set_context(mode=0) args = TrainingArguments(num_train_epochs=1, batch_size=2) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "label", "position_ids", "attention_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "label"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = GLMConfig(num_layers=2, hidden_size=32, inner_hidden_size=None, num_heads=2, position_encoding_2d=True) model = GLMForPreTraining(model_config) self.tokenizer = AutoTokenizer.from_pretrained("glm_6b") self.task_trainer = Trainer(task='text_generation', model=model, model_name='glm_6b', tokenizer=self.tokenizer, args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() # @pytest.mark.run(order=2) # def test_eval(self): # """ # Feature: Trainer.evaluate() # Description: Test trainer for evaluate. # Expectation: TypeError, ValueError, RuntimeError # """ # self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ model_config = GLMConfig(num_layers=1, hidden_size=32, inner_hidden_size=None, num_heads=2, position_encoding_2d=True) model = GLMChatModel(model_config) task_trainer = Trainer(task='text_generation', model=model, tokenizer=self.tokenizer) task_trainer.predict(input_data="你好", max_length=20) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.finetune() pengcheng.mind-test/tests/st/test_model/test_gpt2_lora_model/000077500000000000000000000000001471400134000247505ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_gpt2_lora_model/test_gpt2_lora_trainer.py000066400000000000000000000067531471400134000320110ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt2_lora interface used for mindformers. How to run this: pytest tests/st/test_model/test_gpt2_lora_model/test_gpt2_lora_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models.gpt2.gpt2 import GPT2WithLora from mindformers.pet.pet_config import LoraConfig from mindformers import Trainer, TrainingArguments, GPT2Config ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 65 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 64 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGPT2LoraTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "input_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "input_mask"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = GPT2Config(num_layers=2, embedding_size=32, num_heads=2, seq_length=64) pet_config = LoraConfig(lora_rank=8, lora_alpha=16, lora_dropout=0.05) model_config.pet_config = pet_config model = GPT2WithLora(model_config) self.task_trainer = Trainer(task='text_generation', model=model, pet_method='lora', args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.finetune() def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", max_length=20, repetition_penalty=1, top_k=3, top_p=1) pengcheng.mind-test/tests/st/test_model/test_gpt2_model/000077500000000000000000000000001471400134000237335ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_gpt2_model/__init__.py000066400000000000000000000012541471400134000260460ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test gpt2.""" pengcheng.mind-test/tests/st/test_model/test_gpt2_model/test_autoclass.py000066400000000000000000000072261471400134000273510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_gpt2_model/test_auto_class.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor ms.set_context(mode=0) class TestGPTAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['gpt2'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_llm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_llm_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-test/tests/st/test_model/test_gpt2_model/test_pipeline.py000066400000000000000000000026461471400134000271610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_gpt2_model/test_pipeline.py """ import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestGPTPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['gpt2'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("hello!", top_k=3) pengcheng.mind-test/tests/st/test_model/test_gpt2_model/test_trainer.py000066400000000000000000000074031471400134000270140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_gpt2_model/test_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers import GPT2LMHeadModel, GPT2Config from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 65 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 64 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGPTTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "input_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "input_mask"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = GPT2Config(num_layers=2, hidden_size=32, num_heads=2, seq_length=64) model = GPT2LMHeadModel(model_config) self.task_trainer = Trainer(task='text_generation', model=model, model_name='gpt2', args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() @pytest.mark.run(order=2) def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", max_length=20) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.finetune(finetune_checkpoint=True) pengcheng.mind-test/tests/st/test_model/test_gpt2_model/test_txtcls_model.py000066400000000000000000000047421471400134000300540ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and BertForMultipleChoice class How to run this: windows: pytest .\\tests\\st\\test_model\\test_gpt2_model\\test_txtcls_model.py linux: pytest ./tests/st/test_model/test_gpt2_model/test_txtcls_model.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, GPT2ForSequenceClassification from mindformers.tools import logger class TestModelMethod: """A test class for testing Model classes""" def setup_class(self): """get_input""" # gpt2_txtcls self.gpt2_txtcls_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'gpt2') self.gpt2_txtcls_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'gpt2', 'run_gpt2_txtcls.yaml') self.gpt2_txtcls_config = AutoConfig.from_pretrained('gpt2_txtcls') # save path self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'gpt2') # the first method to load model, AutoModel def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # gpt2_txtcls gpt2_txtcls_model_a = AutoModel.from_pretrained('gpt2_txtcls', download_checkpoint=False) gpt2_txtcls_model_a.save_pretrained(self.save_directory, save_name='gpt2_txtcls') assert isinstance(gpt2_txtcls_model_a, GPT2ForSequenceClassification) pengcheng.mind-test/tests/st/test_model/test_llama_lora_model/000077500000000000000000000000001471400134000251625ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_llama_lora_model/__init__.py000066400000000000000000000012671471400134000273010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test llama with lora.""" pengcheng.mind-test/tests/st/test_model/test_llama_lora_model/test_llama_lora_trainer.py000066400000000000000000000070031471400134000324220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama_lora interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_lora_model/test_llama_lora_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models.llama.llama import LlamaForCausalLMWithLora from mindformers.models.llama.llama_config import LlamaConfig from mindformers.pet.pet_config import LoraConfig from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 513 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) labels = np.ones_like(input_ids) train_data = (input_ids, labels) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 512 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) train_data = (input_ids) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestLlamaTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "labels"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "labels"]) train_dataset = train_dataset.batch(batch_size=4) eval_dataset = eval_dataset.batch(batch_size=4) model_config = LlamaConfig(num_layers=2, hidden_size=32, num_heads=2, seq_length=512) model_config.pet_config = LoraConfig(lora_rank=8, lora_alpha=16, lora_dropout=0.05) model = LlamaForCausalLMWithLora(model_config) self.task_trainer = Trainer(task='text_generation', model=model, model_name='llama_7b_lora', args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.finetune() @pytest.mark.run(order=2) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", max_length=20, repetition_penalty=1, top_k=3, top_p=1) pengcheng.mind-test/tests/st/test_model/test_llama_model/000077500000000000000000000000001471400134000241455ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_llama_model/__init__.py000066400000000000000000000012551471400134000262610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test llama.""" pengcheng.mind-test/tests/st/test_model/test_llama_model/test_autoclass.py000066400000000000000000000072711471400134000275630ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_model/test_auto_class.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor ms.set_context(mode=0) class TestLlamaAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['llama_7b'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_llm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_llm_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-test/tests/st/test_model/test_llama_model/test_pipeline.py000066400000000000000000000026571471400134000273750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_model/test_pipeline.py """ import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestLlamaPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['llama_7b'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("hello!", top_k=3) pengcheng.mind-test/tests/st/test_model/test_llama_model/test_trainer.py000066400000000000000000000076411471400134000272320ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_model/test_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models.llama.llama import LlamaForCausalLM from mindformers.models.llama.llama_config import LlamaConfig from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 513 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) # input_mask = np.ones_like(input_ids) train_data = (input_ids) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 512 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) # input_mask = np.ones_like(input_ids) train_data = (input_ids) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestLlamaTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids"]) train_dataset = train_dataset.batch(batch_size=4) eval_dataset = eval_dataset.batch(batch_size=4) model_config = LlamaConfig(num_layers=2, hidden_size=32, num_heads=2, seq_length=512) model = LlamaForCausalLM(model_config) self.task_trainer = Trainer(task='text_generation', model=model, args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.train() @pytest.mark.run(order=2) def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.model.set_train(False) self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", max_length=20, repetition_penalty=1, top_k=3, top_p=1) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.finetune() pengcheng.mind-test/tests/st/test_model/test_mae_model/000077500000000000000000000000001471400134000236215ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_mae_model/__init__.py000066400000000000000000000012531471400134000257330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test mae.""" pengcheng.mind-test/tests/st/test_model/test_mae_model/test_auto_class.py000066400000000000000000000057561471400134000274040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the swin interface used for mindformers. How to run this: pytest tests/st/test_model/test_mae_model/test_auto_class.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor class TestMaeAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['mae_vit_base_p16'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-test/tests/st/test_model/test_pangualpha_model/000077500000000000000000000000001471400134000251775ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_pangualpha_model/__init__.py000066400000000000000000000012621471400134000273110ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test pangualpha.""" pengcheng.mind-test/tests/st/test_model/test_pangualpha_model/test_pangualpha_autoclass.py000066400000000000000000000072561471400134000330200ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_llm_model/test_auto_class.py """ import os # pylint: disable=W0611 from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor # pylint: disable=W0611 from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor class TestAutoClassMethod: '''A test class for testing Model classes''' def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['pangualpha_2_6b'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_llm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_llm_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-test/tests/st/test_model/test_pangualpha_model/test_pangualpha_pipeline.py000066400000000000000000000026521471400134000326220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_llm_model/test_pipeline.py """ # pylint: disable=W0611 from mindformers import pipeline class TestPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['pangualpha_2_6b'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("今天天气不错，适合", top_k=3) pengcheng.mind-test/tests/st/test_model/test_pangualpha_model/test_pangualpha_trainer.py000066400000000000000000000103061471400134000324540ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the pangualpha interface used for mindformers. How to run this: pytest tests/st/test_model/test_llm_model/test_pangualpha_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models.pangualpha.pangualpha import PanguAlphaHeadModel from mindformers.models.pangualpha.pangualpha_config import PanguAlphaConfig from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 129 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) position_ids = np.ones((seq_len)).astype(np.int32) attention_mask = np.ones((seq_len, seq_len)).astype(np.int32) train_data = (input_ids, position_ids, attention_mask) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 128 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) # input_mask = np.ones_like(input_ids) train_data = (input_ids) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestPanguAlphaTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "position_id", "attention_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids"]) train_dataset = train_dataset.batch(batch_size=4) eval_dataset = eval_dataset.batch(batch_size=4) model_config = PanguAlphaConfig(num_layers=2, hidden_size=128, ffn_hidden_size=128*4, num_heads=2, seq_length=128) model = PanguAlphaHeadModel(model_config) self.task_trainer = Trainer(task='text_generation', model=model, args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.train() def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.model.set_train(False) self.task_trainer.evaluate() def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="今天天气如何？", max_length=20, repetition_penalty=1, top_k=3, top_p=1) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.finetune(finetune_checkpoint=True) pengcheng.mind-test/tests/st/test_model/test_qa_model/000077500000000000000000000000001471400134000234605ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_qa_model/__init__.py000066400000000000000000000012601471400134000255700ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test_qa_model.""" pengcheng.mind-test/tests/st/test_model/test_qa_model/test_qa_model.py000066400000000000000000000075161471400134000266630ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and BertForQuestionAnswering class How to run this: windows: pytest .\\tests\\st\\test_model\\test_qa_model\\test_qa_model.py linux: pytest ./tests/st/test_model/test_qa_model/test_qa_model.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig from mindformers.models import BertForQuestionAnswering, BaseModel from mindformers.tools import logger class TestModelMethod: """A test class for testing Model classes""" def setup_method(self): """get_input""" # fine-tuning self.qa_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'qa') self.qa_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'qa', 'run_qa_bert_base_uncased.yaml') self.qa_config = AutoConfig.from_pretrained('qa_bert_base_uncased') # evaluation and prediction self.qa_squad_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'qa') self.qa_squad_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'qa', 'run_qa_bert_base_uncased.yaml') self.qa_squad_config = AutoConfig.from_pretrained('qa_bert_base_uncased_squad') # save path self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'qa') def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # fine-tuning part # input model name qa_model_a = AutoModel.from_pretrained('qa_bert_base_uncased', download_checkpoint=False) # input yaml path qa_model_c = AutoModel.from_config(self.qa_config_path, download_checkpoint=False) qa_model_a.save_pretrained(self.save_directory, save_name='qa_bert_base_uncased') assert isinstance(qa_model_a, BertForQuestionAnswering) assert isinstance(qa_model_c, BertForQuestionAnswering) assert isinstance(qa_model_a, BaseModel) assert isinstance(qa_model_c, BaseModel) # evaluation and prediction test part # input model name qa_squad_model_a = AutoModel.from_pretrained('qa_bert_base_uncased_squad', download_checkpoint=False) # # input yaml path qa_squad_model_c = AutoModel.from_config(self.qa_squad_config_path, download_checkpoint=False) qa_squad_model_a.save_pretrained(self.save_directory, save_name='qa_bert_base_uncased_squad') assert isinstance(qa_squad_model_a, BertForQuestionAnswering) assert isinstance(qa_squad_model_c, BertForQuestionAnswering) assert isinstance(qa_squad_model_a, BaseModel) assert isinstance(qa_squad_model_c, BaseModel) pengcheng.mind-test/tests/st/test_model/test_sam_model/000077500000000000000000000000001471400134000236375ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_sam_model/__init__.py000066400000000000000000000000001471400134000257360ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_sam_model/test_auto_class.py000066400000000000000000000057501471400134000274140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the swin interface used for mindformers. How to run this: pytest tests/st/test_model/test_swin_model/test_auto_class.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor class TestSamAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['sam_vit_b'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-test/tests/st/test_model/test_sam_model/test_pipeline.py000066400000000000000000000120171471400134000270560ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_model/test_pipeline.py """ import cv2 import numpy as np import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestSamPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['sam_vit_b'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='segment_anything', model=model_type) image = cv2.imread("research/sam/images/truck.jpg") image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # 0.使用task_pipeline.set_image提前抽取图像特征 task_pipeline.set_image(image) # 1. 单点确定一个物体 input_point = np.array([[500, 375]]) input_label = np.array([1]) task_pipeline({"points": input_point, "labels": input_label}, multimask_output=True) # 2.两点确定相同物体 input_point = np.array([[500, 375], [1125, 625]]) input_label = np.array([1, 1]) task_pipeline({"points": input_point, "labels": input_label}, multimask_output=False) # 3.两点确定不同物体 input_point = np.array([[[500, 375]], [[1125, 625]]]) input_label = np.array([[1], [1]]) task_pipeline({"points": input_point, "labels": input_label}, multimask_output=False) # 4.一个前景点和背景点 input_point = np.array([[500, 375], [1125, 625]]) input_label = np.array([1, 0]) task_pipeline({"points": input_point, "labels": input_label}, multimask_output=False) # 5.单框确定一个物体 input_box = np.array([425, 600, 700, 875]) task_pipeline({"boxes": input_box}, multimask_output=False) # 6.框和背景点确定物体 input_box = np.array([425, 600, 700, 875]) input_point = np.array([[575, 750]]) input_label = np.array([0]) task_pipeline({"points": input_point, "labels": input_label, "boxes": input_box}, multimask_output=False) # 7.多组框和点确定不同物体 input_boxes = np.array([[425, 600, 700, 875], [1360, 525, 1680, 780]]) input_points = np.array([[[575, 750]], [[1525, 670]]]) input_labels = np.array([[1], [1]]) task_pipeline({"points": input_points, "labels": input_labels, "boxes": input_boxes}, multimask_output=False) # 8.多个框确定不同物体 input_boxes = np.array([[75, 275, 1725, 850], [425, 600, 700, 875], [1375, 550, 1650, 800], [1240, 675, 1400, 750]]) task_pipeline({"boxes": input_boxes}, multimask_output=False) # 单点确定一个物体: 传入cv2图像和prompt input_point = np.array([[500, 375]]) input_label = np.array([1]) image = cv2.imread("research/sam/images/truck.jpg") image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) task_pipeline({"image": image, "points": input_point, "labels": input_label}, multimask_output=True) # 单点确定一个物体: 传入图像路径和prompt task_pipeline({"image": "research/sam/images/truck.jpg", "points": input_point, "labels": input_label}, multimask_output=True) pengcheng.mind-test/tests/st/test_model/test_swin_model/000077500000000000000000000000001471400134000240375ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_swin_model/__init__.py000066400000000000000000000012541471400134000261520ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test swin.""" pengcheng.mind-test/tests/st/test_model/test_swin_model/test_auto_class.py000066400000000000000000000057561471400134000276220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the swin interface used for mindformers. How to run this: pytest tests/st/test_model/test_swin_model/test_auto_class.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor class TestSwinAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['swin_base_p4w7'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-test/tests/st/test_model/test_t5_model/000077500000000000000000000000001471400134000234075ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_t5_model/__init__.py000066400000000000000000000012521471400134000255200ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test t5.""" pengcheng.mind-test/tests/st/test_model/test_t5_model/test_t5_generation.py000066400000000000000000000043451471400134000275710ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of Generator model class How to run this: linux: pytest ./tests/st/test_model/test_t5_model/test_t5_generation.py """ import pytest import mindspore as ms from mindformers.models import T5ForConditionalGeneration, T5Tokenizer ms.set_context(mode=0) def modify_batch_size(net, batch_size): if hasattr(net, 'batch_size'): net.batch_size = batch_size for cell in net.cells(): modify_batch_size(cell, batch_size) @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGeneratorUseT5: """A test class for testing Model classes""" def setup_class(self): self.t5 = T5ForConditionalGeneration.from_pretrained("t5_small") self.tokenizer = T5Tokenizer.from_pretrained("t5_small") @pytest.mark.parametrize('do_sample', [False, True]) def test_single_inference(self, do_sample): """ Feature: Test input as single example for generator Description: single example inference Expectation: ValueError, AttributeError """ words = "translate the English to the Romanian: UN Chief Says There Is No Military Solution in Syria" words = self.tokenizer(words, max_length=21, padding='max_length')['input_ids'] modify_batch_size(self.t5, batch_size=1) output = self.t5.generate(words, do_sample=do_sample) output = self.tokenizer.decode(output[0], skip_special_tokens=True) assert output == "eful ONU declară că nu există o soluţie militară în Siria" pengcheng.mind-test/tests/st/test_model/test_t5_model/test_t5_tokenizer.py000066400000000000000000000033431471400134000274450ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing Tokenizer class How to run this: linux: pytest ./tests/st/test_model/test_t5_model/test_t5_tokenizer.py """ from mindformers import T5Tokenizer, AutoTokenizer class TestT5TokenizerMethod: """A test class for testing the BertTokenizer""" def setup_method(self): self.tokenizer = T5Tokenizer.from_pretrained('t5_small') self.auto_tokenizer = AutoTokenizer.from_pretrained('t5_small') def test_from_pretrained_tokenizer(self): """ Feature: The T5Tokenizer test using from python class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ self.tokenizer.show_support_list() self.tokenizer("hello world") def test_auto_tokenizer(self): """ Feature: The T5Tokenizer test using auto_class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ self.auto_tokenizer("hello world") pengcheng.mind-test/tests/st/test_model/test_t5_model/test_t5_trainer.py000066400000000000000000000145471471400134000271070ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the t5 trainer used for mindformers. How to run this: pytest tests/st/test_model/test_t5_model/test_t5_trainer.py """ import os import shutil import numpy as np import pytest import mindspore as ms from mindspore.dataset import MindDataset, GeneratorDataset from mindspore.mindrecord import FileWriter from mindformers.trainer import Trainer, TranslationTrainer from mindformers.trainer.config_args import ConfigArguments, \ OptimizerConfig, RunnerConfig from mindformers import T5Config, T5ForConditionalGeneration ms.set_context(mode=0) def generator(src_length=16, target_length=8): """dataset generator""" input_ids = np.random.randint(low=0, high=15, size=(src_length,)).astype(np.int32) attention_mask = np.ones((src_length,)).astype(np.int32) labels = np.random.randint(low=0, high=15, size=(target_length,)).astype(np.int32) for _ in range(2): yield input_ids, attention_mask, labels def write_mindrecord(ds_generator, data_record_path): """Using the generator to get mindrecords""" ds = GeneratorDataset(ds_generator, column_names=["input_ids", "attention_mask", "labels"]) writer = FileWriter(file_name=data_record_path, shard_num=1, overwrite=True) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "attention_mask": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]}} writer.add_schema(data_schema, "test_schema") for item in ds.create_dict_iterator(): for k in item.keys(): item[k] = item[k].asnumpy() writer.write_raw_data([item]) writer.commit() def modify_attrs(net, key, value): if hasattr(net, key): setattr(net, key, value) print(f"Set the net {net.__class__.__name__} with {key}:{value}") for cell in net.cells(): modify_attrs(cell, key, value) def write_raw_text_data(stage, data_record_path): """writes the fake translation data""" source = ["We went through the whole range of emotions during this period.", "The positive reaction of pilots and Federation officials makes me hope that this year we will " "be organizing champions again" " said rally manager, Dan Codreanu."] target = ['Am trecut prin toată gama de trăiri în această perioadă.', "Reacția pozitivă a piloților și oficialilor Federației mă face să sper că vom fi și în acest an " "campion la organizare a spus managerul raliului, Dan Codreanu."] src_path = os.path.join(data_record_path, f'{stage}.source') tgt_path = os.path.join(data_record_path, f'{stage}.target') with open(src_path, 'w') as sfp: with open(tgt_path, 'w') as tfp: for x, y in zip(source, target): sfp.write(x + '\n') tfp.write(y + '\n') @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestTranslationTrainer: """Test Translation Trainer""" def setup_class(self): self.dir_path = os.path.join(os.path.dirname(__file__), 'fake_dataset') os.makedirs(self.dir_path, exist_ok=True) self.abs_path = os.path.join(self.dir_path, 't5_dataset') write_mindrecord(generator(src_length=16, target_length=8), self.abs_path) self.raw_text_path = os.path.join(os.path.dirname(__file__), 'raw_text_dataset') os.makedirs(self.raw_text_path, exist_ok=True) write_raw_text_data(stage='train', data_record_path=self.raw_text_path) def teardown_class(self): shutil.rmtree(self.dir_path, ignore_errors=True) shutil.rmtree(self.raw_text_path, ignore_errors=True) def get_mindfiles_from_path(self, dir_path): dataset_files = [] for r, _, f in os.walk(dir_path): for file in f: if not file.endswith("db"): dataset_files.append(os.path.join(r, file)) return dataset_files @pytest.mark.run(order=1) def test_trainer_with_translation_args_train(self): """ Feature: Create Trainer From Config Description: Test Trainer API to train from config Expectation: TypeError """ batch_size = 1 runner_config = RunnerConfig(epochs=1, batch_size=batch_size) # 运行超参 optim_config = OptimizerConfig(optim_type='AdamWeightDecay', beta1=0.9, learning_rate=0.001) dataset = MindDataset(dataset_files=self.get_mindfiles_from_path(self.dir_path), columns_list=["input_ids", "attention_mask", "labels"]) dataset = dataset.batch(batch_size=batch_size) dataset = dataset.repeat(1) config = ConfigArguments(seed=2022, runner_config=runner_config, optimizer=optim_config) model_config = T5Config(batch_size=batch_size, num_heads=8, num_layers=1, hidden_size=32, seq_length=16, max_decode_length=8) # Model model = T5ForConditionalGeneration(model_config) mim_trainer = Trainer(task='translation', model=model, args=config, train_dataset=dataset) mim_trainer.train() @pytest.mark.run(order=2) def test_trainer_predict(self): """ Feature: Test Predict of the Trainer Description: Test Predict Expectation: TypeError """ # change the length for quick training model_config = T5Config(seq_length=32, max_decode_length=32) model = T5ForConditionalGeneration(model_config) mim_trainer = TranslationTrainer(model_name="t5_small") mim_trainer.predict(input_data="hello world", network=model) pengcheng.mind-test/tests/st/test_model/test_tokcls_model/000077500000000000000000000000001471400134000243565ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_tokcls_model/__init__.py000066400000000000000000000012611471400134000264670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test_ner_model.""" pengcheng.mind-test/tests/st/test_model/test_tokcls_model/test_tokcls_model.py000066400000000000000000000070121471400134000304460ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and BertForTokenClassification class How to run this: windows: pytest .\\tests\\st\\test_model\\test_tokcls_model\\test_tokcls_model.py linux: pytest ./tests/st/test_model/test_tokcls_model/test_tokcls_model.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig from mindformers.models import BertForTokenClassification from mindformers.tools import logger class TestModelMethod: """A test class for testing Model classes""" def setup_class(self): """get_input""" # fine-tuning self.tokcls_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'tokcls') self.tokcls_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'tokcls', 'run_tokcls_bert_base_chinese.yaml') self.tokcls_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese') # evaluation and prediction self.tokcls_cluener_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'tokcls') self.tokcls_cluener_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'tokcls', 'run_tokcls_bert_base_chinese_cluener.yaml') self.tokcls_cluener_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese_cluener') # save path self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'tokcls') # the first method to load model, AutoModel def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # fine-tuning part # input model name tokcls_model_a = AutoModel.from_pretrained('tokcls_bert_base_chinese', download_checkpoint=False) tokcls_model_a.save_pretrained(self.save_directory, save_name='tokcls_bert_base_chinese') assert isinstance(tokcls_model_a, BertForTokenClassification) # evaluation and prediction test part # input model name tokcls_cluener_model_a = \ AutoModel.from_pretrained('tokcls_bert_base_chinese_cluener', download_checkpoint=False) tokcls_cluener_model_a.save_pretrained(self.save_directory, save_name='tokcls_bert_base_chinese_cluener') assert isinstance(tokcls_cluener_model_a, BertForTokenClassification) pengcheng.mind-test/tests/st/test_model/test_tokcls_model/test_tokcls_tokenizer.py000066400000000000000000000051511471400134000313620ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing Tokenizer class How to run this: linux: pytest ./tests/st/test_model/test_tokcls_model/test_tokcls_tokenizer.py """ from mindformers import BertTokenizer, AutoTokenizer class TestBertTokenizerForChineseMethod: """A test class for testing the AutoTokenizer""" def test_from_pretrained(self): """ Feature: The BertTokenizer test using from python class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ tokenizer = BertTokenizer.from_pretrained('tokcls_bert_base_chinese') tokenizer.show_support_list() res = tokenizer.tokenize("你好，世界！") assert isinstance(tokenizer, BertTokenizer) assert res == ['你', '好', '，', '世', '界', '！'] tokenizer = BertTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') tokenizer.show_support_list() res = tokenizer.tokenize("你好，世界！") assert isinstance(tokenizer, BertTokenizer) assert res == ['你', '好', '，', '世', '界', '！'] def test_auto_tokenizer(self): """ Feature: The BertTokenizer test using auto_class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese') tokenizer.show_support_list() res = tokenizer.tokenize("你好，世界！") assert isinstance(tokenizer, BertTokenizer) assert res == ['你', '好', '，', '世', '界', '！'] tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') tokenizer.show_support_list() res = tokenizer.tokenize("你好，世界！") assert isinstance(tokenizer, BertTokenizer) assert res == ['你', '好', '，', '世', '界', '！'] pengcheng.mind-test/tests/st/test_model/test_txtcls_model/000077500000000000000000000000001471400134000244005ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_txtcls_model/__init__.py000066400000000000000000000012641471400134000265140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test_txtcls_model.""" pengcheng.mind-test/tests/st/test_model/test_txtcls_model/test_txtcls_model.py000066400000000000000000000065241471400134000305210ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and BertForMultipleChoice class How to run this: windows: pytest .\\tests\\st\\test_model\\test_txtcls_model\\test_txtcls_model.py linux: pytest ./tests/st/test_model/test_txtcls_model/test_txtcls_model.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig from mindformers.models import BertForMultipleChoice from mindformers.tools import logger class TestModelMethod: """A test class for testing Model classes""" def setup_class(self): """get_input""" # fine-tuning self.txtcls_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'txtcls') self.txtcls_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'txtcls', 'run_txtcls_bert_base_uncased.yaml') self.txtcls_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased') # evaluation and prediction self.txtcls_mnli_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'txtcls') self.txtcls_mnli_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'txtcls', 'run_txtcls_bert_base_uncased_mnli.yaml') self.txtcls_mnli_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased_mnli') # save path self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'txtcls') # the first method to load model, AutoModel def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # fine-tuning part # input model name txtcls_model_a = AutoModel.from_pretrained('txtcls_bert_base_uncased', download_checkpoint=False) txtcls_model_a.save_pretrained(self.save_directory, save_name='txtcls_bert_base_uncased') # evaluation and prediction test part # input model name txtcls_mnli_model_a = AutoModel.from_pretrained('txtcls_bert_base_uncased_mnli', download_checkpoint=False) txtcls_mnli_model_a.save_pretrained(self.save_directory, save_name='txtcls_bert_base_uncased_mnli') assert isinstance(txtcls_mnli_model_a, BertForMultipleChoice) pengcheng.mind-test/tests/st/test_model/test_vit_model/000077500000000000000000000000001471400134000236615ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_vit_model/__init__.py000066400000000000000000000000001471400134000257600ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_model/test_vit_model/test_auto_class.py000066400000000000000000000057531471400134000274410ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the swin interface used for mindformers. How to run this: pytest tests/st/test_model/test_swin_model/test_auto_class.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor class TestVitAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['vit_base_p16'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-test/tests/st/test_resume.py000066400000000000000000000113461471400134000214170ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_resume.py """ from dataclasses import dataclass import os import numpy as np import pytest import mindspore as ms from mindspore.train.callback import LossMonitor, TimeMonitor from mindspore.dataset import GeneratorDataset from mindformers.tools.utils import LOCAL_DEFAULT_PATH from mindformers.trainer import Trainer from mindformers.trainer.config_args import ConfigArguments, \ RunnerConfig from mindformers.models.gpt2 import GPT2LMHeadModel, GPT2Config from mindformers.core.lr import WarmUpDecayLR from mindformers import CheckpointMointor from mindformers.core.optim import FusedAdamWeightDecay ms.set_context(mode=0) def generator(): """dataset generator""" seq_len = 1025 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(32): yield train_data @dataclass class Tempconfig: seed: int = 0 runner_config: RunnerConfig = None data_size: int = 0 train_checkpoint: str = "" @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard def test_gpt_trainer_train_from_instance(): """ Feature: Create Trainer From Instance Description: Test Trainer API to train from self-define instance API. Expectation: TypeError """ # Config definition runner_config = RunnerConfig(epochs=2, batch_size=8, sink_mode=True, sink_size=2) config = ConfigArguments(seed=2022, runner_config=runner_config) # Model model_config = GPT2Config(num_layers=2) gpt_model = GPT2LMHeadModel(model_config) # Dataset and operations dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask"]) dataset = dataset.batch(batch_size=8) # optimizer lr_schedule = WarmUpDecayLR(learning_rate=0.0001, end_learning_rate=0.00001, warmup_steps=0, decay_steps=512) optimizer = FusedAdamWeightDecay(beta1=0.009, beta2=0.999, learning_rate=lr_schedule, params=gpt_model.trainable_params()) # callback loss_cb = LossMonitor(per_print_times=2) time_cb = TimeMonitor() ckpt_cb = CheckpointMointor(directory=os.path.join(LOCAL_DEFAULT_PATH, "test_resume")) callbacks = [loss_cb, time_cb, ckpt_cb] lm_trainer = Trainer(model=gpt_model, args=config, optimizers=optimizer, train_dataset=dataset, callbacks=callbacks, task="text_generation") lm_trainer.train(train_checkpoint=False) checkpoint_dir = os.path.join(LOCAL_DEFAULT_PATH, "test_resume", "checkpoint", "rank_{}".format(int(os.getenv("RANK_ID", "0")))) output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) for _ in range(2): os.remove(os.path.join(checkpoint_dir, output_checkpoint_path.pop())) # Dataset and operations dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask"]) dataset = dataset.batch(batch_size=8) # callback loss_cb = LossMonitor(per_print_times=2) time_cb = TimeMonitor() ckpt_cb = CheckpointMointor(directory=os.path.join(LOCAL_DEFAULT_PATH, "test_resume")) callbacks = [loss_cb, time_cb, ckpt_cb] lm_trainer = Trainer(model=gpt_model, args=config, optimizers=optimizer, train_dataset=dataset, callbacks=callbacks, task="text_generation") lm_trainer.train(train_checkpoint=os.path.join(LOCAL_DEFAULT_PATH, "test_resume", "checkpoint"), resume_training=True) pengcheng.mind-test/tests/st/test_text_generator/000077500000000000000000000000001471400134000225725ustar00rootroot00000000000000pengcheng.mind-test/tests/st/test_text_generator/__init__.py000066400000000000000000000012661471400134000247100ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test text generator.""" pengcheng.mind-test/tests/st/test_text_generator/test_batch_generate.py000066400000000000000000000045541471400134000271460ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the batch infer for text generator. How to run this: pytest tests/st/test_text_generator/test_batch_generate.py """ import mindspore as ms from mindformers import AutoConfig, AutoModel, AutoTokenizer ms.set_context(mode=0) class TestBatchGenerate: """A test class for testing text generate features.""" def setup_method(self): """setup method.""" self.test_model_list = ['glm_6b', 'bloom_560m'] # self.test_model_list = ['glm_6b', 'llama_7b', 'bloom_560m'] def test_batch_generate(self): """ Feature: batch generate. Description: Test batch generate for language models. Expectation: TypeError, ValueError, RuntimeError """ question_list = [ "Hello", "Introduce yourself", "I love Beijing, because", "what color is the sky?" ] batch_size = len(question_list) seq_len = 256 for model_name in self.test_model_list: # set model config config = AutoConfig.from_pretrained(model_name) config.batch_size = batch_size config.seq_length = seq_len config.max_decode_length = seq_len config.use_past = True print(f"config is: {config}") tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModel.from_config(config) inputs = tokenizer(question_list, max_length=seq_len, padding="max_length")["input_ids"] print(f"inputs is: {inputs}") outputs = model.generate(inputs) for output in outputs: print(tokenizer.decode(output)) del tokenizer del model pengcheng.mind-test/tests/st/test_text_generator/test_incremental_generate.py000066400000000000000000000036071471400134000303640ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the incremental generate for text generator. How to run this: pytest tests/st/test_text_generator/test_incremental_generate.py """ import mindspore as ms from mindformers import AutoConfig, AutoModel, AutoTokenizer ms.set_context(mode=0) class TestIncrementalGenerate: """A test class for testing text generate features.""" def setup_method(self): """setup method.""" self.test_model_list = ['glm_6b', 'llama_7b', 'bloom_560m'] def test_incremental_generate(self): """ Feature: incremental generate. Description: Test incremental generate by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_model_list: config = AutoConfig.from_pretrained(model_type) # set incremental infer config config.batch_size = 1 config.use_past = True model = AutoModel.from_config(config) tokenizer = AutoTokenizer.from_pretrained(model_type) input_ids = tokenizer("hello")["input_ids"] output = model.generate(input_ids, max_length=20) print(tokenizer.decode(output)) del tokenizer del model pengcheng.mind-test/tests/st/test_yaml.py000066400000000000000000000033471471400134000210630ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing yaml tools, MindFormerConfig. How to run this: windows: pytest .\\tests\\st\\test_yaml.py linux: pytest ./tests/st/test_yaml.py Note: the name of model yaml file should start with model. the name of model ckpt file should be same with yaml file. example: clip_vit_b_32.yaml starts with clip, and clip_vit_b_32.ckpt is model ckpt file. """ import os import pytest from mindformers import MindFormerBook from mindformers.tools import MindFormerConfig, logger @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training # add for ci @pytest.mark.env_onecard def test_yaml(): """ Feature: MindFormerConfig Description: Test to transform yaml file as MindFormerConfig Expectation: TypeError """ yaml_path = os.path.join( MindFormerBook.get_project_path(), 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') yaml_content = MindFormerConfig(yaml_path) logger.info(yaml_content) assert isinstance(yaml_content, MindFormerConfig) assert isinstance(yaml_content, dict) pengcheng.mind-test/tests/ut/000077500000000000000000000000001471400134000165035ustar00rootroot00000000000000pengcheng.mind-test/tests/ut/__init__.py000066400000000000000000000012451471400134000206160ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """UT.""" pengcheng.mind-test/tests/ut/test_layers_apis.py000066400000000000000000000023051471400134000224270ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ test layers""" import numpy as np from mindspore import Tensor from mindspore.common import dtype from mindspore.common.api import _cell_graph_executor from mindformers.modules.layers import AlibiTensor def test_alibi_tensor(): """ Feature: Test the alibi tensor. Description: Test the forward Expectation: No exception """ model = AlibiTensor(seq_length=128, num_heads=32) input_mask = Tensor(np.ones((10, 128)), dtype.float32) mstype = dtype.float32 _cell_graph_executor.compile(model, input_mask, mstype) pengcheng.mind-test/tests/ut/test_local_block_sparse_attention.py000066400000000000000000000121731471400134000260260ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test local block sparse attention accuracy""" import math import pytest import numpy as np import mindspore as ms from mindspore import Tensor from mindspore import nn import mindspore.ops as P from mindformers.modules.local_block_sparse_attention import LocalBlockSparseAttention class SparseAttention(nn.Cell): """ A Local Sparse Attention """ def __init__(self, seq_len, size_per_head, local_size, block_size): super(SparseAttention, self).__init__() self.seq_len = seq_len self.local_size = local_size self.block_size = block_size self.scale_factor = Tensor(math.sqrt(math.sqrt(size_per_head))) self.real_div = P.RealDiv() self.qk_bmm = ms.ops.BatchMatMul(transpose_b=True) self.mask_mul = P.Mul() self.mask_expand_dims = P.ExpandDims() self.mask_add = P.Add() self.softmax = P.Softmax() self.pv_bmm = ms.ops.BatchMatMul() self.multiply_data = Tensor([-10000.0], dtype=ms.float16) sparse_mask = self.gen_sparse_mask() self.sparse_mask = Tensor(sparse_mask, ms.float16) def gen_sparse_mask(self): """generate a mask for sparse attention""" sparse_mask = np.ones((self.seq_len, self.seq_len)) block_groups = self.seq_len // self.block_size prev_block_num = self.local_size // self.block_size - 1 for i in range(block_groups): row_begin = i * self.block_size row_end = row_begin + self.block_size col_begin = max(0, i - prev_block_num) * self.block_size col_end = col_begin + self.local_size sparse_mask[row_begin:row_end, col_begin:col_end] = 0 return sparse_mask def construct(self, q, k, v, attn_mask=None): """Forward process""" factor = P.Cast()(self.scale_factor, P.DType()(q)) q = self.real_div(q, factor) k = self.real_div(k, factor) sim = self.qk_bmm(q, k) sparse_mask = self.mask_mul(self.sparse_mask, self.multiply_data) sparse_mask = self.mask_expand_dims(sparse_mask, 0) sparse_mask = self.mask_expand_dims(sparse_mask, 1) sim = self.mask_add(sparse_mask, sim) if attn_mask is not None: adder = self.mask_mul(attn_mask, self.multiply_data) adder = self.mask_expand_dims(adder, 1) sim = self.mask_add(adder, sim) probs = self.softmax(sim) out = self.pv_bmm(probs, v) return out def data_compare(ground_truth, predict, diff_thd=0.001, pct_thd=0.001, max_diff_thd=0.1): """compare ground_truth and predict value diff""" total_count = np.prod(ground_truth.shape) greater_than_diff_thd_count = np.sum( np.abs(predict.astype("float32") - ground_truth.astype("float32")) > diff_thd * (np.abs(ground_truth.astype("float32")) + 1e-9)) greater_than_max_diff_thd_count = np.sum( np.abs(predict.astype("float32") - ground_truth.astype("float32")) > max_diff_thd * (np.abs(ground_truth.astype("float32")) + 1e-9)) diff_gt_thd_proportion = greater_than_diff_thd_count / total_count diff_gt_max_thd_proportion = greater_than_max_diff_thd_count / total_count result = "Pass" if diff_gt_thd_proportion > pct_thd or diff_gt_max_thd_proportion > 0: result = "Failed" return result @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_local_block_sparse_attention(): """ Feature: Test Local Block Sparse Attention Description: Test Local Block Sparse Attention Accuracy Expectation: result == "Pass" """ input_shape = (4, 32, 4096, 128) local_size = 1024 block_size = 128 q = Tensor(np.random.random(input_shape).astype("float16")) k = Tensor(np.random.random(input_shape).astype("float16")) v = Tensor(np.random.random(input_shape).astype("float16")) batch_size, seq_len = q.shape[0], q.shape[2] att_mask = Tensor(1.0 - np.tril( np.ones(shape=(batch_size, seq_len, seq_len), dtype=np.float16))) size_per_head = q.shape[-1] model1 = LocalBlockSparseAttention(seq_len, size_per_head, local_size, block_size, dropout_rate=0) model2 = SparseAttention(seq_len, size_per_head, local_size, block_size) out1 = model1(q, k, v, att_mask).asnumpy() out2 = model2(q, k, v, att_mask).asnumpy() result = data_compare(out1, out2) assert result == "Pass" pengcheng.mind-test/tests/ut/test_lr.py000066400000000000000000000055451471400134000205420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test lr schedule.""" import pytest import mindspore as ms from mindspore import Tensor from mindformers.core.lr import LinearWithWarmUpLR,\ CosineWithWarmUpLR, PolynomialWithWarmUpLR, \ CosineWithRestartsAndWarmUpLR, ConstantWarmUpLR @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_lr_schedule(): """ Feature: Test LR Schedule Description: Test Different LR Schedule Expectation: ValueError """ total_steps = 20 warmup_steps = 10 learning_rate = 0.005 lr_end = 0.0000001 constant_with_warmup_lr_std = [0.0005, 0.005, 0.005] linear_lr_std = [0.0005, 0.005, 0.0005] cosine_lr_std = [0.0005, 0.005, 0.00012236] cosine_with_restarts_lr_std = [0.0005, 0.005, 0.00289109] polynomial_lr_std = [0.0005, 0.005, 0.00050009] constant_with_warmup = ConstantWarmUpLR(learning_rate, warmup_steps) linear = LinearWithWarmUpLR(learning_rate, warmup_steps, total_steps) cosine = CosineWithWarmUpLR(learning_rate, warmup_steps, total_steps) cosine_with_restarts = CosineWithRestartsAndWarmUpLR(learning_rate, total_steps, warmup_steps) polynomial = PolynomialWithWarmUpLR(learning_rate, warmup_steps, total_steps, lr_end=lr_end) constant_with_warmup_lr = [] linear_lr = [] cosine_lr = [] cosine_with_restarts_lr = [] polynomial_lr = [] assert_steps = [1, 10, 19] for step in range(0, total_steps): if step in assert_steps: step = Tensor(step, ms.int32) constant_with_warmup_lr.append(constant_with_warmup(step).asnumpy()) linear_lr.append(linear(step).asnumpy()) cosine_lr.append(cosine(step).asnumpy()) cosine_with_restarts_lr.append(cosine_with_restarts(step).asnumpy()) polynomial_lr.append(polynomial(step).asnumpy()) error = 1e-8 for i in range(3): assert abs(constant_with_warmup_lr[i] - constant_with_warmup_lr_std[i]) < error assert abs(linear_lr[i] - linear_lr_std[i]) < error assert abs(cosine_lr[i] - cosine_lr_std[i]) < error assert abs(cosine_with_restarts_lr[i] - cosine_with_restarts_lr_std[i]) < error assert abs(polynomial_lr[i] - polynomial_lr_std[i]) < error pengcheng.mind-test/tests/ut/test_metrics.py000066400000000000000000000047631471400134000215740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test metric schedule.""" import pytest import numpy as np import mindspore as ms from mindspore import Tensor from mindformers.core.metric import PromptAccMetric, EmF1Metric @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_promptacc_metric(): """ Feature: Test PromptAccMetric Description: Test PromptAccMetric Expectation: ValueError """ logits = Tensor(np.array([[[[0.4, -0.9], [0.2, 0.7]], [[0.9, 0.09], [-0.4, 0.4]], [[-0.2, 0.05], [0.6, -0.8]], [[0.6, -0.4], [0.18, -0.56]]]]), ms.float16) input_ids = Tensor(np.array([[0, 1], [3, 7], [6, 2], [4, 4]]), ms.int32) input_mask = Tensor(np.array([[0, 1], [0, 1], [0, 1], [0, 1]]), ms.float32) labels = Tensor(np.array([[0]]), ms.int32) prompt_acc_std = 0 metric = PromptAccMetric() metric.clear() metric.update(logits, input_ids, input_mask, labels) prompt_acc = metric.eval().get("Acc", -1) error = 1e-8 assert abs(prompt_acc - prompt_acc_std) < error @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_emf1_metric(): """ Feature: Test EmF1Metric Description: Test EmF1Metric Expectation: ValueError """ str_pre = ["I love Beijing, because it's beautiful", "Hello world。"] str_label = ["I love Beijing.", "Hello world"] metric = EmF1Metric() metric.clear() for pre, label in zip(str_pre, str_label): metric.update([pre], [label]) result = metric.eval() error = 1e-8 f1_score, em_score = 75.0, 50.0 assert abs(result.get("F1", 0) - f1_score) < error and abs(result.get("Em", 0) - em_score) < error pengcheng.mind-test/tests/ut/test_streamers.py000066400000000000000000000102761471400134000221270ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test generation.streamer schedule.""" from threading import Thread import pytest from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextStreamer, TextIteratorStreamer @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_text_streamer_schedule(): """ Feature: Test Streamer Schedule Description: Test TextStreamer Generation.Streamer Schedule Expectation: ValueError """ tok = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2LMHeadModel.from_pretrained("gpt2") inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) streamer = TextStreamer(tok) # Despite returning the usual output, the streamer will also print the generated text to stdout. result = model.generate(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) result = result[0].tolist() assert result == \ [2025, 3649, 8379, 25, 530, 11, 734, 11, 1115, 11, 1440, 11, 1936, 11, 2237, 11, 3598, 11, 3624, 11] @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_text_iterator_streamer_schedule(): """ Feature: Test Streamer Schedule Description: Test TextIteratorStreamer Generation.Streamer Schedule Expectation: ValueError """ tok = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2LMHeadModel.from_pretrained("gpt2") inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) streamer = TextIteratorStreamer(tok) # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. generation_kwargs = dict(input_ids=inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) thread = Thread(target=model.generate, kwargs=generation_kwargs) thread.start() generated_text = "" for new_text in streamer: generated_text = "".join([generated_text, new_text]) assert generated_text == "An increasing sequence: one, two, three, four, five, six, seven, eight," @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_text_iterator_streamer_schedule_batch(): """ Feature: Test Batch Streamer Schedule Description: Test TextIteratorStreamer Generation.Streamer Schedule, with batch input. Expectation: ValueError """ tok = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2LMHeadModel.from_pretrained("gpt2") text_batch = ["An increasing sequence: one,", "The highest mountain in the world is", "The largest river in China is"] inputs = tok(text_batch, max_length=8, padding='max_length', return_tensors=None, add_special_tokens=False) streamer = TextIteratorStreamer(tok) # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. generation_kwargs = dict(input_ids=inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) thread = Thread(target=model.generate, kwargs=generation_kwargs) thread.start() batch_size = len(text_batch) generated_text = [""] * batch_size for new_text in streamer: for i in range(batch_size): generated_text[i] = "".join([generated_text[i], new_text[i]]) assert generated_text == \ ['An increasing sequence: one, two, three, four, five, six, seven, eight,', 'The highest mountain in the world is the Himalayas, which is the highest mountain in the world', 'The largest river in China is the Yangtze River, which flows through the heart of the country'] pengcheng.mind-test/tests/ut/test_tokenizer.py000066400000000000000000000106551471400134000221350ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ test tokenizer """ # import os import pytest from mindformers import AutoTokenizer @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_all_tokenizers(): """ Feature: Test all tokenizers. Description: Test all tokenizers, including save_pretrained, form_pretrained, tokenizer and return_tensors Expectation: No exception """ # path_name = "./save_vocabulary" # if not os.path.exists(path_name): # os.mkdir(path_name) tokenizer_list = ["gpt2", "bert_base_uncased", "llama_7b", "bloom_560m", "pangualpha_2_6b", "clip_vit_b_32", "glm_6b", "t5_small"] string_list = ["I love Beijing.", "我爱北京。"] return_tensors_sig = ["", "ms", "np"] all_res = [ [40, 1842, 11618, 13], # gpt2 english [22755, 239, 163, 230, 109, 44293, 245, 12859, 105, 16764], # gpt2 chinese [1045, 2293, 7211, 1012], # bert english [1855, 100, 1781, 1755, 1636], # bert chinese [312, 1545, 11285, 31843], # llama english [31822, 233, 139, 148, 234, 139, 180, 232, 143, 154, 231, 189, 175, 230, 131, 133], # llama chinese [44, 19134, 61335, 17], # bloom english [164830, 11385, 420], # bloom chinese [2787, 13, 4, 13, 28420, 13, 4, 37472, 34], # pangualpha english [21, 453, 263, 12], # pangualpha chinese [328, 793, 11796, 269], # clip english [162, 230, 239, 163, 230, 109, 161, 234, 245, 21078, 361, 38000], # clip chinese [115, 703, 8994, 7], # glm english [5, 76202, 64241, 63823], # glm chinese [27, 333, 14465, 5], # t5 english [3, 2] # t5 chinese ] for i, tokenizer_item in enumerate(tokenizer_list): tokenizer = AutoTokenizer.from_pretrained(tokenizer_item) for j, string in enumerate(string_list): for return_tensor_sig in return_tensors_sig: if not return_tensor_sig: result = tokenizer(string, add_special_tokens=False) print("{}: the result of {} is {}".format(tokenizer_item, string, result)) assert result.input_ids == all_res[2*i+j] result = tokenizer(string, padding="max_length", max_length=100, add_special_tokens=False) assert result.input_ids[:len(all_res[2 * i + j])] == all_res[2 * i + j] print("{}: the pad result of {} is {}".format(tokenizer_item, string, result)) else: result = tokenizer(string, return_tensors=return_tensor_sig, add_special_tokens=False) if return_tensor_sig == "ms": assert result.input_ids.asnumpy().tolist() == all_res[2 * i + j] else: assert result.input_ids.tolist() == all_res[2 * i + j] print("{}: the {} result of {} is {}".format(tokenizer_item, return_tensor_sig, string, result)) result = tokenizer(string, padding="max_length", max_length=100, return_tensors=return_tensor_sig, add_special_tokens=False) if return_tensor_sig == "ms": assert result.input_ids.asnumpy().tolist()[:len(all_res[2 * i + j])] == all_res[2 * i + j] else: assert result.input_ids.tolist()[:len(all_res[2 * i + j])] == all_res[2 * i + j] print("{}: the {} pad result of {} is {}".format(tokenizer_item, return_tensor_sig, string, result)) # tokenizer.save_pretrained(save_name=tokenizer_item) # path = os.path.join(path_name, tokenizer_item) # if not os.path.exists(path): # os.mkdir(path) # tokenizer.save_vocabulary(path, tokenizer_item) pengcheng.mind-test/tests/ut/test_transformer_apis.py000066400000000000000000000615631471400134000235050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ test transformer""" import numpy as np import pytest import mindspore from mindspore import Tensor from mindspore.common import dtype from mindspore.ops import operations as ops from mindspore.common.api import _cell_graph_executor from mindformers.core import CrossEntropyLoss from mindformers.modules import MultiHeadAttention, FeedForward, TransformerEncoderLayer, TransformerEncoder, \ TransformerDecoder, TransformerDecoderLayer, Transformer, AttentionMask, FixedSparseAttention class MyActivation(mindspore.nn.Cell): """An example of custom activation""" def __init__(self): super(MyActivation, self).__init__() self.add = ops.Add() def construct(self, x): return self.add(x, 0.1) def activation_shard(self, parallel_config): self.add.shard(((parallel_config.data_parallel, parallel_config.model_parallel), ())) class MyActivationNoShard(mindspore.nn.Cell): def __init__(self): super(MyActivationNoShard, self).__init__() self.add = ops.Add() def construct(self, x): return self.add(x, 0.1) def test_transformer_encoder_only(): """ Feature: Test Transformer API Description: Test Transformer model with encoder only Expectation: No exception """ model = Transformer(batch_size=2, src_seq_length=20, tgt_seq_length=10, encoder_layers=2, decoder_layers=0, hidden_size=64, ffn_hidden_size=64) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_encoder_log_softmax(): """ Feature: Test Transformer API Description: Test Transformer model with unexpected hidden act Expectation: No exception """ with pytest.raises(ValueError): model = Transformer(batch_size=2, src_seq_length=20, tgt_seq_length=10, encoder_layers=2, decoder_layers=0, hidden_act='logsoftmax', hidden_size=64, ffn_hidden_size=64) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_encoder_leakyrelu(): """ Feature: Test Transformer API Description: Test Transformer model with valid hidden act Expectation: No exception """ model = Transformer(batch_size=2, src_seq_length=20, tgt_seq_length=10, encoder_layers=2, decoder_layers=0, hidden_act='leakyrelu', hidden_size=64, ffn_hidden_size=64) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_encoder_and_decoder(): """ Feature: Test Transformer API Description: Test Transformer model with encoder decode input Expectation: No exception """ model = Transformer(batch_size=2, src_seq_length=20, tgt_seq_length=10, encoder_layers=1, decoder_layers=2, hidden_size=64, ffn_hidden_size=64) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask, decoder_input_value, decoder_input_mask, memory_mask) def test_transformer_encoder(): """ Feature: Test Transformer Encoder API Description: Test Transformer Encoder model with valid input Expectation: No exception """ model = TransformerEncoder(batch_size=2, seq_length=16, num_layers=2, hidden_size=8, ffn_hidden_size=64, num_heads=2) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 16, 16)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_encoder_layer(): """ Feature: Test Transformer Encoder Layer API Description: Test Transformer Encoder Layer model with valid input Expectation: No exception """ model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, num_heads=2) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 16, 16)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_encoder_layer_post_ture(): """ Feature: Test Transformer Encoder Layer API Description: Test Transformer Encoder Layer model with post_layernorm_residual True Expectation: No exception """ model = TransformerEncoderLayer(batch_size=2, seq_length=16, hidden_size=8, ffn_hidden_size=64, num_heads=2, post_layernorm_residual=True) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 16, 16)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_decoder(): """ Feature: Test Transformer Decoder API Description: Test Transformer Decoder model. Expectation: No exception """ model = TransformerDecoder(num_layers=1, batch_size=2, src_seq_length=20, tgt_seq_length=10, hidden_size=64, ffn_hidden_size=64, num_heads=2) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) _cell_graph_executor.compile(model, decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) def test_transformer_decoder_layer(): """ Feature: Test Transformer Decoder Layer API Description: Test Transformer DecoderLayer model. Expectation: No exception """ model = TransformerDecoderLayer( batch_size=2, src_seq_length=20, tgt_seq_length=10, hidden_size=64, ffn_hidden_size=64, num_heads=2) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) _cell_graph_executor.compile(model, decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) def test_multihead_attention(): """ Feature: Test MultiHead Attention API Description: Test MultiHead Attention model. Expectation: No exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=2, num_heads=3) from_tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((2, 20, 15)), dtype.float16) attention_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) @pytest.mark.parametrize('batch_size', [1, 2, None, 4]) def test_multihead_attention_wrong_batch(batch_size): """ Feature: Test MultiHeadAttention with wrong batch for training Description: Test the batch size to be any int or None Expectation: No exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=batch_size, num_heads=3) from_tensor = Tensor(np.ones((3, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((3, 20, 15)), dtype.float16) attention_mask = Tensor(np.ones((3, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) @pytest.mark.parametrize('from_tensor,to_tensor', [(Tensor(np.ones((20, 15)), dtype.float32), Tensor(np.ones((20, 15)), dtype.float16)), (Tensor(np.ones((3, 20, 15)), dtype.float32), Tensor(np.ones((3, 20, 15)), dtype.float16))]) def test_multihead_attention_no_mask_2d_or_3d_shape(from_tensor, to_tensor): """ Feature: Test MultiHeadAttention no mask Description: Test MultiHeadAttention no mask and 2d as inputs. Expectation: No exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=None, num_heads=3) _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, None) def test_multihead_attention_fp32_dtype(): """ Feature: Test MultiHeadAttention with float32 as compute dtype Description: Test using float32 as computation for linear layer. Expectation: No exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, compute_dtype=dtype.float32, batch_size=2, num_heads=3) from_tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) attention_mask = Tensor(np.ones((2, 20, 20)), dtype.float32) _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) @pytest.mark.parametrize('batch_size', [1, 2, None, 4]) def test_transformerencoder_wrong_batch(batch_size): """ Feature: Test TransformerEncoderLayer with wrong batch for training Description: Test the batch size to be any int or None Expectation: No exception """ model = TransformerEncoderLayer(batch_size=batch_size, hidden_size=8, ffn_hidden_size=64, seq_length=16, num_heads=2) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 16, 16)), dtype.float16) model(encoder_input_value, encoder_input_mask) @pytest.mark.parametrize('attention_mask', [Tensor(np.ones((2, 16, 16)), dtype.float16), None]) def test_transformerencoder_no_mask(attention_mask): """ Feature: Test TransformerEncoderLayer with no mask Description: Test the attention mask is None Expectation: No exception """ model = TransformerEncoderLayer(batch_size=None, hidden_size=8, ffn_hidden_size=64, seq_length=16, num_heads=2) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) model(encoder_input_value, attention_mask) @pytest.mark.parametrize('shape', [(2, 16, 8), (32, 8)]) def test_transformerencoder_2d_or_3d_shape(shape): """ Feature: Test TransformerEncoderLayer with 2d or 3d inputs Description: Test the attention mask is None Expectation: No exception """ model = TransformerEncoderLayer(batch_size=None, hidden_size=8, ffn_hidden_size=64, seq_length=16, num_heads=2) encoder_input_value = Tensor(np.ones(shape), dtype.float32) model(encoder_input_value, None) @pytest.mark.parametrize('batch_size', [1, 2, None, 4]) def test_transformerdecoder_wrong_batch(batch_size): """ Feature: Test TransformerDecoderLayer with wrong batch for training Description: Test the batch size to be any int or None Expectation: No exception """ model = TransformerDecoderLayer(batch_size=batch_size, hidden_size=64, ffn_hidden_size=64, num_heads=2, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) @pytest.mark.parametrize('decoder_input_mask,memory_mask', [(None, None), (Tensor(np.ones((2, 10, 10)), dtype.float16), None), (None, Tensor(np.ones((2, 10, 20)), dtype.float16))]) def test_transformerdecoder_mask(decoder_input_mask, memory_mask): """ Feature: Test TransformerDecoderLayer with empty mask Description: Test the mask is None Expectation: No exception """ model = TransformerDecoderLayer(batch_size=4, hidden_size=64, ffn_hidden_size=64, num_heads=2, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) @pytest.mark.parametrize('activation', [MyActivation, MyActivationNoShard]) def test_transformerdecoder_custom_activation(activation): """ Feature: Test TransformerDecoderLayer custom activation Description: Test TransformerDecoderLayer custom activation Expectation: No exception """ model = TransformerDecoderLayer(batch_size=4, hidden_size=64, ffn_hidden_size=64, num_heads=2, hidden_act=activation, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) model(decoder_input_value, None, encoder_input_value, None) @pytest.mark.parametrize('activation', [0, None, -1]) def test_transformerdecoder_wrong_activation(activation): """ Feature: Test TransformerDecoderLayer with wrong activation Description: Test TransformerDecoderLayer with wrong activation type Expectation: No exception """ with pytest.raises(TypeError): TransformerDecoderLayer(batch_size=4, hidden_size=64, ffn_hidden_size=64, num_heads=2, hidden_act=activation, src_seq_length=20, tgt_seq_length=10) @pytest.mark.parametrize('encoder_shape,decoder_shape', [((2, 20, 64), (2, 10, 64)), ((20, 64), (10, 64))]) def test_transformerdecoder_2d_or_3d_shape(encoder_shape, decoder_shape): """ Feature: Test TransformerDecoderLayer with 2d or 3d inputs Description: Test the attention mask is None Expectation: No exception """ model = TransformerDecoderLayer(batch_size=None, hidden_size=64, ffn_hidden_size=64, num_heads=2, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones(encoder_shape), dtype.float32) decoder_input_value = Tensor(np.ones(decoder_shape), dtype.float32) model(decoder_input_value, None, encoder_input_value, None) @pytest.mark.parametrize('hidden_act', [MyActivation, "relu"]) def test_transformer_hidden_act(hidden_act): """ Feature: Test Transformer hidden activation with activation or None Description: Test the transformer hidden activation Expectation: No exception """ model = Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, hidden_act=hidden_act, ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) model(encoder_input_value, encoder_input_mask, decoder_input_value, decoder_input_mask, memory_mask) def test_transformer_hidden_act_with_wrong_hidden_act_wrong_lambda_func(): """ Feature: Test Transformer hidden activation with activation or None Description: Test the transformer hidden activation Expectation: No exception """ with pytest.raises(TypeError): Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, hidden_act=lambda x: x, ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) def test_transformer_hidden_act_with_wrong_hidden_act_wrong_str(): """ Feature: Test Transformer hidden activation with wrong activation Description: Test the transformer hidden activation Expectation: No exception """ with pytest.raises(KeyError): Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, hidden_act="no_string", ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) def test_feedforward_layer(): """ Feature: Test Feedforward layer. Description: Test the Feedforward layer Expectation: No exception """ model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, hidden_act='relu') tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) _cell_graph_executor.compile(model, tensor) def test_cross_entroy(): """ Feature: Test cross entory loss. Description: Test the forward Expectation: No exception """ model = CrossEntropyLoss() logits = Tensor(np.array([[3, 5, 6, 9, 12, 33, 42, 12, 32, 72]]), dtype.float32) labels_np = np.array([1]).astype(np.int32) input_mask = Tensor(np.ones(1).astype(np.float32)) labels = Tensor(labels_np) _cell_graph_executor.compile(model, logits, labels, input_mask) def test_attention_mask(): """ Feature: Test the attention mask. Description: Test the forward Expectation: No exception """ model = AttentionMask(seq_length=19) inputs = Tensor(np.ones((2, 19)), dtype.float32) _cell_graph_executor.compile(model, inputs) def test_sparse_attention(): """ Feature: Test the sparse attention layer. Description: Test the forward Expectation: No exception """ model = FixedSparseAttention(batch_size=2, seq_length=1024, size_per_head=64, num_heads=8, block_size=64) q = Tensor(np.ones((2, 1024, 512)), dtype.float16) k = Tensor(np.ones((2, 1024, 512)), dtype.float16) v = Tensor(np.ones((2, 1024, 512)), dtype.float16) mask = Tensor(np.ones((2, 1024, 1024)), dtype.float32) _cell_graph_executor.compile(model, q, k, v, mask) def test_attention_with_wrong_batch_3d_inputs(): """ Feature: Test Transformer batch error when the input's batch size is different Description: Test the input's batch size is different between the tensors. The input is 3d Expectation: Raise a reshape error exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=None, num_heads=3) from_tensor = Tensor(np.ones((3, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((5, 20, 15)), dtype.float16) attention_mask = Tensor(np.ones((3, 20, 20)), dtype.float16) with pytest.raises(ValueError): _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) def test_attention_with_wrong_batch_2d_inputs(): """ Feature: Test Transformer batch error when the input's batch size is different Description: Test the input's batch size is different between the tensors. The inputs is 2d Expectation: Raise a reshape error exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=None, num_heads=3) from_tensor = Tensor(np.ones((60, 15)), dtype.float32) to_tensor = Tensor(np.ones((100, 15)), dtype.float16) attention_mask = Tensor(np.ones((3, 20, 20)), dtype.float16) with pytest.raises(ValueError): _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) def test_incremental_prediction_first_iterator(): """ Feature: Test MultiHeadAttention with incremental prediction Description: Test MultiHeadAttention with incremental prediction in the first iterator Expectation: No Expectation """ # Step 1: set is_first_iteration=True, and input the full sequence length's state. # We need to prepare the memory parameters for saving key and value states firstly. from_tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((2, 20, 15)), dtype.float16) attention_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), dtype.float16) value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), dtype.float16) batch_valid_length = Tensor(np.ones((2,)), dtype.int32) model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, num_heads=3, use_past=True) model.add_flags_recursive(is_first_iteration=True) model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, batch_valid_length) def test_incremental_prediction_second_iterator(): """ Feature: Test MultiHeadAttention with incremental prediction Description: Test MultiHeadAttention with incremental prediction in the second iterator Expectation: No Expectation """ model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, num_heads=3, use_past=True) key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), dtype.float16) value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), dtype.float16) batch_valid_length = Tensor(np.ones((2,)), dtype.int32) # Set is_first_iteration=True to generate the full memory states from_tensor = Tensor(np.ones((2, 1, 15)), dtype.float32) to_tensor = Tensor(np.ones((2, 1, 15)), dtype.float16) attention_mask = Tensor(np.ones((2, 1, 20)), dtype.float16) # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than the # full sequence. model.add_flags_recursive(is_first_iteration=False) model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, batch_valid_length) pengcheng.mind-test/tests/ut/test_ut.py000066400000000000000000000016361471400134000205520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test ut""" import pytest @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_example(): """ Feature: UT class Description: Test the ut Expectation: NotImplementedError, ValueError """ assert 1 < 2 pengcheng.mind-test/tests/ut/test_version_check.py000066400000000000000000000031201471400134000227320ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test version check schedule.""" import pytest from mindformers.tools.utils import is_version_le, is_version_ge @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_check_is_version_ge(): """ Feature: Test IsVerisionGreaterEquale Description: Test IsVersionGreaterEqual Expectation: ValueError """ assert is_version_ge("2.0.0rc1", '1.8.0') assert is_version_ge("1.8.1", '1.8.0') assert is_version_ge("1.8.0", '1.8.0') assert is_version_ge("2.0.0rc1", '2.0.0') assert not is_version_ge("1.8.0", '1.10.0') @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_check_is_version_le(): """ Feature: Test IsVerisionLessEquale Description: Test IsVersionLessEqual Expectation: ValueError """ assert is_version_le("2.0.0rc1", '2.0.0') assert is_version_le("1.8.1", '2.0.0') assert is_version_le("1.8.0", '2.0.0rc1')